prometheus: Prefixing exported labels with `exported_` has problematic edge cases

This is extracted from #5845 .

When Prometheus adds target labels to a scrape, conflicts with existing labels may arise and have to be solved. The docs specify the following: “… label conflicts are resolved by renaming conflicting labels in the scraped data to “exported_<original-label>” (for example “exported_instance”, “exported_job”) and then attaching server-side labels.”

1. What happens if the target labels contain both foo="a" and exported_foo="b" and then a scrape contains its own label foo="c"? End result could be:
   1. `foo=“a”,exported_foo=“b”
   2. `foo=“a”,exported_foo=“c”
   3. `foo=“a”,exported_foo=“b”,exported_exported_foo=“c”
2. What happens if a scrape contains both foo="a" and exported_foo="b" and then there is a target label foo="c"? End result could be:
   1. `foo=“c”,exported_foo=“a”
   2. `foo=“c”,exported_foo=“b”
   3. `foo=“c”,exported_foo=“a”,exported_exported_foo=“b”

In case 2, the current code will do i. But that’s not precisely specified in the docs. One might also expect iii.

Case 1 is more interesting, as the behavior depends on the order the target labels show up in the iteration. It could be i. or iii. That’s in any case an undesired behavior.

For double fun, case 1 and case 2 could be combined (both the scrape and the target labels contain both a foo and an exported_foo label).

While this is unlikely to ever show up, we should do something about it. The most conservative approach would be to specify in the docs that the behavior is undefined if exported labels and/or target labels already have a prefix exported_. The most invasive change would be to go for case iii., essentially prefixing exported_ to exported label as often as necessary to avoid any name collisions. That’s technically a breaking change, but one might argue that’s what the docs so far suggest (in a very vague fashion).

For the record, this has never been reported to be a problem in many years of widespread Prometheus usage.