alloy: When using clustering, exporters may not work correctly due to `instance` label
What’s wrong?
Most embedded Prometheus exporters set the instance label to the hostname where Alloy runs.
This breaks in a subtle, but significant way, the fundamental clustering assumption that all instances have the same configuration. The exporters implicitly inject the hostname as an instance label, but instances usually have different hostnames. This leads to either no scraping of metrics at all, or duplicate scraping with different instance labels (unnecessary).
Steps to reproduce
- Run any exporter in a clustered mode in a cluster of 2+ instances, each running on a different host. Have scraping set up with clustering and a remote write to a metrics DB.
- Observe that some targets will not be scraped at all, some will be scraped multiple times, with different instance labels.
- Observe in the UI that instance label is different in exporters’ targets between instances, indicating different series.
The issue was discussed in this PR, but decided to move the conversation here for better tracking and to provide a place to refer to for workarounds.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 4
- Comments: 18 (10 by maintainers)
Affected our 30+ blackbox probes over ~7 alloy deployed via Helm, we were missing out on random targets, triggering DatasourceNoData in our alerting. The workaround fixed it.
I think you missunderstood, its only a Problem with Exporters that are setting the instance Label based on the alloy running it. So you cant scrape it directly and have to Go the Extra Route via a discovery relabel between.
TBH this proposed rule is not a workaround. Using it breaks multiple dashboard and alert since we can’t distinguish nodes running node-exporter anymore.
We can add the logging warning if you want to. I don’t think it addresses your original concern though:
now the users will need to read about it in the logs or in the docs. But I’m happy to add it to increase the visibility of this potential configuration problem. Can add it to my PR.