graphite_exporter: Handle duplicated metrics gracefully
Summary:
The exporter remembers metrics by Graphite name. It is possible to get into situations where multiple samples (graphite metrics) result in the same Prometheus metric.
At the moment, this makes the exporter catastrophically fail (return error 500 on /metrics). Instead, it should discard the older sample and use the new sample’s value.
Original issue description:
We tried using v0.5.0 in our dev env today, upgrading from 0.4.2.
After the upgrade, metrics seem to be dropping out regularly, as can be clearly seen in this grafana dashboard.
So something is definitively wrong here. There are no changes to the environment or config between 0.4.2 and 0.5.0, and we can not see anything abnormal in the graphite_exporter logfiles.
This looks like a critical bug to us.
It should be noted that we have seen similar behavior before where we had UDP packet drops, but this was resolved by tuning the UDP receive buffer as explained here: https://nav.uninett.no/doc/4.3/howto/tuning_graphite.html
About this issue
- Original URL
- State: open
- Created 5 years ago
- Comments: 30 (20 by maintainers)
Sorry this looks like my carelessness during bumping up client_golang. FYI a similar error happened when running e2e test, hence this fix https://github.com/prometheus/graphite_exporter/commit/3d093347bac809689ebd31a344a4fbe9a10c9b35. Indeed this is a behaviour change in client_golang. I feel it could be addressed by not creating new metric every time in https://github.com/prometheus/graphite_exporter/blob/3d093347bac809689ebd31a344a4fbe9a10c9b35/main.go#L192-L201, instead we can cache the metric instance for same
originalName(like in statsd_exporter).However I do have a question in mind and I’m not very familiar with the graphite protocol: if two metric with the same name and labels arrives between two prometheus scrape, the current behaviour will be exporter only export the later value (since they are Gauge). Is this expected?