statsd_exporter: Handling of conflicting metric values

We had some developers on a new project starting to integrate statsd support in their application. statsd_exporter 0.3.0 is crashing constantly with this error message:

FATA[0000] A change of configuration created inconsistent metrics for "query_timer". You have to restart the statsd_exporter, and you should consider the effects on your monitoring setup. Error: a previously registered descriptor with the same fully-qualified name as Desc{fqName: "query_timer", help: "Metric autogenerated by statsd_exporter.", constLabels: {after_date="2017-01-19 00:00:00-05:00",application="…",component="…",environment="testing",server="…",sub_component="…"}, variableLabels: []} has different label names or a different help string  source=exporter.go:137

If I’m reading that correctly, this due to them having two versions of the app sending inconsistent metric values but while an error in this situation seems reasonable it actually causes the statsd_exporter process to crash.

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 10
Comments: 27 (16 by maintainers)

Commits related to this issue

Don't crash on conflicting metric names This patch simply moves the error message from a log.Fatalf() to a log.Errorf() to continue on. Fixes #63 — committed to jacksontj/statsd_exporter by jacksontj 7 years ago
Don't crash on conflicting metric names This patch simply moves the error message from a log.Fatalf() to a log.Errorf() to continue on. Fixes #63 — committed to jacksontj/statsd_exporter by jacksontj 7 years ago

Most upvoted comments

I don’t know if it worths creating a new issue, so I’ll try here first. We are using datadog statsd python client, and we can’t handle well conflicts today. I am writing this comment because we found out some metrics were’nt been sent to statsd_exporter for a week.

There are no logs ASAIK when a conflict occures. It would be great to have an option to enable these logs, so we will fail early.
There is no way to “ignore” conflicts and duplicate the metric when labels are different.
There is no way to unitest it.

Any suggestion how to handle such cases?

gaizeror on Apr 23, 2019

What about something like this?

mappings:
- match: test.timing.*.*.*
  match_metric_type: counter|gauge|timer
  name: "my_timer"
  labels:
    provider: "$2"
    outcome: "$3"
    job: "${1}_server"

tobiashenkel on Feb 23, 2018