vector: Incorrect Metric count reported to datadog

Thanks @atibdialpad I’ve been able to reproduce it with

[sources.in]
  type = "generator" 
  count = 2
  interval = 10
  lines = ["line"]
  format = "shuffle"

[transforms.add_value]
  type = "remap"
  inputs = ["in"]
  source = '''
    .value_field = 1
  '''

[transforms.to_metric]
  type = "log_to_metric" 
  inputs = ["add_value"] 
  [[transforms.to_metric.metrics]]
    type = "counter"
    field = "value_field"

[sinks.dd_api]
  type = "datadog_metrics" 
  inputs = ["to_metric"] 
  api_key = ...
  region = "eu" 

[sinks.dd_statsd]
  type = "statsd"
  inputs = ["to_metric"]
  address = "127.0.0.1:8125"
  mode = "udp"
  healthcheck.enabled = true

[sinks.outc]
  type = "console" 
  inputs = ["to_metric"]
  encoding = "json"

plus a local Datadog agent needs to be running.

To manifest this behavior same metrics (those with the same name) need to be supplied through two different API’s hence this is an issue in DataDog and how it shows metrics. It’s worth noting that this is only an issue of showing those metrics on the same graph since the values are shown correctly if separated, although Metrics Explorer doesn’t always re update the graph. Most likely Metrics Explorer has to combine intervals for two metrics that it sees as equal, and this combining depends on the set of metrics it has to show so the behavior varies.

I’ve also checked if this is a general issue of sending the same metric with different intervals, but it doesn’t seem so.

Solution

This can be addressed on Datadog side cc. @vladimir-dd

Else, users should avoid sending metric with the same name through different APIs. @atibdialpad can you try this out?

@ktff I did look at the metric summary and you are right. The metric_count is only 3 (1 each via dogstream, vector dd metric sink and vector statsd sink)

I think the flush period and metric normalisation technique is playing a role here. As the doc https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/#count says, all counter types metrics are reported by datadog as RATES (= count / flush period of 10 seconds)

So, as soon as the metric is parsed I see this in metric parser : via dogstream and vector statsd = 0.1, via dd metric sink = 1

and then after a few seconds it changes to [don’t why this happens] via dogstream and vector statsd = 1, via dd metric sink = 10

@atibdialpad

are you able to repro this ?

Nope.

Just trying to make sure that I am not working with a bad config.

That shouldn’t be the case going by the config you posted.

I ran just a single log line to ensure that only 1 metric is emitted (to rule out batching) and still see metric count 10 on the datadog metrics UI.

That narrows it down then. Then… it could be that some averaging or more generally miss interpretation of data is going on somewhere. Something like we are sending 1 count with interval of 100ms and Datadog is extrapolating that to 10 per second. Could you play a bit with Datadog’s Metrics explorer and see if you can get some different outcome, I’ll also try playing with it.