VictoriaMetrics: Staleness_interval for Streaming aggregation does not work corect?

Is your question request related to a specific component?

vmagent

Describe the question in detail

We have few types of job that generate metrics push it to vmagent with streaming aggregation and vmagent writes to the prometheus, one runs every 10 min, second once a hour. To avoid no data in prometheus we setted up staleness_interval. Our config:

      streamaggr.yml: |
        - match: 
          - report_counter
          interval: 2m
          outputs: ["total"]
          without: ["job"]
          staleness_interval: 70m
          keep_metric_names: true
          flush_on_shutdown: true

Problem only that first metric after vmgent start I can see only after staleness_interval, if it 70 min, 70 min will be no data in prometheus, if 15 min then 15 min no data, if 1 day then 1 day nodata. After that everything works fine. Is it a bug or expected behavior?

Troubleshooting docs

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 19 (11 by maintainers)

Commits related to this issue

Most upvoted comments

@AndrewChubatiuk @hagen1778 Reading this PR I’m still a bit at a loss as to why this was added, I understand that it can add “spike” at start but what difference does it make if “spike” is now or after some time? Cardinality will be the same or even larger at the end of the staleness interval.

I have tested streaming aggregation for a while and can assume it was added for the next reason: total output is a counter, it must start from 0 Imagine having total streamaggr output over a constant counter, for instance cpu_limit_total over cpu_limit. Constant here is just for simplicity.

Once vmagent restarted it will receive cpu_limit samples, say 100. vmagent hasn’t seen this timeseries yet and it doesn’t know whether the timeseries even existed before the restart so it has two options:

  1. increase output total to 100 (treat it like a counter reset) and set lastSeenValue (internal state) to 100
  2. keep total 0 and set lastSeenValue to 100

the first option doesn’t work fine: it leads the counter to start from 100 say we want to apply increase to cpu_limit_total, increase assumes counters start from 0 that way increase(cpu_limit_total) will produce a huge spike since the value is increased from 0 to 100

the second option is what IgnoreFirstSampleInterval does vmagent observes the counter and its current value but doesn’t increase total output since it can’t be sure whether it’s a new timeseries or not, it just takes some time to warmup