VictoriaMetrics: Staleness_interval for Streaming aggregation does not work corect?
Is your question request related to a specific component?
vmagent
Describe the question in detail
We have few types of job that generate metrics push it to vmagent with streaming aggregation and vmagent writes to the prometheus, one runs every 10 min, second once a hour. To avoid no data in prometheus we setted up staleness_interval. Our config:
streamaggr.yml: |
- match:
- report_counter
interval: 2m
outputs: ["total"]
without: ["job"]
staleness_interval: 70m
keep_metric_names: true
flush_on_shutdown: true
Problem only that first metric after vmgent start I can see only after staleness_interval, if it 70 min, 70 min will be no data in prometheus, if 15 min then 15 min no data, if 1 day then 1 day nodata. After that everything works fine. Is it a bug or expected behavior?
Troubleshooting docs
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 19 (11 by maintainers)
Commits related to this issue
- lib/streamaggr: add `ignore_first_sample_interval` param for streamaggr cfg (#7313) ### Describe Your Changes As of right now by default aggregated output in streaming aggregation takes a stalene... — committed to VictoriaMetrics/VictoriaMetrics by deleted user 2 years ago
- lib/streamaggr: add `ignore_first_sample_interval` param for streamaggr cfg (#7313) ### Describe Your Changes As of right now by default aggregated output in streaming aggregation takes a stalene... — committed to VictoriaMetrics/VictoriaMetrics by deleted user 2 years ago
- lib/streamaggr: add `ignore_first_sample_interval` param for streamaggr cfg (#7313) ### Describe Your Changes As of right now by default aggregated output in streaming aggregation takes a stalene... — committed to VictoriaMetrics/VictoriaMetrics by deleted user 2 years ago
- lib/streamaggr: add `ignore_first_sample_interval` param for streamaggr cfg (#7313) ### Describe Your Changes As of right now by default aggregated output in streaming aggregation takes a stalene... — committed to truepele/VictoriaMetrics by deleted user 2 years ago
- lib/streamaggr: add `ignore_first_sample_interval` param for streamaggr cfg (#7313) ### Describe Your Changes As of right now by default aggregated output in streaming aggregation takes a stalene... — committed to truepele/VictoriaMetrics by deleted user 2 years ago
I have tested streaming aggregation for a while and can assume it was added for the next reason:
totaloutput is a counter, it must start from 0 Imagine havingtotalstreamaggr output over a constant counter, for instancecpu_limit_totalovercpu_limit. Constant here is just for simplicity.Once vmagent restarted it will receive
cpu_limitsamples, say 100. vmagent hasn’t seen this timeseries yet and it doesn’t know whether the timeseries even existed before the restart so it has two options:totalto 100 (treat it like a counter reset) and setlastSeenValue(internal state) to 100total0 and setlastSeenValueto 100the first option doesn’t work fine: it leads the counter to start from 100 say we want to apply
increasetocpu_limit_total,increaseassumes counters start from 0 that wayincrease(cpu_limit_total)will produce a huge spike since the value is increased from 0 to 100the second option is what
IgnoreFirstSampleIntervaldoes vmagent observes the counter and its current value but doesn’t increasetotaloutput since it can’t be sure whether it’s a new timeseries or not, it just takes some time to warmup