VictoriaMetrics: `histogram_quantile` get incorrect result when samples miss.
Describe the bug
Hello @valyala @YuriKravetc, we find a new issue in our product env.
We get 2 datasource, the left one is our prometheus native storage and the right one is our victoriametrics storage which is remote written by the prometheus.

As the above image shows, victoriametrics sometimes has very large value that could result in wrong alert.
The expression is like:
histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket{instance=~"xxxxx",job=~"xxxx"}[5m])) by (instance, le))
I expand the result by showing all buckets:
| le | left datasource | right datasource |
|---|---|---|
| 0.001 | 0.00351 | 0.00333 |
| 0.002 | 0.0351 | 0.0417 |
| 0.004 | 0.0386 | 0.0367 |
| 0.008 | 0.0421 | 0.0400 |
| 0.016 | 0.0491 | 0.0467 |
| 0.032 | 0.0491 | 0.0467 |
| 0.064 | 0.0491 | 0.0467 |
| 0.128 | 0.0491 | 0.0467 |
| 0.256 | 0.0491 | 0.0467 |
| 0.512 | 0.0491 | 0.0467 |
| 1.024 | 0.0491 | 0.0467 |
| 2.048 | 0.0491 | 0.0467 |
| 4.096 | 0.0491 | 0.0467 |
| 8.192 | 0.0491 | 0.0583 |
| INF | 0.0491 | 0.0467 |
It’s very strange that victoriametrics has a larger value with a smaller le, which should be the reason of incorrect result.

Cause
I plot a new graph for the rate of le=4.096 and the rate of le=8.192 and find the problem: le=8.192 which is the green line in the below picture has a unexpected big value during the problem time range. However, le=8.192 and le=4.096 should have the same rate since they have the same value all over time.
I
I add log for all rollup window’s calculation and find the diff:

The left file is the log for the rate calculation of le=8.192, it’s tEnd stay the same for one more time period. Since the rate is calculated by (vEnd - vPrev) / (tEnd - tPrev), so it’s rate value is larger than what it should be.
For more detail, I print the raw sample timestamp for le=8.192 and le=4.096, there are two points missed with timeseries le=8.192 at the problem time range:

Since victoriametrics will always use the last timestamp in a window rollup value calculation, the tEnd will remain the same in the next window if sample miss:

What we expect
Since sample miss is inevitable in remote write, we hope histogram_quantile calculation should not increase to a very large value in the case. For example: when calculate histogram_quantile, if we keep the value of le=8.192 always larger than or equal to the value of le=4.096 the graph of histogram_quantile will be much more reasonable.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 3
- Comments: 18 (1 by maintainers)
Commits related to this issue
- app/vmselect/promql: properly calculate histogram_quantile over unexpected `le` buckets See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819 — committed to VictoriaMetrics/VictoriaMetrics by valyala 4 years ago
- app/vmselect/promql: properly calculate histogram_quantile over unexpected `le` buckets See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819 — committed to VictoriaMetrics/VictoriaMetrics by valyala 4 years ago
FYI, the changed logic is available in VictoriaMetrics v1.93.16 LTS release.
It would be great if you upgrade to the latest available release - v1.81.1 now - and share the
<query>together with the results of the<query>fromhistogram_quantile(0.9999, <query>)alongside the unexpected results ofhistogram_quantile(0.9999, ...)together with the expected results.The issue should be fixed in the following commits:
These commits will be included in the next release.
@eddietanshuo , could you build VictoriaMetrics from these commits and verify whether this fixes the issue? See build instructions for single-node VictoriaMetrics and build instructions for cluster version of VictoriaMetrics