VictoriaMetrics: `histogram_quantile` get incorrect result when samples miss.

Describe the bug Hello @valyala @YuriKravetc, we find a new issue in our product env. We get 2 datasource, the left one is our prometheus native storage and the right one is our victoriametrics storage which is remote written by the prometheus. image

As the above image shows, victoriametrics sometimes has very large value that could result in wrong alert.

The expression is like:

histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket{instance=~"xxxxx",job=~"xxxx"}[5m])) by (instance, le))

I expand the result by showing all buckets:

le left datasource right datasource
0.001 0.00351 0.00333
0.002 0.0351 0.0417
0.004 0.0386 0.0367
0.008 0.0421 0.0400
0.016 0.0491 0.0467
0.032 0.0491 0.0467
0.064 0.0491 0.0467
0.128 0.0491 0.0467
0.256 0.0491 0.0467
0.512 0.0491 0.0467
1.024 0.0491 0.0467
2.048 0.0491 0.0467
4.096 0.0491 0.0467
8.192 0.0491 0.0583
INF 0.0491 0.0467

It’s very strange that victoriametrics has a larger value with a smaller le, which should be the reason of incorrect result. image

Cause I plot a new graph for the rate of le=4.096 and the rate of le=8.192 and find the problem: le=8.192 which is the green line in the below picture has a unexpected big value during the problem time range. However, le=8.192 and le=4.096 should have the same rate since they have the same value all over time. imageI

I add log for all rollup window’s calculation and find the diff: image

The left file is the log for the rate calculation of le=8.192, it’s tEnd stay the same for one more time period. Since the rate is calculated by (vEnd - vPrev) / (tEnd - tPrev), so it’s rate value is larger than what it should be.

For more detail, I print the raw sample timestamp for le=8.192 and le=4.096, there are two points missed with timeseries le=8.192 at the problem time range: image

Since victoriametrics will always use the last timestamp in a window rollup value calculation, the tEnd will remain the same in the next window if sample miss:

image

What we expect

Since sample miss is inevitable in remote write, we hope histogram_quantile calculation should not increase to a very large value in the case. For example: when calculate histogram_quantile, if we keep the value of le=8.192 always larger than or equal to the value of le=4.096 the graph of histogram_quantile will be much more reasonable.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 3
  • Comments: 18 (1 by maintainers)

Commits related to this issue

Most upvoted comments

FYI, the changed logic is available in VictoriaMetrics v1.93.16 LTS release.

we have the same issue with wrong rates e.g. le=0.5 and le=1 (le=0.5 has greater value than le=1) with histogram_quantile(0.9999 …) on release v1.79.0 it seems to have the same root is it possible to fix it? I could provide more info if needed

It would be great if you upgrade to the latest available release - v1.81.1 now - and share the <query> together with the results of the <query> from histogram_quantile(0.9999, <query>) alongside the unexpected results of histogram_quantile(0.9999, ...) together with the expected results.

The issue should be fixed in the following commits:

  • single-node VictoriaMetrics - f3ece83e67bb6636b042cc1ebad1e7d71e4534cf
  • VictoriaMetrics cluster - 85166705827310062c8cae1b2b1e37bc6ac09530

These commits will be included in the next release.

@eddietanshuo , could you build VictoriaMetrics from these commits and verify whether this fixes the issue? See build instructions for single-node VictoriaMetrics and build instructions for cluster version of VictoriaMetrics