VictoriaMetrics: Query_range wrong last value

Last metrics of query_range are wrong.

When i query VM through query_range the last value is either wrong or not present.

Prometheus query_range:

curl -sg "http://x.x.x.x:9090/api/v1/query_range?query=probe_duration_seconds{instance='xxx'}&start=1627893600&end=1627893900&step=60"
        "values": [
          [
            1627893600,
            "0.09316756"
          ],
          [
            1627893660,
            "0.104327531"
          ],
          [
            1627893720,
            "0.101365639"
          ],
          [
            1627893780,
            "0.09696379"
          ],
          [
            1627893840,
            "0.088306011"
          ],
          [
            1627893900,
            "0.094134059"
          ]

VM query_range :

curl -sg "http://x.x.x.x:8428/api/v1/query_range?query=probe_duration_seconds{instance='xxx'}&start=1627893600&end=1627893900&step=60

        "values": [
          [
            1627893600,
            "0.09316756"
          ],
          [
            1627893660,
            "0.104327531"
          ],
          [
            1627893720,
            "0.101365639"
          ],
          [
            1627893780,
            "0.09696379"
          ],
          [
            1627893840,
            "0.088306011"
          ]

The value with timestamp 1627893900 is missing from VM

Another example:

Prometheus query_range:

curl -sg "http://x.x.x.x:9090/api/v1/query_range?query=count(up)&start=1627893600&end=1627893900&step=60"

        "values": [
          [
            1627893600,
            "931"
          ],
          [
            1627893660,
            "931"
          ],
          [
            1627893720,
            "931"
          ],
          [
            1627893780,
            "931"
          ],
          [
            1627893840,
            "931"
          ],
          [
            1627893900,
            "931"
          ]

VM query_range :

curl -sg "http://x.x.x.x:8428/api/v1/query_range?query=count(up)&start=1627893600&end=1627893900&step=60"

        "values": [
          [
            1627893600,
            "931"
          ],
          [
            1627893660,
            "931"
          ],
          [
            1627893720,
            "931"
          ],
          [
            1627893780,
            "931"
          ],
          [
            1627893840,
            "931"
          ],
          [
            1627893900,
            "778"
          ]

Here the metrics with timestamp 1627893900 is wrong.

You can saw it in grafana too.

last_value_wron,g

metrics late

Querying through api/v1/query is good and through api/v1/export too. Only query_range is returning weird metrics

Version docker exec victoriametrics ps /victoria-metrics-prod -storageDataPath=/victoria-metrics-data -retentionPeriod=12 -dedup.minScrapeInterval=1m -search.latencyOffset=1s

docker exec victoriametrics /victoria-metrics-prod --version victoria-metrics-20210715-111307-tags-v1.63.0-0-g61cc13c16

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 26 (2 by maintainers)

Commits related to this issue

all: add support for Prometheus staleness markers Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748 Updates ... — committed to VictoriaMetrics/VictoriaMetrics by valyala 3 years ago
all: add support for Prometheus staleness markers Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748 Updates ... — committed to VictoriaMetrics/VictoriaMetrics by valyala 3 years ago
lib/promscrape: stop scrapers for the removed targets before starting scrapers for the added targets This should prevent from possible time series overlap when old target is substituted by new target... — committed to VictoriaMetrics/VictoriaMetrics by valyala 3 years ago
lib/promscrape: stop scrapers for the removed targets before starting scrapers for the added targets This should prevent from possible time series overlap when old target is substituted by new target... — committed to VictoriaMetrics/VictoriaMetrics by valyala 3 years ago

Most upvoted comments

VictoriaMetrics was start with -search.latencyOffset=1s. I tried to remove this flags. So now i have 1min scrape_interval, -search.latencyOffset 30s (default value) and -search.maxStepForPointsAdjustment 1m (default value) but i still have that wrong last point

xenofree on Aug 17, 2021

Thanks for the release, I have upgrade to 1.64.0 and i still have issue with count(). /api/v1/query return the right value but last point of query_range is weird

xenofree on Aug 16, 2021

Thanks for your answer. I will not modify all my dashboard with this workaround, i’ll wait for a fix 😃

xenofree on Aug 3, 2021