VictoriaMetrics: vmalert: recording rule metric doesn't seem to match original

Describe the bug The screenshot below shows the difference in the original query, recording rule, and recording rule with a range vector of 1m.

original v:s recording rules

The recording rules are:

groups:
 - name: TestGroup
   rules:
     - record: test:node_cpu_seconds_total:no_range
       expr: >
         sum(rate(node_cpu_seconds_total{instance=~"victoriametrics01.*"}))

     - record: test:node_cpu_seconds_total:1m
       expr: >
         sum(rate(node_cpu_seconds_total{instance=~"victoriametrics01.*"}[1m]))

One difference is that Grafana uses query_range v/s query for recording rules. So I’m not sure how the step is calculated in recording rules v/s Grafana. So I added the 1m range explicitly, but it still doesn’t match.

To Reproduce Add the above rules, and graph to see the difference.

Expected behaviour The original and recorded metric should match.

Version The line returned when passing --version command line flag to binary. For example:

$ ./victoria-metrics-prod --version
victoria-metrics-20200929-211855-tags-v1.42.0-0-ge59de9838

Used command-line flags

$ /usr/local/bin/victoriametrics -loggerDisableTimestamps --storageDataPath=/var/lib/victoriametrics --selfScrapeInterval=0 --retentionPeriod=6 --memory.allowedPercent=60 --search.maxUniqueTimeseries=600000
$ /usr/local/bin/vmalert -loggerDisableTimestamps -datasource.url=http://localhost:8428 -remoteWrite.url=http://localhost:8429 -rule=/etc/victoriametrics/vmalert/*.yml -notifier.url http://locahost:9093 -evaluationInterval=1m0s

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 23

Most upvoted comments

Hi @raags!

One difference is that Grafana uses query_range v/s query for recording rules.

Correct, and Grafana calculates the step value based on the panel size and your display resolution. To override the step set some value to Min step field below the query area.

vmalert execute queries via /query endpoint for instant results. It doesn’t pass the step param, so VM uses the default value for it equal to 5m. So it is expected that rate(metric) (will have default 5m step) isn’t the same as rate(metric[1m]) (will have 1m step). Both won’t match the expression in Grafana from your screenshot, since it is using the Grafana’s defined step (which is likely seconds in this case).