VictoriaMetrics: vmselect failed to query correct result when different values in two replicas
Describe the bug
I have two VM clusters, these two clusters receive the same metric samples. And there are two replicas of VMAgent, i.e, 4 replicas total.
There is a target expose metrics in promethes protocol, its metrics are scraped by vmagent-0-0, vmagent-0-1, vmagent-1-0 and vmagent-1-1. The instance vmagent-0-0 scape and remote write old samples to vmstorage group 0, and the other vmagent instance worked normally.
When our user query metric by grafana with query sum(irate(kitex_server_throughput{cluster="ali-shanghai-prod-01",namespace="xingye-prod",app=~"app-name", pod=~"(app-name-1-7b55d984cb-2t79s)", method="method1"}[1m])) by (pod, method) we got following result. There are many spikes.
When I only query the cluster-0 I get the same graph, but when I only query the cluster-1 I get the correct result, there are no spikes!
When I killed the vmagent-0-0, this problem disappeared.
To Reproduce
- Deploy two VM clusters
- Remote write same metric sample to two VM clusters, but the metric value is different, e.g.
foo 0to cluster 0 andfoo 1to cluster 1 with same timestamp. - Query two VM clusters with one vmselect instance.
Version
vmagent: vmagent-20250124-135527-tags-v1.110.0-0-g4cab63c6a8
vmselect: vmselect-20250124-140048-tags-v1.110.0-cluster-0-g880cd95aa5
vmstorage: vmstorage-20241002-113649-tags-v1.104.0-cluster-0-gdfb2ad4ab4
Logs
No response
Screenshots
No response
Used command-line flags
No response
Additional information
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 20 (17 by maintainers)
Thanks for the raw data. I think the reason of this issue could be related to counter value reset. The second time series data you provided contains the following values:
And you can see the 2nd value is lower than the 1st value. Since you are using
sum(irate, I assume this metrics is a counter, which should be always increasing unless reset. So every time a value is lower than previous value, it will be considered as a counter reset and affect the calculation.If you check the timestamp of these counter reset, they happened 5 times within [05:40, 05:50], which match the first screenshot you provided.
So I think the direction of debugging this issue is: why vmagent scraped such counter reset cases?
/metricsendpoint was cached?No two vmagents have same
cluster=blablablapairYes it’s synchronized.
Also, I’ve checked the core-level CPU usage during this period, there are several cores whose usage is 100%. Other pods with heavy load share the same CPU core as the vmagent pod, which may cause there to be insufficient CPU for vmagent. Since the cpuManagerPolicy of kubelet is static, the CPU request and limit of vmagent are the same, the vmagent pod was bounded to specific cores. But the CPU request and limit of other pods are different, their threads may be scheduled to all CPU cores.
Hi @jiekun , here is raw data export with command
curl 'http://10.32.192.87:8481/select/0/prometheus/api/v1/export?reduce_mem_usage=1' -d 'match[]=kitex_server_throughput{cluster="ali-shanghai-prod-01",namespace="xingye-prod",app=~"weaver-conversation-biz", pod=~"(weaver-conversation-biz-fnr2-7b55d984cb-2t79s)", method="GenerateRedoTalkingMsgList"}' -d 'start=1744954825' -d 'end=1744955485'ping @hagen1778 @valyala