VictoriaMetrics: vmselect failed to query correct result when different values in two replicas

Describe the bug

I have two VM clusters, these two clusters receive the same metric samples. And there are two replicas of VMAgent, i.e, 4 replicas total.

Image

There is a target expose metrics in promethes protocol, its metrics are scraped by vmagent-0-0, vmagent-0-1, vmagent-1-0 and vmagent-1-1. The instance vmagent-0-0 scape and remote write old samples to vmstorage group 0, and the other vmagent instance worked normally.

When our user query metric by grafana with query sum(irate(kitex_server_throughput{cluster="ali-shanghai-prod-01",namespace="xingye-prod",app=~"app-name", pod=~"(app-name-1-7b55d984cb-2t79s)", method="method1"}[1m])) by (pod, method) we got following result. There are many spikes.

Image

When I only query the cluster-0 I get the same graph, but when I only query the cluster-1 I get the correct result, there are no spikes!

Image

When I killed the vmagent-0-0, this problem disappeared.

To Reproduce

  1. Deploy two VM clusters
  2. Remote write same metric sample to two VM clusters, but the metric value is different, e.g. foo 0 to cluster 0 and foo 1 to cluster 1 with same timestamp.
  3. Query two VM clusters with one vmselect instance.

Version

vmagent: vmagent-20250124-135527-tags-v1.110.0-0-g4cab63c6a8
vmselect: vmselect-20250124-140048-tags-v1.110.0-cluster-0-g880cd95aa5
vmstorage: vmstorage-20241002-113649-tags-v1.104.0-cluster-0-gdfb2ad4ab4

Logs

No response

Screenshots

No response

Used command-line flags

No response

Additional information

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 20 (17 by maintainers)

Most upvoted comments

Thanks for the raw data. I think the reason of this issue could be related to counter value reset. The second time series data you provided contains the following values:

    7566016,
    7565902,
    7567597,
    7569185,
    7570783,
    7572405,
    7573958,
    7575480,
    7577050,
    7578701,
    7578575,
    7580280,
    7581834,

And you can see the 2nd value is lower than the 1st value. Since you are using sum(irate, I assume this metrics is a counter, which should be always increasing unless reset. So every time a value is lower than previous value, it will be considered as a counter reset and affect the calculation.

If you check the timestamp of these counter reset, they happened 5 times within [05:40, 05:50], which match the first screenshot you provided.

Image

So I think the direction of debugging this issue is: why vmagent scraped such counter reset cases?

  • Does the previous scrpae happen after the later one?
  • How long does it take to scrape this target?
  • And also regarding your scrape target, is it possible that the response of /metrics endpoint was cached?
  • what happen if multiple requests reached the target concurrently?

Does it mean that both vmagents within the pair have identical cluster label?

No two vmagents have same cluster=blablabla pair

Could you please check local time on machince where vmagents run - is it synchronized?

Yes it’s synchronized.

Also, I’ve checked the core-level CPU usage during this period, there are several cores whose usage is 100%. Other pods with heavy load share the same CPU core as the vmagent pod, which may cause there to be insufficient CPU for vmagent. Since the cpuManagerPolicy of kubelet is static, the CPU request and limit of vmagent are the same, the vmagent pod was bounded to specific cores. But the CPU request and limit of other pods are different, their threads may be scheduled to all CPU cores.

Hi @jiekun , here is raw data export with command curl 'http://10.32.192.87:8481/select/0/prometheus/api/v1/export?reduce_mem_usage=1' -d 'match[]=kitex_server_throughput{cluster="ali-shanghai-prod-01",namespace="xingye-prod",app=~"weaver-conversation-biz", pod=~"(weaver-conversation-biz-fnr2-7b55d984cb-2t79s)", method="GenerateRedoTalkingMsgList"}' -d 'start=1744954825' -d 'end=1744955485'

{
  "metric": {
    "__name__": "kitex_server_throughput",
    "namespace": "xingye-prod",
    "cluster": "ali-shanghai-prod-01",
    "service": "weaver-conversation-biz",
    "job": "app",
    "instance": "10.75.228.184:19191",
    "pod": "weaver-conversation-biz-fnr2-7b55d984cb-2t79s",
    "node": "cn-shanghai.10.74.84.21",
    "app": "weaver-conversation-biz",
    "application": "weaver-conversation-biz",
    "callee": "weaver.conversation.biz",
    "caller": "weaver.gateway.weaver",
    "endpoint": "app-metric",
    "lane": "DEFAULT",
    "method": "GenerateRedoTalkingMsgList",
    "release": "weaver-conversation-biz-fnr2",
    "retry": "0",
    "status": "succeed",
    "vm_account_id": "0",
    "vm_project_id": "0"
  },
  "values": [
    7564484,
    7565903,
    7567356,
    7568747,
    7570166,
    7571606,
    7573003,
    7574382,
    7575788,
    7577179,
    7578575,
    7579957,
    7581316,
    7582683,
    7584011,
    7585391,
    7586747,
    7588092,
    7589448,
    7590837,
    7592178,
    7593487,
    7594787,
    7596099,
    7597467,
    7598753,
    7600040,
    7601368,
    7602643,
    7603938,
    7605291,
    7606621,
    7607899,
    7609179,
    7610419,
    7611651,
    7612923,
    7614207,
    7615453,
    7616732,
    7617979,
    7619296,
    7620506,
    7621727
  ],
  "timestamps": [
    1744954838701,
    1744954853701,
    1744954868701,
    1744954883701,
    1744954898701,
    1744954913701,
    1744954928701,
    1744954943701,
    1744954958701,
    1744954973701,
    1744954988701,
    1744955003701,
    1744955018701,
    1744955033701,
    1744955048701,
    1744955063701,
    1744955078701,
    1744955093701,
    1744955108701,
    1744955123701,
    1744955138701,
    1744955153701,
    1744955168701,
    1744955183701,
    1744955198701,
    1744955213701,
    1744955228701,
    1744955243701,
    1744955258701,
    1744955273701,
    1744955288701,
    1744955303701,
    1744955318701,
    1744955333701,
    1744955348701,
    1744955363701,
    1744955378701,
    1744955393701,
    1744955408701,
    1744955423701,
    1744955438701,
    1744955453701,
    1744955468701,
    1744955483701
  ]
}
{
  "metric": {
    "__name__": "kitex_server_throughput",
    "namespace": "xingye-prod",
    "cluster": "ali-shanghai-prod-01",
    "service": "weaver-conversation-biz",
    "job": "app",
    "instance": "10.75.228.184:19191",
    "pod": "weaver-conversation-biz-fnr2-7b55d984cb-2t79s",
    "node": "cn-shanghai.10.74.84.21",
    "app": "weaver-conversation-biz",
    "application": "weaver-conversation-biz",
    "callee": "weaver.conversation.biz",
    "caller": "weaver.gateway.weaver",
    "endpoint": "app-metric",
    "lane": "DEFAULT",
    "method": "GenerateRedoTalkingMsgList",
    "release": "weaver-conversation-biz-fnr2",
    "retry": "0",
    "status": "succeed",
    "vm_account_id": "0",
    "vm_project_id": "0"
  },
  "values": [
    7566016,
    7565902,
    7567597,
    7569185,
    7570783,
    7572405,
    7573958,
    7575480,
    7577050,
    7578701,
    7578575,
    7580280,
    7581834,
    7583345,
    7584783,
    7586388,
    7587879,
    7589372,
    7590947,
    7590837,
    7592394,
    7593873,
    7595342,
    7597018,
    7598625,
    7600173,
    7600040,
    7601704,
    7603217,
    7604746,
    7606297,
    7607849,
    7609361,
    7609179,
    7610804,
    7612324,
    7613824,
    7615325,
    7616237,
    7616877,
    7617979,
    7619296,
    7620647,
    7622120
  ],
  "timestamps": [
    1744954838705,
    1744954853703,
    1744954868705,
    1744954883705,
    1744954898705,
    1744954913705,
    1744954928705,
    1744954943705,
    1744954958705,
    1744954973705,
    1744954988703,
    1744955003705,
    1744955018705,
    1744955033705,
    1744955048705,
    1744955063705,
    1744955078705,
    1744955093705,
    1744955108705,
    1744955123703,
    1744955138705,
    1744955153705,
    1744955168705,
    1744955183705,
    1744955198705,
    1744955213705,
    1744955228703,
    1744955243705,
    1744955258705,
    1744955273705,
    1744955288705,
    1744955303705,
    1744955318705,
    1744955333703,
    1744955348705,
    1744955363705,
    1744955378705,
    1744955393705,
    1744955408705,
    1744955423705,
    1744955438705,
    1744955453705,
    1744955468705,
    1744955483705
  ]
}