VictoriaMetrics: vmstorage insert slow down during indexDb rotation

Describe the bug

In past two weeks, the ingestion rate of vmstorage decreased during indexDb rotation, also the concurrency increased. This phenomenon occurs every day, and it always appears at eight o’clock in the morning (the time zone is UTC+8).

To Reproduce

do nothing

Version

vmstorage-20241002-113649-tags-v1.104.0-cluster-0-gdfb2ad4ab4

Logs

no error logs

Screenshots


Here are some metrics screenshot of vmstorage and vminsert

churn rate

slow insert

  • percent Clipboard_Screenshot_1733476390

  • vmstorage insert rows Clipboard_Screenshot_1733476081

  • vmstorage slow insert rows Clipboard_Screenshot_1733476125

vmstorage cpu usage

Clipboard_Screenshot_1733475639

vmstorage merge

  • vmstorage active merge Clipboard_Screenshot_1733476295

  • vmstorage merge speed Clipboard_Screenshot_1733476326

vminsert

  • vminsert concurrency Clipboard_Screenshot_1733476026

  • Storage connection saturation Clipboard_Screenshot_1733476473

Used command-line flags

The command is /app/mon/vmstorage-prod -loggerTimezone Asia/Shanghai -retentionPeriod 90d -dedup.minScrapeInterval 15s -storageDataPath /data/vmstorage-data -httpListenAddr :8482 -vminsertAddr :8400 -vmselectAddr :8401

Additional information

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 22 (15 by maintainers)

Most upvoted comments

I found there are two pattern of cpu usage among vmstorage node.

  1. cpu increased right after index rotation Clipboard_Screenshot_1734061677 And I found there are some small/big merge releated logs in the highest cpu usage node. I’m not sure that are these logs related to cpu increasing.
Dec 13 07:54:12 VM-32-202-ubuntu vmstorage-prod[3724376]: 2024-12-13T07:54:12.859+0800        info        VictoriaMetrics/lib/storage/partition.go:1465        merged (15 parts, 158765023 rows, 74532702 blocks, 1331166491 bytes) into (1 part, 150607738 rows, 8585773 blocks, 241875116 bytes) in 41.289 seconds at 3845228 rows/sec to "/data/vmstorage-data/data/small/2024_12/180CE4811DBBA05C"
Dec 13 07:58:49 VM-32-202-ubuntu vmstorage-prod[3724376]: 2024-12-13T07:58:49.872+0800        info        VictoriaMetrics/lib/storage/partition.go:1465        merged (13 parts, 475287921 rows, 85656650 blocks, 1837851089 bytes) into (1 part, 466918315 rows, 8665957 blocks, 671193298 bytes) in 63.066 seconds at 7536394 rows/sec to "/data/vmstorage-data/data/big/2024_12/180CE4811DBBA095"
Dec 13 08:00:36 VM-32-202-ubuntu vmstorage-prod[3724376]: 2024-12-13T08:00:36.966+0800        info        VictoriaMetrics/lib/storage/partition.go:1465        merged (10 parts, 33292328 rows, 33068750 blocks, 519090225 bytes) into (1 part, 29787564 rows, 8546252 blocks, 199165132 bytes) in 32.160 seconds at 1035206 rows/sec to "/data/vmstorage-data/data/small/2024_12/180CE4811DBBA0B4"
Dec 13 08:03:12 VM-32-202-ubuntu vmstorage-prod[3724376]: 2024-12-13T08:03:12.525+0800        info        VictoriaMetrics/lib/storage/partition.go:1465        merged (10 parts, 6459990622 rows, 86674048 blocks, 5844373829 bytes) into (1 part, 6455221528 rows, 9580440 blocks, 2419338068 bytes) in 262.855 seconds at 24576281 rows/sec to "/data/vmstorage-data/data/big/2024_12/180CE4811DBBA0A3"
Dec 13 08:03:20 VM-32-202-ubuntu vmstorage-prod[3724376]: 2024-12-13T08:03:20.233+0800        info        VictoriaMetrics/lib/storage/partition.go:1465        merged (15 parts, 161207326 rows, 81318941 blocks, 1499783513 bytes) into (1 part, 150398002 rows, 8592127 blocks, 242330728 bytes) in 48.494 seconds at 3324271 rows/sec to "/data/vmstorage-data/data/small/2024_12/180CE4811DBBA0D5"
Dec 13 08:08:12 VM-32-202-ubuntu vmstorage-prod[3724376]: 2024-12-13T08:08:12.094+0800        info        VictoriaMetrics/lib/storage/partition.go:1465        merged (13 parts, 165862633 rows, 74121062 blocks, 1371230975 bytes) into (1 part, 157683338 rows, 8584979 blocks, 245754250 bytes) in 40.265 seconds at 4119320 rows/sec to "/data/vmstorage-data/data/small/2024_12/180CE4811DBBA118"
  1. cpu usage remains roughly unchanged Clipboard_Screenshot_1734061723