VictoriaMetrics: vmstorage insert slow down during indexDb rotation
Describe the bug
In past two weeks, the ingestion rate of vmstorage decreased during indexDb rotation, also the concurrency increased. This phenomenon occurs every day, and it always appears at eight o’clock in the morning (the time zone is UTC+8).
To Reproduce
do nothing
Version
vmstorage-20241002-113649-tags-v1.104.0-cluster-0-gdfb2ad4ab4
Logs
no error logs
Screenshots
Here are some metrics screenshot of vmstorage and vminsert
churn rate
slow insert
-
percent
-
vmstorage insert rows
-
vmstorage slow insert rows
vmstorage cpu usage
vmstorage merge
-
vmstorage active merge
-
vmstorage merge speed
vminsert
-
vminsert concurrency
-
Storage connection saturation
Used command-line flags
The command is
/app/mon/vmstorage-prod -loggerTimezone Asia/Shanghai -retentionPeriod 90d -dedup.minScrapeInterval 15s -storageDataPath /data/vmstorage-data -httpListenAddr :8482 -vminsertAddr :8400 -vmselectAddr :8401
Additional information
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 22 (15 by maintainers)
I found there are two pattern of cpu usage among vmstorage node.