VictoriaMetrics: scaling vm-storage leads into crashloop instead of read only mode
Describe the bug Victoria metrics in a cluster mode, deployed into kubernetes.
Scaling victoria metrics storage leads into crashloop of the added replica.
To Reproduce Deploy victoria metrics cluster with 2 storages. Set the PVC to 200 mb. Default limit is 10 000 000 bytes. Wait until both of them are in read only mode.
Then, start scaling. Add 3rd storage. Add the URL of it into vm-select and then also for vm-insert.
Expected behavior New replica 1/1 running and cluster functional again.
Logs
{"ts":"2022-05-17T11:04:45.415Z","level":"panic","caller":"VictoriaMetrics/lib/filestream/filestream.go:222","msg":"FATAL: cannot flush buffered data to file \"/storage/indexdb/16EFDE8D15D469C4/tmp/16EFDFA463C7A159/index.bin\": write /storage/indexdb/16EFDE8D15D469C4/tmp/16EFDFA463C7A159/index.bin: no space left on device"}
Version Using image version v1.75.0-cluster.
Used command-line flags
Args:
--retentionPeriod=1
--storageDataPath=/storage
--envflag.enable=true
--envflag.prefix=VM_
--loggerFormat=json
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 15 (1 by maintainers)
Commits related to this issue
- lib/storage: stop background merge when storage enters read-only mode This should prevent from `no space left on device` errors when VictoriaMetrics under-estimates the additional disk space needed f... — committed to VictoriaMetrics/VictoriaMetrics by valyala 4 years ago
- lib/storage: stop background merge when storage enters read-only mode This should prevent from `no space left on device` errors when VictoriaMetrics under-estimates the additional disk space needed f... — committed to VictoriaMetrics/VictoriaMetrics by valyala 4 years ago
- issue-7599: do not force read-only for legacy idbs (#9355) `isReadOnly` flag does not prevent writes (which we need to perform in case of deletion), it just [disables background merges](https://githu... — committed to VictoriaMetrics/VictoriaMetrics by tIGO a year ago
- issue-7599: do not force read-only for legacy idbs (#9355) `isReadOnly` flag does not prevent writes (which we need to perform in case of deletion), it just [disables background merges](https://githu... — committed to VictoriaMetrics/VictoriaMetrics by rtm0 a year ago
- issue-7599: do not force read-only for legacy idbs (#9355) `isReadOnly` flag does not prevent writes (which we need to perform in case of deletion), it just [disables background merges](https://githu... — committed to VictoriaMetrics/VictoriaMetrics by rtm0 a year ago
- issue-7599: do not force read-only for legacy idbs (#9355) `isReadOnly` flag does not prevent writes (which we need to perform in case of deletion), it just [disables background merges](https://githu... — committed to VictoriaMetrics/VictoriaMetrics by rtm0 a year ago
- issue-7599: do not force read-only for legacy idbs (#9355) `isReadOnly` flag does not prevent writes (which we need to perform in case of deletion), it just [disables background merges](https://githu... — committed to VictoriaMetrics/VictoriaMetrics by rtm0 a year ago
- issue-7599: do not force read-only for legacy idbs (#9355) `isReadOnly` flag does not prevent writes (which we need to perform in case of deletion), it just [disables background merges](https://githu... — committed to VictoriaMetrics/VictoriaMetrics by rtm0 a year ago
I also tried this by deploying 12 storage pods at the same time, with the same configuration as in the previous message, and 1 out of 12 crashed, but just once, and then all were in read only mode and not in crashloop.
@valyala
I did an experiment with this and it looks promising. Initially I had 2 storage pods. The size of pvc was 100 mb and the limit was 10 mb. Load was 1 million timeseries, so they got filled up really fast.
The storage-0 did crash twice, but then it was running for ~30 minutes without new crashes.
This is the last two log lines from the second run, after the first crash:
The storage-1 did crash once, and it also was running just fine after ~30 minutes without new crashes.
Then I scaled out with +2 storage pods. The 3rd did not crash at all, but fourth did crash. These is the last two log lines from the 4th.
Then I added +2 for fifth and sixth, then again +2 for seventh and eighth, and then +4 to get total of 12. No more crashes.
So I now have 12 storage pods that all are in read only mode, but they are not crashlooping. This is much better!
This also happens with version 1.77.
I made an experiment with 10 storage pods, 8 Gb PVC size and then the limit was 800 Mb, and 3 out of 10 storage pods went into crashloop and rest into read only mode.