VictoriaMetrics: vmstorage stop saving any metrics for the whole cluster if one instance cannot write to the disk.
Describe the bug
My cluster consists of bare-metal servers with 10 nodes for insert operations, around 20 storage nodes, and a Prometheus instance that uses remote_write to vminsert through HAProxy. Vminsert is configured with -storageNode=... -replicationFactor 2 to know all storage nodes.
At some point over the weekend, a kernel issue occurred on one of the storage nodes, leading to all RAID operations getting locked. From the perspective of the insert, the vmstorage appeared as follows:
Aug 10, 2024 @ 03:00:17.535 vminsert10 info Victoriastoragerics/app/vminsert/netstorage/netstorage.go:281 successfully dialed -storageNode="vmstorage5:8400"
Aug 10, 2024 @ 03:00:17.497 vminsert10 warn Victoriastoragerics/app/vminsert/netstorage/netstorage.go:320 cannot send 10189366 bytes with 40125 rows to -storageNode="vmstorage5:8400": cannot write data with size 10189366: cannot write data in 3 seconds: cannot flush internal buffer to the underlying writer: write tcp4 10.1.11.142:45098->10.1.193.29:8400: write: connection timed out; closing the connection to storageNode and re-routing this data to healthy storage nodes
Aug 10, 2024 @ 03:00:14.028 vminsert10 info Victoriastoragerics/app/vminsert/netstorage/netstorage.go:281 successfully dialed -storageNode="vmstorage5:8400"
Aug 10, 2024 @ 03:00:13.924 vminsert10 warn Victoriastoragerics/app/vminsert/netstorage/netstorage.go:320 cannot send 30603088 bytes with 118053 rows to -storageNode="vmstorage5:8400": cannot write data with size 30603088: cannot write data in 4 seconds: cannot flush internal buffer to the underlying writer: write tcp4 10.1.11.142:35256->10.1.193.29:8400: write: connection timed out; closing the connection to storageNode and re-routing this data to healthy storage nodes
Aug 10, 2024 @ 03:00:10.454 vminsert10 info Victoriastoragerics/app/vminsert/netstorage/netstorage.go:281 successfully dialed -storageNode="vmstorage5:8400"
This repeated continuously. The storage node vmstorage5 was unable to save anything, and the whole system was at a standstill, while other storage nodes were functioning properly.
What’s strange is that all dashboards displaying metrics through VictoriaMetrics showed nothing. After restarting the problematic host, all metrics appeared after a while. However, there was a significant spike in remote_read freshness, as if all insert nodes were just waiting for this storage node to recover and could not write anything else while it was unhealthy.
To Reproduce
I havent tried to reproduce. I suppose remount as readonly could repro, but not sure.
Version
/usr/bin/vminsert-prod --version vminsert-20240425-150352-tags-v1.101.0-cluster-0-g310d100ed
/usr/bin/vmstorage-prod --version vmstorage-20240425-150402-tags-v1.101.0-cluster-0-g310d100ed
Logs
No response
Screenshots
No response
Used command-line flags
No response
Additional information
I suppose having an exponential backoff could help in this situation, but maybe there is another solution?
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 20 (17 by maintainers)
This would help keeping code simple. If you check lib/fs library it is all filled with
Must.*-like functions, functions which throw panic if syscall failed. This is done due to the following reasons:Another reason why I like throwing panic instead of retries is that NFS is not the target FS for VM. But supporting this switch to RO will affect other FS types like ext4, which are supposed to be reliable.
The startup check makes a lot of sense to me because:
Sounds like a good idea. On start, vmstorage can test its access to FS. If it fails, then vmstorage switches to Read-Only mode with the corresponding log message. If ability to touch FS was lost on already running storage, then it has to throw fatal error and restart. On next start, it should switch to Read-only, if the problem still exists. Thoughts @f41gh7 @zekker6 ?