VictoriaMetrics: scaling vm-storage leads into crashloop instead of read only mode

Describe the bug Victoria metrics in a cluster mode, deployed into kubernetes.

Scaling victoria metrics storage leads into crashloop of the added replica.

To Reproduce Deploy victoria metrics cluster with 2 storages. Set the PVC to 200 mb. Default limit is 10 000 000 bytes. Wait until both of them are in read only mode.

Then, start scaling. Add 3rd storage. Add the URL of it into vm-select and then also for vm-insert.

Expected behavior New replica 1/1 running and cluster functional again.

Logs

{"ts":"2022-05-17T11:04:45.415Z","level":"panic","caller":"VictoriaMetrics/lib/filestream/filestream.go:222","msg":"FATAL: cannot flush buffered data to file \"/storage/indexdb/16EFDE8D15D469C4/tmp/16EFDFA463C7A159/index.bin\": write /storage/indexdb/16EFDE8D15D469C4/tmp/16EFDFA463C7A159/index.bin: no space left on device"}

Version Using image version v1.75.0-cluster.

Used command-line flags

    Args:
      --retentionPeriod=1
      --storageDataPath=/storage
      --envflag.enable=true
      --envflag.prefix=VM_
      --loggerFormat=json

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 15 (1 by maintainers)

Commits related to this issue

Most upvoted comments

I also tried this by deploying 12 storage pods at the same time, with the same configuration as in the previous message, and 1 out of 12 crashed, but just once, and then all were in read only mode and not in crashloop.

@valyala

I did an experiment with this and it looks promising. Initially I had 2 storage pods. The size of pvc was 100 mb and the limit was 10 mb. Load was 1 million timeseries, so they got filled up really fast.

The storage-0 did crash twice, but then it was running for ~30 minutes without new crashes.

This is the last two log lines from the second run, after the first crash:

{
  "ts": "2022-06-02T08:00:38.910Z",
  "level": "warn",
  "caller": "/vmcode/VictoriaMetrics-cluster/lib/storage/storage.go:640",
  "msg": "switching the storage at /storage to read-only mode, since it has less than -storage.minFreeDiskSpaceBytes=10485760 of free space: 31744 bytes left"
}
{
  "ts": "2022-06-02T08:00:38.911Z",
  "level": "panic",
  "caller": "/vmcode/VictoriaMetrics-cluster/lib/fs/fs.go:334",
  "msg": "FATAL: cannot write 14819 bytes: write /storage/indexdb/16F4BEAD7AE6E677/tmp/16F4BEC15A604CF1/items.bin: no space left on device"
}

The storage-1 did crash once, and it also was running just fine after ~30 minutes without new crashes.

{
  "ts": "2022-06-02T07:59:10.210Z",
  "level": "warn",
  "caller": "/vmcode/VictoriaMetrics-cluster/lib/storage/storage.go:640",
  "msg": "switching the storage at /storage to read-only mode, since it has less than -storage.minFreeDiskSpaceBytes=10485760 of free space: 145408 bytes left"
}
{
  "ts": "2022-06-02T07:59:10.314Z",
  "level": "panic",
  "caller": "/vmcode/VictoriaMetrics-cluster/lib/fs/fs.go:334",
  "msg": "FATAL: cannot write 13070 bytes: write /storage/data/small/2022_06/tmp/16F4BEB5826BABE4/index.bin: no space left on device"
}

Then I scaled out with +2 storage pods. The 3rd did not crash at all, but fourth did crash. These is the last two log lines from the 4th.

{
  "ts": "2022-06-02T08:06:56.153Z",
  "level": "warn",
  "caller": "/vmcode/VictoriaMetrics-cluster/lib/storage/storage.go:640",
  "msg": "switching the storage at /storage to read-only mode, since it has less than -storage.minFreeDiskSpaceBytes=10485760 of free space: 1157120 bytes left"
}
{
  "ts": "2022-06-02T08:06:57.027Z",
  "level": "panic",
  "caller": "/vmcode/VictoriaMetrics-cluster/lib/fs/fs.go:334",
  "msg": "FATAL: cannot write 4166 bytes: write /storage/indexdb/16F4BF1B10437A2C/tmp/16F4BF1B1171BDDF/items.bin: no space left on device"
}

Then I added +2 for fifth and sixth, then again +2 for seventh and eighth, and then +4 to get total of 12. No more crashes.

So I now have 12 storage pods that all are in read only mode, but they are not crashlooping. This is much better!

This also happens with version 1.77.

I made an experiment with 10 storage pods, 8 Gb PVC size and then the limit was 800 Mb, and 3 out of 10 storage pods went into crashloop and rest into read only mode.