VictoriaMetrics: VictoriaMetrics breaks on electricity loss

Describe the bug

I am hosting VictoriaMetrics as Docker container on my intel nuc at home. Storage is NFS mountpoint (managed by Docker) on my Synology NAS:

services:
  victoriametrics:
    container_name: victoriametrics
    image: victoriametrics/victoria-metrics:stable
    restart: always
    volumes:
      - victoriametrics:/storage
    command:
      - "--storageDataPath=/storage"

volumes:
  victoriametrics:
    driver_opts:
      type: "nfs4"
      o: "addr=123.123.123.123,rsize=32768,wsize=32768,hard,timeo=600,retrans=2,noresvport,nconnect=16,vers=4.1"
      device: ":/volume1/victoriametrics"

And I live in a somewhat remote location where power loss is usually around once a month or two. My intel nuc has no UPS or battery, so it simply shutdowns together with the NAS and network equipment.

Usually everything starts automatically and just works when power is restored, but this is 2nd time when this happens - VictoriaMetrics does not get up:

        github.com/VictoriaMetrics/VictoriaMetrics/app/victoria-metrics/main.go:74 +0x325
2023-09-15T09:20:37.455Z        info    VictoriaMetrics/lib/logger/flag.go:12   build version: victoria-metrics-20230812-215739-tags-v1.93.0-0-g59f7d810c9
2023-09-15T09:20:37.455Z        info    VictoriaMetrics/lib/logger/flag.go:13   command-line flags
2023-09-15T09:20:37.455Z        info    VictoriaMetrics/lib/logger/flag.go:20     -storageDataPath="/storage"
2023-09-15T09:20:37.455Z        info    VictoriaMetrics/app/victoria-metrics/main.go:70 starting VictoriaMetrics at ":8428"...
2023-09-15T09:20:37.455Z        info    VictoriaMetrics/app/vmstorage/main.go:108       opening storage at "/storage" with -retentionPeriod=1
2023-09-15T09:20:37.457Z        info    VictoriaMetrics/lib/memory/memory.go:42 limiting caches to 20011091558 bytes, leaving 13340727706 bytes to the OS according to -memory.allowedPercent=60
2023-09-15T09:20:37.480Z        info    VictoriaMetrics/lib/storage/storage.go:935      discarding /storage/cache/curr_hour_metric_ids, since it contains outdated hour; got 470638; want 470769
2023-09-15T09:20:37.480Z        info    VictoriaMetrics/lib/storage/storage.go:935      discarding /storage/cache/prev_hour_metric_ids, since it contains outdated hour; got 470637; want 470768
2023-09-15T09:20:37.497Z        info    VictoriaMetrics/lib/storage/storage.go:896      discarding /storage/cache/next_day_metric_ids_v2, since it contains data for stale date; got 19609; want 19615
2023-09-15T09:20:37.554Z        panic   VictoriaMetrics/lib/storage/part_header.go:150  FATAL: cannot parse "/storage/data/small/2023_09/17835E2B9C49FBA7/metadata.json": unexpected end of JSON input
panic: FATAL: cannot parse "/storage/data/small/2023_09/17835E2B9C49FBA7/metadata.json": unexpected end of JSON input

goroutine 1 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logMessage({0xbc255d, 0x5}, {0xc00008f3b0, 0x6e}, 0x2?)
        github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:281 +0xa91
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevelSkipframes(0x1, {0xbc255d, 0x5}, {0xbd1159?, 0xc000388f60?}, {0xc0004dac70?, 0x0?, 0xc0004dabf8?})
        github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:138 +0x194
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevel(...)
        github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:130
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.Panicf(...)
        github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:126
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*partHeader).MustReadMetadata(0xc000388f60, {0xc000388f30, 0x2c})
        github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/part_header.go:150 +0x26c
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenFilePart({0xc000388f30?, 0x1b?})
        github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/part.go:53 +0x65
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenParts({0xc000342220, 0x1b}, {0xc000386000, 0x16, 0x4ca0a0?})
        github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/partition.go:1805 +0x433
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenPartition({0xc000342220?, 0xc0004db258?}, {0xc000342240, 0x19}, 0xc00069f140?)
        github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/partition.go:267 +0x246
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenPartitions({0xc00002a690, 0x13}, {0xc00002a7c8, 0x11}, 0x0?)
        github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/table.go:480 +0x275
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenTable({0xc0006b0080?, 0x1781cdc4ed5ead07?}, 0xc000269a00)
        github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/table.go:103 +0x269
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.MustOpenStorage({0x7ffd4044bf75?, 0xbc1ff6?}, 0x9fa52400, 0x0, 0x0)
        github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/storage.go:275 +0xfe5
github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage.Init(0xd77bf0)
        github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:111 +0x505
main.main()
        github.com/VictoriaMetrics/VictoriaMetrics/app/victoria-metrics/main.go:74 +0x325

Well, this is home lab and the data is not really critical - I can just delete everything and restart it, but I am raising this issue to bring awareness about unclean shutdowns and the way they break VictoriaMetrics.

Anything to suggest? Or can this issue be closed?

To Reproduce

  1. Host VictoriaMetrics (non cluster version), where data is mounted as NFS share via Docker.
  2. Setup some other hosts with Telegraf and write to VictoriaMetrics (as influxdb host, via HTTP port) every 10 seconds.
  3. Switch off electricity
  4. Switch on electricity
  5. Everything boots, but VictoriaMetrics fails with given error logs.

Version

Docker image: victoriametrics/victoria-metrics:stable (victoriametrics/victoria-metrics:v1.93.0) Binaries in Docker image: victoria-metrics-20230812-215739-tags-v1.93.0-0-g59f7d810c9

Logs

Pasted above

Screenshots

N/A

Used command-line flags

Pasted above

Additional information

Everything explained above

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 2
  • Comments: 18 (1 by maintainers)

Most upvoted comments

VictoriaMetrics calls fsync on all the created files and directories in order to make sure the data and the directory structure is successfully written to the underlying storage. If fsync fails, then VictoriaMetrics logs the error and exits with non-zero exit code. It doesn’t remove the source files during the background merge in this case, so the original data isn’t lost after the restart. It automatically drops incompletely stored parts on the next start, if these parts aren’t mentioned in the parts.json file.

The parts.json file is updated atomically after all the parts data is successfully stored and fsync’ed to disk. The atomic update of this file is executed in the following sequence:

  • A temporary file is created with the updated list of parts. This file is fsync’ed to disk in order to make sure all the data is successfully persisted. This file is created in the same directory where the original parts.json file is located.
  • The temporary file is atomically moved to parts.json with rename syscall. This syscall is guaranteed to be atomic:

If newpath already exists, it will be atomically replaced, so that there is no point at which another process attempting to access newpath will find it missing. However, there will probably be a window in which both oldpath and newpath refer to the file being renamed.

I suspect that the NFS implementation you use may have the following issues:

  • It doesn’t guarantee that the data is successfully written to the remote storage after returning from fsync syscall.
  • It doesn’t guarantee that the file is successfully renamed at the remote storage after returning from rename syscall.

Both issues may result in partial data loss and crashes you experience after unclean shutdown (such as power loss). I still think that there are workarounds exist for these issues - we just need to identify the real cause and then figure out the workaround. Probably it is related to NFS mount options.

Probably related NFS issues:

Hi @hagen1778 I partly agree, but a auto-healing of obviously wrong files would be nice to have too. On my side I have basically the same setup as @erkexzcx and had the same fault too. Maybe not VM’s fault, but auto repair would be plenty nice in order not to have another elastic-like DRP scenario

The point here is that VictoriaMetrics should fix itself such files, otherwise there is not much else to do other than drop NFS storage completely and switch to physically attached disks. I don’t see it in any other way. So let’s summarize this discussion/issue thread:

VictoriaMetrics (or VictoriaMetrics storage nodes) are not compatible with remote NFS storage. The only viable solution is to use physically attached disks for storage.

auto-deleting corrupted files hides the root cause of corruption

OP here. This issue is still happening, so eventually I decided to use /dev/shm (in RAM) instead of NFS storage, as I do not want to write to internal SSD on my nuc.

Basically this is not a corruption, but empty files and victoriametrics fails to read them. It’s not really “corruption”, but…okay, it’s corruption. However, I believe VictoriaMetrics should be able to fix itself if such case happens, because what else would you do? Would you fix yourself and continue using? It just doesn’t make sense.

I wish VictoriaMetrics devs would be able to fix such “corruption” when service starts. 🙌