VictoriaMetrics: VictoriaMetrics breaks on electricity loss
Describe the bug
I am hosting VictoriaMetrics as Docker container on my intel nuc at home. Storage is NFS mountpoint (managed by Docker) on my Synology NAS:
services:
victoriametrics:
container_name: victoriametrics
image: victoriametrics/victoria-metrics:stable
restart: always
volumes:
- victoriametrics:/storage
command:
- "--storageDataPath=/storage"
volumes:
victoriametrics:
driver_opts:
type: "nfs4"
o: "addr=123.123.123.123,rsize=32768,wsize=32768,hard,timeo=600,retrans=2,noresvport,nconnect=16,vers=4.1"
device: ":/volume1/victoriametrics"
And I live in a somewhat remote location where power loss is usually around once a month or two. My intel nuc has no UPS or battery, so it simply shutdowns together with the NAS and network equipment.
Usually everything starts automatically and just works when power is restored, but this is 2nd time when this happens - VictoriaMetrics does not get up:
github.com/VictoriaMetrics/VictoriaMetrics/app/victoria-metrics/main.go:74 +0x325
2023-09-15T09:20:37.455Z info VictoriaMetrics/lib/logger/flag.go:12 build version: victoria-metrics-20230812-215739-tags-v1.93.0-0-g59f7d810c9
2023-09-15T09:20:37.455Z info VictoriaMetrics/lib/logger/flag.go:13 command-line flags
2023-09-15T09:20:37.455Z info VictoriaMetrics/lib/logger/flag.go:20 -storageDataPath="/storage"
2023-09-15T09:20:37.455Z info VictoriaMetrics/app/victoria-metrics/main.go:70 starting VictoriaMetrics at ":8428"...
2023-09-15T09:20:37.455Z info VictoriaMetrics/app/vmstorage/main.go:108 opening storage at "/storage" with -retentionPeriod=1
2023-09-15T09:20:37.457Z info VictoriaMetrics/lib/memory/memory.go:42 limiting caches to 20011091558 bytes, leaving 13340727706 bytes to the OS according to -memory.allowedPercent=60
2023-09-15T09:20:37.480Z info VictoriaMetrics/lib/storage/storage.go:935 discarding /storage/cache/curr_hour_metric_ids, since it contains outdated hour; got 470638; want 470769
2023-09-15T09:20:37.480Z info VictoriaMetrics/lib/storage/storage.go:935 discarding /storage/cache/prev_hour_metric_ids, since it contains outdated hour; got 470637; want 470768
2023-09-15T09:20:37.497Z info VictoriaMetrics/lib/storage/storage.go:896 discarding /storage/cache/next_day_metric_ids_v2, since it contains data for stale date; got 19609; want 19615
2023-09-15T09:20:37.554Z panic VictoriaMetrics/lib/storage/part_header.go:150 FATAL: cannot parse "/storage/data/small/2023_09/17835E2B9C49FBA7/metadata.json": unexpected end of JSON input
panic: FATAL: cannot parse "/storage/data/small/2023_09/17835E2B9C49FBA7/metadata.json": unexpected end of JSON input
goroutine 1 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logMessage({0xbc255d, 0x5}, {0xc00008f3b0, 0x6e}, 0x2?)
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:281 +0xa91
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevelSkipframes(0x1, {0xbc255d, 0x5}, {0xbd1159?, 0xc000388f60?}, {0xc0004dac70?, 0x0?, 0xc0004dabf8?})
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:138 +0x194
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevel(...)
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:130
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.Panicf(...)
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:126
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*partHeader).MustReadMetadata(0xc000388f60, {0xc000388f30, 0x2c})
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/part_header.go:150 +0x26c
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenFilePart({0xc000388f30?, 0x1b?})
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/part.go:53 +0x65
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenParts({0xc000342220, 0x1b}, {0xc000386000, 0x16, 0x4ca0a0?})
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/partition.go:1805 +0x433
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenPartition({0xc000342220?, 0xc0004db258?}, {0xc000342240, 0x19}, 0xc00069f140?)
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/partition.go:267 +0x246
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenPartitions({0xc00002a690, 0x13}, {0xc00002a7c8, 0x11}, 0x0?)
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/table.go:480 +0x275
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenTable({0xc0006b0080?, 0x1781cdc4ed5ead07?}, 0xc000269a00)
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/table.go:103 +0x269
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.MustOpenStorage({0x7ffd4044bf75?, 0xbc1ff6?}, 0x9fa52400, 0x0, 0x0)
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/storage.go:275 +0xfe5
github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage.Init(0xd77bf0)
github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:111 +0x505
main.main()
github.com/VictoriaMetrics/VictoriaMetrics/app/victoria-metrics/main.go:74 +0x325
Well, this is home lab and the data is not really critical - I can just delete everything and restart it, but I am raising this issue to bring awareness about unclean shutdowns and the way they break VictoriaMetrics.
Anything to suggest? Or can this issue be closed?
To Reproduce
- Host VictoriaMetrics (non cluster version), where data is mounted as NFS share via Docker.
- Setup some other hosts with Telegraf and write to VictoriaMetrics (as influxdb host, via HTTP port) every 10 seconds.
- Switch off electricity
- Switch on electricity
- Everything boots, but VictoriaMetrics fails with given error logs.
Version
Docker image: victoriametrics/victoria-metrics:stable (victoriametrics/victoria-metrics:v1.93.0)
Binaries in Docker image: victoria-metrics-20230812-215739-tags-v1.93.0-0-g59f7d810c9
Logs
Pasted above
Screenshots
N/A
Used command-line flags
Pasted above
Additional information
Everything explained above
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 2
- Comments: 18 (1 by maintainers)
VictoriaMetrics calls fsync on all the created files and directories in order to make sure the data and the directory structure is successfully written to the underlying storage. If
fsyncfails, then VictoriaMetrics logs the error and exits with non-zero exit code. It doesn’t remove the source files during the background merge in this case, so the original data isn’t lost after the restart. It automatically drops incompletely stored parts on the next start, if these parts aren’t mentioned in theparts.jsonfile.The
parts.jsonfile is updated atomically after all the parts data is successfully stored and fsync’ed to disk. The atomic update of this file is executed in the following sequence:fsync’ed to disk in order to make sure all the data is successfully persisted. This file is created in the same directory where the originalparts.jsonfile is located.parts.jsonwithrenamesyscall. This syscall is guaranteed to be atomic:I suspect that the NFS implementation you use may have the following issues:
fsyncsyscall.renamesyscall.Both issues may result in partial data loss and crashes you experience after unclean shutdown (such as power loss). I still think that there are workarounds exist for these issues - we just need to identify the real cause and then figure out the workaround. Probably it is related to NFS mount options.
Probably related NFS issues:
The point here is that VictoriaMetrics should fix itself such files, otherwise there is not much else to do other than drop NFS storage completely and switch to physically attached disks. I don’t see it in any other way. So let’s summarize this discussion/issue thread:
VictoriaMetrics (or VictoriaMetrics storage nodes) are not compatible with remote NFS storage. The only viable solution is to use physically attached disks for storage.
OP here. This issue is still happening, so eventually I decided to use
/dev/shm(in RAM) instead of NFS storage, as I do not want to write to internal SSD on my nuc.Basically this is not a corruption, but empty files and victoriametrics fails to read them. It’s not really “corruption”, but…okay, it’s corruption. However, I believe VictoriaMetrics should be able to fix itself if such case happens, because what else would you do? Would you fix yourself and continue using? It just doesn’t make sense.
I wish VictoriaMetrics devs would be able to fix such “corruption” when service starts. 🙌
@erkexzcx the mentioned fix https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5017 wasn’t released yet.
@f41gh7 would this change help to mitigate the issue? https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5017