VictoriaMetrics: Breaking change in v1.90.0 is not revertable even when downgrade: `unexpected number of substrings in the part name`
Is your question request related to a specific component?
VictoriaMetrics
Describe the question in detail
Hi dev team,
fyi @vkuznet @nikodemas @leggerf @brij01
We are running VictoriaMetrics in Kubernetes as a single node. (Actually, not as one instance but for this issue I discarded other unrelated details because we’re having this issue in just one specific instance.)
Problem is that:
- The VM instance has been running without any problem with
heads-public-single-node-0-g2ac530eb2image tag in Kubernetes, at least 30 days. - We updated the image tag to the latest one and started to get the
unexpected number of substrings in the part name "1760590103B511E0": got 1; want 5errors. - We rolled back to the previous image tag of
heads-public-single-node-0-g2ac530eb2, however, now it givescannot parse path to part: unexpected number of substrings in the part name "176058FD6ECDC531": got 1; want 3errors.
In the CHANGELOG of v1.90.0[1], it says that we should use version greater than v1.90.0, but it does not work as I described.
Do you have any suggestion about how to solve this problem?
[1] https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.90.0
I am sharing all the k8s pod error logs in each tag trial:
1. ---- (started to use this version and started to get errors) v1.91.0 ----
Image: victoriametrics/victoria-metrics:v1.91.0
mage ID: docker.io/victoriametrics/victoria-metrics@sha256:44a35f77867f05ecbadd6aa8026e234a173cee68b0847b9af54b230a0c29d16a
2023-05-22T12:39:56.833Z info VictoriaMetrics/lib/logger/flag.go:12 build version: victoria-metrics-20230518-212023-tags-v1.91.0-0-g73dfb030d
2023-05-22T12:39:56.833Z info VictoriaMetrics/lib/logger/flag.go:13 command-line flags
2023-05-22T12:39:56.833Z info VictoriaMetrics/lib/logger/flag.go:20 -opentsdbHTTPListenAddr=":4242"
2023-05-22T12:39:56.833Z info VictoriaMetrics/lib/logger/flag.go:20 -retentionPeriod="1"
2023-05-22T12:39:56.833Z info VictoriaMetrics/lib/logger/flag.go:20 -search.maxConcurrentRequests="32"
2023-05-22T12:39:56.833Z info VictoriaMetrics/lib/logger/flag.go:20 -storageDataPath="/tsdb"
2023-05-22T12:39:56.833Z info VictoriaMetrics/app/victoria-metrics/main.go:70 starting VictoriaMetrics at ":8428"...
2023-05-22T12:39:56.833Z info VictoriaMetrics/app/vmstorage/main.go:108 opening storage at "/tsdb" with -retentionPeriod=1
2023-05-22T12:39:56.901Z info VictoriaMetrics/lib/memory/memory.go:42 limiting caches to 8894656512 bytes, leaving 5929771008 bytes to the OS according to -memory.allowedPercent=60
2023-05-22T12:40:08.493Z info VictoriaMetrics/lib/storage/storage.go:874 discarding /tsdb/cache/curr_hour_metric_ids, since it contains outdated hour; got 467217; want 467988
2023-05-22T12:40:08.869Z info VictoriaMetrics/lib/storage/storage.go:874 discarding /tsdb/cache/prev_hour_metric_ids, since it contains outdated hour; got 467216; want 467987
2023-05-22T12:40:08.883Z info VictoriaMetrics/lib/storage/storage.go:834 discarding /tsdb/cache/next_day_metric_ids, since it contains data for stale date; got 19467; want 19499
2023-05-22T12:40:28.361Z panic VictoriaMetrics/lib/storage/part_header.go:142 FATAL: cannot parse metadata from "/tsdb/data/big/2023_05/1760590103B511E0": unexpected number of substrings in the part name "1760590103B511E0": got 1; want 5
panic: FATAL: cannot parse metadata from "/tsdb/data/big/2023_05/1760590103B511E0": unexpected number of substrings in the part name "1760590103B511E0": got 1; want 5
goroutine 1 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logMessage({0xbb6863, 0x5}, {0xc02bb96780, 0x9f}, 0x2?)
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:281 +0xa96
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevelSkipframes(0x1, {0xbb6863, 0x5}, {0xbd6efb?, 0xc02d2a0ba0?}, {0xc000472d40?, 0x0?, 0xc000472c88?})
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:138 +0x1a5
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevel(...)
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:130
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.Panicf(...)
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:126
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*partHeader).MustReadMetadata(0xc00002bf20, {0xc00002bef0, 0x27})
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/part_header.go:142 +0x353
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenFilePart({0xc00002bef0?, 0x16?})
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/part.go:53 +0x6f
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenParts({0xc015db6a68, 0x16}, {0xc0008d2800, 0x1c, 0xaf64c0?})
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/partition.go:1795 +0x42e
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenPartition({0xc015db6a50?, 0x419860?}, {0xc015db6a68, 0x16}, 0xc0004732f0?)
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/partition.go:266 +0x285
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenPartitions({0xc017d3c080, 0x10}, {0xc02befe050, 0xe}, 0x0?)
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/table.go:492 +0x26d
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenTable({0xc000132e00?, 0xb9d700?}, 0xc000320680)
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/table.go:109 +0x292
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.MustOpenStorage({0x7fff2f0ae533?, 0xbb6310?}, 0x9fa52400, 0x0, 0x0)
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/storage.go:245 +0xe56
github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage.Init(0xd64308)
github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:111 +0x4e5
main.main()
github.com/VictoriaMetrics/VictoriaMetrics/app/victoria-metrics/main.go:74 +0x345
2. ---- (rolled-back to the previous working version but now it gives some errors) heads-public-single-node-0-g2ac530eb2 ----
Image: victoriametrics/victoria-metrics:heads-public-single-node-0-g2ac530eb2
Image ID: docker.io/victoriametrics/victoria-metrics@sha256:9cbc9a5d100ab1cb6fe735d0f6614a45f8538addc7785cc3191ca93b64b8d6ba
2023-05-22T12:43:31.626Z info VictoriaMetrics/lib/logger/flag.go:12 build version: victoria-metrics-20230118-091500-heads-public-single-node-0-g2ac530eb2
2023-05-22T12:43:31.626Z info VictoriaMetrics/lib/logger/flag.go:13 command-line flags
2023-05-22T12:43:31.626Z info VictoriaMetrics/lib/logger/flag.go:20 -opentsdbHTTPListenAddr=":4242"
2023-05-22T12:43:31.626Z info VictoriaMetrics/lib/logger/flag.go:20 -retentionPeriod="1"
2023-05-22T12:43:31.626Z info VictoriaMetrics/lib/logger/flag.go:20 -search.maxConcurrentRequests="32"
2023-05-22T12:43:31.626Z info VictoriaMetrics/lib/logger/flag.go:20 -storageDataPath="/tsdb"
2023-05-22T12:43:31.626Z info VictoriaMetrics/app/victoria-metrics/main.go:58 starting VictoriaMetrics at ":8428"...
2023-05-22T12:43:31.626Z info VictoriaMetrics/app/vmstorage/main.go:114 opening storage at "/tsdb" with -retentionPeriod=1
2023-05-22T12:43:31.629Z info VictoriaMetrics/lib/memory/memory.go:42 limiting caches to 8894656512 bytes, leaving 5929771008 bytes to the OS according to -memory.allowedPercent=60
2023-05-22T12:43:31.629Z info VictoriaMetrics/lib/storage/storage.go:1010 loading MetricName->TSID cache from "/tsdb/cache/metricName_tsid"...
2023-05-22T12:43:33.397Z info VictoriaMetrics/lib/storage/storage.go:1015 loaded MetricName->TSID cache from "/tsdb/cache/metricName_tsid" in 1.768 seconds; entriesCount: 4583461; sizeBytes: 3321888768
2023-05-22T12:43:33.397Z info VictoriaMetrics/lib/storage/storage.go:1010 loading MetricID->TSID cache from "/tsdb/cache/metricID_tsid"...
2023-05-22T12:43:33.409Z info VictoriaMetrics/lib/storage/storage.go:1015 loaded MetricID->TSID cache from "/tsdb/cache/metricID_tsid" in 0.012 seconds; entriesCount: 13739; sizeBytes: 33554432
2023-05-22T12:43:33.409Z info VictoriaMetrics/lib/storage/storage.go:1010 loading MetricID->MetricName cache from "/tsdb/cache/metricID_metricName"...
2023-05-22T12:43:33.425Z info VictoriaMetrics/lib/storage/storage.go:1015 loaded MetricID->MetricName cache from "/tsdb/cache/metricID_metricName" in 0.015 seconds; entriesCount: 9121; sizeBytes: 33554432
2023-05-22T12:43:33.425Z info VictoriaMetrics/lib/storage/storage.go:870 loading curr_hour_metric_ids from "/tsdb/cache/curr_hour_metric_ids"...
2023-05-22T12:43:33.435Z info VictoriaMetrics/lib/storage/storage.go:890 discarding /tsdb/cache/curr_hour_metric_ids, since it contains outdated hour; got 467217; want 467988
2023-05-22T12:43:33.435Z info VictoriaMetrics/lib/storage/storage.go:870 loading prev_hour_metric_ids from "/tsdb/cache/prev_hour_metric_ids"...
2023-05-22T12:43:33.449Z info VictoriaMetrics/lib/storage/storage.go:890 discarding /tsdb/cache/prev_hour_metric_ids, since it contains outdated hour; got 467216; want 467987
2023-05-22T12:43:33.449Z info VictoriaMetrics/lib/storage/storage.go:826 loading next_day_metric_ids from "/tsdb/cache/next_day_metric_ids"...
2023-05-22T12:43:33.449Z info VictoriaMetrics/lib/storage/storage.go:846 discarding /tsdb/cache/next_day_metric_ids, since it contains data for stale date; got 19467; want 19499
2023-05-22T12:43:33.449Z info VictoriaMetrics/lib/mergeset/table.go:319 opening table "/tsdb/indexdb/17579B1FC371422B"...
2023-05-22T12:43:33.463Z fatal VictoriaMetrics/app/vmstorage/main.go:119 cannot open a storage at /tsdb with -retentionPeriod=1: cannot open indexdb tables at "/tsdb/indexdb": cannot open curr indexdb table at "/tsdb/indexdb/17579B1FC371422B": cannot open indexDB "/tsdb/indexdb/17579B1FC371422B": cannot open table parts at "/tsdb/indexdb/17579B1FC371422B": cannot open part "/tsdb/indexdb/17579B1FC371422B/176058FD6ECDC531": cannot parse path to part: unexpected number of substrings in the part name "176058FD6ECDC531": got 1; want 3
3. ---- (tried with this version too) v1.89.1 ----
Image: victoriametrics/victoria-metrics:v1.89.1
Image ID: docker.io/victoriametrics/victoria-metrics@sha256:d0f855db66fe2c34455e3ac90c6350e82b3c2aa17eaf38e1fe2b4374f04bec51
2023-05-22T12:41:54.849Z info VictoriaMetrics/lib/logger/flag.go:12 build version: victoria-metrics-20230313-022110-tags-v1.89.1-0-g388d6ee16
2023-05-22T12:41:54.849Z info VictoriaMetrics/lib/logger/flag.go:13 command-line flags
2023-05-22T12:41:54.849Z info VictoriaMetrics/lib/logger/flag.go:20 -opentsdbHTTPListenAddr=":4242"
2023-05-22T12:41:54.849Z info VictoriaMetrics/lib/logger/flag.go:20 -retentionPeriod="1"
2023-05-22T12:41:54.849Z info VictoriaMetrics/lib/logger/flag.go:20 -search.maxConcurrentRequests="32"
2023-05-22T12:41:54.849Z info VictoriaMetrics/lib/logger/flag.go:20 -storageDataPath="/tsdb"
2023-05-22T12:41:54.849Z info VictoriaMetrics/app/victoria-metrics/main.go:61 starting VictoriaMetrics at ":8428"...
2023-05-22T12:41:54.849Z info VictoriaMetrics/app/vmstorage/main.go:107 opening storage at "/tsdb" with -retentionPeriod=1
2023-05-22T12:41:54.855Z info VictoriaMetrics/lib/memory/memory.go:42 limiting caches to 8894656512 bytes, leaving 5929771008 bytes to the OS according to -memory.allowedPercent=60
2023-05-22T12:41:54.855Z info VictoriaMetrics/lib/storage/storage.go:1023 loading MetricName->TSID cache from "/tsdb/cache/metricName_tsid"...
2023-05-22T12:41:56.782Z info VictoriaMetrics/lib/storage/storage.go:1028 loaded MetricName->TSID cache from "/tsdb/cache/metricName_tsid" in 1.927 seconds; entriesCount: 4583461; sizeBytes: 3321888768
2023-05-22T12:41:56.783Z info VictoriaMetrics/lib/storage/storage.go:1023 loading MetricID->TSID cache from "/tsdb/cache/metricID_tsid"...
2023-05-22T12:41:56.801Z info VictoriaMetrics/lib/storage/storage.go:1028 loaded MetricID->TSID cache from "/tsdb/cache/metricID_tsid" in 0.018 seconds; entriesCount: 13739; sizeBytes: 33554432
2023-05-22T12:41:56.801Z info VictoriaMetrics/lib/storage/storage.go:1023 loading MetricID->MetricName cache from "/tsdb/cache/metricID_metricName"...
2023-05-22T12:41:56.817Z info VictoriaMetrics/lib/storage/storage.go:1028 loaded MetricID->MetricName cache from "/tsdb/cache/metricID_metricName" in 0.016 seconds; entriesCount: 9121; sizeBytes: 33554432
2023-05-22T12:41:56.817Z info VictoriaMetrics/lib/storage/storage.go:883 loading curr_hour_metric_ids from "/tsdb/cache/curr_hour_metric_ids"...
2023-05-22T12:41:56.826Z info VictoriaMetrics/lib/storage/storage.go:903 discarding /tsdb/cache/curr_hour_metric_ids, since it contains outdated hour; got 467217; want 467988
2023-05-22T12:41:56.826Z info VictoriaMetrics/lib/storage/storage.go:883 loading prev_hour_metric_ids from "/tsdb/cache/prev_hour_metric_ids"...
2023-05-22T12:41:56.835Z info VictoriaMetrics/lib/storage/storage.go:903 discarding /tsdb/cache/prev_hour_metric_ids, since it contains outdated hour; got 467216; want 467987
2023-05-22T12:41:56.835Z info VictoriaMetrics/lib/storage/storage.go:839 loading next_day_metric_ids from "/tsdb/cache/next_day_metric_ids"...
2023-05-22T12:41:56.835Z info VictoriaMetrics/lib/storage/storage.go:859 discarding /tsdb/cache/next_day_metric_ids, since it contains data for stale date; got 19467; want 19499
2023-05-22T12:41:56.835Z info VictoriaMetrics/lib/mergeset/table.go:319 opening table "/tsdb/indexdb/17579B1FC371422B"...
2023-05-22T12:41:56.856Z fatal VictoriaMetrics/app/vmstorage/main.go:112 cannot open a storage at /tsdb with -retentionPeriod=1: cannot open indexdb tables at "/tsdb/indexdb": cannot open curr indexdb table at "/tsdb/indexdb/17579B1FC371422B": cannot open indexDB "/tsdb/indexdb/17579B1FC371422B": cannot open table parts at "/tsdb/indexdb/17579B1FC371422B": cannot open part "/tsdb/indexdb/17579B1FC371422B/176058FD6ECDC531": cannot parse path to part: unexpected number of substrings in the part name "176058FD6ECDC531": got 1; want 3
Troubleshooting docs
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (7 by maintainers)
Commits related to this issue
- lib/storage: creates parts.json on start-up if it not exists. It fixes migrations from versions below v1.90.0. Previously parts.json was created only after successful merge. But if merge was interrupe... — committed to VictoriaMetrics/VictoriaMetrics by f41gh7 3 years ago
- lib/storage: creates parts.json on start-up if it not exists. (#4450) * lib/storage: creates parts.json on start-up if it not exists. It fixes migrations from versions below v1.90.0. Previously par... — committed to VictoriaMetrics/VictoriaMetrics by f41gh7 3 years ago
- lib/storage: creates parts.json on start-up if it not exists. (#4450) * lib/storage: creates parts.json on start-up if it not exists. It fixes migrations from versions below v1.90.0. Previously parts... — committed to VictoriaMetrics/VictoriaMetrics by f41gh7 3 years ago
- docs/CHANGELOG.md: clarify description for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4336 bugfix This is a follow-up for 5eb5df96e26bae3fbf771afbe9eee313c45ad25d — committed to VictoriaMetrics/VictoriaMetrics by valyala 3 years ago
- docs/CHANGELOG.md: clarify description for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4336 bugfix This is a follow-up for 5eb5df96e26bae3fbf771afbe9eee313c45ad25d — committed to VictoriaMetrics/VictoriaMetrics by valyala 3 years ago
- lib/storage: creates parts.json on start-up if it not exists. (#4450) * lib/storage: creates parts.json on start-up if it not exists. It fixes migrations from versions below v1.90.0. Previously parts... — committed to VictoriaMetrics/VictoriaMetrics by f41gh7 3 years ago
Hello, an issue must be fixed at attached PR.
Workaround for now - to manually delete bad part directory mentioned in the error log. It shouldn’t result in any data loss.
@Amper Sorry for the late reply. For sure it is our mistake to not properly upgrade the VictoriaMetrics. Also it was having resouce problem and its k8s pod was restarted many times.
I went with the hard way by trusting our multi-instances deployment:
Thanks for your help! You may close the issue.