VictoriaMetrics: Breaking change in v1.90.0 is not revertable even when downgrade: `unexpected number of substrings in the part name`

Is your question request related to a specific component?

VictoriaMetrics

Describe the question in detail

Hi dev team,

fyi @vkuznet @nikodemas @leggerf @brij01

We are running VictoriaMetrics in Kubernetes as a single node. (Actually, not as one instance but for this issue I discarded other unrelated details because we’re having this issue in just one specific instance.)

Problem is that:

  • The VM instance has been running without any problem with heads-public-single-node-0-g2ac530eb2 image tag in Kubernetes, at least 30 days.
  • We updated the image tag to the latest one and started to get the unexpected number of substrings in the part name "1760590103B511E0": got 1; want 5 errors.
  • We rolled back to the previous image tag of heads-public-single-node-0-g2ac530eb2, however, now it gives cannot parse path to part: unexpected number of substrings in the part name "176058FD6ECDC531": got 1; want 3 errors.

In the CHANGELOG of v1.90.0[1], it says that we should use version greater than v1.90.0, but it does not work as I described.

Do you have any suggestion about how to solve this problem?

[1] https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.90.0

I am sharing all the k8s pod error logs in each tag trial:

1. ---- (started to use this version and started to get errors) v1.91.0 ----

Image:         victoriametrics/victoria-metrics:v1.91.0
mage ID:      docker.io/victoriametrics/victoria-metrics@sha256:44a35f77867f05ecbadd6aa8026e234a173cee68b0847b9af54b230a0c29d16a

2023-05-22T12:39:56.833Z	info	VictoriaMetrics/lib/logger/flag.go:12	build version: victoria-metrics-20230518-212023-tags-v1.91.0-0-g73dfb030d
2023-05-22T12:39:56.833Z	info	VictoriaMetrics/lib/logger/flag.go:13	command-line flags
2023-05-22T12:39:56.833Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -opentsdbHTTPListenAddr=":4242"
2023-05-22T12:39:56.833Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -retentionPeriod="1"
2023-05-22T12:39:56.833Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -search.maxConcurrentRequests="32"
2023-05-22T12:39:56.833Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -storageDataPath="/tsdb"
2023-05-22T12:39:56.833Z	info	VictoriaMetrics/app/victoria-metrics/main.go:70	starting VictoriaMetrics at ":8428"...
2023-05-22T12:39:56.833Z	info	VictoriaMetrics/app/vmstorage/main.go:108	opening storage at "/tsdb" with -retentionPeriod=1
2023-05-22T12:39:56.901Z	info	VictoriaMetrics/lib/memory/memory.go:42	limiting caches to 8894656512 bytes, leaving 5929771008 bytes to the OS according to -memory.allowedPercent=60
2023-05-22T12:40:08.493Z	info	VictoriaMetrics/lib/storage/storage.go:874	discarding /tsdb/cache/curr_hour_metric_ids, since it contains outdated hour; got 467217; want 467988
2023-05-22T12:40:08.869Z	info	VictoriaMetrics/lib/storage/storage.go:874	discarding /tsdb/cache/prev_hour_metric_ids, since it contains outdated hour; got 467216; want 467987
2023-05-22T12:40:08.883Z	info	VictoriaMetrics/lib/storage/storage.go:834	discarding /tsdb/cache/next_day_metric_ids, since it contains data for stale date; got 19467; want 19499
2023-05-22T12:40:28.361Z	panic	VictoriaMetrics/lib/storage/part_header.go:142	FATAL: cannot parse metadata from "/tsdb/data/big/2023_05/1760590103B511E0": unexpected number of substrings in the part name "1760590103B511E0": got 1; want 5
panic: FATAL: cannot parse metadata from "/tsdb/data/big/2023_05/1760590103B511E0": unexpected number of substrings in the part name "1760590103B511E0": got 1; want 5

goroutine 1 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logMessage({0xbb6863, 0x5}, {0xc02bb96780, 0x9f}, 0x2?)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:281 +0xa96
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevelSkipframes(0x1, {0xbb6863, 0x5}, {0xbd6efb?, 0xc02d2a0ba0?}, {0xc000472d40?, 0x0?, 0xc000472c88?})
	github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:138 +0x1a5
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevel(...)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:130
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.Panicf(...)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:126
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*partHeader).MustReadMetadata(0xc00002bf20, {0xc00002bef0, 0x27})
	github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/part_header.go:142 +0x353
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenFilePart({0xc00002bef0?, 0x16?})
	github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/part.go:53 +0x6f
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenParts({0xc015db6a68, 0x16}, {0xc0008d2800, 0x1c, 0xaf64c0?})
	github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/partition.go:1795 +0x42e
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenPartition({0xc015db6a50?, 0x419860?}, {0xc015db6a68, 0x16}, 0xc0004732f0?)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/partition.go:266 +0x285
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenPartitions({0xc017d3c080, 0x10}, {0xc02befe050, 0xe}, 0x0?)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/table.go:492 +0x26d
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.mustOpenTable({0xc000132e00?, 0xb9d700?}, 0xc000320680)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/table.go:109 +0x292
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.MustOpenStorage({0x7fff2f0ae533?, 0xbb6310?}, 0x9fa52400, 0x0, 0x0)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/storage.go:245 +0xe56
github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage.Init(0xd64308)
	github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:111 +0x4e5
main.main()
	github.com/VictoriaMetrics/VictoriaMetrics/app/victoria-metrics/main.go:74 +0x345


2. ---- (rolled-back to the previous working version but now it gives some errors) heads-public-single-node-0-g2ac530eb2 ----

Image:         victoriametrics/victoria-metrics:heads-public-single-node-0-g2ac530eb2
Image ID:      docker.io/victoriametrics/victoria-metrics@sha256:9cbc9a5d100ab1cb6fe735d0f6614a45f8538addc7785cc3191ca93b64b8d6ba

2023-05-22T12:43:31.626Z	info	VictoriaMetrics/lib/logger/flag.go:12	build version: victoria-metrics-20230118-091500-heads-public-single-node-0-g2ac530eb2
2023-05-22T12:43:31.626Z	info	VictoriaMetrics/lib/logger/flag.go:13	command-line flags
2023-05-22T12:43:31.626Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -opentsdbHTTPListenAddr=":4242"
2023-05-22T12:43:31.626Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -retentionPeriod="1"
2023-05-22T12:43:31.626Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -search.maxConcurrentRequests="32"
2023-05-22T12:43:31.626Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -storageDataPath="/tsdb"
2023-05-22T12:43:31.626Z	info	VictoriaMetrics/app/victoria-metrics/main.go:58	starting VictoriaMetrics at ":8428"...
2023-05-22T12:43:31.626Z	info	VictoriaMetrics/app/vmstorage/main.go:114	opening storage at "/tsdb" with -retentionPeriod=1
2023-05-22T12:43:31.629Z	info	VictoriaMetrics/lib/memory/memory.go:42	limiting caches to 8894656512 bytes, leaving 5929771008 bytes to the OS according to -memory.allowedPercent=60
2023-05-22T12:43:31.629Z	info	VictoriaMetrics/lib/storage/storage.go:1010	loading MetricName->TSID cache from "/tsdb/cache/metricName_tsid"...
2023-05-22T12:43:33.397Z	info	VictoriaMetrics/lib/storage/storage.go:1015	loaded MetricName->TSID cache from "/tsdb/cache/metricName_tsid" in 1.768 seconds; entriesCount: 4583461; sizeBytes: 3321888768
2023-05-22T12:43:33.397Z	info	VictoriaMetrics/lib/storage/storage.go:1010	loading MetricID->TSID cache from "/tsdb/cache/metricID_tsid"...
2023-05-22T12:43:33.409Z	info	VictoriaMetrics/lib/storage/storage.go:1015	loaded MetricID->TSID cache from "/tsdb/cache/metricID_tsid" in 0.012 seconds; entriesCount: 13739; sizeBytes: 33554432
2023-05-22T12:43:33.409Z	info	VictoriaMetrics/lib/storage/storage.go:1010	loading MetricID->MetricName cache from "/tsdb/cache/metricID_metricName"...
2023-05-22T12:43:33.425Z	info	VictoriaMetrics/lib/storage/storage.go:1015	loaded MetricID->MetricName cache from "/tsdb/cache/metricID_metricName" in 0.015 seconds; entriesCount: 9121; sizeBytes: 33554432
2023-05-22T12:43:33.425Z	info	VictoriaMetrics/lib/storage/storage.go:870	loading curr_hour_metric_ids from "/tsdb/cache/curr_hour_metric_ids"...
2023-05-22T12:43:33.435Z	info	VictoriaMetrics/lib/storage/storage.go:890	discarding /tsdb/cache/curr_hour_metric_ids, since it contains outdated hour; got 467217; want 467988
2023-05-22T12:43:33.435Z	info	VictoriaMetrics/lib/storage/storage.go:870	loading prev_hour_metric_ids from "/tsdb/cache/prev_hour_metric_ids"...
2023-05-22T12:43:33.449Z	info	VictoriaMetrics/lib/storage/storage.go:890	discarding /tsdb/cache/prev_hour_metric_ids, since it contains outdated hour; got 467216; want 467987
2023-05-22T12:43:33.449Z	info	VictoriaMetrics/lib/storage/storage.go:826	loading next_day_metric_ids from "/tsdb/cache/next_day_metric_ids"...
2023-05-22T12:43:33.449Z	info	VictoriaMetrics/lib/storage/storage.go:846	discarding /tsdb/cache/next_day_metric_ids, since it contains data for stale date; got 19467; want 19499
2023-05-22T12:43:33.449Z	info	VictoriaMetrics/lib/mergeset/table.go:319	opening table "/tsdb/indexdb/17579B1FC371422B"...
2023-05-22T12:43:33.463Z	fatal	VictoriaMetrics/app/vmstorage/main.go:119	cannot open a storage at /tsdb with -retentionPeriod=1: cannot open indexdb tables at "/tsdb/indexdb": cannot open curr indexdb table at "/tsdb/indexdb/17579B1FC371422B": cannot open indexDB "/tsdb/indexdb/17579B1FC371422B": cannot open table parts at "/tsdb/indexdb/17579B1FC371422B": cannot open part "/tsdb/indexdb/17579B1FC371422B/176058FD6ECDC531": cannot parse path to part: unexpected number of substrings in the part name "176058FD6ECDC531": got 1; want 3


3. ---- (tried with this version too) v1.89.1 ----
Image:         victoriametrics/victoria-metrics:v1.89.1
Image ID:      docker.io/victoriametrics/victoria-metrics@sha256:d0f855db66fe2c34455e3ac90c6350e82b3c2aa17eaf38e1fe2b4374f04bec51


2023-05-22T12:41:54.849Z	info	VictoriaMetrics/lib/logger/flag.go:12	build version: victoria-metrics-20230313-022110-tags-v1.89.1-0-g388d6ee16
2023-05-22T12:41:54.849Z	info	VictoriaMetrics/lib/logger/flag.go:13	command-line flags
2023-05-22T12:41:54.849Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -opentsdbHTTPListenAddr=":4242"
2023-05-22T12:41:54.849Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -retentionPeriod="1"
2023-05-22T12:41:54.849Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -search.maxConcurrentRequests="32"
2023-05-22T12:41:54.849Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -storageDataPath="/tsdb"
2023-05-22T12:41:54.849Z	info	VictoriaMetrics/app/victoria-metrics/main.go:61	starting VictoriaMetrics at ":8428"...
2023-05-22T12:41:54.849Z	info	VictoriaMetrics/app/vmstorage/main.go:107	opening storage at "/tsdb" with -retentionPeriod=1
2023-05-22T12:41:54.855Z	info	VictoriaMetrics/lib/memory/memory.go:42	limiting caches to 8894656512 bytes, leaving 5929771008 bytes to the OS according to -memory.allowedPercent=60
2023-05-22T12:41:54.855Z	info	VictoriaMetrics/lib/storage/storage.go:1023	loading MetricName->TSID cache from "/tsdb/cache/metricName_tsid"...
2023-05-22T12:41:56.782Z	info	VictoriaMetrics/lib/storage/storage.go:1028	loaded MetricName->TSID cache from "/tsdb/cache/metricName_tsid" in 1.927 seconds; entriesCount: 4583461; sizeBytes: 3321888768
2023-05-22T12:41:56.783Z	info	VictoriaMetrics/lib/storage/storage.go:1023	loading MetricID->TSID cache from "/tsdb/cache/metricID_tsid"...
2023-05-22T12:41:56.801Z	info	VictoriaMetrics/lib/storage/storage.go:1028	loaded MetricID->TSID cache from "/tsdb/cache/metricID_tsid" in 0.018 seconds; entriesCount: 13739; sizeBytes: 33554432
2023-05-22T12:41:56.801Z	info	VictoriaMetrics/lib/storage/storage.go:1023	loading MetricID->MetricName cache from "/tsdb/cache/metricID_metricName"...
2023-05-22T12:41:56.817Z	info	VictoriaMetrics/lib/storage/storage.go:1028	loaded MetricID->MetricName cache from "/tsdb/cache/metricID_metricName" in 0.016 seconds; entriesCount: 9121; sizeBytes: 33554432
2023-05-22T12:41:56.817Z	info	VictoriaMetrics/lib/storage/storage.go:883	loading curr_hour_metric_ids from "/tsdb/cache/curr_hour_metric_ids"...
2023-05-22T12:41:56.826Z	info	VictoriaMetrics/lib/storage/storage.go:903	discarding /tsdb/cache/curr_hour_metric_ids, since it contains outdated hour; got 467217; want 467988
2023-05-22T12:41:56.826Z	info	VictoriaMetrics/lib/storage/storage.go:883	loading prev_hour_metric_ids from "/tsdb/cache/prev_hour_metric_ids"...
2023-05-22T12:41:56.835Z	info	VictoriaMetrics/lib/storage/storage.go:903	discarding /tsdb/cache/prev_hour_metric_ids, since it contains outdated hour; got 467216; want 467987
2023-05-22T12:41:56.835Z	info	VictoriaMetrics/lib/storage/storage.go:839	loading next_day_metric_ids from "/tsdb/cache/next_day_metric_ids"...
2023-05-22T12:41:56.835Z	info	VictoriaMetrics/lib/storage/storage.go:859	discarding /tsdb/cache/next_day_metric_ids, since it contains data for stale date; got 19467; want 19499
2023-05-22T12:41:56.835Z	info	VictoriaMetrics/lib/mergeset/table.go:319	opening table "/tsdb/indexdb/17579B1FC371422B"...
2023-05-22T12:41:56.856Z	fatal	VictoriaMetrics/app/vmstorage/main.go:112	cannot open a storage at /tsdb with -retentionPeriod=1: cannot open indexdb tables at "/tsdb/indexdb": cannot open curr indexdb table at "/tsdb/indexdb/17579B1FC371422B": cannot open indexDB "/tsdb/indexdb/17579B1FC371422B": cannot open table parts at "/tsdb/indexdb/17579B1FC371422B": cannot open part "/tsdb/indexdb/17579B1FC371422B/176058FD6ECDC531": cannot parse path to part: unexpected number of substrings in the part name "176058FD6ECDC531": got 1; want 3

Troubleshooting docs

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (7 by maintainers)

Commits related to this issue

Most upvoted comments

Hello, an issue must be fixed at attached PR.

Workaround for now - to manually delete bad part directory mentioned in the error log. It shouldn’t result in any data loss.

@Amper Sorry for the late reply. For sure it is our mistake to not properly upgrade the VictoriaMetrics. Also it was having resouce problem and its k8s pod was restarted many times.

I went with the hard way by trusting our multi-instances deployment:

  • I deleted the directories named with no underscores (3 part requires 2 underscores AFAIU) from both ‘data’ and ‘indexdb’.
  • Then I started VM with previous version before the upgrade.
  • Finally, it started with no error.

Thanks for your help! You may close the issue.