VictoriaMetrics: Importing data from Thanos via vmctl fails
Describe the bug
Trying to import data from Thanos via vmctl prometheus fails.
To Reproduce Copied data from a Thanos S3 bucket to local disk, then running:
vmctl prometheus --prom-snapshot stage/ --vm-addr http://vm-victoria-metrics-single-server.victoria-metrics:8428
fails with the error message:
2022/09/20 04:41:42 import process failed: read failed for block "01FZ80S3MKJDB1RM746X1K9V88": cannot populate chunk 8: invalid chunk encoding "<unknown>"
Expected behavior Import to succeed.
Logs
Prometheus import mode
Prometheus snapshot stats:
blocks found: 1;
blocks skipped by time filter: 0;
min time: 1646265600000 (2022-03-03T00:00:00Z);
max time: 1647475200000 (2022-03-17T00:00:00Z);
samples: 512541101;
series: 69610009.
Found 1 blocks to import. Continue? [Y/n] y
VM worker 0:↖ ? p/s
VM worker 1:↖ ? p/s
Processing blocks: 0 / 1 [▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒] 0.00%
2022/09/20 04:41:42 import process failed: read failed for block "01FZ80S3MKJDB1RM746X1K9V88": cannot populate chunk 8: invalid chunk encoding "<unknown>"
Version vmctl: vmctl version vmctl-20220908-105850-tags-v1.81.2-0-gdd1126dcc
VM server: v1.18.2
Used command-line flags See above.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (5 by maintainers)
Sorry for the late reply. I don’t have an environment right now to test this, but I can reopen the issue if I bump into this again. Thanks!
Well I did a custom build with a up-to-date prometheus lib, but issue is the same. But it seems to be logic, as the blocks are written by Thanos TSDB code, it’s not compatible with the Prometheus TSDB code.
Or the code gets adjusted to use Thanos TSDB code so it can read Thanos written blocks which are compacted/downsampled. Or the import is done via Remote Read.
Blocks that are only compacted and not downsampled can be read by vmctl without problems.