thanos: tools bucket rewrite: invalid memory address or nil pointer dereference

I’m trying the series deletion feature and got the following error.

level=info ts=2021-06-03T04:34:56.077967157Z caller=factory.go:46 msg="loading bucket configuration"
level=info ts=2021-06-03T04:34:56.116875761Z caller=tools_bucket.go:868 msg="downloading block" source=01F6NY6XFBHZSQ159ZYF5FGE61
level=info ts=2021-06-03T04:34:59.814395054Z caller=tools_bucket.go:904 msg="changelog will be available" file=/tmp/thanos-rewrite/01F782EC369F5RPSZQSAZ45CQ5/change.log
level=info ts=2021-06-03T04:34:59.831787434Z caller=tools_bucket.go:919 msg="starting rewrite for block" source=01F6NY6XFBHZSQ159ZYF5FGE61 new=01F782EC369F5RPSZQSAZ45CQ5 toDelete="- matchers: \"{__name__=~\\\"mqtt2tsdb_.*\\\",gateway=\\\"LG01010002012110100\\\"}\""
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x17aba71]

goroutine 101 [running]:
github.com/thanos-io/thanos/pkg/compactv2.(*lazyPopulatableChunk).Bytes(0xc0000a6240, 0x8, 0xc0000d91c8, 0x40db9b)
        /app/pkg/compactv2/chunk_series_set.go:119 +0x31
github.com/prometheus/prometheus/tsdb/chunks.(*Writer).WriteChunks(0xc0000d0960, 0xc0000cc140, 0x5, 0x8, 0xa0, 0xc0000cc140)
        /go/pkg/mod/github.com/prometheus/prometheus@v1.8.2-0.20210421143221-52df5ef7a3be/tsdb/chunks/chunks.go:302 +0x11a
github.com/thanos-io/thanos/pkg/block.(*statsGatheringSeriesWriter).WriteChunks(0xc00003afc0, 0xc0000cc140, 0x5, 0x8, 0x0, 0x0)
        /app/pkg/block/writer.go:172 +0x5f
github.com/thanos-io/thanos/pkg/compactv2.(*Compactor).write(0xc0000d9d68, 0x1fa0228, 0xc000862c00, 0x1f96f50, 0xc0000d0000, 0x1fa0f80, 0xc0000d0050, 0x7f74fb69eff0, 0xc00003afc0, 0x1f6ff20, ...)
        /app/pkg/compactv2/chunk_series_set.go:200 +0x427
github.com/thanos-io/thanos/pkg/compactv2.(*Compactor).WriteSeries(0xc0000d9d68, 0x1fa0228, 0xc000862c00, 0xc0000d9b98, 0x1, 0x1, 0x1fa67b8, 0xc00003afc0, 0x1f6ff20, 0xc000404a80, ...)
        /app/pkg/compactv2/compactor.go:147 +0xb25
main.registerBucketRewrite.func1.1(0x0, 0x0)
        /app/cmd/thanos/tools_bucket.go:920 +0x10f5
github.com/oklog/run.(*Group).Run.func1(0xc0002924e0, 0xc000890a00, 0xc000881b60)
        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xbb

Dry run has no issue. The test is done by thanos v0.20.2 and S3. Following is the output of tools bucket inspect.

level=info ts=2021-06-03T04:05:41.6528167Z caller=factory.go:46 msg="loading bucket configuration"
level=info ts=2021-06-03T04:05:42.1304465Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=461.8639ms cached=187 returned=187 partial=0
|            ULID            |        FROM         |        UNTIL        |     RANGE      |   UNTIL-DOWN    | #SERIES |   #SAMPLES    |  #CHUNKS   | COMP-LEVEL | COMP-FAILED |                                                        LABELS                                                        | RESOLUTION |     SOURCE     |
|----------------------------|---------------------|---------------------|----------------|-----------------|---------|---------------|------------|------------|-------------|----------------------------------------------------------------------------------------------------------------------|------------|----------------|
| 01F11XK5AW6BN0V760D9C4Y5T3 | 04-03-2021 00:00:00 | 18-03-2021 00:00:00 | 335h59m59.864s | -               | 86,658  | 22,437,633    | 286,768    | 4          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 1h0m0s     | compactor      |
| 01F25Z5EGMWKM577FMTYM0718C | 18-03-2021 00:00:00 | 01-04-2021 00:00:00 | 335h59m59.935s | -               | 91,515  | 23,312,170    | 292,946    | 4          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 1h0m0s     | compactor      |
| 01F3A0S4HM9KYV9J5528SM7NS1 | 01-04-2021 00:00:00 | 15-04-2021 00:00:00 | 335h59m59.938s | -               | 106,308 | 25,558,975    | 321,134    | 4          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 1h0m0s     | compactor      |
| 01F4E20BTK668NA3HBNRXN7AKB | 15-04-2021 00:00:00 | 29-04-2021 00:00:00 | 335h59m59.943s | -295h59m59.943s | 100,537 | 3,289,387,764 | 27,425,425 | 4          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 0s         | compactor      |
| 01F4E24RDA6E8WA289YC2KVN8Z | 15-04-2021 00:00:00 | 29-04-2021 00:00:00 | 335h59m59.943s | -95h59m59.943s  | 100,537 | 326,586,223   | 2,366,044  | 4          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 5m0s       | compactor      |
| 01F4E2BZ0Y3XADYFHXEPE0VCN7 | 15-04-2021 00:00:00 | 29-04-2021 00:00:00 | 335h59m59.943s | -               | 100,537 | 27,251,580    | 342,084    | 4          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 1h0m0s     | compactor      |
| 01F5J3JRDFXSFVKJNE3T3JWTW0 | 29-04-2021 00:00:00 | 13-05-2021 00:00:00 | 335h59m59.896s | -295h59m59.896s | 162,678 | 3,312,859,543 | 27,657,855 | 4          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 0s         | compactor      |
| 01F5J3QKAMVCD9VGRXD1AHK8DS | 29-04-2021 00:00:00 | 13-05-2021 00:00:00 | 335h59m59.896s | -95h59m59.896s  | 162,678 | 330,299,454   | 2,463,624  | 4          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 5m0s       | compactor      |
| 01F5J3ZA3JJZ0H7K4G0N0DCYBG | 29-04-2021 00:00:00 | 13-05-2021 00:00:00 | 335h59m59.896s | -               | 162,678 | 27,804,831    | 406,632    | 4          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 1h0m0s     | compactor      |
| 01F5Q8ANWKCJQA7FE34JW6MPT5 | 13-05-2021 00:00:00 | 15-05-2021 00:00:00 | 47h59m59.942s  | 192h0m0.058s    | 84,767  | 41,735,306    | 370,524    | 3          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 5m0s       | compactor      |
| 01F6S0DGVDNJA6FQE3JBXFQ7ZS | 13-05-2021 00:00:00 | 15-05-2021 00:00:00 | 47h59m59.942s  | -7h59m59.942s   | 84,736  | 418,305,020   | 3,489,467  | 3          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 0s         | bucket.rewrite |
| 01F6P52S252GDCNEE6MS6FT1FP | 13-05-2021 00:00:00 | 27-05-2021 00:00:00 | 335h59m59.942s | -295h59m59.942s | 384,795 | 3,841,904,413 | 32,154,121 | 4          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 0s         | compactor      |
| 01F5WD45R5QXQM0HBT98D2Z2AE | 15-05-2021 00:00:00 | 17-05-2021 00:00:00 | 47h59m59.865s  | 192h0m0.135s    | 76,958  | 44,303,448    | 384,616    | 3          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 5m0s       | compactor      |
| 01F61RSC2JKFKAPKSEHVYB0839 | 17-05-2021 00:00:00 | 19-05-2021 00:00:00 | 47h59m59.97s   | 192h0m0.03s     | 118,707 | 45,051,844    | 412,647    | 3          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 5m0s       | compactor      |
| 01F66PPS2E60WH5JTQ0FDFKP7H | 19-05-2021 02:51:34 | 21-05-2021 00:00:00 | 45h8m25.33s    | 194h51m34.67s   | 320,499 | 55,308,708    | 576,107    | 3          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 5m0s       | compactor      |
| 01F6BMMC13E9Y97Z9MKDPAK9H5 | 21-05-2021 00:00:00 | 23-05-2021 00:00:00 | 48h0m0s        | 192h0m0s        | 114,131 | 65,161,317    | 565,736    | 3          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 5m0s       | compactor      |
| 01F6H092T470QK29QPJFRRXYTG | 23-05-2021 00:00:00 | 25-05-2021 00:00:00 | 47h59m59.995s  | 192h0m0.005s    | 115,335 | 65,352,232    | 568,464    | 3          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 5m0s       | compactor      |
| 01F6NY6XFBHZSQ159ZYF5FGE61 | 25-05-2021 00:00:00 | 27-05-2021 00:00:00 | 48h0m0s        | 192h0m0s        | 121,740 | 65,901,813    | 577,159    | 3          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 5m0s       | compactor      |
| 01F6Q0FVT6X0HYATRCS7DY878M | 27-05-2021 00:00:00 | 27-05-2021 08:00:00 | 7h59m59.989s   | 32h0m0.011s     | 116,982 | 110,876,255   | 925,500    | 2          | false       | prometheus=backend/kprom-kube-prometheus-prometheus,prometheus_replica=prometheus-kprom-kube-prometheus-prometheus-0 | 0s         | compactor      |
...

I tried several (but not all) blocks (source==compactor) and only rewriting block 01F6Q0FVT6X0HYATRCS7DY878M has no error.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 4
  • Comments: 22 (4 by maintainers)

Most upvoted comments

Still needed.

Thanks for reporting this issue. Bucket rewrite tool only works for not downsampled blocks (res=0) currently.

We need to mention it in the docs.

There’s still some WIP to do this, so not forgotten, see this recent closed PR: https://github.com/thanos-io/thanos/pull/5725

Still there 😦