thanos: Thanos-compact does not automatically disable compaction for blocks with large index.

Thanos, Prometheus and Golang version used:

thanos, version 0.17.2 (branch: HEAD, revision: 37e6ef61566c7c70793ba6d128f00c4c66cb2402) build user: circleci@2c49ee4f4029 build date: 20201208-10:11:27 go version: go1.15 platform: linux/amd64

Object Storage Provider: GCS

What happened:

thanos-compact halts when index size exceeds 64GiB
thanos-compact seems to continue to halt after marking the blocks as no-compact manually

What you expected to happen:

Compactor automatically disables compaction for this block and continues.
Manually marking a block as no-compact allows thanos-compact to run with out halt.

Full logs to relevant components:

"level=error ts=2021-01-20T05:13:29.348393505Z caller=compact.go:423 msg=\"critical error detected; halting\" err=\"compaction: group 0@7637943154331160108: compact blocks [/var/thanos/compact/compact/0@7637943154331160108/01EBG9FRFSSD8ZEWXBD34NY0XX /var/thanos/compact/compact/0@7637943154331160108/01EBNG1HPA6D0P22KY3ZH89TBY]: \\\"/var/thanos/compact/compact/0@7637943154331160108/01EWEE6FM9YDMQW1RV3MWKMGM2.tmp-for-creation/index\\\" exceeding max size of 64GiB\"\n"
"level=error ts=2021-01-22T00:54:09.703347565Z caller=compact.go:423 msg=\"critical error detected; halting\" err=\"compaction: group 0@7637943154331160108: compact blocks [/var/thanos/compact/compact/0@7637943154331160108/01EBG9FRFSSD8ZEWXBD34NY0XX /var/thanos/compact/compact/0@7637943154331160108/01EBNG1HPA6D0P22KY3ZH89TBY]: \\\"/var/thanos/compact/compact/0@7637943154331160108/01EWK216B7XAZTVQNJ4S5E7GGD.tmp-for-creation/index\\\" exceeding max size of 64GiB\"\n"
"level=error ts=2021-01-23T05:15:31.944994835Z caller=compact.go:423 msg=\"critical error detected; halting\" err=\"compaction: group 0@7637943154331160108: compact blocks [/var/thanos/compact/compact/0@7637943154331160108/01EBG9FRFSSD8ZEWXBD34NY0XX /var/thanos/compact/compact/0@7637943154331160108/01EBNG1HPA6D0P22KY3ZH89TBY]: \\\"/var/thanos/compact/compact/0@7637943154331160108/01EWP4NZG0GG172MVPWVR7XFBG.tmp-for-creation/index\\\" exceeding max size of 64GiB\"\n"
"level=error ts=2021-01-26T04:10:35.300971918Z caller=compact.go:423 msg=\"critical error detected; halting\" err=\"compaction: group 0@7637943154331160108: compact blocks [/var/thanos/compact/compact/0@7637943154331160108/01EBG9FRFSSD8ZEWXBD34NY0XX /var/thanos/compact/compact/0@7637943154331160108/01EBNG1HPA6D0P22KY3ZH89TBY]: \\\"/var/thanos/compact/compact/0@7637943154331160108/01EWXKAAEQ4M0HK0T92GKCAQXN.tmp-for-creation/index\\\" exceeding max size of 64GiB\"\n"

Anything else we need to know: These blocks where marked with no-compact-mark.json via thanos tools bucket on 2021-01-20`

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 2
Comments: 20 (6 by maintainers)

Commits related to this issue

Pad compaction planner size check (#3773) Pad the index size check in the compaction planner by 15% to avoid situations where the sum of index bytes bloats to be larger than the sum of the original... — committed to thanos-io/thanos by SuperQ 3 years ago
Upgrade thanos to release-0.19 (#8) * CHANGELOG.md: add v0.16.0 link (#3697) This small commit fixes the changelog entry for release v0.16.0 so that the section title includes a link to the actual ... — committed to stolostron/thanos by songleo 3 years ago

Most upvoted comments

@kakkoyun It’s needed in addition to a fix for why the no-compact-mark.json file is being igored.

SuperQ on Feb 10, 2021

I’m working on this issue and we have an ongoing progress, and this is expected to be fixed soon 😃

Biswajitghosh98 on May 20, 2021

We’re seeing a similar issue. We noticed some indexes that in bytes don’t go over 64GiB, but when the indexes are merged they bloat to larger than 64GiB. See: https://github.com/thanos-io/thanos/issues/3724

We have been manually marking them with the bucke tool, but it seems to be not reading this marker file consistently. I’ve deployed a patched version with some debug logging to see if we can catch the problem.

Our internal issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3427

SuperQ on Jan 28, 2021