thanos: compact: fails with "invalid plan block" error
Those chunks are from an old pod (we’re using deployments, so the replica label doesn’t exist anymore, in case this could help for debugging)
level=info ts=2018-05-24T22:01:03.10004204Z caller=compact.go:245 msg="starting compact node"
level=info ts=2018-05-24T22:01:03.100144202Z caller=compact.go:131 msg="start sync of metas"
level=debug ts=2018-05-24T22:01:03.426927339Z caller=compact.go:165 msg="download meta" block=01CE75YJXFASBNC40BYG8EHD1N
level=debug ts=2018-05-24T22:01:03.528081002Z caller=compact.go:165 msg="download meta" block=01CE7CTA5BG05CXPDZTTPQZ1WX
level=debug ts=2018-05-24T22:01:03.549836998Z caller=compact.go:165 msg="download meta" block=01CE7KP1DCEYPB1J7SYKNKSXB8
level=debug ts=2018-05-24T22:01:03.572966127Z caller=compact.go:165 msg="download meta" block=01CE7THRND9X4YAHZJP17QP9MY
level=debug ts=2018-05-24T22:01:03.623778574Z caller=compact.go:165 msg="download meta" block=01CE81DFXA83PF5ZA34XJH1GJH
level=debug ts=2018-05-24T22:01:03.645886122Z caller=compact.go:165 msg="download meta" block=01CE88975E33ARVHPQNEX8K8TJ
level=debug ts=2018-05-24T22:01:03.704790623Z caller=compact.go:165 msg="download meta" block=01CE8F4YDGM3CC57HKXFBBSWQ2
level=debug ts=2018-05-24T22:01:03.75381572Z caller=compact.go:165 msg="download meta" block=01CE8P0NMQ16JDFD619390KJQH
level=debug ts=2018-05-24T22:01:03.784011615Z caller=compact.go:165 msg="download meta" block=01CE8WWCW8VR2H1YZ92HTK0WS2
level=debug ts=2018-05-24T22:01:03.806186002Z caller=compact.go:165 msg="download meta" block=01CE8WWCWQK9GZE8K6XWRZ9ME1
level=debug ts=2018-05-24T22:01:03.870331206Z caller=compact.go:165 msg="download meta" block=01CE93R44DQWHR2CFG0CXTP5QJ
level=debug ts=2018-05-24T22:01:03.898889804Z caller=compact.go:165 msg="download meta" block=01CE93R44V6DFZEQY1F4F3NVQH
level=debug ts=2018-05-24T22:01:03.919810068Z caller=compact.go:165 msg="download meta" block=01CE9AKVC52RKEJ7Y3EN4VPEFT
level=debug ts=2018-05-24T22:01:03.940709351Z caller=compact.go:165 msg="download meta" block=01CE9AKVCGCSX7VNDS6HGR9CTY
level=debug ts=2018-05-24T22:01:04.026893239Z caller=compact.go:165 msg="download meta" block=01CE9HFJMGYHTQFP2SWM78H8DG
level=debug ts=2018-05-24T22:01:04.049993424Z caller=compact.go:165 msg="download meta" block=01CE9HFJMJ8GB9JX7FB3CJ9094
level=debug ts=2018-05-24T22:01:04.068987694Z caller=compact.go:165 msg="download meta" block=01CE9RB9WG5C11SHFMBP9524W0
level=debug ts=2018-05-24T22:01:04.090585575Z caller=compact.go:165 msg="download meta" block=01CE9RB9WJ501MZ1FFFG0VH0F7
level=debug ts=2018-05-24T22:01:04.123175132Z caller=compact.go:165 msg="download meta" block=01CE9Z714G2BARSFT68HHKAT4M
level=debug ts=2018-05-24T22:01:04.148953081Z caller=compact.go:165 msg="download meta" block=01CE9Z714JHRXJ7BA4D0J8WAAV
level=info ts=2018-05-24T22:01:04.168912292Z caller=compact.go:137 msg="start of GC"
level=error ts=2018-05-24T22:01:27.662643337Z caller=main.go:147 msg="running command failed" err="compaction: invalid plan block /var/thanos/compact/compact/0@{monitor=\"prometheus\",replica=\"prometheus-thanos-1-86d6c8569d-x22rl\"}/01CE75YJXFASBNC40BYG8EHD1N: No chunks are out of order, but found some outsider blocks. (Blocks that outside of block time range): 4160. Complete: 0"
01CE75YJXFASBNC40BYG8EHD1N/meta.json
{
"version": 1,
"ulid": "01CE75YJXFASBNC40BYG8EHD1N",
"minTime": 1527091200000,
"maxTime": 1527098400000,
"stats": {
"numSamples": 1879211741,
"numSeries": 2858110,
"numChunks": 16550746
},
"compaction": {
"level": 1,
"sources": [
"01CE75YJXFASBNC40BYG8EHD1N"
]
},
"thanos": {
"labels": {
"monitor": "prometheus",
"replica": "prometheus-thanos-1-86d6c8569d-x22rl"
},
"downsample": {
"resolution": 0
}
}
}
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 1
- Comments: 24 (15 by maintainers)
Commits related to this issue
- block: Handle gracefully extra duplicated chunk produced by TSDB issue. TSDB issue: https://github.com/prometheus/tsdb/issues/347 How we handle it? - segregate this issue in special stat entry in ve... — committed to thanos-io/thanos by bwplotka 6 years ago
- block: Handle gracefully extra duplicated chunk produced by TSDB issue. TSDB issue: https://github.com/prometheus/tsdb/issues/347 How we handle it? - segregate this issue in special stat entry in ve... — committed to thanos-io/thanos by bwplotka 6 years ago
- block: Handle gracefully extra duplicated chunk produced by TSDB issue. TSDB issue: https://github.com/prometheus/tsdb/issues/347 How we handle it? - segregate this issue in special stat entry in ve... — committed to thanos-io/thanos by bwplotka 6 years ago
- block: Handle gracefully extra duplicated chunk produced by TSDB issue. TSDB issue: https://github.com/prometheus/tsdb/issues/347 How we handle it? - segregate this issue in special stat entry in ve... — committed to thanos-io/thanos by bwplotka 6 years ago
- block: Handle gracefully extra duplicated chunk produced by TSDB issue. TSDB issue: https://github.com/prometheus/tsdb/issues/347 How we handle it? - segregate this issue in special stat entry in ve... — committed to thanos-io/thanos by bwplotka 6 years ago
- block: Handle gracefully extra duplicated chunk produced by TSDB issue. TSDB issue: https://github.com/prometheus/tsdb/issues/347 How we handle it? - segregate this issue in special stat entry in ve... — committed to thanos-io/thanos by bwplotka 6 years ago
- block: Handle gracefully extra duplicated chunk produced by TSDB issue. TSDB issue: https://github.com/prometheus/tsdb/issues/347 How we handle it? - segregate this issue in special stat entry in ve... — committed to thanos-io/thanos by bwplotka 6 years ago
- block: Handle gracefully extra duplicated chunk produced by TSDB issue. TSDB issue: https://github.com/prometheus/tsdb/issues/347 How we handle it? - segregate this issue in special stat entry in ve... — committed to thanos-io/thanos by bwplotka 6 years ago
- block: Handle gracefully extra duplicated chunk produced by TSDB issue. TSDB issue: https://github.com/prometheus/tsdb/issues/347 How we handle it? - segregate this issue in special stat entry in ve... — committed to thanos-io/thanos by bwplotka 6 years ago
- block: Handle gracefully extra duplicated chunk produced by TSDB issue. (#375) TSDB issue: https://github.com/prometheus/tsdb/issues/347 How we handle it? - segregate this issue in special stat e... — committed to thanos-io/thanos by bwplotka 6 years ago
Thanks all for helping with this issue!
Thanos now handles these blocks gracefully (it repairs them in place). The TSDB it not yet fixed (since it is not straightforward), but when the issue occurs, Thanos handles it well now.
Feel free to use
master-2018-06-15-5b66b72docker tag or above. New release candidate will produce soon.Thanks to @BenoitKnecht who figured the root cause and @clmssz for reporting it! Nice work!
Thanks of @BenoitKnecht we found a root cause and fix is in progress
@povilasv I’m not sure what you mean by the outsiders are before minTime. You seem be in the exact same situation I’m in: the outsider’s minTime (
c.MinTime) is exactly equal to the block’s maxTime (maxTime). So all the outsiders “overflow” the block at the end, not at the start.To make things easier, I’ve been using this script to inspect Prometheus blocks locally, rather than using
thanos bucket verifyto check the blocks after they’ve been uploaded. It gives me this kind of output on affected blocks.Interestingly, out of the four Prometheus instances that I run (all identical, based on the
prom/prometheus:v2.2.1Docker image), only one is affected by this issue, but not on every block:So again, I’m not sure if this is a Prometheus issue (meaning it should never produce blocks with chunks that stick out at the end), or if it’s a Thanos issue (in the sense that it should not error out on blocks with that particular type of outsiders, but just treat them like any other block).