thanos: Thanos Store Panic
Thanos, Prometheus and Golang version used:
Thanos: 0.32.5 Prometheus: 2.48
Object Storage Provider: AWS S3
What happened:
Using Thanos Store with Time Filter from time to time I’m it’s crashing with the following message.
panic: runtime error: slice bounds out of range [:16000] with capacity 1282
What you expected to happen:
Not crash
How to reproduce it (as minimally and precisely as possible):
Actually! I have no idea, this is only happening in one shard.
Full logs to relevant components:
panic: runtime error: slice bounds out of range [:16000] with capacity 1290goroutine 507203 [running]:github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).loadChunks(0x400261e720, {0x26b43f8, 0x407d9c7540}, {0x406284b000, 0x10c, 0x400195e870?}, {0x4102db7748, 0x1, 0x2}, 0x3968440?, ...) /app/pkg/store/bucket.go:3342 +0xe1cgithub.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).load.func4() /app/pkg/store/bucket.go:3270 +0x74golang.org/x/sync/errgroup.(*Group).Go.func1() /go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x58created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 505525 /go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x98
Anything else we need to know:
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 2
- Comments: 24 (14 by maintainers)
We’re also seeing this issue on Thanos: v0.35.1 .
So far we fixed it ourself in pkg.store, line 3532:
From what I was able to debug, source of the problem is chunk size defined in the block meta json. For some reason it states smaller size than reality is, requested buffer is too small and store panics. I guess source of this will be compactor or receiver. We have also “historic stores” serving data created by Thanos v30 and older, there are no such panics, so I assume its combination of the store optimization together with something in receiver or compactor code not filing these values correctly in latest versions.
This will not fix source of this issue but at least fix store code which doesn’t correctly check inputs.
This is also happening to us: Thanos: v0.32.5 (deployed through bitnami helm chart) Prometheus: v2.45.1
The problem only occurs on queries which are a little bit older and only rely on the store s3 data.
I can also provide traces if that would be helpful.
@nicolastakashi Can you share the actual panic stacktrace? I think I saw something similar last week