thanos: Query Push down causes error when using min, max_over_time functions
Thanos, Prometheus and Golang version used: Thanos v0.32.0 Prometheus v2.47.1
Object Storage Provider: GCS
What happened: We enabled query-pushdown feature on Thanos Query
We found out the max_over_time function doesn’t work when Thanos Query pulls data from both Thanos sidecar and store, with error message: execution: vector cannot contain metrics with the same labelset
The max_over_time function works fine when either: a) select time range that fits Thanos Sidecar (Prometheus’s) storage retention, e.g. last 3h. or b) select time range that fits Thanos Store max-time, eg. now-3h
What you expected to happen:
All *_over_time functions should work normally
How to reproduce it (as minimally and precisely as possible):
Full logs to relevant components:
Anything else we need to know:
- Prometheus-operator stack: prometheus shards with thanos-sidecar (same version: v0.32.0)
- Thanos store with GCS bucket as the long-term storage
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 17 (15 by maintainers)
Ill write a quick e2e test and try to reproduce to see if my theory might hold. Can you disable query pushdown in the meantime to remedy the situation for now?