thanos: Query Push down causes error when using min, max_over_time functions

Thanos, Prometheus and Golang version used: Thanos v0.32.0 Prometheus v2.47.1

Object Storage Provider: GCS

What happened: We enabled query-pushdown feature on Thanos Query

We found out the max_over_time function doesn’t work when Thanos Query pulls data from both Thanos sidecar and store, with error message: execution: vector cannot contain metrics with the same labelset

The max_over_time function works fine when either: a) select time range that fits Thanos Sidecar (Prometheus’s) storage retention, e.g. last 3h. or b) select time range that fits Thanos Store max-time, eg. now-3h

What you expected to happen: All *_over_time functions should work normally

How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components:

Anything else we need to know:

  • Prometheus-operator stack: prometheus shards with thanos-sidecar (same version: v0.32.0)
  • Thanos store with GCS bucket as the long-term storage

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 17 (15 by maintainers)

Most upvoted comments

Ill write a quick e2e test and try to reproduce to see if my theory might hold. Can you disable query pushdown in the meantime to remedy the situation for now?