thanos: No metrics displayed with Thanos engine ranged query (go_routines[1d]) in query distributes installation (Prometheus engine returns correct output)

Thanos, Prometheus and Golang version used: Thanos - v0.37.2 Prometheus - v2.55.1 Golang - v1.23.4

Object Storage Provider: Azure Huawei OBS

What happened: I’ve set up Thanos Query in distributed mode with the following stores (aka local queries) in config:

  • query-az.int.zone
  • query-hw.int.zone

extraArgs:

  • –query.timeout=5m
  • –query.mode=distributed

The local queries have Thanos Gateways hostnames in their corresponding configurations (Prometheus installations with Thanos sidecars).

When I query distributed query with range query (e.g. go_threads[1d]) via Thanos engine it returns nothing Image

, Prometheus engine - returns correct output. Image

Querying local query directly also yields correct results with either Thanos and Promethus engines. Image

What you expected to happen: Distributed query returns the same output with Promethues and Thanos engines.

How to reproduce it (as minimally and precisely as possible):

  1. Have Thanos installation with Thanos Gateways.
  2. Set up distributed Query with two local queries in --store extra args.
  3. Search any ranged metric (e.g. go_threads[1d]) with Thanos Enginer.
  4. See nothing

Full logs to relevant components: no errors whatsoever

Anything else we need to know:

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 1
  • Comments: 15 (5 by maintainers)

Most upvoted comments

Flag might be on main and soon 0.38 can you try with 0.38rc and the flag?

My quick and dirty fix was to add 1s in time partitioning (I have 7 days retention in Receive):

      timePartitioning:
        - min: "-26w"
          max: "-7d1s"

Still, some permanent solution would be better