thanos: No metrics displayed with Thanos engine ranged query (go_routines[1d]) in query distributes installation (Prometheus engine returns correct output)
Thanos, Prometheus and Golang version used: Thanos - v0.37.2 Prometheus - v2.55.1 Golang - v1.23.4
Object Storage Provider: Azure Huawei OBS
What happened: I’ve set up Thanos Query in distributed mode with the following stores (aka local queries) in config:
- query-az.int.zone
- query-hw.int.zone
extraArgs:
- –query.timeout=5m
- –query.mode=distributed
The local queries have Thanos Gateways hostnames in their corresponding configurations (Prometheus installations with Thanos sidecars).
When I query distributed query with range query (e.g. go_threads[1d]) via Thanos engine it returns nothing
, Prometheus engine - returns correct output.
Querying local query directly also yields correct results with either Thanos and Promethus engines.
What you expected to happen: Distributed query returns the same output with Promethues and Thanos engines.
How to reproduce it (as minimally and precisely as possible):
- Have Thanos installation with Thanos Gateways.
- Set up distributed Query with two local queries in --store extra args.
- Search any ranged metric (e.g. go_threads[1d]) with Thanos Enginer.
- See nothing
Full logs to relevant components: no errors whatsoever
Anything else we need to know:
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 15 (5 by maintainers)
Flag might be on main and soon 0.38 can you try with 0.38rc and the flag?
My quick and dirty fix was to add 1s in time partitioning (I have 7 days retention in Receive):
Still, some permanent solution would be better