thanos: query: Staleness problem
Thanos, Prometheus and Golang version used: thanos: v0.12.0
Object Storage Provider: private CEPH (S3)
What happened:
See on end_input time and resolution:
Staleness functionality in prometheus library get rid of some points returned from thanos-stores.
What you expected to happen: Return all data from store on any time_range
How to reproduce it (as minimally and precisely as possible): see on screenshots.
Full logs to relevant components:
Anything else we need to know: I think, that we have few ways to resolve problem:
- Update prometheus library and set
LookbackDeltaparameter > 5 min (need check) - Update query and move/duplicate points to needed timestamp. (Interpolate data for PromQL)
- Update prometheus library to return all points from stores.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 22 (20 by maintainers)
Commits related to this issue
- query: chain interpolating iterator (#2608) Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> — committed to krya-kryak/thanos by krya-kryak 4 years ago
- query: introduce dynamic lookback interval Closes #2608 This allows queries with large step to make use of downsampled data. Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> — committed to krya-kryak/thanos by krya-kryak 4 years ago
- query: introduce dynamic lookback interval Closes #2608 This allows queries with large step to make use of downsampled data. Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> — committed to krya-kryak/thanos by krya-kryak 4 years ago
- query: introduce dynamic lookback interval (#3277) * query: introduce dynamic lookback interval Closes #2608 This allows queries with large step to make use of downsampled data. Signed-off-by:... — committed to thanos-io/thanos by krya-kryak 4 years ago
- query: introduce dynamic lookback interval (#3277) * query: introduce dynamic lookback interval Closes #2608 This allows queries with large step to make use of downsampled data. Signed-off-by:... — committed to Oghenebrume50/thanos by krya-kryak 4 years ago
Well, here’s my attempt at it: https://github.com/thanos-io/thanos/pull/3277
We just allow users to configure it on Querier from flag that’s it.
Yes, something like that.
For e.g. sum_over_time you need different data than count_over_time to produce the desired result.