thanos: index out of range [-1]
Thanos Version used: v0.12.2
Object Storage Provider: Ceph S3
What happened: When I run queries against historic dates, for certain dates we get a response
unexpected error: runtime error: index out of range [-1]
{‘field’: ‘subscribed_cores’, ‘date’: datetime.date(2019, 12, 27), ‘runtime’: 1.5100421905517578, ‘text’: ‘{“status”:“error”,“errorType”:“execution”,“error”:“unexpected error: runtime error: index out of range [-1]”}\n’, ‘value’: None, ‘status’: ‘error’, ‘error’: ‘unexpected error: runtime error: index out of range [-1]’}
Other than that we also sometimes get an empty response if not the error but never the actual value.
{‘field’: ‘subscribed_cores’, ‘date’: datetime.date(2019, 12, 27), ‘runtime’: 1.227376699447632, ‘text’: ‘{“status”:“success”,“data”:{“resultType”:“vector”,“result”:[]}}\n’, ‘value’: None, ‘status’: ‘success’, ‘error’: None}
What you expected to happen: Get the value for that date.
CC: @andmatt
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 17 (10 by maintainers)
Commits related to this issue
- querier: Dedup series is now replica label agnostic and simpler. Fixed panic seen when using larger number of replicas and small series. Fixes https://github.com/thanos-io/thanos/issues/2645 Signed... — committed to thanos-io/thanos by bwplotka 4 years ago
- querier: Dedup series is now replica label agnostic and simpler. Fixed panic seen when using larger number of replicas and small series. Fixes https://github.com/thanos-io/thanos/issues/2645 Signed... — committed to thanos-io/thanos by bwplotka 4 years ago
- querier: Dedup series is now replica label agnostic and simpler. Fixed panic seen when using larger number of replicas and small series. Fixes https://github.com/thanos-io/thanos/issues/2645 Signed... — committed to thanos-io/thanos by bwplotka 4 years ago
- querier: Dedup series is now replica label agnostic and simpler. Fixed panic seen when using larger number of replicas and small series. Fixes https://github.com/thanos-io/thanos/issues/2645 Signed... — committed to thanos-io/thanos by bwplotka 4 years ago
- querier: Dedup series is now replica label agnostic and simpler. Fixed panic seen when using larger number of replicas and small series. Fixes https://github.com/thanos-io/thanos/issues/2645 Signed... — committed to thanos-io/thanos by bwplotka 4 years ago
- querier: Dedup series is now replica label agnostic and simpler. Fixed panic seen when using larger number of replicas and small series. (#2728) Fixes https://github.com/thanos-io/thanos/issues/2645 ... — committed to thanos-io/thanos by bwplotka 4 years ago
Ok, got it. The problem is that Querier just have problems with more than 2 replica labels. Fixing.
Thanks for spotting this and quickly describing issue ❤️
And this is the log from the associated
thanos-querypod