VictoriaMetrics: same query works for today data but failed for yesterday data with error "cannot process MetricBlock #1: cannot select more than -search.maxSamplesPerQuery=2000000000 samples"
Is your question request related to a specific component?
vmselect
Describe the question in detail
We have a cluster with 12 insert/2 select/6 storage nodes. The ingesting rate is ~600k/s, and active series is ~20Mil.
We have a query below which queries and calculates histogram_quantile for data in 30 minutes, it works well for today, but failed for history data in yesterday or before.
histogram_quantile(0.9, sum by(le, target_svc) (rate({namespace=~"^(prod|pg)$", result_table_id=~"100147_bcs_custom_metric_result_table_41001|100147_bcs_custom_metric_result_table_40359", __name__="kh_grpc_request_latency_bucket_bucket_value"}[1m] offset -59s999ms)))
Here is trace query output for query today data: query_success.json
Here is the error response for query yesterday data:
422 error occured during search: cannot fetch query results from vmstorage nodes: cannot perform search on vmstorage monitor-vmstorage-3.monitor-kihan-vmstorage.vm.svc.cluster.local:8401: cannot execute funcName="search_v7" on vmstorage "xx.xx.xx.xx:8401": cannot process MetricBlock #1: cannot select more than -search.maxSamplesPerQuery=2000000000 samples; possible solutions: to increase the -search.maxSamplesPerQuery; to reduce time range for the query; to use more specific label filters in order to select lower number of series
Troubleshooting docs
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18 (11 by maintainers)
Thanks for your exaplaining! @hagen1778
If the
maxRowsPerBlockcan be changed, it will be applied to only new blocks, so it can’t help users to query historical data. And compare to the query cost of changing-search.maxSamplesPerQuery, the storage cost of changingmaxRowsPerBlockis much harder to understand and evaluate.