VictoriaMetrics: same query works for today data but failed for yesterday data with error "cannot process MetricBlock #1: cannot select more than -search.maxSamplesPerQuery=2000000000 samples"

Is your question request related to a specific component?

vmselect

Describe the question in detail

We have a cluster with 12 insert/2 select/6 storage nodes. The ingesting rate is ~600k/s, and active series is ~20Mil.

We have a query below which queries and calculates histogram_quantile for data in 30 minutes, it works well for today, but failed for history data in yesterday or before.

histogram_quantile(0.9, sum by(le, target_svc) (rate({namespace=~"^(prod|pg)$", result_table_id=~"100147_bcs_custom_metric_result_table_41001|100147_bcs_custom_metric_result_table_40359", __name__="kh_grpc_request_latency_bucket_bucket_value"}[1m] offset -59s999ms)))

Here is trace query output for query today data: query_success.json

image

Here is the error response for query yesterday data:

422 error occured during search: cannot fetch query results from vmstorage nodes: cannot perform search on vmstorage monitor-vmstorage-3.monitor-kihan-vmstorage.vm.svc.cluster.local:8401: cannot execute funcName="search_v7" on vmstorage "xx.xx.xx.xx:8401": cannot process MetricBlock #1: cannot select more than -search.maxSamplesPerQuery=2000000000 samples; possible solutions: to increase the -search.maxSamplesPerQuery; to reduce time range for the query; to use more specific label filters in order to select lower number of series

Troubleshooting docs

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (11 by maintainers)

Most upvoted comments

Thanks for your exaplaining! @hagen1778

In my opinion, this should be a configurable args for the user to better trade off storage cost and query cost.

If the maxRowsPerBlock can be changed, it will be applied to only new blocks, so it can’t help users to query historical data. And compare to the query cost of changing -search.maxSamplesPerQuery, the storage cost of changing maxRowsPerBlock is much harder to understand and evaluate.