VictoriaMetrics: vmselect returns empty response for range query on interval > 40 days

Describe the bug

For a simple range query, no results are returned when the range is > 40 days (we have 60 s scrape interval). There is nothing relevant in the logs.

Copied from grafana query inspector:

{
  "request": {
    "url": "api/ds/query",
    "method": "POST",
    "data": {
      "queries": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "000000001"
          },
          "expr": "process_cpu_seconds_total{job=\"victoria-metrics\"}",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "CPU cores used",
          "refId": "B",
          "hide": false,
          "editorMode": "code",
          "range": true,
          "exemplar": false,
          "requestId": "57B",
          "utcOffsetSec": 32400,
          "datasourceId": 1,
          "intervalMs": 3600000,
          "maxDataPoints": 857
        }
      ],
      "range": {
        "from": "2023-03-26T05:17:28.942Z",
        "to": "2023-05-06T05:17:28.942Z",
        "raw": {
          "from": "now-71d",
          "to": "now-30d"
        }
      },
      "from": "1679807848942",
      "to": "1683350248942"
    },
    "hideFromInspector": false
  },
  "response": {
    "results": {
      "B": {
        "status": 200,
        "frames": [],
        "refId": "B"
      }
    }
  }
}

To Reproduce

query: process_cpu_seconds_total{job="victoria-metrics"}

with range of 40 days, the query returns result normally

with range of 41 days, the result is empty

Version

single node v1.87.6

Logs

none

Screenshots

No response

Used command-line flags

No response

Additional information

No response

About this issue

Original URL
State: open
Created a year ago
Comments: 22 (13 by maintainers)

Most upvoted comments

I encountered the same problem. My vmstorage version is v1.91.3-cluster. The problem seems to be that vmstorage will clean the indexdb once a month, although my retentionPeriod is 240days. When I query the data from a month ago, the return is empty, but querying the data for multiple months at once can return normally. I speculate that the indexdb has been cleaned up by mistake.

here is my query 30d-60d

here is my query 0d-180d It seems that historical monitoring is not lost, it is just that queries larger than 30d fail due to no index.

Here is the information for index cleaning

I don’t know what I should do. I now feel that even if I upgrade vmstorage, the historical data will be empty because there is no indexdb.

hape2550505 on Nov 20, 2023

I checked vm_missing_metric_names_for_metric_id_total, it’s never been > 0 on our instance.

belm0 on Jun 21, 2023

I’ve reset the cache by reset_cache_on_startup file, but the problem remains.

belm0 on Jun 9, 2023