VictoriaMetrics: victoriametrics metrics missing

Describe the bug

Using victoriametrics v1.99.0, I found that some host indicators could not be collected. Other versions had no problems and no error messages were reported in the logs.Collection is normal through prometheus, but victoriametrics is not collected. (Same node_exporter)

To Reproduce

use victoria-metrics-20240301-013527-tags-v1.99.0-0-g9cd4b0537

Version

[root@geepaas-yunpingtai-infraprometheus ~]# ./victoria-metrics-prod --version victoria-metrics-20240301-013527-tags-v1.99.0-0-g9cd4b0537

Logs

nothing

Screenshots

a3a86178f0c6c4ef75dee191008197de

13ba21e5ba980a2a2b3203978a85cdc3

c093fb9791d0ff37a24d803abbdd2c86

Used command-line flags

VICTORIAMETRICS_OPT=-http.connTimeout=5m
-maxConcurrentInserts=20000
-maxInsertRequestSize=100MB
-maxLabelsPerTimeseries=20000
-insert.maxQueueDuration=5m
-dedup.minScrapeInterval=60s
-retentionPeriod=180d
-search.maxQueryDuration=10m
-search.maxQueryLen=30MB
-search.maxQueueDuration=60s
-search.maxConcurrentRequests=32
-storageDataPath=/usr/local/victoriametrics/data
-promscrape.config=/usr/local/victoriametrics/conf/prometheus.yml
-vmui.defaultTimezone=“Asia/Shanghai” \

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 29 (2 by maintainers)

Commits related to this issue

Most upvoted comments

@valyala @parkjeongryul OK, thank you. I think this bug has been fixed.

I experienced the same issue in v1.99.0, and after upgrade to v1.100.1, I found that the issue is resolved 👍

Would this fix still work when the metric in question has a scrape interval >60s?

The root cause of the issue is that VictoriaMetrics could delete newly registered time series in a few seconds after it has been registered. The time series is deleted because VictoriaMetrics couldn’t find its’ name plus all its labels in the inverted index (aka metricID->metricName entry in the indexdb) when the time series is found in some search query results. VictoriaMetrics assumes that the inverted index is incomplete for the given time series, so it drops the time series, so it could be registered again. The inverted index may miss recently added entries after unclean shutdown (such as out of memory (OOM) crash, kill -9 or hardware power off) or after restoring from snapshot, if the recently added entries aren’t flushed from in-memory storage to persistent storage yet.

The code responsible for deleting the time series if the metricID->metricName isn’t found in indexdb exists in VictoriaMetrics since 2019. Why it triggers with much higher probability starting from v1.99.0? Because of optimizations added in v1.99.0, which improve the registration speed for new time series - 73f0a805e2d8ee55ea4e94b8cb19e3d43c3ac31d and bace9a25011d9175f9fa6861e129f8ed8f9e0521 . As a side effect, these optimizations increase the delay between registering new entries in the indexdb and appearance of these entries in search results. This increased chances that the metricID->metricName entries for newly registered time series aren’t visible for search for a few seconds after their registration.

The bugfix postpones deletion of time series data for up to 60 seconds when it cannot find the corresponding metricID->metricName entry in indexdb. The newly registered metricID->metricName entries become visible for search after a few seconds, so 60 seconds delay should be enough in most cases. This logic doesn’t depend on the scrape interval, so it works with arbitrary scrape intervals.

How to detect whether you are affected by the issue?

Look at the vm_missing_metric_names_for_metric_id_total metric. If you are affected by the issue, then this metric should increase over time with some constant speed. E.g. increase(vm_missing_metric_names_for_metric_id_total[5m]) query should return some non-zero value over long time ranges.

How to detect whether the issue is fixed?

The vm_missing_metric_names_for_metric_id_total metric should stop growing after the upgrade to 76f00cea6b9bf02a262275a112040c2680ed472c or newer commit.

JFYI, the fix was also included in LTS v1.97.4+.

@system51 , @dctrwatson , could you upgrade to v1.100.0 and verify whether this prevents from dropping newly registered time series?

Note that the upgrade doesn’t restore data for old time series, which could be dropped while v1.99.0 was running - it should prevent from dropping newly ingested samples for all the time series - already registered and new ones.

Just noticed this happening. However, vm_missing_metric_names_for_metric_id_total is 0 Even after downgrading to v1.97.1 the metrics still were not available via query. Nor were they available via /api/v1/export.

To make the metrics available going forward, also had to restart the known affected services which changes their labelset. So it seems like since v1.99.0 upgrade there’s an unknown amount of missing data.