VictoriaMetrics: victoriametrics metrics missing
Describe the bug
Using victoriametrics v1.99.0, I found that some host indicators could not be collected. Other versions had no problems and no error messages were reported in the logs.Collection is normal through prometheus, but victoriametrics is not collected. (Same node_exporter)
To Reproduce
use victoria-metrics-20240301-013527-tags-v1.99.0-0-g9cd4b0537
Version
[root@geepaas-yunpingtai-infraprometheus ~]# ./victoria-metrics-prod --version victoria-metrics-20240301-013527-tags-v1.99.0-0-g9cd4b0537
Logs
nothing
Screenshots
Used command-line flags
VICTORIAMETRICS_OPT=-http.connTimeout=5m
-maxConcurrentInserts=20000
-maxInsertRequestSize=100MB
-maxLabelsPerTimeseries=20000
-insert.maxQueueDuration=5m
-dedup.minScrapeInterval=60s
-retentionPeriod=180d
-search.maxQueryDuration=10m
-search.maxQueryLen=30MB
-search.maxQueueDuration=60s
-search.maxConcurrentRequests=32
-storageDataPath=/usr/local/victoriametrics/data
-promscrape.config=/usr/local/victoriametrics/conf/prometheus.yml
-vmui.defaultTimezone=“Asia/Shanghai” \
Additional information
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 29 (2 by maintainers)
Commits related to this issue
- lib/storage: wait for up to 60 seconds before deciding to delete metricID entries from indexdb if metricID->metricName entry is missing during search The metricID->metricName entry can remain invisib... — committed to VictoriaMetrics/VictoriaMetrics by valyala 3 years ago
- lib/storage: wait for up to 60 seconds before deciding to delete metricID entries from indexdb if metricID->metricName entry is missing during search The metricID->metricName entry can remain invisib... — committed to VictoriaMetrics/VictoriaMetrics by valyala 3 years ago
- lib/storage: follow-up for 76f00cea6b9bf02a262275a112040c2680ed472c Store the deadline when the metricID entries must be deleted from indexdb if metricID->metricName entry isn't found after the deadl... — committed to VictoriaMetrics/VictoriaMetrics by valyala 3 years ago
- lib/storage: wait for up to 60 seconds before deciding to delete metricID entries from indexdb if metricID->metricName entry is missing during search The metricID->metricName entry can remain invisib... — committed to possibull/VictoriaMetrics by valyala 3 years ago
- lib/storage: wait for up to 60 seconds before deciding to delete metricID entries from indexdb if metricID->metricName entry is missing during search The metricID->metricName entry can remain invisib... — committed to possibull/VictoriaMetrics by valyala 3 years ago
- lib/storage: follow-up for 76f00cea6b9bf02a262275a112040c2680ed472c Store the deadline when the metricID entries must be deleted from indexdb if metricID->metricName entry isn't found after the deadl... — committed to VictoriaMetrics/VictoriaMetrics by valyala 3 years ago
@valyala @parkjeongryul OK, thank you. I think this bug has been fixed.
I experienced the same issue in
v1.99.0, and after upgrade tov1.100.1, I found that the issue is resolved 👍The root cause of the issue is that VictoriaMetrics could delete newly registered time series in a few seconds after it has been registered. The time series is deleted because VictoriaMetrics couldn’t find its’ name plus all its labels in the inverted index (aka
metricID->metricNameentry in theindexdb) when the time series is found in some search query results. VictoriaMetrics assumes that the inverted index is incomplete for the given time series, so it drops the time series, so it could be registered again. The inverted index may miss recently added entries after unclean shutdown (such as out of memory (OOM) crash,kill -9or hardware power off) or after restoring from snapshot, if the recently added entries aren’t flushed from in-memory storage to persistent storage yet.The code responsible for deleting the time series if the
metricID->metricNameisn’t found inindexdbexists in VictoriaMetrics since 2019. Why it triggers with much higher probability starting from v1.99.0? Because of optimizations added inv1.99.0, which improve the registration speed for new time series - 73f0a805e2d8ee55ea4e94b8cb19e3d43c3ac31d and bace9a25011d9175f9fa6861e129f8ed8f9e0521 . As a side effect, these optimizations increase the delay between registering new entries in theindexdband appearance of these entries in search results. This increased chances that themetricID->metricNameentries for newly registered time series aren’t visible for search for a few seconds after their registration.The bugfix postpones deletion of time series data for up to 60 seconds when it cannot find the corresponding
metricID->metricNameentry inindexdb. The newly registeredmetricID->metricNameentries become visible for search after a few seconds, so 60 seconds delay should be enough in most cases. This logic doesn’t depend on the scrape interval, so it works with arbitrary scrape intervals.How to detect whether you are affected by the issue?
Look at the
vm_missing_metric_names_for_metric_id_totalmetric. If you are affected by the issue, then this metric should increase over time with some constant speed. E.g.increase(vm_missing_metric_names_for_metric_id_total[5m])query should return some non-zero value over long time ranges.How to detect whether the issue is fixed?
The
vm_missing_metric_names_for_metric_id_totalmetric should stop growing after the upgrade to 76f00cea6b9bf02a262275a112040c2680ed472c or newer commit.JFYI, the fix was also included in LTS v1.97.4+.
@system51 , @dctrwatson , could you upgrade to v1.100.0 and verify whether this prevents from dropping newly registered time series?
Note that the upgrade doesn’t restore data for old time series, which could be dropped while
v1.99.0was running - it should prevent from dropping newly ingested samples for all the time series - already registered and new ones.Just noticed this happening. However,
vm_missing_metric_names_for_metric_id_totalis0Even after downgrading to v1.97.1 the metrics still were not available via query. Nor were they available via/api/v1/export.To make the metrics available going forward, also had to restart the known affected services which changes their labelset. So it seems like since v1.99.0 upgrade there’s an unknown amount of missing data.