VictoriaMetrics: Unexpected metrics/labels missing in VictoriaMetrics
Describe the bug
This may be a long story, it will be difficult to describe, but I will try. I have 2 Victoria Metrics intances on two different servers in Docker, let’s call them Victoria1 and Victoria2. Data is sent there from two Prometheus and the remote_write configuration is the same. In general, all the parameters are the same.
So, the problem is that Victoria1 does not have some metrics, or rather, there are no metrics for some labels. For example, there is a metric grpc_server_started_total, it is in both Victorias, but with such labels there is no grpc_server_started_total{hostname="myhost1.mycom ",grpc_service="etcdserverpb.Watch"} on Victoria1.
At the same time, on the second one it is (and on Prometheus too, the values are the same grpc_server_started_total{hostname="myhost1.mycom ",grpc_service="etcdserverpb.Watch"} 4878)
at the same time, on Victoria1 this metric exists for the label grpc_server_started_total{hostname="myhost2.mycom ",grpc_service="etcdserverpb.Watch"} or even grpc_server_started_total{hostname="myhost1.mycom"}.
In general, some metrics for specific labels are missing. In fact, in Victoria1 data all these metrics were (they were approximately from April of this year until May, then disappeared for no reason). During this time I updated both Victorias several times (with deleting the cache https://docs.victoriametrics.com/url-examples/#internalresetrollupresultcache ), but nothing helped, new metrics did not appear in Victoria1.
I actually checked everything I could from all sides, here is the full list:
- upgraded to latest (because found issues in Github regarding metrics missing, like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5948 )
- checked Victoria1 logs, no errors
- startup flags the same on both Victorias
- tested changing some values for flags like -dedup.minScrapeInterval (maybe error with duplication, because both Prometheuses push data to both Victorias and I use global label for all metrics to avoid duplication -
prometheus: global, but seems it’s not the reason) - checked Nginx error logs on Victoria1, no errors about upstream or any 5xx statuses (only during restart)
- Tested push to new instance. I created empty instance (same version and config flags) and push metrics to it - metrics appear
- Tested changing retention (I thought retention somehow removed metrics, but no effect)
- Dropped cache (docker stop victoria1 && rm -rf cache/*), restart Victoria1, no effect
- Dropped cache via api
http://localhost:8428/internal/resetRollupResultCache, no effect - Exported metric from Prometheus and pushed to Victoria1 via import like
curl -v -X POST 'http://localhost:8428/api/v1/import/prometheus' -u '***' -d 'grpc_server_started_total{grpc_service="etcdserverpb.Watch",hostname="myhost1.mycom"} 4878', metric appeared for 10 minutes and was removed from Victoria1 due to unknown for me reason (no logs about that) - Removed completely metric via
curl -u '***' -X POST 'http://localhost:8428/api/v1/admin/tsdb/delete_series' -d 'match[]=grpc_server_started_total{hostname="myhost1.mycom",grpc_service="etcdserverpb.Watch"}', no effect, no new metrics appeared for these labels - Checked prometheus logs, no errors about both victorias, only resharding logs
- Reloaded prometheus
- Checked configs on prometheus side for pushing to Victorias, they are same fro both of them
- Changed parameters for victoria1 for pushing on prometheus side (like max_shards or max_samples_per_send), no effect
- Checked prometheus metrics about errors of sending data to Victoria, didn’t find anything suspicious in prometheus_remote_storage_* metrics
Now I just don’t know what else to do and how to return the metrics. In fact, this is not the only example, some metrics (again for some labels) are also missing, for example, the standard metrics from Node Exporter - node_network_transmit_bytes_total or node_network_receive_bytes_total.
I noticed the problem not so long ago, since I have Victoria1 as a backup and I do not want to delete data from it (about 10 months old). I can assume that it all started with version v1.99.0 (I updated it in March, metrics started to disappear, judging by the data, somewhere around the beginning of May).
To Reproduce
I don’t know how to reproduce it, because the reason for the loss of metrics is unknown.
Version
On both Victorias
./victoria-metrics-prod --version
victoria-metrics-20240801-125729-tags-v1.102.1-0-g996b623585
Prometheus - 2.50.1
Logs
Prometheus logs contain only info about resharding: tail -n 10000 /var/log/docker/prometheus.log | grep victoria1
2024-08-09T13:43:55.354544+00:00 prometheus1 docker/prometheus/3760cdc7f90e/prom/prometheus:v2.50.1[6122]: ts=2024-08-09T13:43:55.354Z caller=dedupe.go:112 component=remote level=info remote_name=90011f url=https://victoria1/api/v1/write msg="Remote storage resharding" from=4 to=6
2024-08-09T13:44:25.353904+00:00 prometheus1 docker/prometheus/3760cdc7f90e/prom/prometheus:v2.50.1[6122]: ts=2024-08-09T13:44:25.353Z caller=dedupe.go:112 component=remote level=info remote_name=90011f url=https://victoria1/api/v1/write msg="Remote storage resharding" from=6 to=4
2024-08-09T13:46:55.353246+00:00 prometheus1 docker/prometheus/3760cdc7f90e/prom/prometheus:v2.50.1[6122]: ts=2024-08-09T13:46:55.353Z caller=dedupe.go:112 component=remote level=info remote_name=90011f url=https://victoria1/api/v1/write msg="Remote storage resharding" from=4 to=6
2024-08-09T13:48:05.353303+00:00 prometheus1 docker/prometheus/3760cdc7f90e/prom/prometheus:v2.50.1[6122]: ts=2024-08-09T13:48:05.353Z caller=dedupe.go:112 component=remote level=info remote_name=90011f url=https://victoria1/api/v1/write msg="Remote storage resharding" from=6 to=4
Victoria1 logs contain only logs about cache and merges:
2024-08-09T13:41:13.914516+00:00 victoria1 docker/victoria-metrics/fb8602e58b4e/victoriametrics/victoria-metrics:v1.102.1[64720]: 2024-08-09T13:41:13.914Z#011info#011VictoriaMetrics/lib/storage/partition.go:1465#011merged (15 parts, 231176199 rows, 176143081 blocks, 2411252145 bytes) into (1 part, 113561638 rows, 19503601 blocks, 431578105 bytes) in 70.846 seconds at 3263098 rows/sec to "/victoria-metrics-data/data/small/2024_08/17E9C1D30055DC99"
2024-08-09T13:41:18.168612+00:00 victoria1 docker/victoria-metrics/fb8602e58b4e/victoriametrics/victoria-metrics:v1.102.1[64720]: 2024-08-09T13:41:18.168Z#011warn#011VictoriaMetrics/app/vmselect/promql/rollup_result_cache.go:77#011resetting rollup result cache because the metric container_network_transmit_errors_total{job=... #omited} (Timestamp=1723210563931, Value=0.000000) has a timestamp older than -search.cacheTimestampOffset=5m0s by 13.069s
2024-08-09T13:41:18.168969+00:00 victoria1 docker/victoria-metrics/fb8602e58b4e/victoriametrics/victoria-metrics:v1.102.1[64720]: 2024-08-09T13:41:18.168Z#011info#011VictoriaMetrics/app/vmselect/promql/rollup_result_cache.go:216#011rollupResult cache has been cleared
2024-08-09T13:41:45.697487+00:00 victoria1 docker/victoria-metrics/fb8602e58b4e/victoriametrics/victoria-metrics:v1.102.1[64720]: 2024-08-09T13:41:45.697Z#011info#011VictoriaMetrics/lib/storage/partition.go:1465#011merged (10 parts, 84359258 rows, 83947992 blocks, 1055850569 bytes) into (1 part, 35761865 rows, 19038960 blocks, 307431145 bytes) in 36.028 seconds at 2341493 rows/sec to "/victoria-metrics-data/data/small/2024_08/17E9C1D30055DCA4"
Almost the same on another Victoria. If I skip all these logs to find something, then nothing:
cat /var/log/docker/victoria-metrics.log | grep -Ev "rollupResult cache has been cleared|search.cacheTimestampOffset|011merged"
Nginx error log only about buffering (on both Victorias), so I don’t think it’s somehow related:
2024/08/09 00:03:41 [warn] 2586365#2586365: *94781939 a client request body is buffered to a temporary file /dev/shm/client_body_temp_path/11/71/0093277111, client: 10.96.1.79, server: victoria1, request: "POST /api/v1/write HTTP/1.1", host: "victoria1"
Screenshots
Screenshots from both Victorias and Prometheus, first - from Victoria1, second - from Victoria2, third - from Prometheus
Used command-line flags
Docker command-line flags (only retention is different, on Victoria2 is 365d, but no such amount of metrics for now):
"Args": [
"-httpListenAddr",
"127.0.0.1:8428",
"-retentionPeriod",
"1095d",
"-httpAuth.username",
"***",
"-httpAuth.password",
"***",
"-search.maxUniqueTimeseries=10000000",
"-search.minStalenessInterval=5m",
"-dedup.minScrapeInterval=60s",
"-maxLabelValueLen=30000",
"-search.latencyOffset=1m",
"-maxLabelsPerTimeseries=50"
],
Additional information
Prometheus config for push (maybe problem somewhere here):
remote_write:
- url: https://victoria1/api/v1/write
basic_auth:
username: ***
password: ***
queue_config:
capacity: 150000
max_samples_per_send: 50000
max_shards: 75
metadata_config:
send_interval: 45s
max_samples_per_send: 50000
- url: https://victoria2/api/v1/write
basic_auth:
username: ***
password: ***
queue_config:
capacity: 150000
max_samples_per_send: 50000
max_shards: 75
metadata_config:
send_interval: 45s
max_samples_per_send: 50000
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 20 (2 by maintainers)
Seems the issue was due to data corruption long time ago. VM1 and VM2 were created long time ago, and VM1 was moved to another servers several times (to not lost metrics). Seems that the data was moved incorrectly (like rsync while Victoria was running). Before I noticed metrics difference, I also moved data from old VM1 server to the new, so corrupted data also was moved. So, I synced data from VM2 (good instance) to VM1 (bad instance) using rsync (while VM2 was stopped). Now I don’t see metrics missing as provided in the issue description and new metrics continue to be saved. If it’s true, strange that no logs about data corruption or something. No exact evidences that data was corrupted, just an assumption, but looks like the true, at least from my side. if it was, maybe there are some checks for data consistency from VM side? And put it explicitly to the logs, if any corruption appears. If data synchronization was completed incorrectly, I saw that:
panic: FATAL: part "/victoria-metrics-data/indexdb/17647E6429E8EF17/17E8BE40FF287598" is listed in "/victoria-metrics-data/indexdb/17647E6429E8EF17/parts.json", but is missing on disk; ensure "/victoria-metrics-data/indexdb/17647E6429E8EF17/parts.json" contents is not corrupted; remove "/victoria-metrics-data/indexdb/17647E6429E8EF17/parts.json" to rebuild its' content from the list of existing parts' content from the list of existing partsbut in this case VM won’t start. But in my case VM worked good all the time. Either data corruption wasn’t found for long time or something else… I will continue monitor metrics in both VMs and let you know later@hagen1778 I added new endpoint to remote_write section with same config on one Prometheus server (remember I have two Prometheus servers with the same configs).
I also tested case if there is some kind of metrics duplication - shut down one prometheus - no effect, no metrics in VM1
About ingestion rate, here for last 30 days:
Blue color - VM2 (good) Red color - VM1 (not good)
rate(vm_rows_inserted_total{type="promremotewrite"}[1m])rate is about 1.5M, but for VM2 there are some spikes about 1.8M and higher.rate(vm_http_requests_total{path="/api/v1/write"}[1m])rate is about 30 for both VMSIf I understood right, you asked about difference between these metrics. Here my query for VM1:
sum by (instance)(prometheus_remote_storage_highest_timestamp_in_seconds{instance=~".*-prometheus-.*"}) - sum by (instance)( prometheus_remote_storage_queue_highest_sent_timestamp_seconds{instance=~".*-prometheus-.*", url="https://victoria1/api/v1/write"})And for VM2:
Results: VM1 - max difference 10 seconds VM2 - max difference 43 seconds
For VM2 looks worse than for VM1, but VM2 is good with metrics) I think, that big difference for VM2 somehow related to the load, CPU load is higher and this server has less cpu cores, but maybe I am wrong. Anyway, for VM1 situation with such metrics are better( correct me please if I am wrong).
Regarding
rollupResult cache has been cleared- I also see this log for both VMs, and all these logs related to k8s metrics, like this:I don’t know, maybe it’s just because k8s metrics are not collected very often. Should I increase
-search.cacheTimestampOffset=10m0sin that case? (maybe this is not related to the current problem)Currently 2.50.1, but I want to update it to latest in the near future
Ok, after vmagent tests, I see the metric has appeared.
Now I’m completely confused why I don’t see the same using Prometheus.
vmagent-prod --version vmagent-20240801-125405-tags-v1.102.1-0-g996b623585vmagent startup flags:
vmagent-prod -promscrape.config vmagent.yaml -remoteWrite.url https://***:***@victoria1/api/v1/write -promscrape.config.strictParse=falsevmagent.yaml:
Nginx log:
And as I mentioned, problem can be not only in that metric, faced the same behaviour with another metrics
@hagen1778 regarding
vm_missing_metric_names_for_metric_id_totalno, I don’t see increasing now. In the screenshot above, spikes were several months ago. At that time I changed retention so probably some old metrics were deleted due old retention period. I have a data from 1st October 2023, so in the picture spikes started in the beginning of May 2024, and at that time retention flag was-retentionPeriod 180d. I changed to 365d and no more spikes, so probably it’s just related to retention. I found in the linked issues a mention about that metric, that’s why I also checked it.Little increasing of
vm_missing_metric_names_for_metric_id_total2 weeks ago - I tested importing problematic metrics toapi/v1/import/prometheusand Victoria deleted that data (I mentioned it in the issue description). Version was v1.99.0 at that time, after that I updated to latest v1.100.0 and after that to latest v1.102.1