VictoriaMetrics: Victorial metrics is showing data for nodes, that don't have label in filter
Describe the bug
When I filter data for containers on host based on host label, I am getting some amount of data for containers, that not part of that host. I suppose that this is linked with the situation, that different service are using the same IP for node from time to time (nodes are creating and deleted dynamically)
Metric is container_cpu_usage_seconds_total
I am expecting to exclude such metrics in the filter, because they are not really part of existing node( red line on screenhot )
To Reproduce
Create host with label=1 and container_1 on board, wait for data for this host in monitoring. Delete host and immediately recreate it with the same ip and label=2 and container_2 on board. Filter container_cpu_usage_seconds_total{label='2'}.
You will see the small amount of data with container_1 for host, where we have container_2 only
Version
1.102.0
Logs
No response
Screenshots
Used command-line flags
No response
Additional information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (10 by maintainers)
I do see in in
Discovered targetsunhealthy target with ip10.10.10.45withlabel=1and healthy with the same ip andlabel=2. When this happened, I see the data on graph, as on screenhot. After some time unhealthy target is removed fromDiscovered targets, but data left on grapth. So I suppose faster removing of unhealthy targets may solve an issue, but I don’t know how to enforce victoria to do thisWe have
gce_sd_configs, that looks for__meta_gce_metadata_labeland usingrelabel_configsit creates the label for victoria with keylabel, and value, that it contains. In our case it islabel=2May I ask how
label=2is attached to the metrics? Is it attached by service discovery or by cadvisor_exporter?It’s likely that the labels have changed for service discovery, but incorrect container metrics still exist on cadvisor_exporter, which vmagent seems unable to handle.