VictoriaMetrics: Victorial metrics is showing data for nodes, that don't have label in filter

Describe the bug

When I filter data for containers on host based on host label, I am getting some amount of data for containers, that not part of that host. I suppose that this is linked with the situation, that different service are using the same IP for node from time to time (nodes are creating and deleted dynamically) image

Metric is container_cpu_usage_seconds_total

I am expecting to exclude such metrics in the filter, because they are not really part of existing node( red line on screenhot )

To Reproduce

Create host with label=1 and container_1 on board, wait for data for this host in monitoring. Delete host and immediately recreate it with the same ip and label=2 and container_2 on board. Filter container_cpu_usage_seconds_total{label='2'}. You will see the small amount of data with container_1 for host, where we have container_2 only

Version

1.102.0

Logs

No response

Screenshots

image

Used command-line flags

No response

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22 (10 by maintainers)

Most upvoted comments

I do see in in Discovered targets unhealthy target with ip 10.10.10.45 with label=1 and healthy with the same ip and label=2. When this happened, I see the data on graph, as on screenhot. After some time unhealthy target is removed from Discovered targets, but data left on grapth. So I suppose faster removing of unhealthy targets may solve an issue, but I don’t know how to enforce victoria to do this

We have gce_sd_configs, that looks for __meta_gce_metadata_label and using relabel_configs it creates the label for victoria with key label, and value, that it contains. In our case it is label=2

May I ask how label=2 is attached to the metrics? Is it attached by service discovery or by cadvisor_exporter?

You will see the small amount of data with container_1 for host, where we have container_2

It’s likely that the labels have changed for service discovery, but incorrect container metrics still exist on cadvisor_exporter, which vmagent seems unable to handle.