VictoriaMetrics: Prometheus ServiceMonitors not working with VM

Is your question request related to a specific component?

vmagent

Describe the question in detail

Hey Team,

We’re planning to migrate from Prometheus to VictoriaMetrics. Prometheus is installed thro’ kube-prom-stack helm chart which includes all the service monitors and pod monitors by default. I read that VMAgent can scrape all these metrics if there’s a ServiceMonitor obj. present for the component. It’s also scrapping some metrics but some metrics from the node exporter are not being scrapped.

Below are our pods

NAME                                                            READY   STATUS    RESTARTS   AGE
prometheus-kube-prometheus-operator-57b7ddd6cc-t5jgr            1/1     Running   0          137m
prometheus-kube-state-metrics-6d9f56f868-rtgnz                  1/1     Running   0          137m
prometheus-prometheus-kube-prometheus-prometheus-0              3/3     Running   0          138m
prometheus-prometheus-kube-prometheus-prometheus-1              3/3     Running   0          138m
prometheus-prometheus-kube-prometheus-prometheus-2              3/3     Running   0          138m
prometheus-prometheus-node-exporter-42llw                       1/1     Running   0          96m
prometheus-prometheus-node-exporter-5bt2w                       1/1     Running   0          4h43m
prometheus-prometheus-node-exporter-779hx                       1/1     Running   0          56m
prometheus-prometheus-node-exporter-czskg                       1/1     Running   0          137m
prometheus-prometheus-node-exporter-dgjgg                       1/1     Running   0          4h43m
prometheus-prometheus-node-exporter-jgl52                       1/1     Running   0          4h42m
prometheus-prometheus-node-exporter-rx892                       1/1     Running   0          4h42m
prometheus-prometheus-node-exporter-svxb9                       1/1     Running   0          42m
prometheus-prometheus-node-exporter-vtwjm                       1/1     Running   0          4h42m
thanos-compactor-0                                              1/1     Running   0          137m
thanos-query-f967c9cc4-8x4bq                                    1/1     Running   0          137m
thanos-store-0                                                  1/1     Running   0          138m
vm-stack-victoria-metrics-operator-6ff65c78cf-vn4ht             1/1     Running   0          137m
vmagent-vm-stack-victoria-metrics-k8s-stack-5886bdbf5f-rd7th    2/2     Running   0          137m
vminsert-vm-stack-victoria-metrics-k8s-stack-65887f874c-2lh7w   1/1     Running   0          138m
vminsert-vm-stack-victoria-metrics-k8s-stack-65887f874c-zf8rb   1/1     Running   0          137m
vmselect-vm-stack-victoria-metrics-k8s-stack-0                  1/1     Running   0          138m
vmselect-vm-stack-victoria-metrics-k8s-stack-1                  1/1     Running   0          137m
vmselect-vm-stack-victoria-metrics-k8s-stack-2                  1/1     Running   0          138m
vmstorage-vm-stack-victoria-metrics-k8s-stack-0                 1/1     Running   0          138m
vmstorage-vm-stack-victoria-metrics-k8s-stack-1                 1/1     Running   0          138m
vmstorage-vm-stack-victoria-metrics-k8s-stack-2                 1/1     Running   0          138m

Below are our service monitors

➜  ~ kubectl get servicemonitors.monitoring.coreos.com -A
NAMESPACE      NAME                                                 AGE
istio-system   istio-component-monitor                              47d
monitoring     node-exporter-monitor                                9h
monitoring     prom-thanos-query                                    47d
monitoring     prometheus-kube-prometheus-apiserver                 47d
monitoring     prometheus-kube-prometheus-kube-controller-manager   47d
monitoring     prometheus-kube-prometheus-kube-dns                  47d
monitoring     prometheus-kube-prometheus-kube-etcd                 47d
monitoring     prometheus-kube-prometheus-kube-proxy                47d
monitoring     prometheus-kube-prometheus-kube-scheduler            47d
monitoring     prometheus-kube-prometheus-kubelet                   47d
monitoring     prometheus-kube-prometheus-operator                  47d
monitoring     prometheus-kube-prometheus-prometheus                47d
monitoring     prometheus-kube-prometheus-thanos-sidecar            47d
monitoring     prometheus-kube-state-metrics                        47d
monitoring     prometheus-oper-istio-controlplane                   47d
monitoring     prometheus-oper-istio-dataplane                      47d
monitoring     prometheus-prometheus-node-exporter                  47d
monitoring     thanos-compactor                                     47d
monitoring     thanos-store                                         47d

I also enabled some of the VMScrapes from the helm chart

➜  ~ kubectl get vmservicescrapes.operator.victoriametrics.com -A
NAMESPACE    NAME                                                          AGE
monitoring   vm-stack-victoria-metrics-k8s-stack-apiserver                 25m
monitoring   vm-stack-victoria-metrics-k8s-stack-kube-controller-manager   25m
monitoring   vm-stack-victoria-metrics-k8s-stack-kube-etcd                 25m
monitoring   vm-stack-victoria-metrics-k8s-stack-kube-proxy                25m
monitoring   vm-stack-victoria-metrics-k8s-stack-kube-scheduler            25m
monitoring   vm-stack-victoria-metrics-k8s-stack-kubedns                   25m
monitoring   vm-stack-victoria-metrics-k8s-stack-node-exporter             14m
monitoring   vm-stack-victoria-metrics-k8s-stack-operator                  5h20m
monitoring   vmagent-vm-stack-victoria-metrics-k8s-stack                   5h20m
monitoring   vminsert-vm-stack-victoria-metrics-k8s-stack                  5h20m
monitoring   vmselect-vm-stack-victoria-metrics-k8s-stack                  5h20m
monitoring   vmstorage-vm-stack-victoria-metrics-k8s-stack                 5h20m

I also tried to mimic and create a node-exporter VMScrape like below but no luck.

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
  annotations:
    meta.helm.sh/release-name: vm-stack
    meta.helm.sh/release-namespace: monitoring
  generation: 1
  labels:
    app.kubernetes.io/instance: vm-stack
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: victoria-metrics-k8s-stack
    app.kubernetes.io/version: v1.94.0
    helm.sh/chart: victoria-metrics-k8s-stack-0.18.6
  name: vm-stack-victoria-metrics-k8s-stack-node-exporter
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    port: http-metrics
    scheme: https
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      serverName: kubernetes
  jobLabel: jobLabel
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app.kubernetes.io/instance: prometheus
      app.kubernetes.io/name: prometheus-node-exporter

Our Grafana Dashboards work well with Prometheus as datasource but not with VM. image image

I also noticed, some metrics are present in Prometheus datasource while with VM Datasource those are not present. like kube_pod_info for example. How do we fix this ?

Troubleshooting docs

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 18 (8 by maintainers)

Most upvoted comments

Hey @Haleygo,

Below is the log of vmagent and i see some errors for scrapping kubelet service.

2023-11-07T12:19:15.061Z	info	VictoriaMetrics/lib/logger/flag.go:12	build version: vmagent-20231002-215021-tags-v1.94.0-0-gf13a96f42
2023-11-07T12:19:15.061Z	info	VictoriaMetrics/lib/logger/flag.go:13	command-line flags
2023-11-07T12:19:15.061Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -httpListenAddr=":8429"
2023-11-07T12:19:15.061Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -promscrape.config="/etc/vmagent/config_out/vmagent.env.yaml"
2023-11-07T12:19:15.061Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -promscrape.streamParse="true"
2023-11-07T12:19:15.061Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -remoteWrite.maxDiskUsagePerURL="1073741824"
2023-11-07T12:19:15.061Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -remoteWrite.tmpDataPath="/tmp/vmagent-remotewrite-data"
2023-11-07T12:19:15.061Z	info	VictoriaMetrics/lib/logger/flag.go:20	  -remoteWrite.url="secret"
2023-11-07T12:19:15.061Z	info	VictoriaMetrics/app/vmagent/main.go:121	starting vmagent at ":8429"...
2023-11-07T12:19:15.062Z	info	VictoriaMetrics/lib/memory/memory.go:42	limiting caches to 314572800 bytes, leaving 209715200 bytes to the OS according to -memory.allowedPercent=60
2023-11-07T12:19:15.071Z	info	VictoriaMetrics/lib/persistentqueue/fastqueue.go:60	opened fast persistent queue at "/tmp/vmagent-remotewrite-data/persistent-queue/1_ACAE21DBDABF66A7" with maxInmemoryBlocks=200, it contains 0 pending bytes
2023-11-07T12:19:15.087Z	info	VictoriaMetrics/app/vmagent/remotewrite/client.go:200	initialized client for -remoteWrite.url="1:secret-url"
2023-11-07T12:19:15.087Z	info	VictoriaMetrics/app/vmagent/main.go:147	started vmagent in 0.026 seconds
2023-11-07T12:19:15.087Z	info	VictoriaMetrics/lib/promscrape/scraper.go:113	reading Prometheus configs from "/etc/vmagent/config_out/vmagent.env.yaml"
2023-11-07T12:19:15.140Z	info	VictoriaMetrics/lib/httpserver/httpserver.go:96	starting http server at http://127.0.0.1:8429/
2023-11-07T12:19:15.140Z	info	VictoriaMetrics/lib/httpserver/httpserver.go:97	pprof handlers are exposed at http://127.0.0.1:8429/debug/pprof/
2023-11-07T12:19:15.258Z	info	VictoriaMetrics/lib/promscrape/config.go:132	starting service discovery routines...
2023-11-07T12:19:15.258Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:525	started pod watcher for "https://10.116.0.1:443/api/v1/namespaces/default/pods"
2023-11-07T12:19:15.267Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:703	reloaded 1 objects from "https://10.116.0.1:443/api/v1/namespaces/default/pods?resourceVersion=0&resourceVersionMatch=NotOlderThan" in 0.009s; updated=0, removed=0, added=1, resourceVersion="62558580"
2023-11-07T12:19:15.267Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:525	started service watcher for "https://10.116.0.1:443/api/v1/namespaces/default/services"
2023-11-07T12:19:15.269Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:703	reloaded 1 objects from "https://10.116.0.1:443/api/v1/namespaces/default/services?resourceVersion=0&resourceVersionMatch=NotOlderThan" in 0.002s; updated=0, removed=0, added=1, resourceVersion="62558527"
2023-11-07T12:19:15.269Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:525	started endpoints watcher for "https://10.116.0.1:443/api/v1/namespaces/default/endpoints"
2023-11-07T12:19:15.276Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:703	reloaded 2 objects from "https://10.116.0.1:443/api/v1/namespaces/default/endpoints?resourceVersion=0&resourceVersionMatch=NotOlderThan" in 0.007s; updated=0, removed=0, added=2, resourceVersion="62558586"
2023-11-07T12:19:15.276Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:525	started pod watcher for "https://10.116.0.1:443/api/v1/namespaces/kube-system/pods"
2023-11-07T12:19:15.458Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:703	reloaded 55 objects from "https://10.116.0.1:443/api/v1/namespaces/kube-system/pods?resourceVersion=0&resourceVersionMatch=NotOlderThan" in 0.182s; updated=0, removed=0, added=55, resourceVersion="62558580"
2023-11-07T12:19:15.458Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:525	started service watcher for "https://10.116.0.1:443/api/v1/namespaces/kube-system/services"
2023-11-07T12:19:15.460Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:703	reloaded 13 objects from "https://10.116.0.1:443/api/v1/namespaces/kube-system/services?resourceVersion=0&resourceVersionMatch=NotOlderThan" in 0.003s; updated=0, removed=0, added=13, resourceVersion="62558527"
2023-11-07T12:19:15.460Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:525	started endpoints watcher for "https://10.116.0.1:443/api/v1/namespaces/kube-system/endpoints"
2023-11-07T12:19:15.461Z	info	VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:192	reloading relabel configs pointed by -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig
2023-11-07T12:19:15.461Z	info	VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:203	successfully reloaded relabel configs
2023-11-07T12:19:15.473Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:703	reloaded 16 objects from "https://10.116.0.1:443/api/v1/namespaces/kube-system/endpoints?resourceVersion=0&resourceVersionMatch=NotOlderThan" in 0.012s; updated=0, removed=0, added=16, resourceVersion="62558586"
2023-11-07T12:19:15.474Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:525	started pod watcher for "https://10.116.0.1:443/api/v1/namespaces/monitoring/pods"
2023-11-07T12:19:15.569Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:703	reloaded 28 objects from "https://10.116.0.1:443/api/v1/namespaces/monitoring/pods?resourceVersion=0&resourceVersionMatch=NotOlderThan" in 0.094s; updated=0, removed=0, added=28, resourceVersion="62558580"
2023-11-07T12:19:15.569Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:525	started service watcher for "https://10.116.0.1:443/api/v1/namespaces/monitoring/services"
2023-11-07T12:19:15.572Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:703	reloaded 15 objects from "https://10.116.0.1:443/api/v1/namespaces/monitoring/services?resourceVersion=0&resourceVersionMatch=NotOlderThan" in 0.003s; updated=0, removed=0, added=15, resourceVersion="62558527"
2023-11-07T12:19:15.572Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:525	started endpoints watcher for "https://10.116.0.1:443/api/v1/namespaces/monitoring/endpoints"
2023-11-07T12:19:15.642Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:703	reloaded 15 objects from "https://10.116.0.1:443/api/v1/namespaces/monitoring/endpoints?resourceVersion=0&resourceVersionMatch=NotOlderThan" in 0.070s; updated=0, removed=0, added=15, resourceVersion="62558586"
2023-11-07T12:19:15.646Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:525	started node watcher for "https://10.116.0.1:443/api/v1/nodes"
2023-11-07T12:19:15.664Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:703	reloaded 10 objects from "https://10.116.0.1:443/api/v1/nodes?resourceVersion=0&resourceVersionMatch=NotOlderThan" in 0.018s; updated=0, removed=0, added=10, resourceVersion="62558491"
2023-11-07T12:19:15.665Z	info	VictoriaMetrics/lib/promscrape/config.go:138	started service discovery routines in 0.407 seconds
2023-11-07T12:19:15.666Z	info	VictoriaMetrics/lib/promscrape/scraper.go:156	SIGHUP received; reloading Prometheus configs from "/etc/vmagent/config_out/vmagent.env.yaml"
2023-11-07T12:19:15.952Z	info	VictoriaMetrics/lib/promscrape/config.go:143	restarting service discovery routines...
2023-11-07T12:19:16.052Z	info	VictoriaMetrics/lib/promscrape/scraper.go:430	kubernetes_sd_configs: added targets: 44, removed targets: 0; total targets: 44
2023-11-07T12:19:16.055Z	info	VictoriaMetrics/lib/promscrape/config.go:186	restarted service discovery routines in 0.103 seconds, stopped=0, started=1, restarted=0
2023-11-07T12:19:16.142Z	info	VictoriaMetrics/lib/promscrape/scraper.go:200	found changes in "/etc/vmagent/config_out/vmagent.env.yaml"; applying these changes
2023-11-07T12:19:16.155Z	info	VictoriaMetrics/lib/promscrape/scraper.go:430	kubernetes_sd_configs: added targets: 1, removed targets: 0; total targets: 45
2023-11-07T12:19:55.749Z	warn	VictoriaMetrics/lib/promscrape/scrapework.go:387	cannot scrape target "http://10.112.3.2:10054/metrics" ({container="sidecar",endpoint="http-metrics-dnsmasq",instance="10.112.3.2:10054",job="kube-dns",namespace="kube-system",pod="kube-dns-566c4c6b8-q4bkh",service="vm-stack-victoria-metrics-k8s-stack-kubedns"}) 1 out of 2 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "http://10.112.3.2:10054/metrics": Get "http://10.112.3.2:10054/metrics": dial tcp4 10.112.3.2:10054: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses
2023-11-07T12:19:57.742Z	warn	VictoriaMetrics/lib/promscrape/scrapework.go:387	cannot scrape target "http://10.112.3.3:10054/metrics" ({container="sidecar",endpoint="http-metrics-dnsmasq",instance="10.112.3.3:10054",job="kube-dns",namespace="kube-system",pod="kube-dns-566c4c6b8-bb52z",service="vm-stack-victoria-metrics-k8s-stack-kubedns"}) 1 out of 3 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "http://10.112.3.3:10054/metrics": Get "http://10.112.3.3:10054/metrics": dial tcp4 10.112.3.3:10054: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses
2023-11-07T12:20:06.346Z	warn	VictoriaMetrics/lib/promscrape/scrapework.go:387	cannot scrape target "http://10.112.1.35:8080/metrics" ({container="victoria-metrics-operator",endpoint="http",instance="10.112.1.35:8080",job="vm-stack-victoria-metrics-operator",namespace="monitoring",pod="vm-stack-victoria-metrics-operator-6ff65c78cf-6vlkb",service="vm-stack-victoria-metrics-operator"}) 1 out of 3 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "http://10.112.1.35:8080/metrics": Get "http://10.112.1.35:8080/metrics": dial tcp4 10.112.1.35:8080: connect: no route to host; try -enableTCP6 command-line flag if you scrape ipv6 addresses
2023-11-07T12:20:15.754Z	info	VictoriaMetrics/lib/promscrape/scraper.go:430	kubernetes_sd_configs: added targets: 5, removed targets: 5; total targets: 45
2023-11-07T12:25:14.168Z	warn	VictoriaMetrics/lib/promscrape/scrapework.go:387	cannot scrape target "https://10.128.15.192:10250/metrics/cadvisor" ({addon_gke_io_node_local_dns_ds_ready="true",beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="e2-custom-10-49152",beta_kubernetes_io_os="linux",cloud_google_com_gke_boot_disk="pd-standard",cloud_google_com_gke_container_runtime="con..gke-justcall-testapps-infra-node-pool-926e87bd-t6zp",node_kubernetes_io_instance_type="e2-custom-10-49152",teamname="infra",topology_gke_io_zone="us-central1-a",topology_kubernetes_io_region="us-central1",topology_kubernetes_io_zone="us-central1-a"}) 1 out of 12 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "https://10.128.15.192:10250/metrics/cadvisor": Get "https://10.128.15.192:10250/metrics/cadvisor": context deadline exceeded
2023-11-07T12:25:14.801Z	warn	VictoriaMetrics/lib/promscrape/scrapework.go:387	cannot scrape target "https://10.128.15.192:10250/metrics/probes" ({addon_gke_io_node_local_dns_ds_ready="true",beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="e2-custom-10-49152",beta_kubernetes_io_os="linux",cloud_google_com_gke_boot_disk="pd-standard",cloud_google_com_gke_container_runtime="con..gke-justcall-testapps-infra-node-pool-926e87bd-t6zp",node_kubernetes_io_instance_type="e2-custom-10-49152",teamname="infra",topology_gke_io_zone="us-central1-a",topology_kubernetes_io_region="us-central1",topology_kubernetes_io_zone="us-central1-a"}) 1 out of 12 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "https://10.128.15.192:10250/metrics/probes": Get "https://10.128.15.192:10250/metrics/probes": context deadline exceeded
2023-11-07T12:25:39.019Z	info	VictoriaMetrics/lib/promscrape/scraper.go:156	SIGHUP received; reloading Prometheus configs from "/etc/vmagent/config_out/vmagent.env.yaml"
2023-11-07T12:25:39.034Z	info	VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:192	reloading relabel configs pointed by -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig
2023-11-07T12:25:39.034Z	info	VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:203	successfully reloaded relabel configs
2023-11-07T12:25:39.052Z	info	VictoriaMetrics/lib/promscrape/config.go:143	restarting service discovery routines...
2023-11-07T12:25:39.053Z	info	VictoriaMetrics/lib/promscrape/config.go:186	restarted service discovery routines in 0.001 seconds, stopped=0, started=0, restarted=1
2023-11-07T12:25:39.058Z	info	VictoriaMetrics/lib/promscrape/scraper.go:200	found changes in "/etc/vmagent/config_out/vmagent.env.yaml"; applying these changes
2023-11-07T12:25:39.145Z	warn	VictoriaMetrics/lib/promscrape/scrapework.go:387	cannot scrape target "https://10.128.15.192:10250/metrics" ({addon_gke_io_node_local_dns_ds_ready="true",beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="e2-custom-10-49152",beta_kubernetes_io_os="linux",cloud_google_com_gke_boot_disk="pd-standard",cloud_google_com_gke_container_runtime="con..gke-justcall-testapps-infra-node-pool-926e87bd-t6zp",node_kubernetes_io_instance_type="e2-custom-10-49152",teamname="infra",topology_gke_io_zone="us-central1-a",topology_kubernetes_io_region="us-central1",topology_kubernetes_io_zone="us-central1-a"}) 1 out of 13 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "https://10.128.15.192:10250/metrics": Get "https://10.128.15.192:10250/metrics": context canceled
2023-11-07T12:25:39.442Z	info	VictoriaMetrics/lib/promscrape/scraper.go:430	kubernetes_sd_configs: added targets: 0, removed targets: 3; total targets: 42
2023-11-07T12:25:49.140Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:737	stopped pod watcher for "https://10.116.0.1:443/api/v1/namespaces/default/pods"
2023-11-07T12:25:49.140Z	info	VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:737	stopped service watcher for "https://10.116.0.1:443/api/v1/namespaces/default/services"
2023-11-07T12:34:09.364Z	warn	VictoriaMetrics/lib/promscrape/scrapework.go:387	cannot scrape target "https://10.128.15.205:10250/metrics/probes" ({addon_gke_io_node_local_dns_ds_ready="true",beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="e2-standard-2",beta_kubernetes_io_os="linux",cloud_google_com_gke_boot_disk="pd-standard",cloud_google_com_gke_container_runtime="containe..metrics/probes",node="gke-justcall-testapp-justcall-testapp-1c21a94b-l4rc",node_kubernetes_io_instance_type="e2-standard-2",topology_gke_io_zone="us-central1-c",topology_kubernetes_io_region="us-central1",topology_kubernetes_io_zone="us-central1-c"}) 1 out of 30 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "https://10.128.15.205:10250/metrics/probes": Get "https://10.128.15.205:10250/metrics/probes": context deadline exceeded
2023-11-07T12:34:19.439Z	warn	VictoriaMetrics/lib/promscrape/scrapework.go:387	cannot scrape target "https://10.128.15.205:10250/metrics/cadvisor" ({addon_gke_io_node_local_dns_ds_ready="true",beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="e2-standard-2",beta_kubernetes_io_os="linux",cloud_google_com_gke_boot_disk="pd-standard",cloud_google_com_gke_container_runtime="containe..trics/cadvisor",node="gke-justcall-testapp-justcall-testapp-1c21a94b-l4rc",node_kubernetes_io_instance_type="e2-standard-2",topology_gke_io_zone="us-central1-c",topology_kubernetes_io_region="us-central1",topology_kubernetes_io_zone="us-central1-c"}) 1 out of 30 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "https://10.128.15.205:10250/metrics/cadvisor": Get "https://10.128.15.205:10250/metrics/cadvisor": context deadline exceeded
2023-11-07T12:34:21.958Z	warn	VictoriaMetrics/lib/promscrape/scrapework.go:387	cannot scrape target "https://10.128.15.205:10250/metrics" ({addon_gke_io_node_local_dns_ds_ready="true",beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="e2-standard-2",beta_kubernetes_io_os="linux",cloud_google_com_gke_boot_disk="pd-standard",cloud_google_com_gke_container_runtime="containe..path="/metrics",node="gke-justcall-testapp-justcall-testapp-1c21a94b-l4rc",node_kubernetes_io_instance_type="e2-standard-2",topology_gke_io_zone="us-central1-c",topology_kubernetes_io_region="us-central1",topology_kubernetes_io_zone="us-central1-c"}) 1 out of 31 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "https://10.128.15.205:10250/metrics": Get "https://10.128.15.205:10250/metrics": context deadline exceeded
2023-11-07T12:34:39.364Z	warn	VictoriaMetrics/lib/promscrape/scrapework.go:387	cannot scrape target "https://10.128.15.205:10250/metrics/probes" ({addon_gke_io_node_local_dns_ds_ready="true",beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="e2-standard-2",beta_kubernetes_io_os="linux",cloud_google_com_gke_boot_disk="pd-standard",cloud_google_com_gke_container_runtime="containe..metrics/probes",node="gke-justcall-testapp-justcall-testapp-1c21a94b-l4rc",node_kubernetes_io_instance_type="e2-standard-2",topology_gke_io_zone="us-central1-c",topology_kubernetes_io_region="us-central1",topology_kubernetes_io_zone="us-central1-c"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "https://10.128.15.205:10250/metrics/probes": Get "https://10.128.15.205:10250/metrics/probes": context deadline exceeded
2023-11-07T12:34:45.745Z	warn	VictoriaMetrics/lib/promscrape/scrapework.go:387	cannot scrape target "https://10.128.15.205:10250/metrics/cadvisor" ({addon_gke_io_node_local_dns_ds_ready="true",beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="e2-standard-2",beta_kubernetes_io_os="linux",cloud_google_com_gke_boot_disk="pd-standard",cloud_google_com_gke_container_runtime="containe..trics/cadvisor",node="gke-justcall-testapp-justcall-testapp-1c21a94b-l4rc",node_kubernetes_io_instance_type="e2-standard-2",topology_gke_io_zone="us-central1-c",topology_kubernetes_io_region="us-central1",topology_kubernetes_io_zone="us-central1-c"}) 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: cannot read data: cannot scrape "https://10.128.15.205:10250/metrics/cadvisor": Get "https://10.128.15.205:10250/metrics/cadvisor": context canceled
2023-11-07T12:34:45.949Z	info	VictoriaMetrics/lib/promscrape/scraper.go:430	kubernetes_sd_configs: added targets: 0, removed targets: 3; total targets: 39

How do i add scrape config to vmagent thro’ victoria-metrics-k8s-stack helm chart.