VictoriaMetrics: Prometheus ServiceMonitors not working with VM
Is your question request related to a specific component?
vmagent
Describe the question in detail
Hey Team,
We’re planning to migrate from Prometheus to VictoriaMetrics. Prometheus is installed thro’ kube-prom-stack helm chart which includes all the service monitors and pod monitors by default. I read that VMAgent can scrape all these metrics if there’s a ServiceMonitor obj. present for the component. It’s also scrapping some metrics but some metrics from the node exporter are not being scrapped.
Below are our pods
NAME READY STATUS RESTARTS AGE
prometheus-kube-prometheus-operator-57b7ddd6cc-t5jgr 1/1 Running 0 137m
prometheus-kube-state-metrics-6d9f56f868-rtgnz 1/1 Running 0 137m
prometheus-prometheus-kube-prometheus-prometheus-0 3/3 Running 0 138m
prometheus-prometheus-kube-prometheus-prometheus-1 3/3 Running 0 138m
prometheus-prometheus-kube-prometheus-prometheus-2 3/3 Running 0 138m
prometheus-prometheus-node-exporter-42llw 1/1 Running 0 96m
prometheus-prometheus-node-exporter-5bt2w 1/1 Running 0 4h43m
prometheus-prometheus-node-exporter-779hx 1/1 Running 0 56m
prometheus-prometheus-node-exporter-czskg 1/1 Running 0 137m
prometheus-prometheus-node-exporter-dgjgg 1/1 Running 0 4h43m
prometheus-prometheus-node-exporter-jgl52 1/1 Running 0 4h42m
prometheus-prometheus-node-exporter-rx892 1/1 Running 0 4h42m
prometheus-prometheus-node-exporter-svxb9 1/1 Running 0 42m
prometheus-prometheus-node-exporter-vtwjm 1/1 Running 0 4h42m
thanos-compactor-0 1/1 Running 0 137m
thanos-query-f967c9cc4-8x4bq 1/1 Running 0 137m
thanos-store-0 1/1 Running 0 138m
vm-stack-victoria-metrics-operator-6ff65c78cf-vn4ht 1/1 Running 0 137m
vmagent-vm-stack-victoria-metrics-k8s-stack-5886bdbf5f-rd7th 2/2 Running 0 137m
vminsert-vm-stack-victoria-metrics-k8s-stack-65887f874c-2lh7w 1/1 Running 0 138m
vminsert-vm-stack-victoria-metrics-k8s-stack-65887f874c-zf8rb 1/1 Running 0 137m
vmselect-vm-stack-victoria-metrics-k8s-stack-0 1/1 Running 0 138m
vmselect-vm-stack-victoria-metrics-k8s-stack-1 1/1 Running 0 137m
vmselect-vm-stack-victoria-metrics-k8s-stack-2 1/1 Running 0 138m
vmstorage-vm-stack-victoria-metrics-k8s-stack-0 1/1 Running 0 138m
vmstorage-vm-stack-victoria-metrics-k8s-stack-1 1/1 Running 0 138m
vmstorage-vm-stack-victoria-metrics-k8s-stack-2 1/1 Running 0 138m
Below are our service monitors
➜ ~ kubectl get servicemonitors.monitoring.coreos.com -A
NAMESPACE NAME AGE
istio-system istio-component-monitor 47d
monitoring node-exporter-monitor 9h
monitoring prom-thanos-query 47d
monitoring prometheus-kube-prometheus-apiserver 47d
monitoring prometheus-kube-prometheus-kube-controller-manager 47d
monitoring prometheus-kube-prometheus-kube-dns 47d
monitoring prometheus-kube-prometheus-kube-etcd 47d
monitoring prometheus-kube-prometheus-kube-proxy 47d
monitoring prometheus-kube-prometheus-kube-scheduler 47d
monitoring prometheus-kube-prometheus-kubelet 47d
monitoring prometheus-kube-prometheus-operator 47d
monitoring prometheus-kube-prometheus-prometheus 47d
monitoring prometheus-kube-prometheus-thanos-sidecar 47d
monitoring prometheus-kube-state-metrics 47d
monitoring prometheus-oper-istio-controlplane 47d
monitoring prometheus-oper-istio-dataplane 47d
monitoring prometheus-prometheus-node-exporter 47d
monitoring thanos-compactor 47d
monitoring thanos-store 47d
I also enabled some of the VMScrapes from the helm chart
➜ ~ kubectl get vmservicescrapes.operator.victoriametrics.com -A
NAMESPACE NAME AGE
monitoring vm-stack-victoria-metrics-k8s-stack-apiserver 25m
monitoring vm-stack-victoria-metrics-k8s-stack-kube-controller-manager 25m
monitoring vm-stack-victoria-metrics-k8s-stack-kube-etcd 25m
monitoring vm-stack-victoria-metrics-k8s-stack-kube-proxy 25m
monitoring vm-stack-victoria-metrics-k8s-stack-kube-scheduler 25m
monitoring vm-stack-victoria-metrics-k8s-stack-kubedns 25m
monitoring vm-stack-victoria-metrics-k8s-stack-node-exporter 14m
monitoring vm-stack-victoria-metrics-k8s-stack-operator 5h20m
monitoring vmagent-vm-stack-victoria-metrics-k8s-stack 5h20m
monitoring vminsert-vm-stack-victoria-metrics-k8s-stack 5h20m
monitoring vmselect-vm-stack-victoria-metrics-k8s-stack 5h20m
monitoring vmstorage-vm-stack-victoria-metrics-k8s-stack 5h20m
I also tried to mimic and create a node-exporter VMScrape like below but no luck.
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
annotations:
meta.helm.sh/release-name: vm-stack
meta.helm.sh/release-namespace: monitoring
generation: 1
labels:
app.kubernetes.io/instance: vm-stack
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: victoria-metrics-k8s-stack
app.kubernetes.io/version: v1.94.0
helm.sh/chart: victoria-metrics-k8s-stack-0.18.6
name: vm-stack-victoria-metrics-k8s-stack-node-exporter
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
port: http-metrics
scheme: https
tlsConfig:
caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
serverName: kubernetes
jobLabel: jobLabel
namespaceSelector:
matchNames:
- default
selector:
matchLabels:
app.kubernetes.io/instance: prometheus
app.kubernetes.io/name: prometheus-node-exporter
Our Grafana Dashboards work well with Prometheus as datasource but not with VM.
I also noticed, some metrics are present in Prometheus datasource while with VM Datasource those are not present. like kube_pod_info for example. How do we fix this ?
Troubleshooting docs
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 18 (8 by maintainers)
Hey @Haleygo,
Below is the log of vmagent and i see some errors for scrapping kubelet service.
How do i add scrape config to vmagent thro’
victoria-metrics-k8s-stackhelm chart.