prometheus-kubernetes: kubelet Kubernetes node labels are missing

@camilb I’m facing the same issue with the node-exporter.

Hi guys, probably this is a more cleaner way to do this (on the serviceMonitor object)

prometheus-node-exporter:
  prometheus:
    monitor:
      enabled: true
      relabelings:
        - sourceLabels: [__meta_kubernetes_endpoint_node_name]
          targetLabel: node

These values are for kube-prometheus-stack helm chart. Check service discovery on prometheus for more labels. most of the useful ones are available.

I managed to sort out this issue. There are several things worth mentioning. The Prometheus Operator ServiceMonitor object/kind/crd does not provide access to the kubernetes_sd_config role=node

  kubernetes_sd_configs:
  - api_server: null
    role: node

And therefor non of the __meta_kubernetes_node_* labels are available. This can be worked around using additionalScrapeConfigs in the PrometheusSpec (https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusspec).

Trying to use targetLabels in the service monitor will fail, becasue they will translate into __meta_kubernetes_service_label_<labelname> as the role of targets created by the automated processes in the operator always are service.

  kubernetes_sd_configs:
  - api_server: null
    role: service

So I ended up removing the kubelet ServiceMonitor (prometheus-k8s-service-monitor-kubelet.yaml), and replacing this with a custom scraping config. To get this to work you have to do several things.

1. Alter the rbac config to allow node access a the cluster scope

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: prometheus-k8s
  namespace: default
rules:
- apiGroups: [""]
  resources:
  - nodes/metrics  
  - nodes  
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"] 
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups: [""]
  resources:
  - nodes/metrics
  - nodes  
  - endpoints
  - pods
  - services
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

1. Create a yaml file with your custom scraping configs kubelet.yaml

- job_name: 'kubernetes-nodes'
  scheme: https
  metrics_path: /metrics
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - source_labels: [__meta_kubernetes_node_address_InternalIP]      
    target_label: __address__
    regex: (.+)
    replacement: ${1}:10250

- job_name: 'kubernetes-cadvisor'
  scheme: https
  metrics_path: /metrics/cadvisor
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - source_labels: [__meta_kubernetes_node_address_InternalIP]      
    target_label: __address__
    regex: (.+)
    replacement: ${1}:10250  

1. Create a secret with content from the custom scraping config yaml kubectl create secret generic additional-scrape-configs --from-file=kubelet.yaml=.\kubelet.yaml

2. Modify the Prometheus config to include the customs scraping configs

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  externalUrl: xxx
  replicas: 2
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: prometheus-rulefiles
  serviceAccountName: prometheus-k8s
  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: kubelet.yaml
  serviceMonitorSelector:
    matchExpressions:
    - key: k8s-app
      operator: Exists
  version: v2.2.1

All node labels are then propagated to metrics like container_memory_working_set_bytes, machine_memory_bytes etc.