prometheus-adapter: Prometheus adapter pod unable to get node metrics

This issue is still present in v0.10.0 edit: Got this working. Running v0.10.0 on EKS 1.23, using kube-prometheus-stack. I needed to add a relabeling config (https://github.com/prometheus-community/helm-charts/blob/0b928f341240c76d8513534035a825686ed28a4b/charts/kube-prometheus-stack/values.yaml#L471) to the ServiceMonitor for node-exporter

  prometheus-node-exporter:
    prometheus:
      monitor:
        relabelings:
          - sourceLabels: [__meta_kubernetes_pod_node_name]
            separator: ;
            regex: ^(.*)$
            targetLabel: node
            replacement: $1
            action: replace

After that I used this form of the query (https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/deploy/manifests/config-map.yaml)

For the cpu query, the labelMatchers should match node and not instance. As for memory, we have some relabeling in place in kube-prometheus for node-exporter: https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/node-exporter-serviceMonitor.yaml

I’ll try to reproduce with your query, but with the one from kube-prometheus I wasn’t able to so far.

I was facing the same problem with Amazon EKS version 1.21-eks.2 with both, prometheus-server and prometheus-adapter installed using the community Charts, using the example in the README. The version are as below:

$ helm ls -n prometheus-system 
NAME                    NAMESPACE               REVISION        UPDATED                                 STATUS          CHART                       APP VERSION
prometheus              prometheus-system       1               2021-10-12 01:27:02.127990141 +0000 UTC deployed        prometheus-14.9.2           2.26.0     
prometheus-adapter      prometheus-system       6               2021-10-14 19:20:03.800011211 +0000 UTC deployed        prometheus-adapter-2.17.0   v0.9.0     

Following the workaround proposed by @junaid-ali, I was able to make it work changing the association of resource nodes to the label instance (instead of the original node). My value file is currently like this:

prometheus:
  path: ""
  port: 80
  url: http://prometheus-server.prometheus-system.svc
rules:
  resource:
    cpu:
      containerLabel: container
      containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>,
        container!=""}[3m])) by (<<.GroupBy>>)
      nodeQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>, id='/'}[3m]))
        by (<<.GroupBy>>)
      resources:
        overrides:
          instance:
            resource: node
          namespace:
            resource: namespace
          pod:
            resource: pod
    memory:
      containerLabel: container
      containerQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>, container!=""})
        by (<<.GroupBy>>)
      nodeQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>,id='/'})
        by (<<.GroupBy>>)
      resources:
        overrides:
          instance:
            resource: node
          namespace:
            resource: namespace
          pod:
            resource: pod
    window: 3m

After that, I’m now able to query resource metrics with kubectl top nodes/pods:

$ kubectl top nodes
NAME                          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-10-1-10-60.ec2.internal    47m          2%     657Mi           9%        
ip-10-1-14-239.ec2.internal   76m          3%     1121Mi          16%       

$ kubectl top pods -A
NAMESPACE           NAME                                             CPU(cores)   MEMORY(bytes)   
kube-system         aws-node-2h4k6                                   3m           47Mi            
kube-system         aws-node-fdspx                                   3m           48Mi            
kube-system         coredns-66cb55d4f4-7g7x4                         0m           10Mi            
kube-system         coredns-66cb55d4f4-7wzsc                         1m           9Mi             
kube-system         kube-proxy-fd9ps                                 0m           13Mi            
kube-system         kube-proxy-fsbwq                                 0m           13Mi            
prometheus-system   prometheus-adapter-8bcbbfb8b-gv8m8               10m          39Mi            
prometheus-system   prometheus-alertmanager-787f86875f-x9skk         0m           12Mi            
prometheus-system   prometheus-kube-state-metrics-58c5cd6ddb-666td   0m           11Mi            
prometheus-system   prometheus-node-exporter-4rh98                   0m           7Mi             
prometheus-system   prometheus-node-exporter-5xv2s                   0m           7Mi             
prometheus-system   prometheus-pushgateway-6bd6fcd9b8-m4nmg          0m           7Mi             
prometheus-system   prometheus-server-648c978678-9dbbx               13m          370Mi  

i undid the node overrides and put it back to how the readme has it, seems to have resolved the issue for me

@nicraMarcin I don’t declare any additional rules

@dgrisonnet it’s only happening for nodes. Also, it always returns error: metrics not available yet for nodes (and not an intermittent issue); on re-creating the Prometheus Adapter pod, the issue goes away.