prometheus-adapter: Resources list is empty at /apis/custom.metrics.k8s.io/v1beta1/

Hi there! I have the following metrics in Prometheus: http_requests_total{namespace!="",pod!=""} -> http_requests_total{endpoint="http",instance="100.96.3.10:8080",job="sample-app",namespace="default",pod="sample-app-74f4657c87-5s54g",service="sample-app"}

My config is:

apiVersion: v1
kind: ConfigMap
metadata:
  name: adapter-config
  namespace: custom-metrics
data:
  config.yaml: |
    rules:
    - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
      resources:
        overrides:
          namespace: {resource: "namespace"}
          pod: {resource: "pod"}
      name:
        matches: "^(.*)_total"
        as: "${1}_per_second"
      metricsQuery: 'http_requests_total{<<.LabelMatchers>>}'

When I check I get:

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/ --v=6
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[]}

What can be wrong? Now I’m just trying to make it work Use image directxman12/k8s-prometheus-adapter-amd64 Is there way I can debug? Or which logs should I check?

k8s version:

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:05:37Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 2
Comments: 20 (3 by maintainers)

Most upvoted comments

I have right URL specified. But still facing with the same issue.

$ kubectl -n monitoring describe po prometheus-adapter-66fc7797fd-r79zc
...
    Args:
      --cert-dir=/var/run/serving-cert
      --config=/etc/adapter/config.yaml
      --logtostderr=true
      --metrics-relist-interval=1m
      --prometheus-url=http://prometheus-k8s.monitoring.svc:9090/
      --secure-port=6443
...

$ kubectl -n monitoring get svc
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
alertmanager-main       ClusterIP   192.96.41.38     <none>        9093/TCP            17h
alertmanager-operated   ClusterIP   None             <none>        9093/TCP,6783/TCP   17h
grafana                 ClusterIP   192.96.108.241   <none>        3000/TCP            17h
kube-state-metrics      ClusterIP   None             <none>        8443/TCP,9443/TCP   17h
node-exporter           ClusterIP   None             <none>        9100/TCP            17h
prometheus-adapter      ClusterIP   192.96.229.170   <none>        443/TCP             17h
prometheus-k8s          ClusterIP   192.96.9.10      <none>        9090/TCP            17h
prometheus-operated     ClusterIP   None             <none>        9090/TCP            17h
prometheus-operator     ClusterIP   None             <none>        8080/TCP            17h

$ kubectl exec curl-86d6c5f87-8gld4 -- curl http://prometheus-k8s.monitoring.svc:9090/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    29  100    29    0     0   5115      0 --:--:-- --:--:-- --:--:--  5800
<a href="/graph">Found</a>.

$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[]}

FrostyLeaf on Jan 30, 2019

Ok, I figured it out: prometheus URL was wrong

kchuriumova on Oct 9, 2018

Hi! I had the same problem with empty at /apis/custom.metrics.k8s.io/v1beta1/ but prometheus URL was right in my case but not completely correct: url need the http:// and adapter works as excpect.

vldmrdev on May 25, 2022

logs:

$ kubectl -n monitoring logs -f prometheus-adapter-66fc7797fd-r79zc
I0130 02:32:54.484069       1 adapter.go:91] successfully using in-cluster auth
I0130 02:32:55.244684       1 serving.go:273] Generated self-signed cert (/var/run/serving-cert/apiserver.crt, /var/run/serving-cert/apiserver.key)
I0130 02:32:55.567418       1 serve.go:96] Serving securely on [::]:6443
E0130 02:35:20.002006       1 provider.go:186] unable to fetch CPU metrics for pod gitlab-prod/saturn-admin-master-65dd6c86c5-ccmjt, skipping
E0130 02:35:20.002032       1 reststorage.go:144] unable to fetch pod metrics for pod gitlab-prod/saturn-admin-master-65dd6c86c5-ccmjt: no metrics known for pod
E0130 02:35:30.074320       1 provider.go:186] unable to fetch CPU metrics for pod gitlab-prod/saturn-merchant-master-7d77b579fd-dmnrt, skipping
E0130 02:35:30.074343       1 reststorage.go:144] unable to fetch pod metrics for pod gitlab-prod/saturn-merchant-master-7d77b579fd-dmnrt: no metrics known for pod
E0130 02:35:30.090615       1 provider.go:186] unable to fetch CPU metrics for pod gitlab-prod/saturn-api-master-7685b6d8c9-qnvdg, skipping
E0130 02:35:30.090636       1 provider.go:186] unable to fetch CPU metrics for pod gitlab-prod/saturn-api-master-7685b6d8c9-nq2rs, skipping

FrostyLeaf on Jan 30, 2019