k3s: Unable to get pod metrics
Version:
$ k3s -v
k3s version v1.0.0-rc1 (670d4b41)
k3s install
$ curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.0.0-rc1 INSTALL_K3S_EXEC="server" sh -s - --docker --kube-apiserver-arg=enable-admission-plugins=LimitRanger
[INFO] Using v1.0.0-rc1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.0.0-rc1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.0.0-rc1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Skipping /usr/local/bin/kubectl symlink to k3s, command exists in PATH at /usr/bin/kubectl
[INFO] Skipping /usr/local/bin/crictl symlink to k3s, command exists in PATH at /usr/bin/crictl
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
$ export KUBECONFIG=$(locate k3s.yaml)
$ sudo chmod 777 $KUBECONFIG
Describe the bug
Pods won’t display metrics with kube-metrics installed from https://github.com/kubernetes-sigs/metrics-server
Horizontal Pod Autoscaling also doesn’t resolve metrics, however kubectl top nodes resolves fine
To Reproduce
Install and configure kube-metrics
$ git clone https://github.com/kubernetes-incubator/metrics-server Cloning into 'metrics-server'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 11349 (delta 0), reused 1 (delta 0), pack-reused 11345
Receiving objects: 100% (11349/11349), 12.18 MiB | 7.15 MiB/s, done.
Resolving deltas: 100% (5912/5912), done.
$ kubectl apply -f metrics-server/deploy/1.8+/
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.apps/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
$ kubectl edit deploy -n kube-system metrics-server
deployment.apps/metrics-server edited
$ kubectl get deploy -n kube-system metrics-server -o json | jq .spec.template.spec.containers[].args
[
"--cert-dir=/tmp",
"--secure-port=4443",
"--kubelet-insecure-tls=true",
"--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname"
]
Verify functionality
$ kubectl get apiservices
NAME SERVICE AVAILABLE AGE
v1. Local True 3m50s
v1.admissionregistration.k8s.io Local True 3m50s
v1beta1.admissionregistration.k8s.io Local True 3m50s
v1beta1.apiextensions.k8s.io Local True 3m50s
v1.apiextensions.k8s.io Local True 3m50s
v1.apps Local True 3m50s
v1beta1.authentication.k8s.io Local True 3m50s
v1.authentication.k8s.io Local True 3m50s
v1.authorization.k8s.io Local True 3m50s
v2beta1.autoscaling Local True 3m50s
v1beta1.authorization.k8s.io Local True 3m50s
v2beta2.autoscaling Local True 3m50s
v1.batch Local True 3m50s
v1.autoscaling Local True 3m50s
v1beta1.batch Local True 3m50s
v1beta1.certificates.k8s.io Local True 3m50s
v1.coordination.k8s.io Local True 3m50s
v1beta1.coordination.k8s.io Local True 3m50s
v1beta1.events.k8s.io Local True 3m50s
v1.networking.k8s.io Local True 3m50s
v1beta1.extensions Local True 3m50s
v1beta1.networking.k8s.io Local True 3m50s
v1beta1.policy Local True 3m50s
v1.rbac.authorization.k8s.io Local True 3m50s
v1beta1.node.k8s.io Local True 3m50s
v1beta1.scheduling.k8s.io Local True 3m50s
v1beta1.rbac.authorization.k8s.io Local True 3m50s
v1.scheduling.k8s.io Local True 3m50s
v1.storage.k8s.io Local True 3m50s
v1beta1.storage.k8s.io Local True 3m50s
v1.k3s.cattle.io Local True 3m24s
v1.helm.cattle.io Local True 3m24s
v1beta1.metrics.k8s.io kube-system/metrics-server True 93s
$ kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
local-path-provisioner-58fb86bdfd-6czp5 1/1 Running 0 3m46s
coredns-d798c9dd-x782x 1/1 Running 0 3m46s
helm-install-traefik-fsj6q 0/1 Completed 0 3m46s
traefik-65bccdc4bd-pcmbb 1/1 Running 0 2m20s
svclb-traefik-fbtj7 3/3 Running 0 2m20s
metrics-server-b5655b66c-gjt75 1/1 Running 0 76s
$ kubectl top no
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
ayanami 765m 9% 9449Mi 80%
Using https://www.digitalocean.com/community/tutorials/how-to-autoscale-your-workloads-on-digitalocean-kubernetes as a reference
Create a deployment with resource limits
$ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
$ kubectl edit deploy nginx
deployment.apps/nginx edited
$ kubectl get deploy nginx -o json | jq .spec.template.spec.containers[].resources
{
"limits": {
"cpu": "300m"
},
"requests": {
"cpu": "100m",
"memory": "250Mi"
}
}
Expected behavior For pod metrics to be displayed, the node metrics work fine
Actual behavior Some error snippets:
the HPA was unable to compute the replica count: unable to get metrics for resource cpu: no metrics returned from resource metrics API
---
horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Create an HPA and test pod metrics
$ kubectl autoscale deploy nginx --min=1 --max=5 --cpu-percent=50
horizontalpodautoscaler.autoscaling/nginx autoscaled
$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx Deployment/nginx <unknown>/50% 1 5 1 2m32s
$ kubectl top po
W1113 16:18:41.392771 7082 top_pod.go:266] Metrics not available for pod default/nginx-7bfff5fd9f-rbklh, age: 2m20.39276264s
error: Metrics not available for pod default/nginx-7bfff5fd9f-rbklh, age: 2m20.39276264s
$ kubectl describe hpa nginx
Name: nginx
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Wed, 13 Nov 2019 16:18:33 -0800
Reference: Deployment/nginx
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 50%
Min replicas: 1
Max replicas: 5
Deployment pods: 1 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 4s (x10 over 2m20s) horizontal-pod-autoscaler unable to get metrics for resource cpu: no metrics returned from resource metrics API
Warning FailedComputeMetricsReplicas 4s (x10 over 2m20s) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
$ journalctl -eu k3s | tail -n 2
Nov 13 16:20:05 Ayanami k3s[29018]: I1113 16:20:05.194601 29018 event.go:255] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"nginx", UID:"ee5c9d7b-9251-4563-bed8-ebd5103ea906", APIVersion:"autoscaling/v2beta2", ResourceVersion:"1054", FieldPath:""}): type: 'Warning' reason: 'FailedGetResourceMetric' unable to get metrics for resource cpu: no metrics returned from resource metrics API
Nov 13 16:20:05 Ayanami k3s[29018]: I1113 16:20:05.194669 29018 event.go:255] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"nginx", UID:"ee5c9d7b-9251-4563-bed8-ebd5103ea906", APIVersion:"autoscaling/v2beta2", ResourceVersion:"1054", FieldPath:""}): type: 'Warning' reason: 'FailedComputeMetricsReplicas' invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Also following https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#create-horizontal-pod-autoscaler
Walkthrough autoscale
$ kubectl run php-apache --image=k8s.gcr.io/hpa-example --requests=cpu=200m --limits=cpu=500m --expose --port=80
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
service/php-apache created
deployment.apps/php-apache created
$ kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
horizontalpodautoscaler.autoscaling/php-apache autoscaled
$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx Deployment/nginx <unknown>/50% 1 5 1 32m
php-apache Deployment/php-apache <unknown>/50% 1 10 1 23s
$ kubectl top no
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
ayanami 867m 10% 9209Mi 78%
$ kubectl top po
W1113 16:53:11.127447 28186 top_pod.go:266] Metrics not available for pod default/nginx-7bfff5fd9f-rbklh, age: 36m50.127437786s
error: Metrics not available for pod default/nginx-7bfff5fd9f-rbklh, age: 36m50.127437786s
$ kubectl logs -n kube-system metrics-server-b5655b66c-gjt75 | tail -n 4
E1114 00:58:47.254980 1 reststorage.go:160] unable to fetch pod metrics for pod default/nginx-7bfff5fd9f-rbklh: no metrics known for pod
E1114 00:58:47.258093 1 reststorage.go:160] unable to fetch pod metrics for pod default/php-apache-79544c9bd9-f6m8t: no metrics known for pod
E1114 00:59:02.267962 1 reststorage.go:160] unable to fetch pod metrics for pod default/nginx-7bfff5fd9f-rbklh: no metrics known for pod
E1114 00:59:02.279683 1 reststorage.go:160] unable to fetch pod metrics for pod default/php-apache-79544c9bd9-f6m8t: no metrics known for pod
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 3
- Comments: 28 (8 by maintainers)
@erikwilson I just found out that the issue happens (to me at least) only when running in docker mode. export INSTALL_K3S_EXEC=“–docker” Otherwise it works.
Does that gives you any pointers?
My docker version is 19.03.1.
The problem is, that I must use Docker mode… 😦
Same issue
Install and create pod/hpa
@edenreich If your issue is with
kubectl top podswhen using Docker, I believe this was fixed in https://github.com/rancher/k3s/pull/1627 - you might try updating to latest stable. This issue is about the HPA which does not sound like what you’re running into. Hard to tell with limited information though.It looks like you didn’t specify a cpu resource request for that pod. ‘top pods’ only cares what it’s using, while the HPA wants to compare utilization to requests to figure out if it needs to scale up or down.
I’ve gone through and added cpu and memory requests to all my pods. Even if you’re not using limits or autoscaling, the scheduler works better if it knows what to expect. The QoS stuff also requires requests (and limits, if you want to use the guaranteed class).
I’ve got same problem. #1149 Everything is ok, but metrics cannot get pod’s values of cpu and memory. Version: k3s version v1.0.0 (18bd921)
Same issue:
Also some logs that might help
Well after lots of time solution was adding the cn kubernetes-proxy to allowed, as described here: https://github.com/kubernetes-sigs/metrics-server/issues/292
I would like to chime in – I’ve installed a k3s cluster with three x86 agent nodes and a raspberry pi as the master. The agents I installed with:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent --node-label=istio=enabled --token-file=/home/cfox/kube/token.txt --server https://knodemaster.localdomain:6443" sh -When I try ‘top node’ I get this:
(top pod returns an identical error)