metrics-server: kubectl top pod responding with "Metrics not available for pod xxx"
What happened:
After applying the deployment for metrics-server, and adding the --kubelet-insecure-tls arg to the container, I am seeing “Metrics not available for pod xxx” as output to kubectl top pod.
What you expected to happen: Metrics to successfully be reported for pods.
Anything else we need to know?:
PS C:\Users\nater\source\repos\local-kube> kubectl top pod -A
error: Metrics not available for pod default/kube-auto-apply-deployment-b7bb5c779-kwbpd, age: 2m8.0911182s
I am able to successfully see the metrics for my node(via kubectl top node), but nothing else.
Environment:
-
Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.): Docker Desktop 4.10.1
-
Container Network Setup (flannel, calico, etc.): None
-
Kubernetes version (use
kubectl version): 1.24 -
Metrics Server version: v0.6.1
-
Metrics Server manifest
spoiler for Metrics Server manifest:
- Kubelet config:
spoiler for Kubelet config:
- Metrics server logs:
spoiler for Metrics Server logs:
I0718 22:09:16.573813 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0718 22:09:16.573938 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0718 22:09:16.573963 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0718 22:09:16.573971 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0718 22:09:16.573984 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0718 22:09:16.573995 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0718 22:09:16.574655 1 dynamic_serving_content.go:131] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
I0718 22:09:16.575678 1 secure_serving.go:266] Serving securely on [::]:4443
I0718 22:09:16.575724 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
W0718 22:09:16.575841 1 shared_informer.go:372] The sharedIndexInformer has started, run more than once is not allowed
I0718 22:09:16.674734 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0718 22:09:16.674754 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0718 22:09:16.674760 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
- Status of Metrics API:
spolier for Status of Metrics API:
kubectl describe apiservice v1beta1.metrics.k8s.io
Name: v1beta1.metrics.k8s.io
Namespace:
Labels: k8s-app=metrics-server
Annotations: <none>
API Version: apiregistration.k8s.io/v1
Kind: APIService
Metadata:
Creation Timestamp: 2022-07-17T18:30:39Z
Managed Fields:
API Version: apiregistration.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
.:
k:{"type":"Available"}:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
Manager: kube-apiserver
Operation: Update
Subresource: status
Time: 2022-07-17T18:30:39Z
API Version: apiregistration.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:labels:
.:
f:k8s-app:
f:spec:
f:group:
f:groupPriorityMinimum:
f:insecureSkipTLSVerify:
f:service:
.:
f:name:
f:namespace:
f:port:
f:version:
f:versionPriority:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-07-17T18:30:39Z
Resource Version: 1187400
UID: 118eff22-1c09-417a-af48-7cb661511225
Spec:
Group: metrics.k8s.io
Group Priority Minimum: 100
Insecure Skip TLS Verify: true
Service:
Name: metrics-server
Namespace: kube-system
Port: 443
Version: v1beta1
Version Priority: 100
Status:
Conditions:
Last Transition Time: 2022-07-18T22:09:43Z
Message: all checks passed
Reason: Passed
Status: True
Type: Available
Events: <none>
/kind bug
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (4 by maintainers)
I met the same bug, after some searching, I found a related issue.
Root cause: This problem is caused by a bug in Mirantis/cri-dockerd, which is the replacement of the deprecated dockershim. Specifically, this bug is fixed in cri-dockerd v2.3.0, (see this commit 5576d3c5d4b030b2feb24ebb79640932c5838f6a). However, the cri-dockered integrated in OSX Docker Desktop 4.10/11 is v2.1.0.
Solution: Download cri-dockerd v2.3.0 from the releases page and overwrite the lower-version binary in Docker VM. Commands:
Step 1: Login Docker VM:
Step2: Download cri-dockerd v2.3.0
Step3: Find cri-dockerd binaries
These binaries are of old version (0.2.1)
Step4: Overwrite found all cri-dockerd binaries with v2.3.0, (actually I think it’s unnecessary to cover all of them, do it just for simplicity)
Step 5: Restart Docker Desktop.
Finally,
kubectl top pods -Aworks!So with the new Docker Desktop 4.12.0 release, this should no longer be an issue, since they upgraded the cri-dockerd version to 0.2.5 which includes the fix @weicao mentioned.
However, I had to do a full restart of the Kubernetes cluster under Docker Desktop --> Kubernetes --> Reset Kubernetes Cluster. Before the restart I was still having the error of pod metrics not being available.
If there are someone struggling with this when using minikube. Don’t be dazed by all above it’s not ur solution… just enable metric server from minikube with
minikube addons enable metrics-server…@yangjunmyfm192085 the command you provided returns data in the pods section with cpu and memory values.
kubectl get --raw /api/v1/nodes/docker-desktop/proxy/metrics/resource