cri-o: Kubernetes metrics doesn't work with K8s v1.30.x and cgroup-driver=cgroupfs (the default)
What happened?
When using K8s v1.30.0-rc.2 and crio then pod metrics never become available:
# kubectl top pods
error: Metrics not available for pod default/tserver-6854544b4b-fjmmg, age: 8m4.082260059s
Tested with crio 1.28.1 and 1.29.2 with the same result.
Metric-server v0.7.1 is used, and seem to run without problems:
# kubectl get pod -n kube-system metrics-server-d994c478f-sxbf4
NAME READY STATUS RESTARTS AGE
metrics-server-d994c478f-sxbf4 1/1 Running 0 12m
No errors in the metrics-server logs.
What did you expect to happen?
“kubectl top pods” should work. It works with K8s v1.29.x
How can we reproduce it (as minimally and precisely as possible)?
- Install K8s v1.30.1 (or K8s built on master) with --cgroup-driver unset, or
cgroupfs(the default) - Install the metrics-server. I use the
components.yamlmethod - Wait until the metrics-server is running 1/1, and yet some time
- Do “kubectl top pods” in some namespace with pods
Anything else we need to know?
It works with K8s v1.29.3 and cri-o.
It also works with K8s v1.30.0-rc.2 and containerd 1.7.11 and 2.0.0-beta.2.
CRI-O and Kubernetes version
See above
OS version
Linux 6.8.0
Additional environment details (AWS, VirtualBox, physical, etc.)
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 45 (8 by maintainers)
Update. Metrics stared working again after upgrading kubernetes clusters to 1.32.1.
I’d say cadvisor 😃
It must be something is my environment then, but I can’t understand what. Since I get it working with other combinations, it must be something that happened in K8s v1.30.0, but may be unrelated to crio.
My run differs from yours:
It gets evicted after a minute or so, goes to “Error” and never restarts.
BTW, I built crio from source (on master), but get the same problem. But I couldn’t even start containers with “crun” (my default), but “runc” worked. My guess was that new crio needs a newer crun, but I didn’t investigate further.
Naturally @aojea has made a gist on howto use crio in KinD 😄