cri-o: Kubernetes metrics doesn't work with K8s v1.30.x and cgroup-driver=cgroupfs (the default)

What happened?

When using K8s v1.30.0-rc.2 and crio then pod metrics never become available:

# kubectl top pods
error: Metrics not available for pod default/tserver-6854544b4b-fjmmg, age: 8m4.082260059s

Tested with crio 1.28.1 and 1.29.2 with the same result.

Metric-server v0.7.1 is used, and seem to run without problems:

# kubectl get pod -n kube-system   metrics-server-d994c478f-sxbf4
NAME                             READY   STATUS    RESTARTS   AGE
metrics-server-d994c478f-sxbf4   1/1     Running   0          12m

No errors in the metrics-server logs.

What did you expect to happen?

“kubectl top pods” should work. It works with K8s v1.29.x

How can we reproduce it (as minimally and precisely as possible)?

  • Install K8s v1.30.1 (or K8s built on master) with --cgroup-driver unset, or cgroupfs (the default)
  • Install the metrics-server. I use the components.yaml method
  • Wait until the metrics-server is running 1/1, and yet some time
  • Do “kubectl top pods” in some namespace with pods

Anything else we need to know?

It works with K8s v1.29.3 and cri-o. It also works with K8s v1.30.0-rc.2 and containerd 1.7.11 and 2.0.0-beta.2.

CRI-O and Kubernetes version

See above

OS version

Linux 6.8.0

Additional environment details (AWS, VirtualBox, physical, etc.)

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 45 (8 by maintainers)

Most upvoted comments

Update. Metrics stared working again after upgrading kubernetes clusters to 1.32.1.

The K8s project? Or cadvisor? Or cri-o?

I’d say cadvisor 😃

It must be something is my environment then, but I can’t understand what. Since I get it working with other combinations, it must be something that happened in K8s v1.30.0, but may be unrelated to crio.

My run differs from yours:

# kubectl get pods
NAME               READY   STATUS   RESTARTS   AGE
test-pod-evicted   0/1     Error    0          3m36s

It gets evicted after a minute or so, goes to “Error” and never restarts.

BTW, I built crio from source (on master), but get the same problem. But I couldn’t even start containers with “crun” (my default), but “runc” worked. My guess was that new crio needs a newer crun, but I didn’t investigate further.

Naturally @aojea has made a gist on howto use crio in KinD 😄