k3s: cAdvisor metrics stopped working correctly in K3s 1.20

Environmental Info: K3s Version: k3s version v1.20.0+k3s2 (2ea6b163) go version go1.15.5

Node(s) CPU architecture, OS, and Version: Linux kube-master0 5.4.0-54-generic #60-Ubuntu SMP Fri Nov 6 10:37:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration: 1 master, 2 workers, embedded containerd (no docker, no custom CRI)

Describe the bug: cAdvisor is unable to connect to containerd resulting in mostly empty labels (container, image, name) in metrics.

Adding --kubelet-arg containerd=/run/k3s/containerd/containerd.sock to k3s launch args fixes the issue.

Steps To Reproduce:

k3s server

curl -k --cert /var/lib/rancher/k3s/server/tls/client-admin.crt --key /var/lib/rancher/k3s/server/tls/client-admin.key https://127.0.0.1:6443/api/v1/nodes/NODE_NAME/proxy/metrics/cadvisor

Expected behavior: Metrics from cAdvisor have non-empty labels:

container_cpu_load_average_10s{container="fluent-bit",id="/kubepods/burstable/pod227a7799-c04b-419a-9d96-98b5ca911666/67d16f0aa8f6e214914c9769a55e8154a9f133646b36974d03b8a3f185ae3e38",image="docker.io/fluent/fluent-bit:1.6",name="67d16f0aa8f6e214914c9769a55e8154a9f133646b36974d03b8a3f185ae3e38",namespace="logging",pod="fluent-bit-vw9cg"} 0 1611138060164

Actual behavior: cAdvisor metric’s labels are empty

container_cpu_load_average_10s{container="",id="/kubepods/burstable/pod227a7799-c04b-419a-9d96-98b5ca911666/67d16f0aa8f6e214914c9769a55e8154a9f133646b36974d03b8a3f185ae3e38",image="",name="",namespace="",pod=""} 0 1611135942449

Additional context / logs: Most likely regression caused by https://github.com/k3s-io/k3s/commit/5b318d093f3905e0b4d7a5592b64f2bbb981cdd3

That value is used for both argsMap["container-runtime-endpoint"] and argsMap["containerd"] and it seems the containerd one cannot be an URI

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 15 (8 by maintainers)

Most upvoted comments

You don’t need to add the containerd arg on v1.20.4.

Upgrading to v1.20.2+k3s1 (1d4adb03) didn’t help, still need that extra kubelet arg for proper labels.

The only change I noticed was presence of additional metrics, presumably added by https://github.com/kubernetes/kubernetes/pull/97006