kubernetes: Kubelet cannot determine CPU online state

What happened:

Every 5 minutes, Kubelet on my Raspberry Pi logs the following error:

Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.261966   21177 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.262434   21177 sysfs.go:348] unable to read /sys/devices/system/cpu/cpu0/online: open /sys/devices/system/cpu/cpu0/online: no such file or directory
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.262513   21177 sysfs.go:348] unable to read /sys/devices/system/cpu/cpu1/online: open /sys/devices/system/cpu/cpu1/online: no such file or directory
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.262575   21177 sysfs.go:348] unable to read /sys/devices/system/cpu/cpu2/online: open /sys/devices/system/cpu/cpu2/online: no such file or directory
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.262637   21177 sysfs.go:348] unable to read /sys/devices/system/cpu/cpu3/online: open /sys/devices/system/cpu/cpu3/online: no such file or directory
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263083   21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu0 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263137   21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu1 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263180   21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu2 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263223   21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu3 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: E0924 17:01:14.263240   21177 machine.go:72] Cannot read number of physical cores correctly, number of cores set to 0
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263414   21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu0 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263456   21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu1 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263498   21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu2 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263542   21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu3 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: E0924 17:01:14.263558   21177 machine.go:86] Cannot read number of sockets correctly, number of sockets set to 0

I think the error may be related to the fact that Raspberry Pi OS is compiled without the CONFIG_CFS_BANDWIDTH flag. See this Github issue for more info: https://github.com/raspberrypi/linux/issues/2298

I tried setting cpuCFSQuota: false in /var/lib/kubelet/config.yaml and restarting Kubelet, but it is still logging this error. Here is the full config.yaml file:

apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuCFSQuota: false
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

Is there a way I can either resolve or silence this error?

What you expected to happen:

I do not expect to see these errors in Kubelet’s logs.

How to reproduce it (as minimally and precisely as possible):

On Raspberry Pi OS 64-bit beta, install Docker, then kubernetes using kubeadm:

kubeadm init --config kubeadm.yml

My kubeadm.yml has the following contents:

---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
  podSubnet: 10.244.0.0/16

My CNI layer is flannel.

Anything else we need to know?:

I raised this first on the Kubeadm Github: https://github.com/kubernetes/kubeadm/issues/2300

Environment:

  • Kubernetes version (use kubectl version): 1.19.1
  • Cloud provider or hardware configuration: Raspberry Pi 4, Raspberry Pi OS 64-bit beta
  • OS (e.g: cat /etc/os-release): Debian GNU/Linux 10 (buster)
  • Kernel (e.g. uname -a): Linux 5.4.51-v8+ aarch64 GNU/Linux
  • Install tools: kubeadm 1.19.1
  • Network plugin and version (if this is a network-related bug): 0.12.0
  • Others: Kubelet version: 1.19.1, Docker version: 19.03.13

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 19
  • Comments: 44 (16 by maintainers)

Most upvoted comments

Bug will be fixed in 1.21: https://github.com/kubernetes/kubernetes/pull/99875. I will work towards backporting it to 1.20 too.

It seems like with closure they do not intend to fix this. I do not know what to do next, but to me this issue seems important for many RPi users

same here with 1.19.3 / Ubuntu 20.04.1 LTS / 5.4.0-1022-raspi

same for me on 1.19.3 / Ubuntu 20.04.1 LTS / 5.4.0-1022-raspi

This is a known issue in cAdvisor: https://github.com/google/cadvisor/issues/2743

Seeing this on 1.20.1 using kernel 5.10.1-talos on a Raspberry Pi 4B 4GB