kubernetes: Kubelet cannot determine CPU online state
What happened:
Every 5 minutes, Kubelet on my Raspberry Pi logs the following error:
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.261966 21177 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.262434 21177 sysfs.go:348] unable to read /sys/devices/system/cpu/cpu0/online: open /sys/devices/system/cpu/cpu0/online: no such file or directory
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.262513 21177 sysfs.go:348] unable to read /sys/devices/system/cpu/cpu1/online: open /sys/devices/system/cpu/cpu1/online: no such file or directory
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.262575 21177 sysfs.go:348] unable to read /sys/devices/system/cpu/cpu2/online: open /sys/devices/system/cpu/cpu2/online: no such file or directory
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.262637 21177 sysfs.go:348] unable to read /sys/devices/system/cpu/cpu3/online: open /sys/devices/system/cpu/cpu3/online: no such file or directory
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263083 21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu0 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263137 21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu1 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263180 21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu2 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263223 21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu3 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: E0924 17:01:14.263240 21177 machine.go:72] Cannot read number of physical cores correctly, number of cores set to 0
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263414 21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu0 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263456 21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu1 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263498 21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu2 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: W0924 17:01:14.263542 21177 machine.go:253] Cannot determine CPU /sys/bus/cpu/devices/cpu3 online state, skipping
Sep 24 17:01:14 homepi-01 kubelet[21177]: E0924 17:01:14.263558 21177 machine.go:86] Cannot read number of sockets correctly, number of sockets set to 0
I think the error may be related to the fact that Raspberry Pi OS is compiled without the CONFIG_CFS_BANDWIDTH
flag. See this Github issue for more info: https://github.com/raspberrypi/linux/issues/2298
I tried setting cpuCFSQuota: false
in /var/lib/kubelet/config.yaml
and restarting Kubelet, but it is still logging this error.
Here is the full config.yaml file:
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 0s
cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuCFSQuota: false
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
Is there a way I can either resolve or silence this error?
What you expected to happen:
I do not expect to see these errors in Kubelet’s logs.
How to reproduce it (as minimally and precisely as possible):
On Raspberry Pi OS 64-bit beta, install Docker, then kubernetes using kubeadm:
kubeadm init --config kubeadm.yml
My kubeadm.yml has the following contents:
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
podSubnet: 10.244.0.0/16
My CNI layer is flannel.
Anything else we need to know?:
I raised this first on the Kubeadm Github: https://github.com/kubernetes/kubeadm/issues/2300
Environment:
- Kubernetes version (use
kubectl version
): 1.19.1 - Cloud provider or hardware configuration: Raspberry Pi 4, Raspberry Pi OS 64-bit beta
- OS (e.g:
cat /etc/os-release
): Debian GNU/Linux 10 (buster) - Kernel (e.g.
uname -a
): Linux 5.4.51-v8+ aarch64 GNU/Linux - Install tools: kubeadm 1.19.1
- Network plugin and version (if this is a network-related bug): 0.12.0
- Others: Kubelet version: 1.19.1, Docker version: 19.03.13
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 19
- Comments: 44 (16 by maintainers)
Bug will be fixed in 1.21: https://github.com/kubernetes/kubernetes/pull/99875. I will work towards backporting it to 1.20 too.
It seems like with closure they do not intend to fix this. I do not know what to do next, but to me this issue seems important for many RPi users
same here with 1.19.3 / Ubuntu 20.04.1 LTS / 5.4.0-1022-raspi
same for me on 1.19.3 / Ubuntu 20.04.1 LTS / 5.4.0-1022-raspi
This is a known issue in cAdvisor: https://github.com/google/cadvisor/issues/2743
Seeing this on 1.20.1 using kernel
5.10.1-talos
on a Raspberry Pi 4B 4GB