kubernetes: kubelet 1.23.0 (cAdvisor 0.43) panic on arm64 / Raspberry PI OS32

What happened?

Configuration:

  • OS: Raspberry PI OS 32 bullseye
  • arch: Linux raspberrypixxx 5.10.63-v8+ #1488 SMP PREEMPT Thu Nov 18 16:16:16 GMT 2021 aarch64 GNU/Linux /boot/config.txt with arm_64bits=1

kubelet panic at https://github.com/kubernetes/kubernetes/blob/release-1.23/vendor/github.com/google/cadvisor/manager/container.go#L676

$ journalctl -eu | grep -C10 panic
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: I1211 09:42:41.697571   10354 kuberuntime_manager.go:248] "Container runtime initialized" containerRuntime="containerd" version="1.4.5~ds1" apiVersion="v1alpha2"
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: I1211 09:42:41.699202   10354 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-server-current.pem".
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: I1211 09:42:41.703243   10354 server.go:1231] "Started kubelet"
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: I1211 09:42:41.706678   10354 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: E1211 09:42:41.712014   10354 cri_stats_provider.go:455] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs"
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: E1211 09:42:41.712128   10354 kubelet.go:1351] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: I1211 09:42:41.715137   10354 volume_manager.go:291] "Starting Kubelet Volume Manager"
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: I1211 09:42:41.715516   10354 desired_state_of_world_populator.go:147] "Desired state populator starts to run"
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: I1211 09:42:41.718436   10354 server.go:150] "Starting to listen" address="0.0.0.0" port=10250
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: I1211 09:42:41.720825   10354 server.go:410] "Adding debug handlers to kubelet server"
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: panic: unaligned 64-bit atomic operation
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: goroutine 309 [running]:
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: runtime/internal/atomic.panicUnaligned()
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]:         /usr/local/go/src/runtime/internal/atomic/unaligned.go:8 +0x24
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: runtime/internal/atomic.Load64(0x70ea16c)
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]:         /usr/local/go/src/runtime/internal/atomic/atomic_arm.s:286 +0x14
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats(0x70ea000)
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]:         /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:676 +0x438
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0x70ea000, 0x712af40, 0x5f5e100)
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]:         /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:587 +0x104
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping(0x70ea000)
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]:         /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:535 +0x3c0
Dec 11 09:42:41 raspberrypicluster06 kubelet[10354]: created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.(*containerData).Start

What did you expect to happen?

kubelet working

How can we reproduce it (as minimally and precisely as possible)?

Start the rpi with arm_64bits=1. Kubelet fails with panic.

Anything else we need to know?

See https://github.com/google/cadvisor/issues/3027

Kubernetes version

```console $ kubectl version Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/arm"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:42:41Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/arm"} ```

Cloud provider

baremetal

OS version

$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)"
NAME="Raspbian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
$ uname -a
Linux raspberrypi-xxx 5.10.63-v8+ #1459 SMP PREEMPT Wed Oct 6 16:42:49 BST 2021 aarch64 GNU/Linux

Install tools

Container runtime (CRI) and and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)

/sig node

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 23 (13 by maintainers)

Commits related to this issue

Most upvoted comments

Can confirm that the binary soider shared works with a rpi3 with no issues in my test cluster.

+5
ke1satsu on Dec 16, 2021

@vescoc

To check that the problem is solved you need to have an armv8 rpi 3 or 4 with Raspberry PI OS 32 Bullseye (arm_64bit=1 in /boot/config.txt).

I have VM with custom ubuntu image so I don’t have this /boot/config.txt Any other ways to validate running kernel mode? The closest I have is

cat /boot/config-5.10.0-9-armmp-lpae  |grep -i 64bit
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y

Can I get a preview of the kubelet build? I can try it on my device / hardware configuration.

Sure, I don’t really have any hosting, but I can share it via dropbox https://www.dropbox.com/s/zb5viqzmo1t3x12/kubelet_armv7?dl=0

➜  Dropbox file kubelet_armv7
kubelet_armv7: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, Go BuildID=zlAMKCZsaexwu7F1t1-O/dUh5HW5S8nZtMJr4XNMg/auRR_0qCTP3KR3Mf2ps0/sgatOzOhQfIeOydXSiHG, stripped
➜  Dropbox md5 kubelet_armv7
MD5 (kubelet_armv7) = 47e4b1f531c2980c6920dbf039f38e79
+2
mikhail-sakhnov on Dec 14, 2021

@jonyhy96 fyi, I tried using the same patch in k0sproject release for 1.23 (https://github.com/k0sproject/k0s/pull/1340/commits/953909e91663b6a167945284b40af44f59a1441a) it seems to be working on armv7, at least kubelet doesn’t panic and generally passed our test suite.

+2
mikhail-sakhnov on Dec 14, 2021

/close

fixed in master which will show up in 1.24 when we get that out the door.

+1
dims on Apr 3, 2022

google/cadvisor#3029 has been merged.

+1
iwankgb on Dec 26, 2021

Hello guys.

Run into the same error messeages yesterday on my RPI4. OS: Debian Buster.

Can I help you in any way with a testing?

Done already. With the attached version my RPI was able to join to the cluster.

Thank you guys!

+1
zolbarna on Dec 16, 2021
Read more comments on GitHub
← kubernetes: Kubelet 1.17.8 breaks pod/container startup
kubernetes: kubelet (1.27.1) fails to mount a hostPath volume for a static pod →