rancher: Rancher doesn't work with arm32v7 (Raspberry Pi4 Cluster)

Hi, I just imported a k3s RasPi 4 cluster into Rancher and it’s stuck in pending state. It looks like it’s stuck “pending” because of an image pulling error.

kubectl get pods --all-namespaces
NAMESPACE       NAME                                   READY   STATUS             RESTARTS   AGE
cattle-system   cattle-cluster-agent-5d5c498cf-mbt28   0/1     ImagePullBackOff   0          32m
cattle-system   cattle-node-agent-5w2dw                0/1     ImagePullBackOff   0          32m
cattle-system   cattle-node-agent-j5wv6                0/1     ImagePullBackOff   0          32m
cattle-system   cattle-node-agent-npt96                0/1     ImagePullBackOff   0          32m
cattle-system   cattle-node-agent-sfqlb                0/1     ImagePullBackOff   0          32m
kube-system     coredns-b7464766c-hqw8t                1/1     Running            2          48m
kube-system     helm-install-traefik-lrh2z             0/1     CrashLoopBackOff   9          48m
kubectl get events --all-namespaces
NAMESPACE       LAST SEEN   TYPE      REASON                    OBJECT                                      MESSAGE
cattle-system   29m         Normal    Scheduled                 pod/cattle-cluster-agent-5d5c498cf-mbt28    Successfully assigned cattle-system/cattle-cluster-agent-5d5c498cf-mbt28 to node-3
cattle-system   28m         Normal    Pulling                   pod/cattle-cluster-agent-5d5c498cf-mbt28    Pulling image "rancher/rancher-agent:v2.3.0"
cattle-system   28m         Warning   Failed                    pod/cattle-cluster-agent-5d5c498cf-mbt28    Failed to pull image "rancher/rancher-agent:v2.3.0": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/rancher/rancher-agent:v2.3.0": fail
ed to unpack image on snapshotter overlayfs: no match for platform in manifest sha256:5933fb5d6669c790aa65d0937f92f48d8143ae60b5bd181ef1da3fa41e0fc4e4: not found
cattle-system   28m         Warning   Failed                    pod/cattle-cluster-agent-5d5c498cf-mbt28    Error: ErrImagePull
cattle-system   4m44s       Normal    BackOff                   pod/cattle-cluster-agent-5d5c498cf-mbt28    Back-off pulling image "rancher/rancher-agent:v2.3.0"
cattle-system   27m         Warning   Failed                    pod/cattle-cluster-agent-5d5c498cf-mbt28    Error: ImagePullBackOff
cattle-system   29m         Normal    SuccessfulCreate          replicaset/cattle-cluster-agent-5d5c498cf   Created pod: cattle-cluster-agent-5d5c498cf-mbt28
cattle-system   29m         Normal    ScalingReplicaSet         deployment/cattle-cluster-agent             Scaled up replica set cattle-cluster-agent-5d5c498cf to 1
cattle-system   29m         Normal    Scheduled                 pod/cattle-node-agent-5w2dw                 Successfully assigned cattle-system/cattle-node-agent-5w2dw to node-4
cattle-system   28m         Normal    Pulling                   pod/cattle-node-agent-5w2dw                 Pulling image "rancher/rancher-agent:v2.3.0"
cattle-system   28m         Warning   Failed                    pod/cattle-node-agent-5w2dw                 Failed to pull image "rancher/rancher-agent:v2.3.0": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/rancher/rancher-agent:v2.3.0": fail
ed to unpack image on snapshotter overlayfs: no match for platform in manifest sha256:5933fb5d6669c790aa65d0937f92f48d8143ae60b5bd181ef1da3fa41e0fc4e4: not found
cattle-system   28m         Warning   Failed                    pod/cattle-node-agent-5w2dw                 Error: ErrImagePull
cattle-system   4m37s       Normal    BackOff                   pod/cattle-node-agent-5w2dw                 Back-off pulling image "rancher/rancher-agent:v2.3.0"
cattle-system   14m         Warning   Failed                    pod/cattle-node-agent-5w2dw                 Error: ImagePullBackOff
cattle-system   29m         Normal    Scheduled                 pod/cattle-node-agent-j5wv6                 Successfully assigned cattle-system/cattle-node-agent-j5wv6 to node-2
cattle-system   28m         Normal    Pulling                   pod/cattle-node-agent-j5wv6                 Pulling image "rancher/rancher-agent:v2.3.0"
cattle-system   28m         Warning   Failed                    pod/cattle-node-agent-j5wv6                 Failed to pull image "rancher/rancher-agent:v2.3.0": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/rancher/rancher-agent:v2.3.0": fail
ed to unpack image on snapshotter overlayfs: no match for platform in manifest sha256:5933fb5d6669c790aa65d0937f92f48d8143ae60b5bd181ef1da3fa41e0fc4e4: not found
cattle-system   28m         Warning   Failed                    pod/cattle-node-agent-j5wv6                 Error: ErrImagePull
cattle-system   4m32s       Normal    BackOff                   pod/cattle-node-agent-j5wv6                 Back-off pulling image "rancher/rancher-agent:v2.3.0"
cattle-system   24m         Warning   Failed                    pod/cattle-node-agent-j5wv6                 Error: ImagePullBackOff
cattle-system   29m         Normal    Scheduled                 pod/cattle-node-agent-npt96                 Successfully assigned cattle-system/cattle-node-agent-npt96 to node-1
cattle-system   28m         Normal    Pulling                   pod/cattle-node-agent-npt96                 Pulling image "rancher/rancher-agent:v2.3.0"

Does the image not exists?

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 18
  • Comments: 44 (3 by maintainers)

Most upvoted comments

As listed the issue is you are not running the 64bit version of Raspbian. I’m running three PI4’s with SSD only.

Steps I followed to get my cluster up:

Adjust Raspbian to run 64bit, adding this line informs rpi-update to update with the 64bit version.

Update: /boot/config.txt with arm_64bit=1

Run rpi-update to update the pi to 64bit

To confirm run uname -a

(Source: https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=250730)

1 - Setup Master run the following to install K3S: curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644

Validate that it’s up: sudo kubectl get nodes sudo systemctl status k3s.service

Make sure the cluster is up and happy before adding the workers.

2 - Setup the Worker On the master export the token from: sudo cat /var/lib/rancher/k3s/server/node-token

On the worker: export K3S_TOKEN=“paste above token here” export K3S_URL=https://<your master ip>:6443 curl -sfL https://get.k3s.io | sh -

Then create the cluster in Rancher, Import custom.

Run the two commands to add the user and add the master that Rancher gives you. kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user pi sudo kubectl apply -f https://<rancher server ip>/v3/import/2r3131231231231231231231231313123.yaml

Reopening this issue because we recently had someone in the Rancher Users Slack asking for help importing an ARM cluster as well.

I’m also seeing this issue on a RPi 4 with the latest Rasperry Pi OS with bullseye. Pointing to the image tage v2.6.2-linux-arm64 allows me to download the image, however, it will not run on k3s due to an incompatible architecture:

standard_init_linux.go:228: exec user process caused: exec format error

Found this blog:

Since I already have a rancher server, I have to import an rpi3 k3s cluster. With above instruction, I can deploy the rancher-agent:v2.5.3-linux-arm64 successfully on my raspberry pi 3/B+ k3s cluster to import it into my Rancher server.

More detailed instructions: >>

1. Prepare (Optional)

  1. Get k3os-rootfs-arm64.tar.gz from https://github.com/rancher/k3os.
  2. Create your own config.yaml for server/master or agents/workers.
  • for server
...
hostname: pi1.local.net
k3os:
  k3s_args:
  - server
...
  • for workers
...
hostname: pi2.local.net
k3os:
  server_url: https://pi1.local.net:6443
  token: $NODE_TOKEN_FROM_SERVER
  k3s_args:
  - agent
...

2. K3s server on a raspberry pi 3 b+

  1. Create the bootable SD card with 2020-02-13-raspbian-buster-lite.img.
  2. (Optional) touch /boot/ssh
  3. Add a “arm_64bit=1” line to /boot/config.txt
  4. Append the “cgroup_memory=1 cgroup_enable=memory” options to /boot/cmdline.txt
  5. Boot up the rpi3 b+.
  6. Login (or SSH) into the pi.
  7. Get k3os-rootfs-arm64.tar.gz
  8. sudo tar xzvf k3os-rootfs-arm64.tar.gz --strip-components=1 -C /
  9. Create or cp /k3os/system/config.yaml for server
  10. sudo rpi-update. (check if this can be skipped.)
  11. Reboot
  12. Login (or SSH) into the pi1.local.net server.
  13. If everything goes fine, you could see followings.
Welcome to k3OS!

Refer to https://github.com/rancher/k3os for README and issues

By default mode of k3OS is to run a single node cluster. Use "kubectl"
to access it.  The node token in /var/lib/rancher/k3s/server/node-token
can be used to join agents to this server.

3. Import k3s cluster into your Rancher server.

  1. Browse to your Rancher server’s web UI.
  2. Select Add Cluser > Import an existing cluster then check the notifications.
  3. Login (or SSH) into the pi1.local.net server.
  4. curl --insecure -sfL https://your.rancher.server/v3/import/XXXXX.yaml > rancher.yaml
  5. vi rancher.yaml
:%s,image: rancher/rancher-agent:v2.3.5,image: rancher/rancher-agent:v2.3.5-linux-arm64,g
  1. kubectl apply -f rancher.yaml
  2. Check this k3s cloud is imported successfully.

4. Add some workers

  1. Login (or SSH) into the pi1.local.net server.
  2. cat /var/lib/rancher/k3s/server/node-token
  3. Do the same things 2.1~2.11 except 2.9 (config.yaml) on other rpi 3 b+ nodes.
  4. Edit config.yaml for worker nodes to change k3os.token with above node-token contents.
  5. Reboot worker after rpi-update.
  6. After a few minutes, you can see your worker nodes on the rancher server web UI.

That’s all folks.

Good luck!

Please find a solution. We love rancher!

To append onto what @SolaceZA and @felts94 suggested, I was able to use my existing Rancher 2.4.5 to adopt a current 1.16 K3S cluster.

I was able to convert existing pi’s to arm64 using @SolaceZA suggested steps. This process seems to have worked nicely without damaging anything I was paying attention to.

to @felts94 point I did find(though I am not sure if this is a sequencing problem or an actual issue) that I needed to edit BOTH the deployment/cattle-cluster-agent and daemonset/cattle-node-agent and explicitly force the use of *-linux-arm64 images. From here the cluster seems okay and I can confirm that I have been able to add nodes repeatedly using this method.

Ok this worked,

I also changed the deployment file to the image name rancher/rancher-agent:v2.5.1-linux-arm64 in this step

curl --insecure -sfL https:// /v3/import/dlzwxczz5vcrx4xk5kq4t97s8l2bm6zwf2cptx946txjjsbtxhv4kk.yaml | kubectl apply -f -

k3os and rpi-update might be the magic. But please check if rpi-update is a must.

Current version of k3os arm64 images does NOT support rpi 3 b+. See https://github.com/rancher/k3os/blob/master/README.md#arm-overlay-installation.

And of course, you can prepare or remaster the SD card. I’ll be glad to see your successful. 😉

最后咋解决的啊

NAME                                        READY   STATUS             RESTARTS   AGE
pod/cattle-node-agent-26zqp                 0/1     ImagePullBackOff   0          6m25s
pod/cattle-cluster-agent-745db996cc-2lwzr   0/1     ImagePullBackOff   0          6m25s

NAME                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/cattle-node-agent   1         1         0       1            0           <none>          6m26s

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cattle-cluster-agent   0/1     1            0           6m26s

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/cattle-cluster-agent-745db996cc   1         1         0       6m26s

yes, just rancher/rancher-agent no rancher-agent-armhf image only linux-amd64,linux-arm64,windows-amd64

ubuntu arm64 K3S docker :No container started