kind: Rancher-Desktop [Alpine] can't create cluster with v0.20.0 [Previously Also Colima]

Yeah, same issue here. brew install doesn’t support kind@0.19.0 so I had to install it through the go approach. Running go install sigs.k8s.io/kind@v0.19.0 seems to have temporarily fixed the issue.

Experiencing the same on Rancher Desktop. Downgrading to kind 0.19.0 fixes the issue for now.

Would be great to get a fix for 0.20.0.

The issue I see on Rancher Desktop using Kind 0.20.0 is the following:

$ kind create cluster --name test-cluster --image kindest/node:v1.27.3
Boostrapping cluster…
Creating cluster "test-cluster" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 ✗ Preparing nodes 📦  
Deleted nodes: ["eks-cluster-control-plane"]
ERROR: failed to create cluster: command "docker run --name test-cluster-control-plane --hostname test-cluster-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --label io.x-k8s.kind.cluster=test-cluster --net kind --restart=on-failure:1 --init=false --cgroupns=private --publish=127.0.0.1:50566:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node:v1.27.3" failed with error: exit status 125
Command Output: 82623b67d511c7e10ed075323e621ec66befa9047e3c7b56647ca99fd78e0db6
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "cgroup" to rootfs at "/sys/fs/cgroup": mount cgroup:/sys/fs/cgroup/openrc (via /proc/self/fd/7), flags: 0xe, data: openrc: invalid argument: unknown.

Colima v0.6.0 supports kind https://github.com/abiosoft/colima/releases/tag/v0.6.0

Do not want to duplicate issues. Running on MacOS Ventura 13.5.1.

Kind version

⚠️  kind --version
> kind version 0.20.0

 $ kind create cluster --config=config/kind/main.yaml
>
Creating cluster "kind-local" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 ✗ Preparing nodes 📦 📦
Deleted nodes: ["kind-local-control-plane" "kind-local-worker"]
ERROR: failed to create cluster: command "docker run --name kind-local-control-plane --hostname kind-local-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --label io.x-k8s.kind.cluster=kind-local --net kind --restart=on-failure:1 --init=false --cgroupns=private --publish=0.0.0.0:30070:30080/TCP --publish=127.0.0.1:62681:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node:v1.27.3@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72" failed with error: exit status 125
Command Output: a7174e21d76791171c521a8b7fd09e4fd2122f8f602d0735204f58073478078f
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "cgroup" to rootfs at "/sys/fs/cgroup": mount cgroup:/sys/fs/cgroup/openrc (via /proc/self/fd/7), flags: 0xe, data: openrc: invalid argument: unknown.

Docker info

⚠️  docker info
Client:
 Version:    24.0.2-rd
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.0
    Path:     /Users/ik/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.19.0
    Path:     /Users/ik/.docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 22
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1fbd70374134b891f97ce19c70b6e50c7b9f4e0d
 runc version: 860f061b76bb4fc671f0f9e900f7d80ff93d4eb7
 init version:
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 6.1.32-0-virt
 Operating System: Alpine Linux v3.18
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 5.798GiB
 Name: lima-rancher-desktop
 ID: JL2Y:IUE7:SXIV:CD7T:LS7D:PUWN:PAUE:TB6O:ELJP:7JVT:K67A:OSBM
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Rollback to 0.19

$ go install sigs.k8s.io/kind@v0.19.0
$ kind --version
> kind version 0.19.0
$ kind create cluster --config=config/kind/main.yaml
> Creating cluster "kind-local" ...
 ✓ Ensuring node image (kindest/node:v1.27.1) 🖼
 ✓ Preparing nodes 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-kind-local"
You can now use your cluster with:

kubectl cluster-info --context kind-kind-local

Switching to an Ubuntu image with regular lima instead of colima worked for me:

limactl start template://docker

Same error happens with Rancher Desktop that is using lima under the hood

You can start an ubuntu VM with https://github.com/lima-vm/lima instead (which colima is built on), please see previous comments #3277 (comment).

Thanks, switching to limactl start template://docker fixed my issue. I am now again able to build docker images without errors.

I’ve been able to switch Alpine to use the unified cgroups v2 layout, which seems to fix the buildkitd issue.

And it fixes the initial problem with kind as well, but fails with a different problem right after:

$ docker logs kind-control-plane
INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v2
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: faking /sys/class/dmi/id/product_name to be "kind"
INFO: setting iptables to detected mode: legacy
INFO: detected IPv4 address: 172.18.0.2
INFO: detected IPv6 address: fc00:f853:ccd:e793::2
INFO: starting init
systemd 247.3-7+deb11u2 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Debian GNU/Linux 11 (bullseye)!

Set hostname to <kind-control-plane>.
Failed to create /init.scope control group: Operation not supported
Failed to allocate manager object: Operation not supported
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...
INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v2
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: faking /sys/class/dmi/id/product_name to be "kind"
INFO: setting iptables to detected mode: legacy
INFO: detected IPv4 address: 172.18.0.2
INFO: detected old IPv4 address: 172.18.0.2
INFO: detected IPv6 address: fc00:f853:ccd:e793::2
INFO: detected old IPv6 address: fc00:f853:ccd:e793::2
INFO: starting init
systemd 247.3-7+deb11u2 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Debian GNU/Linux 11 (bullseye)!

Set hostname to <kind-control-plane>.
Failed to create /init.scope control group: Operation not supported
Failed to allocate manager object: Operation not supported
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...

I guess the issue is that cgroups are not writable inside the container.

the output of that command will give info on how to use docker CLI with it, which is all kind needs.

You can use the docker.lima (or podman.lima, or kubectl.lima) wrappers to do all the setup for you.

It was deemed unnecessary to have a all-in-one example of kind (or k3d), in addition to kubeadm (and k3s).

Right, colima and rancher desktop don’t have or need kind specific examples either to my knowledge.

kind just needs docker (or podman), so just the example for running docker with a functioning VM guest distro is sufficient.

The standard docker template currently uses ubuntu and is reported to work fine in an earlier comment https://github.com/kubernetes-sigs/kind/issues/3277#issuecomment-1680876276, as I understand it

Depending on your use case, it may make sense to use the kubeadm or K3s templates instead, but that’s a little out of scope here 😅

limactl start template://docker is briefly mentioned in https://github.com/lima-vm/lima#advanced-usage, and the output of that command will give info on how to use docker CLI with it, which is all kind needs. https://github.com/lima-vm/lima/blob/7b7b84a7983a7c26138660ad2db6ca9269963894/examples/docker.yaml#L80-L85

P.S. Thanks for your contributions, lima is a cool project 😃

Lima has support for running containerd, and Docker, and Podman, and Kubernetes out-of-the-box…

It was deemed unnecessary to have a all-in-one example of kind (or k3d), in addition to kubeadm (and k3s).

But that is also possible, if you want to run kind but don’t have access to Docker Engine or Podman Engine:

• https://github.com/lima-vm/lima/issues/1184#issuecomment-1365147754

@BenTheElder what are the current options on Mac given that colima and rancher-desktop are based on Alpine and don’t support cgroup v2? is it just pinning kind to v0.19.0 and waiting for one of these projects to fix the issue?

The tool both colima and rancher-desktop are built on, lima, supports other distros / templates, and should work fine. Aside from e.g. docker desktop or running docker in other VM tools that are not pinned to Alpine. Podman desktop also supports kind, though kind needs some improvements around podman still.

limactl start template://docker should work https://github.com/kubernetes-sigs/kind/issues/3277#issuecomment-1680876276

the other is more control over the k8s version used.

You can pick any k8s (k3s) version you want in Rancher Desktop and you can also upgrade to any new version and see how it affects your deployed workloads:

I’m not actually sure if versions prior to 1.19 still work properly, but all the more recent releases should be fully functional.

Off-topic question, but why not use Rancher Desktop’s Kubernetes? 😄

For me the only reason to use k3d is when I want to have a multi-node cluster to play around with pod placement strategies like taints and affinity, to make sure the manifests work as expected.

Eventually there should be a config setting in Rancher Desktop to allow multiple nodes. Personally I’ve also wanted a mixed-architecture cluster with both amd64 and arm64 nodes, but that is more for fun than actual need… 😄

I think I will stick with Ubuntu LTS for the default kubeadm template (k8s.yaml), even if Debian is also a possibility.

https://github.com/rancher-sandbox/rancher-desktop/issues/5363

I understand that with https://github.com/kubernetes-sigs/kind/pull/3241 the ship might have already sailed but perhaps we might still consider using the provider info Cgroup2 field and set the --cgroupns flag only when cgroupv2 is available?

The point of setting this flag is to ensure that this is set on cgroupv1 hosts. cgroupv2 hosts already default to this.

cgroupv1 hosts are the problem. On hosts other than apline/colima/rancher desktop this works great. Alpine and colima / rancher desktop use an unusual init system that doesn’t seem to set this up properly.

We also have a lot of DNS issues with Lima due to use Alpine. I really wish they would move away from a musl based operating system.

Lima defaults to Ubuntu…

limactl start template://docker

Using Alpine is a choice by downstream, mostly for size reasons. I don’t know of an apk distro using systemd/glibc instead of openrc/musl, but I suppose it is possible (or maybe use Debian, it is also smaller)

EDIT: updating this early comment to note that Colima is fixed via https://github.com/kubernetes-sigs/kind/issues/3277#issuecomment-1807235030, just upgrade to v0.6.0 colima

This is an issue with the host environment presumably with --cgroupns=private.

colima is @abiosoft