kind: Can't run KIND on ChromeOS Linux VM

Hi! 👋

K8s ports are not on the existing list of default ports

you can configure kind to use a particular port with kind create cluster --config=config.yaml and config.yaml:

apiVersion: kind.sigs.k8s.io/v1alpha3
kind: Config
networking:
  apiServerPort: 6443

Beyond that there is also the use of a privileged container - which I understood was not supported on Crostini? This is more of an issue and one which I do not think will be enabled due to security concerns.

That’s a bit more of a blocker.

Each kind “node” docker container is --privileged in order to run containerd / pods on the inside, each kubelet needs its own container runtime. To run pods we need proc, cgroups, mounting …

It might be possible to hack up kind to use usermode Linux instead but that’s going to be fairly involved.

If I read the logs from @adrianludwin correctly though, we actually manage to create those containers, but it fails once we try to use mount inside them. Besides the re-mounting we do*, I believe kubelet needs permission to mount in order to setup pods etc.

* (changing /sys to read only, docker forces /sys to a read-write mount if you specify --privileged),

I would like to inform you I can run KinD on ChromeOS: either with docker or with podman (running from root).

First I enabled nesting mode for LXC container: it is requirement for Docker on ChromeOS. After ctrl+alt+t in Chrome:

vmc launch termina
lxc config set penguin security.nesting true
lxc restart penguin

Then:

cat > cluster.yaml << END 
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  KubeletInUserNamespace: true
END
kind create cluster --config cluster.yaml

The reason why KinD fails is LXC container misses some devices (/dev/kmsg, /dev/mapper/control) then kubelet panics.

Also rootless setup doesn’t work because either it doesn’t have bridge-nf-call-iptables or cgroups v2. I think it is not a problem if KubeletInUserNamespace is used and docker or podman are started from root.

oh, it works after changing the /etc/libvirt/qemu.conf and rebooting the chrome book! Thanks @cbandy ! add these lines: user = “root” group = “root” remember_owner = 0

@maxamillion I’m sorry. It works for me: very fresh ChromeOS with bookworm (fresh install, not an upgrade) with Docker 24.0.7 and kind v0.20.0:

dex4er@penguin:~/tmp$ cat cluster.yaml 
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  KubeletInUserNamespace: true
dex4er@penguin:~/tmp$ kind create cluster --config cluster.yaml 
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/
dex4er@penguin:~/tmp$ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS                       NAMES
862e3ffebc57   kindest/node:v1.27.3   "/usr/local/bin/entr…"   54 seconds ago   Up 52 seconds   127.0.0.1:41295->6443/tcp   kind-control-plane
dex4er@penguin:~/tmp$ kubectl version
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.3

You can try docker logs -f kind-control-plane for troubleshooting. Also:

$ docker exec -it kind-control-plane bash
root@kind-control-plane:/# journalctl -u kubelet

Thanks @keyonjie I updated my blog with the additional steps.

I used ChromeOS debian (Crostini) environment, no additional changes were required. I used a Pixelbook i7 16Gb and a Asus CN60 i7 16Gb to test. I posted the blog post over a year ago, so may require some changes?

sn’t use VMs (it uses container no

Thanks for sharing @rosera . I am going to try what you have shared. Can you share more details about your environment? e.g. we should perform all commands on ChromeOS/crostini/debian (or ubuntu? 20.04?)

FYI - ChromeOS supports minikube. I havent tried Kind, so not sure if this works. I wrote how to run Minikube here https://richrose.dev/posts/chromeos/productivity/chromeos-minikube/

Hey, I am highly interested in getting this working on ChromeOS (I raised the original ticket reference https://bugs.chromium.org/p/chromium/issues/detail?id=878034). Thanks very much for the tips regarding debugging - very helpful.

I plan to investigate a bit more over the coming weekend. My thoughts at the moment relate to the container being restricted on the accessible ports available to Crostini. K8s ports are not on the existing list of default ports. I believe you can enable them, so it might be trial and error for this to work. Beyond that there is also the use of a privileged container - which I understood was not supported on Crostini? This is more of an issue and one which I do not think will be enabled due to security concerns.

Running the docker run command on its own, without -d, has the same result I got initially:

aludwin@penguin:~$ docker run -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname kind-control-plane --name kind-control-plane --label io.k8s.sigs.kind.cluster=kind --label io.k8s.sigs.kind.role=control-plane --expose 43283 --publish=127.0.0.1:43283:6443 kindest/node:v1.15.0@sha256:b4d092fd2b507843dd096fe6c85d06a27a0cbd740a0b32a880fe61aba24bb478
mount: /sys: permission denied.

Re #462: I did see that bug but I thought this was something different. My computer successfully got past the “preparing nodes” step but died on the next “Creating kubeadm config” step.

Can I provide any more details that would help with this? Agreed that there’s a chance ChromeOS won’t want to “fix” this but I’m motivated to push on it a bit 😃

Here’s the result of the debug logs:

aludwin@penguin:~$ kind create cluster --loglevel=debug
DEBU[12:04:09] Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}}] 
Creating cluster "kind" ...
DEBU[12:04:09] Running: /usr/bin/docker [docker inspect --type=image kindest/node:v1.15.0] 
INFO[12:04:09] Image: kindest/node:v1.15.0 present locally  
 ✓ Ensuring node image (kindest/node:v1.15.0) 🖼
DEBU[12:04:09] Running: /usr/bin/docker [docker info --format '{{json .SecurityOptions}}'] 
DEBU[12:04:09] Running: /usr/bin/docker [docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname kind-control-plane --name kind-control-plane --label io.k8s.sigs.kind.cluster=kind --label io.k8s.sigs.kind.role=control-plane --expose 43283 --publish=127.0.0.1:43283:6443 kindest/node:v1.15.0@sha256:b4d092fd2b507843dd096fe6c85d06a27a0cbd740a0b32a880fe61aba24bb478] 
 ✓ Preparing nodes 📦 
DEBU[12:04:13] Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}} --filter label=io.k8s.sigs.kind.cluster=kind] 
DEBU[12:04:13] Running: /usr/bin/docker [docker inspect -f {{index .Config.Labels "io.k8s.sigs.kind.role"}} kind-control-plane] 
DEBU[12:04:14] Running: /usr/bin/docker [docker exec --privileged kind-control-plane cat /kind/version] 
 ✗ Creating kubeadm config 📜 
DEBU[12:04:14] Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}} --filter label=io.k8s.sigs.kind.cluster=kind] 
DEBU[12:04:14] Running: /usr/bin/docker [docker rm -f -v kind-control-plane] 
Error: failed to create cluster: failed to get kubernetes version from node: failed to get file: exit status 1

I’ll try to reproduce this more precisely.