kind: Create cluster fails - kind-control-plane does not work on zfs

What happened: cluster create failed

failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged kind-control-plane kubeadm init --ignore-preflight-errors=all --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1

What you expected to happen: successful cluster creation

How to reproduce it (as minimally and precisely as possible):

$ kind create cluster 

Anything else we need to know?:

I’m using ZFS. I have read the other (now closed as resolved) issue about using zfs and I can see that /dev/mapper is bind-mounted into the kind-control-plane container. I’m running 0.8.1 so I believe that I should have the version that should work with ZFS.

Environment:

• kind version: (use kind version): kind v0.8.1 go1.14.2 linux/amd64

• Kubernetes version: (use kubectl version): Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"archive", BuildDate:"2020-04-23T22:11:11Z", GoVersion:"go1.14.2", Compiler:"gc", Platform:"linux/amd64"}

• Docker version: (use docker info): Server Version: 19.03.8-ce containerd version: d76c121f76a5fc8a462dc64594aea72fe18e1178.m

• OS (e.g. from /etc/os-release): Arch Linux 5.6.11-arch1-1 #1 SMP PREEMPT Wed, 06 May 2020 17:32:37 +0000 x86_64 GNU/Linux

To capture more information, I ran with kind create cluster --loglevel=debug --retain. Here is some further log data…

In kind-control-plane/containerd.log

Jul 08 12:53:09 kind-control-plane containerd[127]: time="2020-07-08T12:53:09.553579553Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-apiserver-kind-control-plane,Uid:350cc499f8fb9468ea828e0b13035d50,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to mount rootfs component &{overlay overlay [workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/82/work upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/82/fs lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs]}: invalid argument: unknown"

Also it appears to be trying to use zfs but doesn’t have the zfs executable:

Jul 09 14:57:36 kind-control-plane containerd[161]: time="2020-07-09T14:57:36.850165016Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
Jul 09 14:57:36 kind-control-plane containerd[161]: time="2020-07-09T14:57:36.850516545Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="exec: \"zfs\": executable file not found in $PATH: \"zfs fs list -Hp -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset system/storage/docker\" => "
Jul 09 14:57:36 kind-control-plane containerd[161]: time="2020-07-09T14:57:36.850555778Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
Jul 09 14:57:36 kind-control-plane containerd[161]: time="2020-07-09T14:57:36.850594684Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="exec: \"zfs\": executable file not found in $PATH: \"zfs fs list -Hp -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset system/storage/docker\" => "