kind: Error Starting Control Plane w/ Podman on Fedora 33

What happened: Hello, when trying to create a kind cluster via kind create cluster on Fedora 33 w/ podman resulted in an error deploying the control plane.

What you expected to happen: I was hoping for a cluster to be created.

How to reproduce it (as minimally and precisely as possible):

Fedora 33 w/ podman as underlying OS
Execute kind create cluster

Anything Else Here are further log details

➜  kind kind create cluster                                                                                                   
enabling experimental podman provider                                                                                         
Creating cluster "kind" ...                                                                                                   
 ✓ Ensuring node image (kindest/node:v1.20.2) 🖼                                                                               
 ✓ Preparing nodes 📦                                                                                                         
 ✓ Writing configuration 📜                                                                                                   
 ✗ Starting control-plane 🕹️             
                                                                                      
ERROR: failed to create cluster: failed to init node with kubeadm: command "podman exec --privileged kind-control-plane kubead
m init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1         
Command Output: I0425 20:27:13.948213      98 initconfiguration.go:201] loading configuration from "/kind/kubeadm.conf"       
[config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta2, Kind=JoinConfiguration                  

 ...

[kubelet-check] It seems like the kubelet isn't running or healthy.          
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:102
48/healthz": dial tcp [::1]:10248: connect: connection refused. 
                               
        Unfortunately, an error has occurred:     
                timed out waiting for the condition

Environment:

kind version: (use kind version): kind v0.11.0-alpha go1.15.8 linux/amd64
Kubernetes version: (use kubectl version): 1.21
Docker version: (use docker info):N/A
Podman version: (use podman info):3.1.2
OS (e.g. from /etc/os-release): Fedora 33

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 3
Comments: 28 (14 by maintainers)

Most upvoted comments

Same problem with Fedora 34.

alanjholiveira on May 21, 2021

weird, are you using fedora? it is complaining about the kernel module for ip6ables, you have to load it

sudo modprobe -v ip6_tables
insmod /lib/modules/5.4.132-1.el8.elrepo.x86_64/kernel/net/ipv6/netfilter/ip6_tables.ko.xz 
$ sudo lsmod | grep ip6
ip6_tables             36864  0

aojea on Oct 7, 2021

I’m facing the same issue with a Fedora 34 virtual machine. Going through other issues, #2112 makes me wonder if this issue is related to btrfs. I’m not sure other Fedora users under this issue are using btrfs as it’s default since Fedora 33. Out of curiosity, I made another vm with same Fedora 34 workstation installation and same 5.13.6 kernel version but choosing xfs manually instead of btrfs. And there is no issue on that.

(Please refer to the attachment if you need log files from my kind export logs output) logs.zip

seraphlive on Aug 3, 2021

After ugrading podman recently I have run into problems myself. When using the v1.21.2 kind node image that contains the proper iptables wrapper script I appear to be reverting to having similar issues.

$ podman run --hostname kind-control-plane --name kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume bf9ca67764a451fd098017594578ea5b2e35808047316865ed43bcbf89c649ba:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro --detach --tty --net kind --label io.x-k8s.kind.cluster=kind -e container=podman --publish=127.0.0.1:34813:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node:v1.21.2
ERRO[0002] error loading cached network config: network "kind" not found in CNI cache
WARN[0002] falling back to loading from existing plugins on disk
ERRO[0003] Error tearing down partially created network namespace for container 8779c1e9e30e664da84cc7d40537a104b44f99cffa07c44ed4354ce2d9644f34: error removing pod kind-control-plane_kind-control-plane from CNI network "kind": running [/usr/bin/iptables -t nat -D POSTROUTING -s 10.88.2.13 -j CNI-60d270dcc022372e968c855b -m comment --comment name: "kind" id: "8779c1e9e30e664da84cc7d40537a104b44f99cffa07c44ed4354ce2d9644f34" --wait]: exit status 2: iptables v1.8.7 (legacy): Couldn't load target `CNI-60d270dcc022372e968c855b':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
Error: error configuring network namespace for container 8779c1e9e30e664da84cc7d40537a104b44f99cffa07c44ed4354ce2d9644f34: error adding pod kind-control-plane_kind-control-plane to CNI network "kind": failed to list chains: running [/usr/bin/ip6tables -t nat -S --wait]: exit status 3: modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted
ip6tables v1.8.7 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.

Sorry for replying to closed issue:

I’m just replying here as this is where google sent me when googling modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted when trying to start kind with podman. Just sudo to root and run modprobe ip6_tables and retry and it works for me.

kind version 0.18.0 kindest/node:v1.26.3 podman version 4.4.4

lookcrabs on Apr 4, 2023

I’m going to close this issue to avoid more confusion, no worries about that.

Your error seems related to that https://github.com/kubernetes-sigs/kind/issues/2445, AFAIK podman machine doesn’t work /close

aojea on Oct 7, 2021

probably:

/assign @aojea

BenTheElder on Apr 26, 2021