kind: Kind not working on RHEL8

What happened:

Cluster did not start.

What you expected to happen:

Cluster would start

How to reproduce it (as minimally and precisely as possible):

kind create cluster Anything else we need to know?:

[root@rhel8 ci-tools]# kind create cluster
enabling experimental podman provider
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼 
 ✗ Preparing nodes 📦  
ERROR: failed to create cluster: podman run error: command "podman run --hostname kind-control-plane --name kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume 55514369f9cc8993328e1cc4e90c178b7f28d6959c0480f473aae19596460a52:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro --detach --tty --net kind --label io.x-k8s.kind.cluster=kind -e container=podman --publish=127.0.0.1:38363:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6" failed with error: exit status 126
Command Output: time="2021-08-20T07:02:56Z" level=error msg="error loading cached network config: network \"kind\" not found in CNI cache"
time="2021-08-20T07:02:56Z" level=warning msg="falling back to loading from existing plugins on disk"
time="2021-08-20T07:02:56Z" level=error msg="Error tearing down partially created network namespace for container 82f418157e4710f79828e9e187a801311250086672938abf272aa1ec01be8bff: error removing pod kind-control-plane_kind-control-plane from CNI network \"kind\": running [/usr/sbin/iptables -t nat -D POSTROUTING -s 10.89.0.5 -j CNI-54b084262fabc3428409d3dd -m comment --comment name: \"kind\" id: \"82f418157e4710f79828e9e187a801311250086672938abf272aa1ec01be8bff\" --wait]: exit status 2: iptables v1.8.4 (nf_tables): Chain 'CNI-54b084262fabc3428409d3dd' does not exist\nTry `iptables -h' or 'iptables --help' for more information.\n"
Error: error configuring network namespace for container 82f418157e4710f79828e9e187a801311250086672938abf272aa1ec01be8bff: error adding pod kind-control-plane_kind-control-plane to CNI network "kind": failed to set bridge addr: could not add IP address to "cni-podman1": permission denied

Environment:

  • kind version: (use kind version):
[root@rhel8 ci-tools]# kind version
kind v0.11.1 go1.16.4 linux/amd64
  • Kubernetes version: (use kubectl version):
[root@rhel8 ci-tools]# podman  version
Version:      3.2.3
API Version:  3.2.3
Go Version:   go1.15.7
Built:        Tue Jul 27 07:29:39 2021
OS/Arch:      linux/amd64
  • Docker version: (use docker info):
[root@rhel8 ci-tools]# podman info
host:
  arch: amd64
  buildahVersion: 1.21.3
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.26-3.module+el8.4.0+11311+9da8acfb.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: a35bb9ea67d5a83c7da53202f2fcd505c036d29c'
  cpus: 1
  distribution:
    distribution: '"rhel"'
    version: "8.3"
  eventLogger: file
  hostname: rhel8.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-240.el8.x86_64
  linkmode: dynamic
  memFree: 1141030912
  memTotal: 1880039424
  ociRuntime:
    name: runc
    package: runc-1.0.0-73.rc93.module+el8.4.0+11311+9da8acfb.x86_64
    path: /usr/bin/runc
    version: |-
      runc version spec: 1.0.2-dev
      go: go1.15.13
      libseccomp: 2.4.3
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 2195595264
  swapTotal: 2210394112
  uptime: 24m 57.57s
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 44
    paused: 0
    running: 0
    stopped: 44
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 317
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.2.3
  Built: 1627370979
  BuiltTime: Tue Jul 27 07:29:39 2021
  GitCommit: ""
  GoVersion: go1.15.7
  OsArch: linux/amd64
  Version: 3.2.3
  • OS (e.g. from etc/os-release):
[root@rhel8 ci-tools]# cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.3 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.3"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.3"

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 24 (8 by maintainers)

Most upvoted comments

I had the same problem trying to make kind work on Fedora 35 w/ rootless Podman. After trying everything else I could possibly find, the fix was to install rootless Docker. I have no idea why it works, but putting it here in case it helps someone else.

curl -fsSL https://get.docker.com/rootless | sh

(thanks https://mohitgoyal.co/2021/04/26/create-kubernetes-clusters-with-kind-rootless-docker-and-rootless-podman/)

Update: I tried using docker client and with podman socket, it didn’t work. But when I tried using docker socket with docker client, it worked (sudo kind create cluster):

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., 0.0.0+unknown)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: journald
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: v1.5.4
 runc version:
 init version:
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.71
 Operating System: NixOS 21.05 (Okapi)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.13GiB
 Name: alexlaptop
 ID: MQRS:KJRE:5XZL:75F6:VJZK:WHWZ:D3PH:LVRE:RP6N:3ZZ6:TWBM:7REE
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true