kind: [fedora firewall issue] Fail to create an IPv6 multinode cluster - it hangs when it is Joining the workers.

What happened: When I am trying to run a multinode or multinode HA cluster with IPv6 it just hangs in there for so many minutes that I have stopped counting,

What you expected to happen: The nodes should join the cluster after less than 2 minutes and I have waited 45 minutes and nothing.

How to reproduce it (as minimally and precisely as possible):

cat <<EOF >./kind_multinode_ipv6.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
networking:
  ipFamily: ipv6
EOF

$ time kind create cluster --name multinode_ipv6 --config kind_multinode_ipv6.yaml

Anything else we need to know?: I tried global IPv6 routing space and IPv6 unique local address and I’ve got the same results. when I was trying to run the same YAML for IPv4 only there are no problems.

Environment:

kind version: (use kind version): kind v0.7.0 go1.13.5 linux/amd64
Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.2”, GitCommit:“59603c6e503c87169aea6106f57b9f242f64df89”, GitTreeState:“clean”, BuildDate:“2020-01-18T23:30:10Z”, GoVersion:“go1.13.5”, Compiler:“gc”, Platform:“linux/amd64”}

Docker version: (use docker info):

Client: Debug Mode: false

Server: Containers: 3 Running: 0 Paused: 0 Stopped: 3 Images: 18 Server Version: 19.03.5 Storage Driver: btrfs Build Version: Btrfs v5.2.1 Library Version: 102 Logging Driver: json-file Cgroup Driver: cgroupfs

Client: Docker Engine - Community Version: 19.03.5 API version: 1.40 Go version: go1.12.12 Git commit: 633a0ea838 Built: Wed Nov 13 07:26:43 2019 OS/Arch: linux/amd64 Experimental: false

Server: Docker Engine - Community Engine: Version: 19.03.5 API version: 1.40 (minimum version 1.12) Go version: go1.12.12 Git commit: 633a0ea838 Built: Wed Nov 13 07:24:37 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.2.10 GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339 runc: Version: 1.0.0-rc8+dev GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 docker-init: Version: 0.18.0 GitCommit: fec3683

OS (e.g. from /etc/os-release):

NAME=Fedora VERSION=“31 (Workstation Edition)” ID=fedora VERSION_ID=31 VERSION_CODENAME=“” PLATFORM_ID=“platform:f31” PRETTY_NAME=“Fedora 31 (Workstation Edition)” ANSI_COLOR=“0;34” LOGO=fedora-logo-icon CPE_NAME=“cpe:/o:fedoraproject:fedora:31” HOME_URL=“https://fedoraproject.org/” DOCUMENTATION_URL=“https://docs.fedoraproject.org/en-US/fedora/f31/system-administrators-guide/” SUPPORT_URL=“https://fedoraproject.org/wiki/Communicating_and_getting_help” BUG_REPORT_URL=“https://bugzilla.redhat.com/” REDHAT_BUGZILLA_PRODUCT=“Fedora” REDHAT_BUGZILLA_PRODUCT_VERSION=31 REDHAT_SUPPORT_PRODUCT=“Fedora” REDHAT_SUPPORT_PRODUCT_VERSION=31 PRIVACY_POLICY_URL=“https://fedoraproject.org/wiki/Legal:PrivacyPolicy” VARIANT=“Workstation Edition” VARIANT_ID=workstation

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 26 (17 by maintainers)

Most upvoted comments

Todo listo no hay problemas, I made a test with 2 new VMs and I issued the firewalld commands you wrote. Nothing more to report.

ricardo-rod on Jan 31, 2020

@aojea yes the cluster works and the pods are running and the containers too, when I shutdown the firewalld I got the same effect everything works, but when the daemon firewalld is started after the creation of the pods, services and dashboard do not get through.

was this rule added by the firewalld-cmd command ‘/usr/sbin/iptables -w10 -t nat -A DOCKER -p tcp -d ::1 --dport 32770 -j DNAT --to-destination 172.17.0.2:6443 ! -i docker0’? Yes, it was issued by the daemon firewalld

if the cluster works docker network inspect bridge should show the containers attached to the bridge and I can’t see any.

docker network inspect bridge

[ { "Name": "bridge", "Id": "6cdc9a4f1e2c09a9d70893a0aaad82406868426851b61eb78b1902e82ae97b47", "Created": "2020-01-31T11:15:29.918267531-04:00", "Scope": "local", "Driver": "bridge", "EnableIPv6": true, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "172.17.0.1/20", "IPRange": "172.17.0.0/20", "Gateway": "172.17.0.1" }, { "Subnet": "fc00:dead:beef::/64", "Gateway": "fc00:dead:beef::1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": { "3628b28fb6273868142fd7c2a4f5c6277ec11dca6e7b955c4d3748a6dba28edb": { "Name": "multinode_ipv6-worker", "EndpointID": "697ef0e71f6989c4e9e9d7ba5319ff5aae75b6193a3f616db552848433695bb1", "MacAddress": "02:42:ac:11:00:03", "IPv4Address": "172.17.0.3/20", "IPv6Address": "fc00:dead:beef::242:ac11:3/64" }, "397f20f81cc540743d8dd4ace77908a076a244de932004be930cebe0f40c0df0": { "Name": "multinode_ipv6-worker2", "EndpointID": "6dfdb5d643f7e2afc6ba5327269ae8a34b4893f25e4414a55c03f2e34bd67741", "MacAddress": "02:42:ac:11:00:02", "IPv4Address": "172.17.0.2/20", "IPv6Address": "fc00:dead:beef::242:ac11:2/64" }, "e2e301bb783cf7ef1508b8ebc6929bd0527b7534322c5c83954c2e40448596d1": { "Name": "multinode_ipv6-control-plane", "EndpointID": "304322cf8be332ee11321251b88686b75c4388e1e2e41bc0d3ad230f597c1682", "MacAddress": "02:42:ac:11:00:04", "IPv4Address": "172.17.0.4/20", "IPv6Address": "fc00:dead:beef::242:ac11:4/64" } }, "Options": { "com.docker.network.bridge.default_bridge": "true", "com.docker.network.bridge.enable_icc": "true", "com.docker.network.bridge.enable_ip_masquerade": "true", "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0", "com.docker.network.bridge.name": "docker0", "com.docker.network.driver.mtu": "1500" }, "Labels": {} } ]

kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME multinodeipv6-control-plane Ready master 128m v1.17.0 fc00:dead:beef::242:ac11:4 <none> Ubuntu 19.10 5.4.13-201.fc31.x86_64 containerd://1.3.2 multinodeipv6-worker Ready <none> 127m v1.17.0 fc00:dead:beef::242:ac11:3 <none> Ubuntu 19.10 5.4.13-201.fc31.x86_64 containerd://1.3.2 multinodeipv6-worker2 Ready <none> 127m v1.17.0 fc00:dead:beef::242:ac11:2 <none> Ubuntu 19.10 5.4.13-201.fc31.x86_64 containerd://1.3.2 Waiting further results.

ricardo-rod on Jan 31, 2020