kind: Cluster does not initialize properly, kindnet times out and crashes.

What happened:

CoreDNS fails to start, and Kindnet has timeouts in their logs:

I0402 17:39:58.069948       1 main.go:64] hostIP = 172.17.0.2
podIP = 172.17.0.2
I0402 17:40:33.166911       1 main.go:104] Failed to get nodes, retrying after error: Get https://10.96.0.1:443/api/v1/nodes: dial tcp 10.96.0.1:443: i/o timeout
I0402 17:41:03.167486       1 main.go:104] Failed to get nodes, retrying after error: Get https://10.96.0.1:443/api/v1/nodes: dial tcp 10.96.0.1:443: i/o timeout
I0402 17:41:34.167858       1 main.go:104] Failed to get nodes, retrying after error: Get https://10.96.0.1:443/api/v1/nodes: dial tcp 10.96.0.1:443: i/o timeout
I0402 17:42:06.168271       1 main.go:104] Failed to get nodes, retrying after error: Get https://10.96.0.1:443/api/v1/nodes: dial tcp 10.96.0.1:443: i/o timeout
I0402 17:42:39.168601       1 main.go:104] Failed to get nodes, retrying after error: Get https://10.96.0.1:443/api/v1/nodes: dial tcp 10.96.0.1:443: i/o timeout
panic: Reached maximum retries obtaining node list: Get https://10.96.0.1:443/api/v1/nodes: dial tcp 10.96.0.1:443: i/o timeout

What you expected to happen:

CoreDNS containers to start, and kindnet not to crash

How to reproduce it (as minimally and precisely as possible):

kind create cluster kubectl -n kube-system logs kindnet-xxxxx

NAME                                         READY   STATUS             RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
coredns-6955765f44-tvtk5                     0/1     Pending            0          20m   <none>       <none>               <none>           <none>
coredns-6955765f44-vbpxc                     0/1     Pending            0          20m   <none>       <none>               <none>           <none>
etcd-kind-control-plane                      1/1     Running            0          20m   172.17.0.2   kind-control-plane   <none>           <none>
kindnet-wgsbc                                0/1     CrashLoopBackOff   5          20m   172.17.0.2   kind-control-plane   <none>           <none>
kube-apiserver-kind-control-plane            1/1     Running            0          20m   172.17.0.2   kind-control-plane   <none>           <none>
kube-controller-manager-kind-control-plane   1/1     Running            0          20m   172.17.0.2   kind-control-plane   <none>           <none>
kube-proxy-gwjfp                             1/1     Running            0          20m   172.17.0.2   kind-control-plane   <none>           <none>
kube-scheduler-kind-control-plane            1/1     Running            0          20m   172.17.0.2   kind-control-plane   <none>           <none>

Environment:

kind version: (use kind version): 0.7.0
Kubernetes version: (use kubectl version): 1.17.0
Docker version: (use docker info): 19.03.8
OS (e.g. from /etc/os-release): Gentoo

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 22 (13 by maintainers)

Most upvoted comments

I encountered the same problem with running kind on Gentoo Linux. I discovered that kind-proxy requires

CONFIG_IP_VS_RR
CONFIG_IP_VS_WRR
CONFIG_NETFILTER_XT_MATCH_STATISTIC
CONFIG_NETFILTER_XT_MATCH_COMMENT

I think it is not exhaustive list of course, but may be helpful for someone.

Tiberivs on Jan 16, 2021

I encountered the same problem. kind requires iptables to support comment match, so I have enabled CONFIG_NETFILTER_XT_MATCH_COMMENT in my kernel config to resolve the problem.

Most distributions probably enable it by default, but in the case of distributions like Gentoo Linux, you should configure it properly.

ntoofu on Dec 20, 2020

I encountered the same problem with running kind on Gentoo Linux. I discovered that kind-proxy requires
CONFIG_IP_VS_RR
CONFIG_IP_VS_WRR
CONFIG_NETFILTER_XT_MATCH_STATISTIC
CONFIG_NETFILTER_XT_MATCH_COMMENT
I think it is not exhaustive list of course, but may be helpful for someone.

I know it’ve been 2 year but thanks man, i try to setup k8s and kind on gentoo, it come to the point I want to switch my os but you saved me

tin-pham on Jun 7, 2023

That seems the problem, is missing all the kube-proxy rules with the services.

What is kube-proxy logging kubectl logs -n kube-system kube-proxy-pk8km?

aojea on Apr 2, 2020

docker info:

Client:
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc.)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 35bd7a5f69c13e1563af8a93431411cd9ecf5021
 runc version: 
 init version: fec3683b971d9c3ef73f284f176672c44b448662
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.5.14-gentoo
 Operating System: Gentoo/Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.58GiB
 Name: thundercloud
 ID: DYOX:QA2L:J7ZA:WS2R:RRHG:5OCY:ZHGS:VBNH:HCJU:PZQJ:S4FC:HD65
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

/etc/resolv.conf

# Generated by NetworkManager
search lan
nameserver 192.168.1.20
nameserver 2604:5500:c248:6b00:134f:7b72:6bb3:71d2
nameserver 2604:5500:c248:6b00::1

I do have legacy iptables:

eselect iptables list

Available iptables symlink targets:
  [1]   xtables-legacy-multi *`

iptables --version

iptables v1.8.4 (legacy)

salasrod on Apr 2, 2020