kind: Kind clusters using ipvs kubeProxyMode doesn't work On v1.21.1

What happened:

Clusters using image kindest/node:v1.21.1 cannot resolve DNS names. It works, for example, with kindest/node:v1.20.7

What you expected to happen:

DNS should work

How to reproduce it (as minimally and precisely as possible):

mnencia@sirio  ➤  kind create cluster --image=kindest/node:v1.21.1
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋
mnencia@sirio  ➤  kubectl run -i -t busybox --image=busybox --restart=Never --rm
If you don't see a command prompt, try pressing enter.
/ # ping wwww.google.com
ping: bad address 'wwww.google.com'
/ #

Anything else we need to know?:

As a workaround, you can patch the CoreDNS configuration and restart the CoreDNS deployment

--- coredns.yaml.orig	2021-06-24 11:40:42.000000000 +0200
+++ coredns.yaml	2021-06-24 11:41:00.000000000 +0200
@@ -1,30 +1,30 @@
 apiVersion: v1
 data:
   Corefile: |
     .:53 {
         errors
         health {
            lameduck 5s
         }
         ready
         kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
         }
         prometheus :9153
-        forward . /etc/resolv.conf {
+        forward . dns://1.1.1.1 {
            max_concurrent 1000
         }
         cache 30
         loop
         reload
         loadbalance
     }
 kind: ConfigMap
 metadata:
   creationTimestamp: "2021-06-24T09:28:09Z"
   name: coredns
   namespace: kube-system
   resourceVersion: "238"
   uid: c2def70f-5890-41cb-8184-b4dd33e8cece
mnencia@sirio  ➤  kubectl apply -f coredns.yaml
Warning: resource configmaps/coredns is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
configmap/coredns configured
mnencia@sirio  ➤  kubectl rollout restart -n kube-system deployment coredns
deployment.apps/coredns restarted
mnencia@sirio  ➤  kubectl run -i -t busybox --image=busybox --restart=Never --rm
If you don't see a command prompt, try pressing enter.
/ # ping www.google.com
PING www.google.com (142.250.180.68): 56 data bytes
64 bytes from 142.250.180.68: seq=0 ttl=36 time=11.237 ms
64 bytes from 142.250.180.68: seq=1 ttl=36 time=59.229 ms
64 bytes from 142.250.180.68: seq=2 ttl=36 time=9.575 ms
64 bytes from 142.250.180.68: seq=3 ttl=36 time=10.221 ms
^C
--- www.google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 9.575/22.565/59.229 ms
/ #
pod "busybox" deleted

Environment:

  • kind version:
    kind v0.11.1 go1.16.4 darwin/amd64
    
  • Kubernetes version:
    Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"darwin/amd64"}
    Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-18T01:10:20Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
    
  • Docker version:
    Client:
     Context:    default
     Debug Mode: false
     Plugins:
      buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
      compose: Docker Compose (Docker Inc., 2.0.0-beta.3)
      scan: Docker Scan (Docker Inc., v0.8.0)
    
    Server:
     Containers: 2
      Running: 2
      Paused: 0
      Stopped: 0
     Images: 10
     Server Version: 20.10.7
     Storage Driver: overlay2
      Backing Filesystem: extfs
      Supports d_type: true
      Native Overlay Diff: true
      userxattr: false
     Logging Driver: json-file
     Cgroup Driver: cgroupfs
     Cgroup Version: 1
     Plugins:
      Volume: local
      Network: bridge host ipvlan macvlan null overlay
      Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
     Swarm: inactive
     Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
     Default Runtime: runc
     Init Binary: docker-init
     containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
     runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
     init version: de40ad0
     Security Options:
      seccomp
       Profile: default
     Kernel Version: 5.10.25-linuxkit
     Operating System: Docker Desktop
     OSType: linux
     Architecture: x86_64
     CPUs: 6
     Total Memory: 3.844GiB
     Name: docker-desktop
     ID: MVJ3:ZO5L:5OPX:VUGX:64A7:SHR5:3CUR:GS47:446R:UGPZ:Y2BS:2MRN
     Docker Root Dir: /var/lib/docker
     Debug Mode: true
      File Descriptors: 58
      Goroutines: 57
      System Time: 2021-06-24T09:37:41.819583449Z
      EventsListeners: 3
     HTTP Proxy: http.docker.internal:3128
     HTTPS Proxy: http.docker.internal:3128
     Registry: https://index.docker.io/v1/
     Labels:
     Experimental: true
     Insecure Registries:
      127.0.0.0/8
     Registry Mirrors:
      https://mirror.gcr.io/
     Live Restore Enabled: false
    
  • OS:
    ProductName:	macOS
    ProductVersion:	11.4
    BuildVersion:	20F71
    

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 33 (20 by maintainers)

Commits related to this issue

Most upvoted comments

I assume that the description has a typo, the url has an extra w

/ # ping wwww.google.com
ping: bad address 'wwww.google.com'

Uff, you are right. I did it many times and I didn’t notice that this very instance had a typo. The result is the same.

what is the IP 192.168.65.2 ?

It is the DNS that is set by docker. However, it get rewritten on the host by iptables.

# iptables-save | grep 192.168.65
-A PREROUTING -d 192.168.65.2/32 -j DOCKER_OUTPUT
-A OUTPUT -d 192.168.65.2/32 -j DOCKER_OUTPUT
-A POSTROUTING -d 192.168.65.2/32 -j DOCKER_POSTROUTING
-A DOCKER_OUTPUT -d 192.168.65.2/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 127.0.0.11:33719
-A DOCKER_OUTPUT -d 192.168.65.2/32 -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.11:49489
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p tcp -m tcp --sport 33719 -j SNAT --to-source 192.168.65.2:53
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p udp -m udp --sport 49489 -j SNAT --to-source 192.168.65.2:53

can you also paste the resolv.conf of one of the kind nodes?

# head /etc/resolv.conf*
==> /etc/resolv.conf <==
nameserver 192.168.65.2
options ndots:0

==> /etc/resolv.conf.original <==
nameserver 127.0.0.11
options ndots:0

I’ve tried both it with the default image (without specifying the version) and using the current one (sha256:f4eaad3d61dfe1228b43407dc9db980cd0a2e254588d133762750e605f5f368a)

Regarding the version of Docker Desktop, I’ve tried with 3.5.0, 3.4.0, 3.2.2, and 3.0.3. The result is always the same.