minikube: CoreDNS fails on minions on multi-node clusters. Can't resolve external DNS from non-master pods.

So, I already fixed this and lost some of the logs. But it’s pretty straight-forward.

Make a cluster

minikube start --vm-driver=kvm2 --cpus=2 --nodes 3 --network-plugin=cni \
--addons registry --enable-default-cni=false \
--insecure-registry "10.0.0.0/24" --insecure-registry "192.168.39.0/24" \
--extra-config=kubeadm.pod-network-cidr=10.244.0.0/16 \
--extra-config=kubelet.network-plugin=cni
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

n.b. I built from head a couple days ago

minikube version: v1.10.0-beta.2
commit: 80c3324b6f526911d46033721df844174fe7f597

make a pod on master and a pod on a node
from node pod: curl google.com
from master pod: curl google.com

CoreDNS was crashing per https://github.com/kubernetes/kubernetes/issues/75414

Fixed with

kubectl patch deployment coredns -n kube-system --patch '{"spec":{"template":{"spec":{"volumes":[{"name":"emptydir-tmp","emptyDir":{}}],"containers":[{"name":"coredns","volumeMounts":[{"name":"emptydir-tmp","mountPath":"/tmp"}]}]}}}}'

Edit: had wrong flannel yaml listed.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 23 (5 by maintainers)

Most upvoted comments

I also still see problems with multi node clusters and kvm2. This happens on first creation of the cluster but also on restarting of the cluster. Here u see the logs when I restart a 3 node cluster.

minikube start --cpus=6 --memory=8g --disk-size=18g --driver=kvm2 --kubernetes-version=latest --nodes=3
😄  minikube v1.16.0 on Ubuntu 20.04
✨  Using the kvm2 driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing kvm2 VM for "minikube" ...
🐳  Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
❗  The cluster minikube already exists which means the --nodes parameter will be ignored. Use "minikube node add" to add nodes to an existing cluster.
👍  Starting node minikube-m02 in cluster minikube
🔄  Restarting existing kvm2 VM for "minikube-m02" ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.39.52
🐳  Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
    ▪ env NO_PROXY=192.168.39.52
🔎  Verifying Kubernetes components...
👍  Starting node minikube-m03 in cluster minikube
🔄  Restarting existing kvm2 VM for "minikube-m03" ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.39.52,192.168.39.217
🐳  Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
    ▪ env NO_PROXY=192.168.39.52
    ▪ env NO_PROXY=192.168.39.52,192.168.39.217
🔎  Verifying Kubernetes components...
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

CoreDNS pod is running but the problem seems that its started too early. Logs of CoreDNS pod.

E0117 11:59:13.235020       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: no route to host
E0117 11:59:13.235021       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Get "https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: no route to host
E0117 11:59:13.235026       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: no route to host
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d

After restarting the CoreDNS pod, there are no more erros visible in the logs and DNS starts working.

.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d

@tstromberg can we reopen this issue or create a new one for it?

leozilla on Jan 17, 2021

Okay, I think I’ve figured something out. I’m going to open a new ticket. This is all based on problems in the iptables. I’ll add a link to the new ticket when I get it put together.

applitect on Feb 25, 2021

This issue seems to be closed by mistake.

If I understood correctly @tstromberg wrote “Does not fix #…” in his PR and issue got closed automatically w/o taking “Does not” part into consideration 😃

btw I can confirm that the issue persists on latest MacOS and minikube v1.17.1 (latest), when I run it like this: minikube start --nodes 2 --vm-driver=hyperkit

DNS resolves fine inside minikube nodes, but containers fail to resolve.

YektaLeblebici on Feb 14, 2021

After testing, I can confirm that resolution of Kubernetes hosts from non-master pods is broken. I was not able to replicate issues with DNS resolution, however.

In a nutshell, I believe that the issue of CoreDNS access from non-master nodes is a sign of a broken CNI configuration. I’ll continue to investigate.

tstromberg on Jun 17, 2020

My tests were based on https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution.

My env Ubuntu 19.10 Minikube v1.11.0 Multi-node KVM2

Scenario 1

minikube start -p dns --cpus=2 --memory=2g --nodes=2 --driver=kvm2 --extra-config=kubelet.resolv-conf=/run/systemd/resolve/resolv.conf

CoreDNS started normally. Just an event.
- Event: Warning FailedScheduling 36m default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn’t tolerate.
DNS did not work internally in a pod.
- kubectl exec -ti dnsutils – nslookup kubernetes.default
delete DNS pods, k8s recreates DNS PODS. Then DNS works normally.

Scenario 2

minikube start -p dns --cpus=2 --memory=2g --nodes=2 --driver=kvm2 --enable-default-cni=false --network-plugin=cni

CoreDNS started normally. Just an event.
- Event: Event: Warning FailedScheduling 36m default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn’t tolerate.
DNS did not work internally in a pod.
- kubectl exec -ti dnsutils – nslookup kubernetes.default
delete DNS pods, k8s recreates DNS PODS. Then DNS works normally in pods.

Conclusion:

Inicially the DNS pods were hosted on master node. And DNS was not working in PODs.
in both situation I had to delete de DNS pods. Then DNS pods were spread to the nodes. DNS worked in PODS
Forced DNS Pods run only in master node. It worked normally.
It’s somehow cluster’s startup related.

turnes on Jun 9, 2020

I checked connectivity in the pods via launching a pod on each node and trying to connect to each other with nc.

workers work. master connectivity is not.

I deleted the coredns pods and they restarted on the non master nodes. and dns started working.

So something is not working with kindnet on the master.

kfox1111 on May 16, 2020