kubeadm: CoreDNS Crashloop Error - 1.12.1

@bravinash please, confirm that the issue is the same on your setup.

If it’s the same than you can try to point kubelet to the original resolv.conf this way:

$ echo -e '[Service]\nEnvironment="KUBELET_EXTRA_ARGS=--resolv-conf=/run/systemd/resolve/resolv.conf"\n' | sudo tee /etc/systemd/system/kubelet.service.d/99-local.conf
$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet

and remove coredns pods:

$ kubectl get pods -n kube-system -oname |grep coredns |xargs kubectl delete -n kube-system

Kubelet will start them again with new configuration.

At least this fixed the issue in my setup:

$ kubectl get pods -n kube-system
NAME                                READY   STATUS    RESTARTS   AGE
calico-node-cpsqz                   2/2     Running   0          5m20s
coredns-576cbf47c7-2prpv            1/1     Running   0          18s
coredns-576cbf47c7-xslbx            1/1     Running   0          18s
etcd-ed-ipv6-1                      1/1     Running   0          17m
kube-apiserver-ed-ipv6-1            1/1     Running   0          17m
kube-controller-manager-ed-ipv6-1   1/1     Running   0          17m
kube-proxy-8dkqt                    1/1     Running   0          17m
kube-scheduler-ed-ipv6-1            1/1     Running   0          17m

I shared the solution that has worked for me here: https://stackoverflow.com/a/53414041/1005102

Try 1.4.0 to see if that behaves any differently. Could be related to the glog/klog issue we fixed in 1.4.0.

@utkuozdemir

ps auxww | grep kubelet you might see a line like: /usr/bin/kubelet … --resolv-conf=/run/systemd/resolve/resolv.conf

alternatively: cat /var/lib/kubelet/kubeadm-flags.env kubeadm writes the --resolv-conf flag in that file each time it runs join and init.

I will close this issue for now as this is not reproducible anymore.

I’d propose to close this issue for 2 reasons:

• it’s more coredns than Kubeadm issue
• it’s not reproducible anymore

@bravinash you can reopen this issue if you see it again

On my system original resolv.conf also triggered the same issue. It started to work after copying it to another file, removing one of 3 nameserver lines from it and changing --resolv-conf option in /var/lib/kubelet/kubeadm-flags.env

Anyway, we need confirmation from @bravinash that the issue is the same. It doesn’t look like Kubeadm issue for me.