coredns: Plugin [loop] not work with systemd-resolved running

When systemd-resolved is running, nameserver in /etc/resolved.conf default to 127.0.0.53; The plugin loop detects two DNS query, and finally coredns fails to start.

Environment:

Ubuntu 18.04.1
Kubernetes 1.11.2
CoreDNS 1.2.2

Error log:

docker1.node ➜  kubectl logs coredns-55f86bf584-7sbtj -n kube-system
.:53
2018/09/06 13:02:45 [INFO] CoreDNS-1.2.2
2018/09/06 13:02:45 [INFO] linux/amd64, go1.11, eb51e8b
CoreDNS-1.2.2
linux/amd64, go1.11, eb51e8b
2018/09/06 13:02:45 [INFO] plugin/reload: Running configuration MD5 = 86e5222d14b17c8b907970f002198e96
2018/09/06 13:02:45 [FATAL] plugin/loop: Seen "HINFO IN 2050421060481615995.5620656063561519376." more than twice, loop detected

Deploy with deploy.sh

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 2
Comments: 32 (15 by maintainers)

Links to this issue

CoreDNS CrashLoopBackOff on fresh install

Most upvoted comments

e.g.

kubectl -n kube-system edit configmap coredns

Then delete the line that says loop, and save the configuration. It can take several minutes for k8s to propagate the config change to the coredns pods.

+30

chrisohaver on Oct 23, 2018

this from syslog; looks like it’s not being passed as expected (maybe my expectations, set by https://kubernetes.io/docs/setup/minikube/#quickstart, are incorrect) Nov 19 16:10:53 ubuntu kubelet[16413]: F1119 16:10:53.060353 16413 server.go:145] unknown flag: --ResolverConfig

this gave me an idea to try this… ubuntu % sudo minikube start --vm-driver=none --extra-config=kubelet.resolv-conf=/var/run/systemd/resolve/resolv.conf

and that seems to have worked; coredns and kube-dns now much happier

thanks for the nudge…

bwillcox on Nov 19, 2018

I shared the solution that has worked for me here: https://stackoverflow.com/a/53414041/1005102

utkuozdemir on Nov 27, 2018

No there isn’t any effects. It only checks for loops.

On Sat, 20 Oct 2018, 11:51 ahalimkara, notifications@github.com wrote:

Removing loop plugin is worked for me, is there any side effect of removing loop from the coredns configuration?

If you doubt there is a loop, you may try removing the loop detection (remove loop from the coredns configuration), and then test DNS resolution from pods (i.e. test resolution to external domains from the command line of a pod running in the cluster).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/coredns/coredns/issues/2087#issuecomment-431569875, or mute the thread https://github.com/notifications/unsubscribe-auth/AAVkW22-5k2eaQOBRVoS9V7r6U82Eg3Oks5umwAUgaJpZM4Wc6xv .

miekg on Oct 20, 2018

Removing loop plugin is worked for me, is there any side effect of removing loop from the coredns configuration?

If you doubt there is a loop, you may try removing the loop detection (remove loop from the coredns configuration), and then test DNS resolution from pods (i.e. test resolution to external domains from the command line of a pod running in the cluster).

ahalimkara on Oct 20, 2018

This is working as intended. The loop plugin has detected a forwarding loop, caused by systemd-resolved. If CoreDNS didn’t exit, it would loop “forever” on the first upstream query it receives and get OOM killed.

The best fix is to add a flag to kubelet, to let it know that it should use the original resolv.conf… --resolv-conf=/run/systemd/resolve/resolv.conf, then restart coredns pods

chrisohaver on Sep 6, 2018