kubeadm: Install on a system using `systemd-resolved` leads to broken DNS
What keywords did you search in kubeadm issues before filing this one?
systemd resolved dns
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Versions
kubeadm version (use kubeadm version): v1.6.3
Environment:
- Kubernetes version (use
kubectl version): v1.6.3 - Cloud provider or hardware configuration: bare metal
- OS (e.g. from /etc/os-release): Ubuntu 17.04
- Kernel (e.g.
uname -a): Linux gjc-XPS-8500 4.10.0-21-generic #23-Ubuntu SMP Fri Apr 28 16:14:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux - Others:
What happened?
Installed kubernetes on bare metal using kubeadm. Dns inside pods did not work.
What you expected to happen?
Would expect dns inside pods to work.
How to reproduce it (as minimally and precisely as possible)?
Anything else we need to know?
As noted in kubernetes/kubernetes#45828, the problem is due to the fact that on a normal Ubuntu desktop (and maybe other desktop Linux OSes), /etc/resolve.conf contains 127.0.0.35, which doesn’t work inside Pods.
The correct thing to do is to add --resolv-conf=/run/systemd/resolve/resolv.conf to the kubelet config in case systemd-resolved is running with DNSStubListener and /etc/resolv.conf is configured with the local resolver (solution suggested by @antoineco and @thockin).
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 18 (10 by maintainers)
Commits related to this issue
- Merge pull request #63691 from detiber/warn_systemd-resolved Automatic merge from submit-queue (batch tested with PRs 63673, 63712, 63691, 63684). If you want to cherry-pick this change to another br... — committed to kubernetes/kubernetes by deleted user 6 years ago
- Fixed #1 Make sure kubeletes use `/run/systemd/resolve/resolv.conf` and not `/etc/resolv.conf` to make sure that any dnsmasq / resolved installed on the workers does not interfere with the clusters DN... — committed to asksven/kubernetes-the-hard-way-vagrant by asksven 6 years ago
- add workaround for k8s issue in ubuntu 18.04 https://github.com/kubernetes/kubeadm/issues/273 https://github.com/kelseyhightower/kubernetes-the-hard-way/issues/356#issuecomment-410470246 — committed to lukasmacko/vpp by lukasmacko 6 years ago
- Fix for systemd-resolved DNS incompatibility This problem occurs because systems using systemd-resolved copy 127.0.0.53 from the host's /etc/resolv.conf. More discussion here: https://github.com/kub... — committed to platform9/nodeadm by vannrt 6 years ago
- Fix for systemd-resolved DNS incompatibility This problem occurs because systems using systemd-resolved copy 127.0.0.53 from the host's /etc/resolv.conf. More discussion here: https://github.com/kub... — committed to platform9/nodeadm by vannrt 6 years ago
- Fix for systemd-resolved DNS incompatibility This problem occurs because kube-dns on systems using systemd-resolved copy 127.0.0.53 from the host's /etc/resolv.conf. Since 127.0.0.53 is a loopback a... — committed to platform9/nodeadm by vannrt 6 years ago
- Fix for systemd-resolved DNS incompatibility This problem occurs because kube-dns on systems using systemd-resolved copy 127.0.0.53 from the host's /etc/resolv.conf. Since 127.0.0.53 is a loopback a... — committed to platform9/nodeadm by vannrt 6 years ago
- kubelet resolv.conf set explicitly https://github.com/kubernetes/kubeadm/issues/273 — committed to wkandek/kwth-vbox by wkandek 4 years ago
I’ve hit the very same issue with kubeadm 1.10.0 and CoreDNS - with even worse results, as CoreDNS asked to resolve any external name starts looping to itself, consuming all allowed RAM and getting OOM-killed.
Obviously it can be fixed either by
kubelet --resolv-confparam (as mentioned above), or by editing config map withCorefile, but it takes a moment to realise what’s failing and why. It’s unfortunate that default setup fails so miserably.I’ve raised an issue in CoreDNS tracker for better handling of such a misconfiguration on CoreDNS side: https://github.com/coredns/coredns/issues/1647
As an FYI: as I commented on https://github.com/kubernetes/kubernetes/issues/45828, I don’t believe that over-riding kubelet’s resolv.conf reference will work anyway. This will just dump a broken (referencing 127.0.0.53) resolv.conf into all the pods and bypass cluster-local resolution. The current state of affairs is that just external resolution is broken because kube-dns has a broken upstream, but it is able to stub the cluster-local zones off to k8s. The only fix I can see is adding / editing config to kube-dns / CoreDNS.
NB
seems like a duplicate of https://github.com/kubernetes/kubeadm/issues/787 which is being worked on.
I’m hitting this when I try to use kubeadm with GCE’s ubuntu-1710 image so it looks like it’s not limited to the desktop install.