kubernetes: DNS is broken in Ubuntu 18.04 due to CNI hostport rules

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened: In Ubuntu 18.04 DNS queries are being served via systemd-resolved daemon which is listening on 127.0.0.53:53 and serves queries only from 127.0.0.1/8. The only line in /etc/resolv.conf is nameserver 127.0.0.53, and this file is managed by systemd. Once any pod with hostPort notation appears on the host, kubelet fires the iptables rule in POSTROUTING chain of nat table:

-A POSTROUTING -s 127.0.0.0/8 -o lo -m comment --comment "SNAT for localhost access to hostports" -j MASQUERADE

See commit https://github.com/kubernetes/kubernetes/commit/aabdaa984f3cff92421bb87197e1767d51407333 and lines https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/network/hostport/hostport.go#L137-L138 for details.

Therefore all queries to 127.0.0.53:53 goes not from 127.0.0.0/8, but from interface with default route due to masquerading, and systemd-resolved rejects all of these requests with

systemd-resolved[21366]: Got packet on unexpected IP range, refusing.

How to reproduce it (as minimally and precisely as possible):

Just install Kubernetes with CNI on Ubuntu-18.04 (with kubeadm, kubespray, chef-kubernetes, whatever) and run any pod with hostPort. Voila! You just lost DNS resolving.

Environment:

Kubernetes version (use kubectl version): v1.11.0
Cloud provider or hardware configuration: Digital Ocean (but reproduceable on any configuration)
OS (e.g. from /etc/os-release): Ubuntu 18.04 LTS
Kernel (e.g. uname -a): Linux kube01 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 18:02:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Install tools: chef-kubernetes (but reproduceable with any install tool)
Others:
- CNI: 0.6.0
- CNI plugins: 0.7.1
- CNI daemon: weave

/sig network

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 28
Comments: 17 (4 by maintainers)

Most upvoted comments

@kostyrev Please if you have nothing useful to add, just move along.

The workaround that I have settled on, is to run the following command on your Ubuntu 18.04 system during your bootstrapping:

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

This ^^ solves the issue for me.

+12

KptnKMan on Dec 24, 2018

@freehan but it is still broken, why did you close issue?

Bregor on May 5, 2019

Bug still present in kubernetes 1.16.0

randy-deb on Sep 24, 2019

Adding an SNAT rule above the rule generated by kubelet fixes it (replace the 4 with whatever the slot is above the offending rule for you):

iptables -t nat -I POSTROUTING 4 -d 127.0.0.53 -o lo -m comment --comment "SNAT for systemd-resolved" -j SNAT --to-source 127.0.0.1

I’m going to look at upstreaming this fix.

erhudy on Feb 27, 2019

Hi, is anyone managed to figure a workaround for this?

KptnKMan on Nov 14, 2018

Update: After updating cri-o from 1.15 to 1.16 the problem vanished.

matihost on Jan 15, 2020

I don’t know if people still encounter this issue. For kops users, the solution I’ve found is to add this to cluster spec:

spec:
  kubelet:
    resolvConf: "/run/systemd/resolve/resolv.conf"

This will actually tell kubelets to use the systemd resolv.conf insted of the default one in /etc.

For the users not using kops, you can just set --resolv-conf="/run/systemd/resolve/resolv.conf" in the kublet options.

betinro on Dec 5, 2019

Fixed in #80591

riking on Sep 14, 2019

during your bootstrapping

Can you explain what that means? Thank you!

cjbottaro on Dec 25, 2018