k3s: DNS resolution fails with dnsPolicy: ClusterFirstWithHostNet and hostNetwork: true

Version:

k3s version v1.17.4+k3s1 (3eee8ac3) ubuntu 20.04

K3s arguments:

–no-deploy traefik --no-deploy=servicelb --kubelet-arg containerd=/run/k3s/containerd/containerd.sock

Describe the bug

The dns resolution does not work for my container which is running using these settings:

      dnsPolicy: ClusterFirstWithHostNet
      hostNetwork: true

The dns resolution works just fine if I do not use hostNetwork and don’t change the dns policy.

The core dns service looks fine:

kubectl describe service kube-dns -n kube-system                                                                                                                     ±[master]
Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=CoreDNS
                   objectset.rio.cattle.io/hash=bce283298811743a0386ab510f2f67ef74240c57
Annotations:       objectset.rio.cattle.io/applied:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"objectset.rio.cattle.io/id":"","objectset.rio.cattle.io/owner-gvk":"k3s.ca...
                   objectset.rio.cattle.io/id: 
                   objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
                   objectset.rio.cattle.io/owner-name: coredns
                   objectset.rio.cattle.io/owner-namespace: kube-system
                   prometheus.io/port: 9153
                   prometheus.io/scrape: true
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP:                10.43.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         10.42.0.141:53,10.42.1.117:53,10.42.3.202:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         10.42.0.141:53,10.42.1.117:53,10.42.3.202:53
Port:              metrics  9153/TCP
TargetPort:        9153/TCP
Endpoints:         10.42.0.141:9153,10.42.1.117:9153,10.42.3.202:9153
Session Affinity:  None
Events:            <none>

As you can see I can sucessfuly query the single instances of coredns but the cluster ip access fails:

bash-5.0# dig @10.42.1.117 www.heise.de

; <<>> DiG 9.14.8 <<>> @10.42.1.117 www.heise.de
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59239
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.heise.de.                  IN      A

;; ANSWER SECTION:
www.heise.de.           30      IN      A       193.99.144.85

;; Query time: 23 msec
;; SERVER: 10.42.1.117#53(10.42.1.117)
;; WHEN: Mon May 25 21:43:20 UTC 2020
;; MSG SIZE  rcvd: 69

bash-5.0# dig @10.42.3.202 www.heise.de

; <<>> DiG 9.14.8 <<>> @10.42.3.202 www.heise.de
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36239
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.heise.de.                  IN      A

;; ANSWER SECTION:
www.heise.de.           30      IN      A       193.99.144.85

;; Query time: 14 msec
;; SERVER: 10.42.3.202#53(10.42.3.202)
;; WHEN: Mon May 25 21:43:31 UTC 2020
;; MSG SIZE  rcvd: 69

bash-5.0# dig @10.43.0.10 www.heise.de

; <<>> DiG 9.14.8 <<>> @10.43.0.10 www.heise.de
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

To Reproduce

Run a pod with host network and dns policy ClusterFirstWithHostNet.

Expected behavior

DNS resolution should work fine

Actual behavior

DNS resolution does not work at all

Additional context / logs

DNS resolution works fine with container network:

~ $ nslookup  www.heise.de
nslookup: can't resolve '(null)': Name does not resolve

Name:      www.heise.de
Address 1: 193.99.144.85 www.heise.de
Address 2: 2a02:2e0:3fe:1001:7777:772e:2:85 www.heise.de

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 5
  • Comments: 16 (4 by maintainers)

Commits related to this issue

Most upvoted comments

I have three amd64 nodes who suffer from this issue…

+1, though this seems to happen exclusively on my arm64 nodes.

I’m running image: homeassistant/home-assistant:0.110.2 which requires hostNetwork: true and dnsPolicy: clusterFirstWithHostNet for discovery of local network smart-home devices.

To clarify, I’ve had this problem for a while now, but I face no such issue on my armv7l devices, it only happens to arm64 devices.

Node 1 information

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 10 (buster)
Release:	10
Codename:	buster
$ uname -r
5.4.32-rockchip64
$ sudo iptables -V
iptables v1.8.2 (legacy)
$ lscpu
Architecture:        aarch64
Byte Order:          Little Endian
CPU(s):              6
On-line CPU(s) list: 0-5
Thread(s) per core:  1
Core(s) per socket:  3
Socket(s):           2
NUMA node(s):        1
Vendor ID:           ARM
Model:               4
Model name:          Cortex-A53
Stepping:            r0p4
CPU max MHz:         2016.0000
CPU min MHz:         408.0000
BogoMIPS:            48.00
NUMA node0 CPU(s):   0-5
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid

Node 2 Information

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
Codename:	bionic
$ uname -r
5.4.28-rockchip64
$ sudo iptables -V
iptables v1.6.1
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
Codename:	bionic
pi@l2:~$ lscpu
Architecture:        aarch64
Byte Order:          Little Endian
CPU(s):              6
On-line CPU(s) list: 0-5
Thread(s) per core:  1
Core(s) per socket:  3
Socket(s):           2
NUMA node(s):        1
Vendor ID:           ARM
Model:               4
Model name:          Cortex-A53
Stepping:            r0p4
CPU max MHz:         2016.0000
CPU min MHz:         408.0000
BogoMIPS:            48.00
NUMA node0 CPU(s):   0-5
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid