cilium: DNS lookup fails in IPv6 with DNS policy

Bug report

I’ve been asked here to share my findings as an issue.

In short - any DNS L7 rule prevents DNS responces from being forwarded to the client in our installation.

cilium status --all-redirects (sed-ed) on the node with both coredns and the client pod.

KVStore:                Ok   Disabled
Kubernetes:             Ok   1.20 (v1.20.2) [linux/amd64]
Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:   Strict   [bond0 (Direct Routing)]
Cilium:                 Ok       OK
NodeMonitor:            Listening for events on 64 CPUs with 64x4096 of shared memory
Cilium health daemon:   Ok
IPAM:                   , IPv6: 5/281474976710655 allocated from xx:yy:zz:1150:2::/80
BandwidthManager:       Disabled
Host Routing:           BPF
Masquerading:           BPF   [bond0]
Controller Status:      29/29 healthy
Proxy Status:           OK, ip <nil>, 2 redirects active on ports 10000-20000
  Protocol            Redirect             Proxy Port
  cilium-dns-egress   1736:egress:TCP:53   39479
  cilium-dns-egress   1736:egress:UDP:53   39479
Hubble:           Ok              Current/Max Flows: 4096/4096 (100.00%), Flows/s: 4.04   Metrics: Disabled
Cluster health:   3/3 reachable   (2021-02-12T16:55:48Z)

Cilium helm chart values (version 1.9.[34] - same results for both versions):

USER-SUPPLIED VALUES:
autoDirectNodeRoutes: true
cleanBpfState: true
cleanState: true
containerRuntime:
  integration: crio
debug:
  enabled: true
enableIPv6Masquerade: false
extraArgs:
  tofqdns-dns-reject-response-code: nameError
hubble:
  listenAddress: :4244
  relay:
    enabled: true
  ui:
    enabled: true
ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv6MaskSize: 80
    clusterPoolIPv6PodCIDR: xx:yy:zz:1150::/64
ipv4:
  enabled: false
ipv6:
  enabled: true
k8sServiceHost: xx:yy:zz:110a:ef02:1:0:2
k8sServicePort: 6443
kubeProxyReplacement: strict
nativeRoutingCIDR: xx:yy:zz:1150::/64
tunnel: disabled

CNP - a bit modified relevant part of a rule from connectivity-check:

  egress:
  - toFQDNs:
    - matchPattern: '*.google.com'
  - toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: kube-system
        k8s:k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: '*'

Dump on client (tcpdump -i eth0 while inside netns of the pod):

16:24:46.901310 IP6 xx:yy:zz:1150:2::e7b4.53604 > xx:yy:zz:1150:2::f2ec.53: 20996+ AAAA? www.google.com.cilitest.svc.cluster.local. (59)
16:24:46.906696 IP6 xx:yy:zz:1150:2::f2ec.53 > xx:yy:zz:1150:2::e7b4.53604: 20996 NXDomain*- 0/1/0 (152)
16:24:51.906609 IP6 xx:yy:zz:1150:2::e7b4.53604 > xx:yy:zz:1150:2::f2ec.53: 20996+ AAAA? www.google.com.cilitest.svc.cluster.local. (59)
16:24:51.911786 IP6 xx:yy:zz:1150:2::f2ec.53 > xx:yy:zz:1150:2::e7b4.53604: 20996 NXDomain*- 0/1/0 (152)
16:24:56.912023 IP6 xx:yy:zz:1150:2::e7b4.38837 > xx:yy:zz:1150:2::f2ec.53: 4304+ AAAA? www.google.com. (32)
16:24:56.919237 IP6 xx:yy:zz:1150:2::f2ec.53 > xx:yy:zz:1150:2::e7b4.38837: 4304 4/0/0 AAAA 2a00:1450:4010:c05::68, AAAA 2a00:1450:4010:c05::69, AAAA 2a00:1450:4010:c05::63, AAAA 2a00:1450:4010:c05::67 (200)
16:25:01.917370 IP6 xx:yy:zz:1150:2::e7b4.38837 > xx:yy:zz:1150:2::f2ec.53: 4304+ AAAA? www.google.com. (32)
16:25:01.923853 IP6 xx:yy:zz:1150:2::f2ec.53 > xx:yy:zz:1150:2::e7b4.38837: 4304 4/0/0 AAAA 2a00:1450:4010:c0b::68, AAAA 2a00:1450:4010:c0b::67, AAAA 2a00:1450:4010:c0b::63, AAAA 2a00:1450:4010:c0b::69 (200)
16:25:06.922958 IP6 xx:yy:zz:1150:2::e7b4.42367 > xx:yy:zz:1150:2::f2ec.53: 11883+ A? www.google.com.cilitest.svc.cluster.local. (59)
16:25:06.928068 IP6 xx:yy:zz:1150:2::f2ec.53 > xx:yy:zz:1150:2::e7b4.42367: 11883 NXDomain*- 0/1/0 (152)
16:25:11.928231 IP6 xx:yy:zz:1150:2::e7b4.42367 > xx:yy:zz:1150:2::f2ec.53: 11883+ A? www.google.com.cilitest.svc.cluster.local. (59)
16:25:11.933339 IP6 xx:yy:zz:1150:2::f2ec.53 > xx:yy:zz:1150:2::e7b4.42367: 11883 NXDomain*- 0/1/0 (152)
16:25:16.933637 IP6 xx:yy:zz:1150:2::e7b4.45658 > xx:yy:zz:1150:2::f2ec.53: 9963+ A? www.google.com. (32)
16:25:16.940448 IP6 xx:yy:zz:1150:2::f2ec.53 > xx:yy:zz:1150:2::e7b4.45658: 9963 6/0/0 A 173.194.222.103, A 173.194.222.106, A 173.194.222.105, A 173.194.222.99, A 173.194.222.147, A 173.194.222.104 (212)
16:25:21.937194 IP6 xx:yy:zz:1150:2::e7b4.45658 > xx:yy:zz:1150:2::f2ec.53: 9963+ A? www.google.com. (32)

cilium monitor --related-to 1736 (sed-ed)

Policy verdict log: flow 0x653b1027 local EP ID 1736, remote ID 15030, proto 17, egress, action redirect, match L3-L4, [xx:yy:zz:1150:2::e7b4]:52236 -> [xx:yy:zz:1150:2::f2ec]:53 udp
-> proxy flow 0x653b1027 identity 41733->0 state new ifindex 0 orig-ip 0.0.0.0: [xx:yy:zz:1150:2::e7b4]:52236 -> [xx:yy:zz:1150:2::f2ec]:53 udp
-> Request dns from 1736 ([k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.cilium.k8s.policy.cluster=default]) to 0 ([]), identity 41733->0, verdict Forwarded DNS Query: www.google.com.cilitest.svc.cluster.local. AAAA
-> Response dns to 1736 ([k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.cilium.k8s.policy.cluster=default]) from 0 ([]), identity 41733->0, verdict Forwarded DNS Proxy: www.google.com.cilitest.svc.cluster.local. AAAA TTL: 4294967295 Answer: ''
-> endpoint 1736 flow 0x7d68e065 identity 15030->41733 state reply ifindex lxc6aff95651e57 orig-ip xx:yy:zz:1150:2::f2ec: [xx:yy:zz:1150:2::f2ec]:53 -> [xx:yy:zz:1150:2::e7b4]:52236 udp
-> proxy flow 0x653b1027 identity 41733->0 state established ifindex 0 orig-ip 0.0.0.0: [xx:yy:zz:1150:2::e7b4]:52236 -> [xx:yy:zz:1150:2::f2ec]:53 udp
-> Request dns from 1736 ([k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.cilium.k8s.policy.cluster=default]) to 0 ([]), identity 41733->0, verdict Forwarded DNS Query: www.google.com.cilitest.svc.cluster.local. AAAA
-> Response dns to 1736 ([k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.cilium.k8s.policy.cluster=default]) from 0 ([]), identity 41733->0, verdict Forwarded DNS Proxy: www.google.com.cilitest.svc.cluster.local. AAAA TTL: 4294967295 Answer: ''
-> endpoint 1736 flow 0x7d68e065 identity 15030->41733 state reply ifindex lxc6aff95651e57 orig-ip xx:yy:zz:1150:2::f2ec: [xx:yy:zz:1150:2::f2ec]:53 -> [xx:yy:zz:1150:2::e7b4]:52236 udp
Policy verdict log: flow 0x6ff17b88 local EP ID 1736, remote ID 15030, proto 17, egress, action redirect, match L3-L4, [xx:yy:zz:1150:2::e7b4]:34812 -> [xx:yy:zz:1150:2::f2ec]:53 udp
-> proxy flow 0x6ff17b88 identity 41733->0 state new ifindex 0 orig-ip 0.0.0.0: [xx:yy:zz:1150:2::e7b4]:34812 -> [xx:yy:zz:1150:2::f2ec]:53 udp
-> Request dns from 1736 ([k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.cilium.k8s.policy.cluster=default]) to 0 ([]), identity 41733->0, verdict Forwarded DNS Query: www.google.com. AAAA
-> Response dns to 1736 ([k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.cilium.k8s.policy.cluster=default k8s:name=pod-to-external-fqdn-allow-google-cnp]) from 0 ([]), identity 41733->0, verdict Forwarded DNS Proxy: www.google.com. AAAA TTL: 30 Answer: '2a00:1450:4010:c0b::68,2a00:1450:4010:c0b::69,2a00:1450:4010:c0b::63,2a00:1450:4010:c0b::93'
-> endpoint 1736 flow 0x7d68e065 identity 15030->41733 state reply ifindex lxc6aff95651e57 orig-ip xx:yy:zz:1150:2::f2ec: [xx:yy:zz:1150:2::f2ec]:53 -> [xx:yy:zz:1150:2::e7b4]:34812 udp
-> proxy flow 0x6ff17b88 identity 41733->0 state established ifindex 0 orig-ip 0.0.0.0: [xx:yy:zz:1150:2::e7b4]:34812 -> [xx:yy:zz:1150:2::f2ec]:53 udp
-> Request dns from 1736 ([k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.cilium.k8s.policy.cluster=default]) to 0 ([]), identity 41733->0, verdict Forwarded DNS Query: www.google.com. AAAA
-> Response dns to 1736 ([k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.cilium.k8s.policy.cluster=default]) from 0 ([]), identity 41733->0, verdict Forwarded DNS Proxy: www.google.com. AAAA TTL: 25 Answer: '2a00:1450:4010:c0b::93,2a00:1450:4010:c0b::68,2a00:1450:4010:c0b::63,2a00:1450:4010:c0b::69'
-> endpoint 1736 flow 0x7d68e065 identity 15030->41733 state reply ifindex lxc6aff95651e57 orig-ip xx:yy:zz:1150:2::f2ec: [xx:yy:zz:1150:2::f2ec]:53 -> [xx:yy:zz:1150:2::e7b4]:34812 udp
Policy verdict log: flow 0x92104851 local EP ID 1736, remote ID 15030, proto 17, egress, action redirect, match L3-L4, [xx:yy:zz:1150:2::e7b4]:40971 -> [xx:yy:zz:1150:2::f2ec]:53 udp
-> proxy flow 0x92104851 identity 41733->0 state new ifindex 0 orig-ip 0.0.0.0: [xx:yy:zz:1150:2::e7b4]:40971 -> [xx:yy:zz:1150:2::f2ec]:53 udp
-> Request dns from 1736 ([k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.cilium.k8s.policy.cluster=default]) to 0 ([]), identity 41733->0, verdict Forwarded DNS Query: www.google.com.cilitest.svc.cluster.local. A
-> Response dns to 1736 ([k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.cilium.k8s.policy.cluster=default k8s:name=pod-to-external-fqdn-allow-google-cnp]) from 0 ([]), identity 41733->0, verdict Forwarded DNS Proxy: www.google.com.cilitest.svc.cluster.local. A TTL: 4294967295 Answer: ''
-> endpoint 1736 flow 0x7d68e065 identity 15030->41733 state reply ifindex lxc6aff95651e57 orig-ip xx:yy:zz:1150:2::f2ec: [xx:yy:zz:1150:2::f2ec]:53 -> [xx:yy:zz:1150:2::e7b4]:40971 udp
-> proxy flow 0x92104851 identity 41733->0 state established ifindex 0 orig-ip 0.0.0.0: [xx:yy:zz:1150:2::e7b4]:40971 -> [xx:yy:zz:1150:2::f2ec]:53 udp
-> Request dns from 1736 ([k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.cilium.k8s.policy.cluster=default]) to 0 ([]), identity 41733->0, verdict Forwarded DNS Query: www.google.com.cilitest.svc.cluster.local. A
-> Response dns to 1736 ([k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.cilium.k8s.policy.cluster=default]) from 0 ([]), identity 41733->0, verdict Forwarded DNS Proxy: www.google.com.cilitest.svc.cluster.local. A TTL: 4294967295 Answer: ''
-> endpoint 1736 flow 0x7d68e065 identity 15030->41733 state reply ifindex lxc6aff95651e57 orig-ip xx:yy:zz:1150:2::f2ec: [xx:yy:zz:1150:2::f2ec]:53 -> [xx:yy:zz:1150:2::e7b4]:40971 udp
Policy verdict log: flow 0xede38533 local EP ID 1736, remote ID 15030, proto 17, egress, action redirect, match L3-L4, [xx:yy:zz:1150:2::e7b4]:60988 -> [xx:yy:zz:1150:2::f2ec]:53 udp
-> proxy flow 0xede38533 identity 41733->0 state new ifindex 0 orig-ip 0.0.0.0: [xx:yy:zz:1150:2::e7b4]:60988 -> [xx:yy:zz:1150:2::f2ec]:53 udp
-> Request dns from 1736 ([k8s:io.cilium.k8s.policy.cluster=default k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default]) to 0 ([]), identity 41733->0, verdict Forwarded DNS Query: www.google.com. A
-> Response dns to 1736 ([k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.cilium.k8s.policy.cluster=default]) from 0 ([]), identity 41733->0, verdict Forwarded DNS Proxy: www.google.com. A TTL: 30 Answer: '64.233.162.147,64.233.162.106,64.233.162.99,64.233.162.103,64.233.162.105,64.233.162.104'
-> endpoint 1736 flow 0x7d68e065 identity 15030->41733 state reply ifindex lxc6aff95651e57 orig-ip xx:yy:zz:1150:2::f2ec: [xx:yy:zz:1150:2::f2ec]:53 -> [xx:yy:zz:1150:2::e7b4]:60988 udp
-> proxy flow 0xede38533 identity 41733->0 state established ifindex 0 orig-ip 0.0.0.0: [xx:yy:zz:1150:2::e7b4]:60988 -> [xx:yy:zz:1150:2::f2ec]:53 udp
-> Request dns from 1736 ([k8s:io.cilium.k8s.policy.cluster=default k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default]) to 0 ([]), identity 41733->0, verdict Forwarded DNS Query: www.google.com. A
-> Response dns to 1736 ([k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.kubernetes.pod.namespace=cilitest k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.cilium.k8s.policy.cluster=default]) from 0 ([]), identity 41733->0, verdict Forwarded DNS Proxy: www.google.com. A TTL: 25 Answer: '64.233.162.106,64.233.162.147,64.233.162.105,64.233.162.99,64.233.162.104,64.233.162.103'
-> endpoint 1736 flow 0x7d68e065 identity 15030->41733 state reply ifindex lxc6aff95651e57 orig-ip xx:yy:zz:1150:2::f2ec: [xx:yy:zz:1150:2::f2ec]:53 -> [xx:yy:zz:1150:2::e7b4]:60988 udp

Lookup result (empty):

root@pod-to-external-fqdn-allow-google-cnp-my-6ccb45d95b-qb85r:/# getent hosts www.google.com
root@pod-to-external-fqdn-allow-google-cnp-my-6ccb45d95b-qb85r:/#

(since neither nslookup, dig or host gives any useful info I stick with getent because it’s always present regardless of container image I’m using)

cilium policy trace says traffic is allowed.

But when you remove rules:dns:matchPattern: '*' from cnp everything works as intended. Dump:

16:34:33.992016 IP6 xx:yy:zz:1150:2::e7b4.44920 > xx:yy:zz:1150:2::f2ec.53: 21047+ AAAA? www.google.com.cilitest.svc.cluster.local. (59)
16:34:33.992434 IP6 xx:yy:zz:1150:2::f2ec.53 > xx:yy:zz:1150:2::e7b4.44920: 21047 NXDomain*- 0/1/0 (152)
16:34:33.992909 IP6 xx:yy:zz:1150:2::e7b4.32888 > xx:yy:zz:1150:2::f2ec.53: 22612+ AAAA? www.google.com.svc.cluster.local. (50)
16:34:33.993419 IP6 xx:yy:zz:1150:2::f2ec.53 > xx:yy:zz:1150:2::e7b4.32888: 22612 NXDomain*- 0/1/0 (143)
16:34:33.993668 IP6 xx:yy:zz:1150:2::e7b4.46599 > xx:yy:zz:1150:2::f2ec.53: 48221+ AAAA? www.google.com.cluster.local. (46)
16:34:33.994022 IP6 xx:yy:zz:1150:2::f2ec.53 > xx:yy:zz:1150:2::e7b4.46599: 48221 NXDomain*- 0/1/0 (139)
16:34:33.994221 IP6 xx:yy:zz:1150:2::e7b4.44804 > xx:yy:zz:1150:2::f2ec.53: 21621+ AAAA? www.google.com.<companydomain>. (50)
16:34:33.998764 IP6 xx:yy:zz:1150:2::f2ec.53 > xx:yy:zz:1150:2::e7b4.44804: 21621 NXDomain 0/1/0 (127)
16:34:33.999114 IP6 xx:yy:zz:1150:2::e7b4.47525 > xx:yy:zz:1150:2::f2ec.53: 23233+ AAAA? www.google.com. (32)

Lookup result:

root@pod-to-external-fqdn-allow-google-cnp-my-6ccb45d95b-qb85r:/# getent hosts www.google.com
2a00:1450:4010:c05::63 www.google.com
2a00:1450:4010:c05::68 www.google.com
2a00:1450:4010:c05::67 www.google.com
2a00:1450:4010:c05::6a www.google.com

General Information Tested on 2 versions of alpine and latest stable debian, so it’s not an image issue. At first we thought It was related to known musl resolver issues, but neither fix (–tofqdns-dns-reject-response-code=nameError) nor replacing alpine with debian-based curl image worked for us. Tried latest k8s v1.19.x and v1.20.1 to the same result. Requester pod and coredns pod (replicaset scaled to 1) are placed on the same node, so it’s not a bird/ospf related issue. Everything else works fine.

• Cilium version (run cilium version)

Client: 1.9.4 07b62884c 2021-02-03T11:45:44-08:00 go version go1.15.7 linux/amd64
Daemon: 1.9.4 07b62884c 2021-02-03T11:45:44-08:00 go version go1.15.7 linux/amd64

• Kernel version (run uname -a)

5.10.12-hardened1

• Orchestration system version in use (e.g. kubectl version, Mesos, …)

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-14T05:15:04Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

• Link to relevant artifacts (policies, deployments scripts, …) see above
• Generate and upload a system zip:

curl -sLO https://git.io/cilium-sysdump-latest.zip && python cilium-sysdump-latest.zip

https://drive.google.com/file/d/1gVPjgAz5GFHB6IKmCVP8ASwbdNcrRFzC/view?usp=sharing (rearchived with tar.bz2 after sed)

• Kernel config https://pastebin.com/raw/y0v7qsSE

How to reproduce the issue

See above