cilium: Inconsistent behavior of policy matching link-local address of node-local-dns

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

I tried to modify the pod-to-external-fqdn-allow-google-cnp policy from connectivity-check tests to allow connections to node-local-dns bound to link-local address 169.254.20.10:53 running in the host network namespace.

What I found out is that whether my policy works or not depends on the order in which the pods start:

  • if the node-local-dns pod starts before the cilium pod, the following policy allowing CIDR 169.254.0.0/16 works,
  • if the node-local-dns starts after the cilium pod, the following policy does not work. After I restart the cilium pod on the node, it starts working.

Policy:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: pod-to-external-fqdn-allow-google-cnp
  labels:
    name: pod-to-external-fqdn-allow-google-cnp
    topology: any
    component: policy-check
    traffic: external
    quarantine: "false"
    type: autocheck
spec:
  endpointSelector:
    matchLabels:
      name: pod-to-external-fqdn-allow-google-cnp
  egress:
  - toFQDNs:
    - matchPattern: '*.google.com'
  - toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: '*'
    toCIDR:
    - 169.254.0.0/16
    - fd00::/8

Steps to reproduce:

  • Deploy a cluster with CIlium CNI, without node-local DNS
  • AFTER Cilium pods are running, add node-local-dns daemonset that runs in the host network namesapce and binds to 169.254.20.10:53
  • In this case, the above mentioned policy does not allow connection to the node-local-dns:
# hubble observe --pod cilium-test/pod-to-external-fqdn-allow-google-cnp-5ff4986c89-pm5zj
Jan 27 15:50:12.716: cilium-test/pod-to-external-fqdn-allow-google-cnp-5ff4986c89-pm5zj:38346 <> 169.254.20.10:53 Policy denied DROPPED (UDP)
Jan 27 15:50:12.792: cilium-test/pod-to-external-fqdn-allow-google-cnp-5ff4986c89-pm5zj:39429 <> 169.254.20.10:53 Policy denied DROPPED (UDP)


# cilium monitor --related-to 401
Policy verdict log: flow 0x0 local EP ID 401, remote ID host, proto 17, egress, action deny, match none, 172.25.1.159:59035 -> 169.254.20.10:53 udp
xx drop (Policy denied) flow 0x0 to endpoint 0, , identity 12828->host: 172.25.1.159:59035 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x0 local EP ID 401, remote ID host, proto 17, egress, action deny, match none, 172.25.1.159:59035 -> 169.254.20.10:53 udp
xx drop (Policy denied) flow 0x0 to endpoint 0, , identity 12828->host: 172.25.1.159:59035 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x0 local EP ID 401, remote ID host, proto 17, egress, action deny, match none, 172.25.1.159:53883 -> 169.254.20.10:53 udp
xx drop (Policy denied) flow 0x0 to endpoint 0, , identity 12828->host: 172.25.1.159:53883 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x0 local EP ID 401, remote ID host, proto 17, egress, action deny, match none, 172.25.1.159:53883 -> 169.254.20.10:53 udp
xx drop (Policy denied) flow 0x0 to endpoint 0, , identity 12828->host: 172.25.1.159:53883 -> 169.254.20.10:53 udp


# ip addr
29: nodelocaldns: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default 
    link/ether 3a:d8:82:97:d8:2b brd ff:ff:ff:ff:ff:ff
    inet 169.254.20.10/32 brd 169.254.20.10 scope global nodelocaldns
       valid_lft forever preferred_lft forever

Then restart the Cilium pod - without any changes, the policy starts working:

# hubble observe --pod cilium-test/pod-to-external-fqdn-allow-google-cnp-5ff4986c89-pm5zj -f
Jan 27 15:54:30.366: cilium-test/pod-to-external-fqdn-allow-google-cnp-5ff4986c89-pm5zj:54119 -> 169.254.20.10:53 L3-L4 REDIRECTED (UDP)
Jan 27 15:54:30.366: cilium-test/pod-to-external-fqdn-allow-google-cnp-5ff4986c89-pm5zj:54119 -> 169.254.20.10:53 to-proxy FORWARDED (UDP)
Jan 27 15:54:30.367: cilium-test/pod-to-external-fqdn-allow-google-cnp-5ff4986c89-pm5zj:54119 -> 169.254.20.10:53 to-proxy FORWARDED (UDP)
Jan 27 15:54:30.367: cilium-test/pod-to-external-fqdn-allow-google-cnp-5ff4986c89-pm5zj:54119 -> 169.254.20.10:53 dns-request FORWARDED (DNS Query www.google.com.cilium-test.svc.cluster.local. A)
Jan 27 15:54:30.367: cilium-test/pod-to-external-fqdn-allow-google-cnp-5ff4986c89-pm5zj:54119 -> 169.254.20.10:53 dns-request FORWARDED (DNS Query www.google.com.cilium-test.svc.cluster.local. AAAA)
Jan 27 15:54:30.372: cilium-test/pod-to-external-fqdn-allow-google-cnp-5ff4986c89-pm5zj:54119 <- 169.254.20.10:53 dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy www.google.com.cilium-test.svc.cluster.local. AAAA))
Jan 27 15:54:30.372: cilium-test/pod-to-external-fqdn-allow-google-cnp-5ff4986c89-pm5zj:54119 <- 169.254.20.10:53 to-endpoint FORWARDED (UDP)
Jan 27 15:54:30.373: cilium-test/pod-to-external-fqdn-allow-google-cnp-5ff4986c89-pm5zj:54119 <- 169.254.20.10:53 dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy www.google.com.cilium-test.svc.cluster.local. A))

# cilium monitor --related-to 401
Policy verdict log: flow 0x0 local EP ID 401, remote ID 16777217, proto 17, egress, action redirect, match L3-L4, 172.25.1.159:54053 -> 169.254.20.10:53 udp
-> proxy port 37761 flow 0x0 , identity 12828->unknown state new ifindex 0 orig-ip 0.0.0.0: 172.25.1.159:54053 -> 169.254.20.10:53 udp
-> proxy port 37761 flow 0x0 , identity 12828->unknown state established ifindex 0 orig-ip 0.0.0.0: 172.25.1.159:54053 -> 169.254.20.10:53 udp
-> Request dns from 401 ([k8s:io.kubernetes.pod.namespace=cilium-test k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-test k8s:io.cilium.k8s.policy.cluster=default k8s:io.cilium.k8s.policy.serviceaccount=default]) to 0 ([reserved:world]), identity 12828->2, verdict Forwarded DNS Query: www.google.com.cilium-test.svc.cluster.local. AAAA
-> Request dns from 401 ([k8s:io.kubernetes.pod.namespace=cilium-test k8s:name=pod-to-external-fqdn-allow-google-cnp k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-test k8s:io.cilium.k8s.policy.cluster=default k8s:io.cilium.k8s.policy.serviceaccount=default]) to 0 ([reserved:world]), identity 12828->2, verdict Forwarded DNS Query: www.google.com.cilium-test.svc.cluster.local. A
-> Response dns to 401 ([k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-test k8s:io.cilium.k8s.policy.cluster=default k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.kubernetes.pod.namespace=cilium-test k8s:name=pod-to-external-fqdn-allow-google-cnp]) from 0 ([reserved:world]), identity 12828->2, verdict Forwarded DNS Proxy: www.google.com.cilium-test.svc.cluster.local. AAAA TTL: 4294967295 Answer: ''
-> endpoint 401 flow 0x0 , identity 16777217->12828 state reply ifindex lxcc7c4aaa47af9 orig-ip 169.254.20.10: 169.254.20.10:53 -> 172.25.1.159:54053 udp
-> Response dns to 401 ([k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-test k8s:io.cilium.k8s.policy.cluster=default k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.kubernetes.pod.namespace=cilium-test k8s:name=pod-to-external-fqdn-allow-google-cnp]) from 0 ([reserved:world]), identity 12828->2, verdict Forwarded DNS Proxy: www.google.com.cilium-test.svc.cluster.local. A TTL: 4294967295 Answer: ''

Cilium Version

cilium image (running): v1.11.0

Kernel Version

Linux 5.11.0-1027-aws #30~20.04.1-Ubuntu SMP Thu Jan 13 11:46:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

v1.21.8

Sysdump

cilium-sysdump-20220127-165230.zip

Relevant log output

No response

Anything else?

No response

Code of Conduct

  • I agree to follow this project’s Code of Conduct

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 20 (12 by maintainers)

Most upvoted comments

I think that it’s fair to say that CiliumLocalRedirectPolicy is the intended way to use node-local-dns with Cilium today, so that path is probably the most reliable. I don’t recall any known issues with that feature, but if the beta status is an issue then perhaps it’s time for the community to revisit that classification and consider whether to promote it to GA.

Another thought, given that the behaviour changes depending on the order of running the hostNetwork pod… I’m assuming that the reason for this is that if you start the hostNetwork pod first, then it creates the virtual device, attaches this IP address, and this is all done in the host netns. Then Cilium starts up, detects the IPs in the host netns (including from this pod), and consider each IP as “host”. If the ordering is the other way around, then the address isn’t yet assigned when Cilium runs. Cilium doesn’t find the address. Then presumably there is no trigger in Cilium to learn about the new address once the pod does eventually start up & assign the address.

So taking this thought further, another option beyond the above two would be to re-trigger the code that I linked above somehow. Perhaps either by subscribing to some Linux notification mechanism that would allow Cilium to learn about newly attached IP addresses, or alternatively perhaps by monitoring for hostNetwork pods deployed on the same node, and then triggering the logic from there just in case the hostNetwork pods do anything special. For the latter idea though, it could be a bit racy - there’s no way to know when the node-local-dns pod assigns the address.

+3
joestringer on Mar 8, 2022

@JBOClara You don’t need the hard-coded 169.254 address in node-local-dns.yaml if you are deploying with LRP. I would suggest following these exact instructions - https://docs.cilium.io/en/latest/gettingstarted/local-redirect-policy/#node-local-dns-cache.

But I’ve got errors too on a test cluster with this installation

I didn’t follow - did you follow the LRP guide, and run into errors? What are the errors?

Yes, I tested both. There is error with https://docs.cilium.io/en/latest/gettingstarted/local-redirect-policy/#node-local-dns-cache too.

For LRP deployment, you need to add k8s-app: node-local-dns to the connectivity test policy endpoint selector label. Here is the cilium-cli PR - https://github.com/cilium/cilium-cli/pull/995.

+2
aditighag on Jul 27, 2022

I encountered the same issue; where deploying node-local-dns with a bind on 169.254.20.10 on the hostNetwork would sometimes work, sometimes it wouldn’t. I worked around this by deploying node-local-dns with the CiliumLocalRedirectPolicy and without the bind on 169.254.20.10 on the hostNetwork, as per the Cilium docs, and adding a secondary CLRP to forward traffic destined for the legacy address of 169.254.20.10 to node-local-dns as well.

---
apiVersion: cilium.io/v2
kind: CiliumLocalRedirectPolicy
metadata:
  name: node-local-dns
  namespace: kube-system
spec:
  redirectFrontend:
    serviceMatcher:
      serviceName: kube-dns
      namespace: kube-system
  redirectBackend:
    localEndpointSelector:
      matchLabels:
        k8s-app: node-local-dns
    toPorts:
      - port: "53"
        name: dns
        protocol: UDP
      - port: "53"
        name: dns-tcp
        protocol: TCP
---
apiVersion: cilium.io/v2
kind: CiliumLocalRedirectPolicy
metadata:
  name: node-local-dns-legacy
  namespace: kube-system
spec:
  redirectFrontend:
    addressMatcher:
      ip: "169.254.20.10"
      toPorts:
        - port: "53"
          name: dns
          protocol: UDP
        - port: "53"
          name: dns-tcp
          protocol: TCP
  redirectBackend:
    localEndpointSelector:
      matchLabels:
        k8s-app: node-local-dns
    toPorts:
      - port: "53"
        name: dns
        protocol: UDP
      - port: "53"
        name: dns-tcp
        protocol: TCP
+1
mhulscher on Feb 16, 2022

Could this be related to https://github.com/cilium/cilium/issues/17839?

+1
aanm on Feb 14, 2022
Read more comments on GitHub
← cilium: In K8s dual-stack traffic from an external source to a loadBalancerIP does not work
cilium: Inconsistent behavior related to VRRP packets drops when host firewall is enabled in v1.11 vs v1.10 →