talos: Enabling `forwardKubeDNSToHost` results in `i/o timeout`

Bug Report

Follow up to https://github.com/siderolabs/talos/issues/8698

Description

I have tried enabling forwardKubeDNSToHost, and then restarting coredns however I am greeted with i/o timeout on DNS look ups. All nodes are handed out DNS via DHCP which is my routers IP address which has a DNS server.

Talos configuration (forwardKubeDNSToHost: true is not commited but applied): https://github.com/onedr0p/home-ops/tree/08b4e8f4807e46dfe48c27bf4dc738c9ee13dd29/kubernetes/main/bootstrap/talos/assets

Cilium configuration: https://github.com/onedr0p/home-ops/blob/08b4e8f4807e46dfe48c27bf4dc738c9ee13dd29/kubernetes/main/apps/kube-system/cilium/app/helm-values.yaml

Coredns configuration: https://github.com/onedr0p/home-ops/blob/08b4e8f4807e46dfe48c27bf4dc738c9ee13dd29/kubernetes/main/apps/kube-system/coredns/app/helm-values.yaml

Logs

❯ k get services -n kube-system host-dns
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
host-dns   ClusterIP   10.43.0.9    <none>        53/UDP,53/TCP   5m
coredns-579c8898d5-mnzf2 coredns [INFO] 10.42.5.166:37488 - 61834 "AAAA IN api.github.com. udp 32 false 512" - - 0 2.001392651s
coredns-579c8898d5-mnzf2 coredns [ERROR] plugin/errors: 2 api.github.com. AAAA: read udp 10.42.4.60:40192->10.43.0.9:53: i/o timeout
coredns-579c8898d5-mnzf2 coredns [INFO] 10.42.5.166:37488 - 61656 "A IN api.github.com. udp 32 false 512" - - 0 2.001732239s
coredns-579c8898d5-mnzf2 coredns [ERROR] plugin/errors: 2 api.github.com. A: read udp 10.42.4.60:43338->10.43.0.9:53: i/o timeout

Environment

  • Talos version: v1.7.4
  • Kubernetes version: v1.30.1
  • Platform: Metal
  • CNI: Cilium 1.15.5

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 6
  • Comments: 19 (8 by maintainers)

Commits related to this issue

Most upvoted comments

Is that the solution? Because I’d rather not disable those features.

Currently it is. We use a separate Service which says 10.96.0.9 (for example, where 10.96.0.0 is the Kubernetes service network) is a ClusterIP with a single endpoint 10.96.0.9:53. When the pod with CNI networking reaches out to 10.96.0.9:53, we expect the traffic to do no-op NAT and reach the host networking where host dns is located. On that guys from Cilium say that:

We’re fairly aggressive about rolling out BPF-Host routing, even though there are some known setups where it’s not compatible. So the traffic would exit the pod, and while doing so get noop-translated in BPF (based on the service definition). We would then fail to match the “backend IP” (still 10.96.0.9, because noop) against any known endpoint, and BPF Host-Routing then redirects it straight to the external network.

Last paragraph is reworded a bit.