talos: Enabling `forwardKubeDNSToHost` results in `i/o timeout`
Bug Report
Follow up to https://github.com/siderolabs/talos/issues/8698
Description
I have tried enabling forwardKubeDNSToHost, and then restarting coredns however I am greeted with i/o timeout on DNS look ups. All nodes are handed out DNS via DHCP which is my routers IP address which has a DNS server.
Talos configuration (forwardKubeDNSToHost: true is not commited but applied): https://github.com/onedr0p/home-ops/tree/08b4e8f4807e46dfe48c27bf4dc738c9ee13dd29/kubernetes/main/bootstrap/talos/assets
Cilium configuration: https://github.com/onedr0p/home-ops/blob/08b4e8f4807e46dfe48c27bf4dc738c9ee13dd29/kubernetes/main/apps/kube-system/cilium/app/helm-values.yaml
Coredns configuration: https://github.com/onedr0p/home-ops/blob/08b4e8f4807e46dfe48c27bf4dc738c9ee13dd29/kubernetes/main/apps/kube-system/coredns/app/helm-values.yaml
Logs
❯ k get services -n kube-system host-dns
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
host-dns ClusterIP 10.43.0.9 <none> 53/UDP,53/TCP 5m
coredns-579c8898d5-mnzf2 coredns [INFO] 10.42.5.166:37488 - 61834 "AAAA IN api.github.com. udp 32 false 512" - - 0 2.001392651s
coredns-579c8898d5-mnzf2 coredns [ERROR] plugin/errors: 2 api.github.com. AAAA: read udp 10.42.4.60:40192->10.43.0.9:53: i/o timeout
coredns-579c8898d5-mnzf2 coredns [INFO] 10.42.5.166:37488 - 61656 "A IN api.github.com. udp 32 false 512" - - 0 2.001732239s
coredns-579c8898d5-mnzf2 coredns [ERROR] plugin/errors: 2 api.github.com. A: read udp 10.42.4.60:43338->10.43.0.9:53: i/o timeout
Environment
- Talos version: v1.7.4
- Kubernetes version: v1.30.1
- Platform: Metal
- CNI: Cilium 1.15.5
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 6
- Comments: 19 (8 by maintainers)
Commits related to this issue
- fix(talos): Disable forwardKubeDNSToHost https://github.com/siderolabs/talos/issues/8836 — committed to jfroy/flatops by jfroy 2 years ago
- fix(talos): Disable forwardKubeDNSToHost https://github.com/siderolabs/talos/issues/8836 — committed to jfroy/flatops by jfroy 2 years ago
Currently it is. We use a separate
Servicewhich says10.96.0.9(for example, where10.96.0.0is the Kubernetes service network) is aClusterIPwith a single endpoint10.96.0.9:53. When the pod with CNI networking reaches out to10.96.0.9:53, we expect the traffic to do no-op NAT and reach the host networking where host dns is located. On that guys from Cilium say that:Last paragraph is reworded a bit.