talos: 1.7.1 with hostdns and forwardKubeDNSToHost doesn't resolve anything

Bug Report

Description

This is on a cluster that has been upgraded (1.6.x->1.7.x), not fresh

after applying the following patch:

machine:
  features:
    hostDNS:
      enabled: true
      forwardKubeDNSToHost: true

nothing seems to resolve dns, either in the cluster or externally

getupstream:

talosctl -n 192.168.5.10,192.168.5.11,192.168.5.12,192.168.5.15 get dnsupstream
NODE           NAMESPACE   TYPE          ID            VERSION   HEALTHY   ADDRESS
192.168.5.10   network     DNSUpstream   192.168.5.1   1         true      192.168.5.1:53
192.168.5.11   network     DNSUpstream   192.168.5.1   1         true      192.168.5.1:53
192.168.5.12   network     DNSUpstream   192.168.5.1   1         true      192.168.5.1:53
192.168.5.15   network     DNSUpstream   192.168.5.1   1         true      192.168.5.1:53

resolv.conf

 talosctl -n 192.168.5.10 read /system/resolved/resolv.conf
nameserver 10.96.0.9
talosctl -n 192.168.5.10 read /etc/resolv.conf
nameserver 127.0.0.53

resolvers

 talosctl -n 192.168.5.10 get resolvers
NODE           NAMESPACE   TYPE             ID          VERSION   RESOLVERS
192.168.5.10   network     ResolverStatus   resolvers   2         ["192.168.5.1"]

CoreDNS was restarted twice after applying the patch.

Logs

[ERROR] plugin/errors: 2 radarr.media.svc. AAAA: read udp 10.244.2.33:48571->10.96.0.9:53: i/o timeout
[INFO] 10.244.0.142:37010 - 44799 "AAAA IN radarr.media.svc. udp 34 false 512" - - 0 2.001171487s
[ERROR] plugin/errors: 2 radarr.media.svc. AAAA: read udp 10.244.0.228:41133->10.96.0.9:53: i/o timeout
[INFO] 10.244.0.142:37010 - 44353 "A IN radarr.media.svc. udp 34 false 512" - - 0 2.001187098s
[ERROR] plugin/errors: 2 radarr.media.svc. A: read udp 10.244.0.228:49164->10.96.0.9:53: i/o timeout
[INFO] 10.244.0.30:55153 - 65462 "AAAA IN sonarr.media.svc. udp 34 false 512" - - 0 2.001133409s
[INFO] 10.244.0.30:55153 - 65275 "A IN sonarr.media.svc. udp 34 false 512" - - 0 2.001014136s
[ERROR] plugin/errors: 2 sonarr.media.svc. A: read udp 10.244.2.33:38186->10.96.0.9:53: i/o timeout
[ERROR] plugin/errors: 2 sonarr.media.svc. AAAA: read udp 10.244.2.33:57661->10.96.0.9:53: i/o timeout
[INFO] 10.244.3.16:57244 - 50161 "AAAA IN api.allegion.yonomi.cloud. udp 43 false 512" - - 0 2.001230715s
[ERROR] plugin/errors: 2 api.allegion.yonomi.cloud. AAAA: read udp 10.244.0.228:33430->10.96.0.9:53: i/o timeout
[INFO] 10.244.3.16:57244 - 49553 "A IN api.allegion.yonomi.cloud. udp 43 false 512" - - 0 2.001237302s
[ERROR] plugin/errors: 2 api.allegion.yonomi.cloud. A: read udp 10.244.0.228:47070->10.96.0.9:53: i/o timeout
[INFO] 10.244.1.242:47829 - 1031 "AAAA IN api.doppler.com. udp 44 false 1232" - - 0 2.001031405s
[INFO] 10.244.1.242:50138 - 44842 "A IN api.doppler.com. udp 44 false 1232" - - 0 2.001066446s
[ERROR] plugin/errors: 2 api.doppler.com. AAAA: read udp 10.244.0.228:52401->10.96.0.9:53: i/o timeout
[ERROR] plugin/errors: 2 api.doppler.com. A: read udp 10.244.0.228:48637->10.96.0.9:53: i/o timeout

Environment

  • Talos version: 1.7.1
  • Kubernetes version: 1.30.0
  • Platform: dell 5060/7060 nodes

Reverting the patch (false/false) fixes dns again.

FWIW, here’s my coredns configmap:

.:53 {
    errors
    health {
        lameduck 5s
    }
    ready
    log . {
        class error
    }
    prometheus :9153

    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
    }
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 16 (6 by maintainers)

Commits related to this issue

Most upvoted comments

The fix is coming, thanks for reporting it, it’s indeed the TTL. It’s only related to fowardKubeDNSToHost option which is not enabled by default in Talos 1.7 (only enabled for Docker-based clusters).

Greetings! Can you provide talosctl -n 192.168.5.10,192.168.5.11,192.168.5.12,192.168.5.15 logs dns-resolve-cache output?