talos: 1.7.1 with hostdns and forwardKubeDNSToHost doesn't resolve anything
Bug Report
Description
This is on a cluster that has been upgraded (1.6.x->1.7.x), not fresh
after applying the following patch:
machine:
features:
hostDNS:
enabled: true
forwardKubeDNSToHost: true
nothing seems to resolve dns, either in the cluster or externally
getupstream:
talosctl -n 192.168.5.10,192.168.5.11,192.168.5.12,192.168.5.15 get dnsupstream
NODE NAMESPACE TYPE ID VERSION HEALTHY ADDRESS
192.168.5.10 network DNSUpstream 192.168.5.1 1 true 192.168.5.1:53
192.168.5.11 network DNSUpstream 192.168.5.1 1 true 192.168.5.1:53
192.168.5.12 network DNSUpstream 192.168.5.1 1 true 192.168.5.1:53
192.168.5.15 network DNSUpstream 192.168.5.1 1 true 192.168.5.1:53
resolv.conf
talosctl -n 192.168.5.10 read /system/resolved/resolv.conf
nameserver 10.96.0.9
talosctl -n 192.168.5.10 read /etc/resolv.conf
nameserver 127.0.0.53
resolvers
talosctl -n 192.168.5.10 get resolvers
NODE NAMESPACE TYPE ID VERSION RESOLVERS
192.168.5.10 network ResolverStatus resolvers 2 ["192.168.5.1"]
CoreDNS was restarted twice after applying the patch.
Logs
[ERROR] plugin/errors: 2 radarr.media.svc. AAAA: read udp 10.244.2.33:48571->10.96.0.9:53: i/o timeout
[INFO] 10.244.0.142:37010 - 44799 "AAAA IN radarr.media.svc. udp 34 false 512" - - 0 2.001171487s
[ERROR] plugin/errors: 2 radarr.media.svc. AAAA: read udp 10.244.0.228:41133->10.96.0.9:53: i/o timeout
[INFO] 10.244.0.142:37010 - 44353 "A IN radarr.media.svc. udp 34 false 512" - - 0 2.001187098s
[ERROR] plugin/errors: 2 radarr.media.svc. A: read udp 10.244.0.228:49164->10.96.0.9:53: i/o timeout
[INFO] 10.244.0.30:55153 - 65462 "AAAA IN sonarr.media.svc. udp 34 false 512" - - 0 2.001133409s
[INFO] 10.244.0.30:55153 - 65275 "A IN sonarr.media.svc. udp 34 false 512" - - 0 2.001014136s
[ERROR] plugin/errors: 2 sonarr.media.svc. A: read udp 10.244.2.33:38186->10.96.0.9:53: i/o timeout
[ERROR] plugin/errors: 2 sonarr.media.svc. AAAA: read udp 10.244.2.33:57661->10.96.0.9:53: i/o timeout
[INFO] 10.244.3.16:57244 - 50161 "AAAA IN api.allegion.yonomi.cloud. udp 43 false 512" - - 0 2.001230715s
[ERROR] plugin/errors: 2 api.allegion.yonomi.cloud. AAAA: read udp 10.244.0.228:33430->10.96.0.9:53: i/o timeout
[INFO] 10.244.3.16:57244 - 49553 "A IN api.allegion.yonomi.cloud. udp 43 false 512" - - 0 2.001237302s
[ERROR] plugin/errors: 2 api.allegion.yonomi.cloud. A: read udp 10.244.0.228:47070->10.96.0.9:53: i/o timeout
[INFO] 10.244.1.242:47829 - 1031 "AAAA IN api.doppler.com. udp 44 false 1232" - - 0 2.001031405s
[INFO] 10.244.1.242:50138 - 44842 "A IN api.doppler.com. udp 44 false 1232" - - 0 2.001066446s
[ERROR] plugin/errors: 2 api.doppler.com. AAAA: read udp 10.244.0.228:52401->10.96.0.9:53: i/o timeout
[ERROR] plugin/errors: 2 api.doppler.com. A: read udp 10.244.0.228:48637->10.96.0.9:53: i/o timeout
Environment
- Talos version: 1.7.1
- Kubernetes version: 1.30.0
- Platform: dell 5060/7060 nodes
Reverting the patch (false/false) fixes dns again.
FWIW, here’s my coredns configmap:
.:53 {
errors
health {
lameduck 5s
}
ready
log . {
class error
}
prometheus :9153
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 16 (6 by maintainers)
Commits related to this issue
- chore: fix incorrect ttl for pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes #8698. Signed-off-by: Dmitriy Matrenichev <dmitry.ma... — committed to DmitriyMV/talos by DmitriyMV 3 years ago
- chore: enable forwardKubeDNSToHost by default and fix incorrect ttl for pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. It also enables by default the usage of our host ... — committed to DmitriyMV/talos by DmitriyMV 3 years ago
- chore: enable forwardKubeDNSToHost by default and fix incorrect ttl for pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. It also enables by default the usage of our host ... — committed to DmitriyMV/talos by DmitriyMV 3 years ago
- fix: increase host dns packet ttl for pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes #8698. Signed-off-by: Dmitriy Matrenichev <... — committed to DmitriyMV/talos by DmitriyMV 3 years ago
- chore: enable forwardKubeDNSToHost by default and fix incorrect ttl for pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. It also enables by default the usage of our host ... — committed to DmitriyMV/talos by DmitriyMV 3 years ago
- chore: enable forwardKubeDNSToHost by default and fix incorrect ttl for pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. It also enables by default the usage of our host ... — committed to DmitriyMV/talos by DmitriyMV 3 years ago
- chore: enable forwardKubeDNSToHost by default and fix incorrect ttl for pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. It also enables by default the usage of our host ... — committed to DmitriyMV/talos by DmitriyMV 3 years ago
- fix: increase host dns packet ttl for pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes #8698. Signed-off-by: Dmitriy Matrenichev <... — committed to DmitriyMV/talos by DmitriyMV 3 years ago
- fix: increase host dns packet ttl for pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes #8698. Signed-off-by: Dmitriy Matrenichev <... — committed to DmitriyMV/talos by DmitriyMV 3 years ago
- fix: increase host dns packet ttl for pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes #8698. Signed-off-by: Dmitriy Matrenichev <... — committed to DmitriyMV/talos by DmitriyMV 3 years ago
- fix: increase host dns packet ttl for pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes #8698. Signed-off-by: Dmitriy Matrenichev <... — committed to DmitriyMV/talos by DmitriyMV 3 years ago
- fix: increase host dns packet ttl for pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes #8698. Signed-off-by: Dmitriy Matrenichev <... — committed to DmitriyMV/talos by DmitriyMV 3 years ago
- fix: increase host dns packet ttl for pods This PR fixes incorrect packet TTL if `forwardKubeDNSToHost` is enabled. Credits go to Julian Wiedmann. Closes #8698. Signed-off-by: Dmitriy Matrenichev <... — committed to smira/talos by DmitriyMV 3 years ago
The fix is coming, thanks for reporting it, it’s indeed the TTL. It’s only related to
fowardKubeDNSToHostoption which is not enabled by default in Talos 1.7 (only enabled for Docker-based clusters).Greetings! Can you provide
talosctl -n 192.168.5.10,192.168.5.11,192.168.5.12,192.168.5.15 logs dns-resolve-cacheoutput?