talos: QEMU dev environment unable to bring up NTP because of failure to reach 8.8.8.8 DNS
Bug Report
QEMU test cluster ran from the guide fails because of inability to reach the 8.8.8.8 DNS resolver. 8.8.8.8 can be pinged from the host thus I believe it’s a VM config issue or a compatibility problem.
Description
validating CIDR and reserving IPs
generating PKI and tokens
creating "/tmp/tl-dev-home/.talos/cni/bin"
creating "/tmp/tl-dev-home/.talos/cni/cache"
creating "/tmp/tl-dev-home/.talos/cni/conf.d"
downloading CNI bundle from "https://github.com/siderolabs/talos/releases/download/v1.7.0-alpha.0/talosctl-cni-bundle-amd64.tar.gz" to "/tmp/tl-dev-home/.talos/cni/bin"
creating state directory in "/tmp/tl-dev-home/.talos/clusters/talos-default"
creating network talos-default
creating load balancer
creating dhcpd
creating controlplane nodes
creating worker nodes
waiting for API
^CSignal received, aborting, press Ctrl+C once again to abort immediately...
bootstrap error: 3 error(s) occurred:
rpc error: code = DeadlineExceeded desc = context deadline exceeded
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.5.0.2:50000: connect: no route to host"
context canceled
HOME=/tmp/tl-dev-home sudo --preserve-env=HOME _out/talosctl-linux-amd64 cluster create \ [±main ✓]
--provisioner=qemu \
--registry-mirror docker.io=http://172.20.0.1:5000 \
--registry-mirror registry.k8s.io=http://172.20.0.1:5001 \
--registry-mirror gcr.io=http://172.20.0.1:5003 \
--registry-mirror ghcr.io=http://172.20.0.1:5004 \
--registry-mirror 127.0.0.1:5005=http://172.20.0.1:5005 \
--install-image=127.0.0.1:5005/siderolabs/installer:v1.7.0-alpha.0-19-g5324d3916 \
--controlplanes 1 \
--workers 2 \
--with-bootloader=false \
--extra-uefi-search-paths "/tmp"
HOME is set to tmp to make installation temporary (I typically do one-time stuff and build artifacts on tmpfs). Extra search path includes OVMF_CODE.fd and OVMF_VARS.fd because provisioner couldn’t find them in locations SUSE puts those in by default.
Logs
==> /tmp/tl-dev-home/.talos/clusters/talos-default/talos-default-worker-1.log <==
[ 33.843549] [talos] failed looking up "pool.ntp.org", ignored {"component": "controller-runtime", "controller": "time.SyncController", "error": "lookup pool.ntp.org on 8.8.8.8:53: dial udp 8.8.8.8:53: connect: network is unreachable"}
==> /tmp/tl-dev-home/.talos/clusters/talos-default/talos-default-controlplane-1.log <==
[ 34.823032] [talos] failed looking up "pool.ntp.org", ignored {"component": "controller-runtime", "controller": "time.SyncController", "error": "lookup pool.ntp.org on 8.8.8.8:53: dial udp 8.8.8.8:53: connect: network is unreachable"}
==> /tmp/tl-dev-home/.talos/clusters/talos-default/talos-default-worker-2.log <==
[ 34.657992] [talos] failed looking up "pool.ntp.org", ignored {"component": "controller-runtime", "controller": "time.SyncController", "error": "lookup pool.ntp.org on 8.8.8.8:53: dial udp 8.8.8.8:53: connect: network is unreachable"}
==> /tmp/tl-dev-home/.talos/clusters/talos-default/talos-default-worker-1.log <==
[ 34.844676] [talos] failed looking up "pool.ntp.org", ignored {"component": "controller-runtime", "controller": "time.SyncController", "error": "lookup pool.ntp.org on 8.8.8.8:53: dial udp 8.8.8.8:53: connect: network is unreachable"}
==> /tmp/tl-dev-home/.talos/clusters/talos-default/talos-default-controlplane-1.log <==
[ 35.824837] [talos] failed looking up "pool.ntp.org", ignored {"component": "controller-runtime", "controller": "time.SyncController", "error": "lookup pool.ntp.org on 8.8.8.8:53: dial udp 8.8.8.8:53: connect: network is unreachable"}
==> /tmp/tl-dev-home/.talos/clusters/talos-default/talos-default-worker-2.log <==
[ 35.660024] [talos] failed looking up "pool.ntp.org", ignored {"component": "controller-runtime", "controller": "time.SyncController", "error": "lookup pool.ntp.org on 8.8.8.8:53: dial udp 8.8.8.8:53: connect: network is unreachable"}
Environment
- Talos version: [
talosctl version --nodes <problematic nodes>] main 5324d391671dfbf918aee1bd6b095adffadecf8e - Kubernetes version: [
kubectl version --short] 1.28 perhaps - Platform: openSUSE Tumbleweed: SELinux enforcing, firewall active with defaults
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 1
- Comments: 17 (11 by maintainers)
Okay, this should fix it, for users using firewalld
I recommend against baking detection for firewalld into talosctl. There are more firewall solutions out there, e.g. ufw, and there are people out there using plain nftables/iptables as well.
Taking into consideration that you may even hit this issue without any firewall in place on the machine you run talosctl when behind a proxy beyond one’s control, I think it better to document the symptoms one runs into when the default DNS and default NTP cannot be reached, what might cause that to happen and how to override the defaults to match one’s network peculiarities.