k3s: Error "no default routes found" when using Air-gap installation

Version: v1.0.0 Describe the bug errors when using Air-gap installing.

FATA[2019-11-28T09:04:09.931286843+08:00] apiserver exited: unable to find suitable network address.error=‘no default routes found in “/proc/net/route” or “/proc/net/ipv6_route”’. Try to set the AdvertiseAddress directly or provide a valid BindAddress to fix this

To Reproduce

[aiops@7 ~]$ sudo INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh
[INFO]  Skipping k3s download and verify
which: no kubectl in (/sbin:/bin:/usr/sbin:/usr/bin)
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
which: no crictl in (/sbin:/bin:/usr/sbin:/usr/bin)
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink from /etc/systemd/system/multi-user.target.wants/k3s.service to /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
Job for k3s.service failed because the control process exited with error code. See "systemctl status k3s.service" and "journalctl -xe" for details.
[aiops@7 ~]$ sudo systemctl status -l k3s.service
● k3s.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since 四 2019-11-28 09:03:14 CST; 4s ago
     Docs: https://k3s.io
  Process: 30109 ExecStart=/usr/local/bin/k3s server (code=exited, status=1/FAILURE)
  Process: 30106 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 30104 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
 Main PID: 30109 (code=exited, status=1/FAILURE)

11月 28 09:03:14 7 k3s[30109]: --log-file-max-size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
11月 28 09:03:14 7 k3s[30109]: --log-flush-frequency duration     Maximum number of seconds between log flushes (default 5s)
11月 28 09:03:14 7 k3s[30109]: --logtostderr                      log to standard error instead of files (default true)
11月 28 09:03:14 7 k3s[30109]: --skip-headers                     If true, avoid header prefixes in the log messages
11月 28 09:03:14 7 k3s[30109]: --skip-log-headers                 If true, avoid headers when opening log files
11月 28 09:03:14 7 k3s[30109]: --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
11月 28 09:03:14 7 k3s[30109]: -v, --v Level                          number for the log level verbosity
11月 28 09:03:14 7 k3s[30109]: --version version[=true]           Print version information and quit
11月 28 09:03:14 7 k3s[30109]: --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging
11月 28 09:03:14 7 k3s[30109]: time="2019-11-28T09:03:14.937465904+08:00" level=fatal msg="apiserver exited: unable to find suitable network address.error='no default routes found in \"/proc/net/route\" or \"/proc/net/ipv6_route\"'. Try to set the AdvertiseAddress directly or provide a valid BindAddress to fix this"

Expected behavior The K3s server should be started

Actual behavior failed with error “no default routes found”

Additional context

ifconfig output

[aiops@7 ~]$ ifconfig -a
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:c8ff:fe4e:cadd  prefixlen 64  scopeid 0x20<link>
        ether 02:42:c8:4e:ca:dd  txqueuelen 0  (Ethernet)
        RX packets 82  bytes 8449 (8.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 86  bytes 23010 (22.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.7  netmask 255.255.255.0  broadcast 192.168.100.255
        inet6 fe80::e255:ed27:fe49:370e  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::4ddc:440f:1a22:8def  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::7ef5:38f0:1116:c541  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:90:dc:83  txqueuelen 1000  (Ethernet)
        RX packets 2825735  bytes 815531129 (777.7 MiB)
        RX errors 0  dropped 883  overruns 0  frame 0
        TX packets 50330  bytes 4290449 (4.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ip route output

[aiops@7 ~]$ ip route
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.100.0/24 dev ens192 proto kernel scope link src 192.168.100.7 metric 100

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 19 (7 by maintainers)

Most upvoted comments

after add sudo ip route add default via 192.168.100.1, the installation passed. but it should be a bug because this default route is meaningless in my env. thx also， the default route info will be missing after reboot

floydljy on Nov 28, 2019

I find

maybe alter that to
ip link add dummy0 type dummy
ip link set dummy0 up
ip addr add 169.254.255.254/31 dev dummy0
ip route add default via 169.254.255.255 dev dummy0 metric 1000
You may have to play with it a bit - I haven’t tested this myself recently.

hello, We saw a similar issue to this.But follow that step can not solve the problem.As I setted the dummy route and then restart k3s, it failed to start k3s and report a fatal message “unable to select an IP from default routes.” this is my route info: `[root@master route-test]# ip route

default via 169.254.255.255 dev dummy0 metric 1000 10.42.0.0/24 dev cni0 proto kernel scope link src 10.42.0.1 169.254.255.254/31 dev dummy0 proto kernel scope link src 169.254.255.254 192.168.169.0/24 dev ens33 proto kernel scope link src 192.168.169.12 metric 100`

And after I checked the source code of k3s and k8s , I found when k3s want to get hostname and ip, it will use func ChooseHostInterface() in “k8s.io/apimachinery/pkg/util/net” package and in this package it will detect the default route and check the ip by func IsGlobalUnicast() in net package, but now IsGlobalUnicast() will return false when using 169.xx.xx.xx.So I think maybe the change of func IsGlobalUnicast() that make the dummy route not work. Is there any other way to solve this problem?

x-coder-L on Jul 5, 2023

Following modification of work-around worked for me:

$ sudo ip link add dummy0 type dummy $ sudo ip link set dummy0 up $ sudo ip addr add 192.168.3.254/31 dev dummy0 $ sudo ip route add default via 192.168.3.255 dev dummy0 metric 1000

Also, for coredns, any 127.0.0.x entries should be commented out in /etc/resolv.conf and instead

nameserver 192.168.3.254

should be added.

taleodor on Oct 12, 2022