tailscale: UDM Pro requires fwmark rules on table 201
What is the issue?
I happen to be running tailscale on a device that puts the default route on another table (table 201) instead of putting it on the main table. Specifically, i’m running on a unifi UDM pro. If I manually re-add the default route to the main table, tailscaled is able to connect to the control servers just fine.
I’m on version 1.20.4
Below is some relevant log output:
2022/02/27 19:45:28 trying bootstrapDNS("derp8d.tailscale.com", "178.62.44.132") for "controlplane.tailscale.com" ...
2022/02/27 19:45:28 bootstrapDNS("derp8d.tailscale.com", "178.62.44.132") for "controlplane.tailscale.com" error: Get "https://derp8d.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp 178.62.44.132:443: connect: network is unreachable
2022/02/27 19:45:28 trying bootstrapDNS("derp5.tailscale.com", "2001:19f0:5801:10b7:5400:2ff:feaa:284c") for "controlplane.tailscale.com" ...
2022/02/27 19:45:28 bootstrapDNS("derp5.tailscale.com", "2001:19f0:5801:10b7:5400:2ff:feaa:284c") for "controlplane.tailscale.com" error: Get "https://derp5.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp [2001:19f0:5801:10b7:5400:2ff:feaa:284c]:443: connect: network is unreachable
2022/02/27 19:45:28 trying bootstrapDNS("derp6.tailscale.com", "68.183.90.120") for "controlplane.tailscale.com" ...
2022/02/27 19:45:28 bootstrapDNS("derp6.tailscale.com", "68.183.90.120") for "controlplane.tailscale.com" error: Get "https://derp6.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp 68.183.90.120:443: connect: network is unreachable
2022/02/27 19:45:28 trying bootstrapDNS("derp8b.tailscale.com", "2a03:b0c0:1:d0::ec1:e001") for "controlplane.tailscale.com" ...
2022/02/27 19:45:28 bootstrapDNS("derp8b.tailscale.com", "2a03:b0c0:1:d0::ec1:e001") for "controlplane.tailscale.com" error: Get "https://derp8b.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp [2a03:b0c0:1:d0::ec1:e001]:443: connect: network is unreachable
2022/02/27 19:45:28 Received error: fetch control key: Get "https://controlplane.tailscale.com/key": dial tcp 52.29.8.43:443: connect: network is unreachable
2022/02/27 19:45:28 control: authRoutine: backoff: 15144 msec
Steps to reproduce
remove default route, place it on another adjacent table, and verify other services (like ping) can still reach the internet but tailscaled should be constantly unable to reach any control servers with the message “network is unreachable”.
Are there any recent changes that introduced the issue?
No response
OS
Linux
OS version
kernel 4.19.152-al-linux-v10.2.0-v1.11.0.3921-f2e3fac, aarch64, unifi-os 1.11.0
Tailscale version
1.20.4
Bug report
BUG-c989981553e78e7a212129e04c3ed835b8c95a422272bd264c5ce12f3940c8a8-20220227200449Z-e441c8451363aa7d
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 19 (13 by maintainers)
Commits related to this issue
- net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 5 years ago
- net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 5 years ago
- net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 5 years ago
- net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 5 years ago
- net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 5 years ago
- net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 5 years ago
- net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 5 years ago
- net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 5 years ago
- add a couple more ip rules for unifi udm-pro Fixes: #4038 Signed-off-by: Jason Barnett <J@sonBarnett.com> — committed to jasonwbarnett/tailscale by jasonwbarnett 3 years ago
- wgengine/router: add ip rules for unifi udm-pro Fixes: #4038 Signed-off-by: Jason Barnett <J@sonBarnett.com> — committed to jasonwbarnett/tailscale by jasonwbarnett 3 years ago
- wgengine/router: add ip rules for unifi udm-pro Fixes: #4038 Signed-off-by: Jason Barnett <J@sonBarnett.com> — committed to jasonwbarnett/tailscale by jasonwbarnett 3 years ago
- wgengine/router: add ip rules for unifi udm-pro Fixes: #4038 Signed-off-by: Jason Barnett <J@sonBarnett.com> — committed to jasonwbarnett/tailscale by jasonwbarnett 3 years ago
- wgengine/router: add ip rules for unifi udm-pro Fixes: #4038 Signed-off-by: Jason Barnett <J@sonBarnett.com> — committed to jasonwbarnett/tailscale by jasonwbarnett 3 years ago
- wgengine/router: add ip rules for unifi udm-pro Fixes: #4038 Signed-off-by: Jason Barnett <J@sonBarnett.com> — committed to jasonwbarnett/tailscale by jasonwbarnett 3 years ago
- wgengine/router: add ip rules for unifi udm-pro Fixes: #4038 Signed-off-by: Jason Barnett <J@sonBarnett.com> — committed to jasonwbarnett/tailscale by jasonwbarnett 3 years ago
- wgengine/router: add ip rules for unifi udm-pro Fixes: #4038 Signed-off-by: Jason Barnett <J@sonBarnett.com> — committed to jasonwbarnett/tailscale by jasonwbarnett 3 years ago
- wgengine/router: add ip rules for unifi udm-pro Fixes: #4038 Signed-off-by: Jason Barnett <J@sonBarnett.com> — committed to jasonwbarnett/tailscale by jasonwbarnett 3 years ago
- wgengine/router: add ip rules for unifi udm-pro Fixes: #4038 Signed-off-by: Jason Barnett <J@sonBarnett.com> — committed to jasonwbarnett/tailscale by jasonwbarnett 3 years ago
- wgengine/router: add ip rules for unifi udm-pro Fixes: #4038 Signed-off-by: Jason Barnett <J@sonBarnett.com> — committed to jasonwbarnett/tailscale by jasonwbarnett 3 years ago
- wgengine/router: add ip rules for unifi udm-pro Fixes: #4038 Signed-off-by: Jason Barnett <J@sonBarnett.com> — committed to jasonwbarnett/tailscale by jasonwbarnett 3 years ago
Running the following script on my UDMP now. It’s first looking at the table used in rule 32766 and sets up an fwmark rule as mentioned above. Afterwards, it’s monitoring the
message.logfor WAN failover messages, adjusting the rule accordingly.Oh, it seems to be fixed in #14452. I can confirm it works by using (unstable) tailscale 1.79.130
Thank you @docbobo! My reply is a bit late, but I just setup your script on my UDM Pro today and it works perfectly! Perfect workaround for now until the issue is completely fixed.
Another update: apparently, tailscaled is adding the following rules:
Since UDM doesn’t have the default route in the default table but in 201 (and/or 202 in case if multiple WAN interfaces) that looked problematic to me. In my case, the following command solved the problem of not being able to connect to the control plane:
Once this was executed, tailscaled started up correctly and I was able to ping the hosts and subnets that were added to table 52 - at least from the UDM itself. What didn’t yet work was pinging those systems from different machines.
Thanks for debugging! We don’t want to start using the
ipcommand in general (we’ve been migrating away from it and other such tools and using netlink directly), but we can use netlink to get the default route.