tailscale: UDM Pro requires fwmark rules on table 201

What is the issue?

I happen to be running tailscale on a device that puts the default route on another table (table 201) instead of putting it on the main table. Specifically, i’m running on a unifi UDM pro. If I manually re-add the default route to the main table, tailscaled is able to connect to the control servers just fine.

I’m on version 1.20.4

Below is some relevant log output:

2022/02/27 19:45:28 trying bootstrapDNS("derp8d.tailscale.com", "178.62.44.132") for "controlplane.tailscale.com" ...
2022/02/27 19:45:28 bootstrapDNS("derp8d.tailscale.com", "178.62.44.132") for "controlplane.tailscale.com" error: Get "https://derp8d.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp 178.62.44.132:443: connect: network is unreachable
2022/02/27 19:45:28 trying bootstrapDNS("derp5.tailscale.com", "2001:19f0:5801:10b7:5400:2ff:feaa:284c") for "controlplane.tailscale.com" ...
2022/02/27 19:45:28 bootstrapDNS("derp5.tailscale.com", "2001:19f0:5801:10b7:5400:2ff:feaa:284c") for "controlplane.tailscale.com" error: Get "https://derp5.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp [2001:19f0:5801:10b7:5400:2ff:feaa:284c]:443: connect: network is unreachable
2022/02/27 19:45:28 trying bootstrapDNS("derp6.tailscale.com", "68.183.90.120") for "controlplane.tailscale.com" ...
2022/02/27 19:45:28 bootstrapDNS("derp6.tailscale.com", "68.183.90.120") for "controlplane.tailscale.com" error: Get "https://derp6.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp 68.183.90.120:443: connect: network is unreachable
2022/02/27 19:45:28 trying bootstrapDNS("derp8b.tailscale.com", "2a03:b0c0:1:d0::ec1:e001") for "controlplane.tailscale.com" ...
2022/02/27 19:45:28 bootstrapDNS("derp8b.tailscale.com", "2a03:b0c0:1:d0::ec1:e001") for "controlplane.tailscale.com" error: Get "https://derp8b.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp [2a03:b0c0:1:d0::ec1:e001]:443: connect: network is unreachable
2022/02/27 19:45:28 Received error: fetch control key: Get "https://controlplane.tailscale.com/key": dial tcp 52.29.8.43:443: connect: network is unreachable
2022/02/27 19:45:28 control: authRoutine: backoff: 15144 msec

Steps to reproduce

remove default route, place it on another adjacent table, and verify other services (like ping) can still reach the internet but tailscaled should be constantly unable to reach any control servers with the message “network is unreachable”.

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

kernel 4.19.152-al-linux-v10.2.0-v1.11.0.3921-f2e3fac, aarch64, unifi-os 1.11.0

Tailscale version

1.20.4

Bug report

BUG-c989981553e78e7a212129e04c3ed835b8c95a422272bd264c5ce12f3940c8a8-20220227200449Z-e441c8451363aa7d

About this issue

Original URL
State: open
Created 2 years ago
Comments: 16 (10 by maintainers)

Commits related to this issue

net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 2 years ago
net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 2 years ago
net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 2 years ago
net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 2 years ago
net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 2 years ago
net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 2 years ago
net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 2 years ago
net/interfaces: get Linux default route from netlink as fallback If it's in a non-standard table, as it is on Unifi UDM Pro, apparently. Updates #4038 (probably fixes, but don't have hardware to ver... — committed to tailscale/tailscale by bradfitz 2 years ago

Most upvoted comments

Running the following script on my UDMP now. It’s first looking at the table used in rule 32766 and sets up an fwmark rule as mentioned above. Afterwards, it’s monitoring the message.log for WAN failover messages, adjusting the rule accordingly.

RULE_PRIORITY="5225"

function getDefaultTable() {
        /sbin/ip rule list priority 32766 | cut -d " " -f 4
}

function updateTailscaleRule() {
        /sbin/ip rule del priority $RULE_PRIORITY
        /sbin/ip rule add priority $RULE_PRIORITY from all fwmark 0x80000 lookup $1
}

echo Routing table with default route is $(getDefaultTable)
updateTailscaleRule $(getDefaultTable)

tail -Fn 0  /var/log/messages | while read line; do
        table=`echo $line | grep -e "ubios-udapi-server: wanFailover"`
        if [[ "$table" != ""  ]]
        then
                echo Detected WAN failover: Routing table with default route is $(getDefaultTable) now, adjusting rule
                updateTailscaleRule $(getDefaultTable)
        fi
done

docbobo on Apr 8, 2022

Thank you @docbobo! My reply is a bit late, but I just setup your script on my UDM Pro today and it works perfectly! Perfect workaround for now until the issue is completely fixed.

jtcressy on May 11, 2022

Thanks for debugging! We don’t want to start using the ip command in general (we’ve been migrating away from it and other such tools and using netlink directly), but we can use netlink to get the default route.

bradfitz on Feb 28, 2022