tailscale: UDM Pro requires fwmark rules on table 201

What is the issue?

I happen to be running tailscale on a device that puts the default route on another table (table 201) instead of putting it on the main table. Specifically, i’m running on a unifi UDM pro. If I manually re-add the default route to the main table, tailscaled is able to connect to the control servers just fine.

I’m on version 1.20.4

Below is some relevant log output:

2022/02/27 19:45:28 trying bootstrapDNS("derp8d.tailscale.com", "178.62.44.132") for "controlplane.tailscale.com" ...
2022/02/27 19:45:28 bootstrapDNS("derp8d.tailscale.com", "178.62.44.132") for "controlplane.tailscale.com" error: Get "https://derp8d.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp 178.62.44.132:443: connect: network is unreachable
2022/02/27 19:45:28 trying bootstrapDNS("derp5.tailscale.com", "2001:19f0:5801:10b7:5400:2ff:feaa:284c") for "controlplane.tailscale.com" ...
2022/02/27 19:45:28 bootstrapDNS("derp5.tailscale.com", "2001:19f0:5801:10b7:5400:2ff:feaa:284c") for "controlplane.tailscale.com" error: Get "https://derp5.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp [2001:19f0:5801:10b7:5400:2ff:feaa:284c]:443: connect: network is unreachable
2022/02/27 19:45:28 trying bootstrapDNS("derp6.tailscale.com", "68.183.90.120") for "controlplane.tailscale.com" ...
2022/02/27 19:45:28 bootstrapDNS("derp6.tailscale.com", "68.183.90.120") for "controlplane.tailscale.com" error: Get "https://derp6.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp 68.183.90.120:443: connect: network is unreachable
2022/02/27 19:45:28 trying bootstrapDNS("derp8b.tailscale.com", "2a03:b0c0:1:d0::ec1:e001") for "controlplane.tailscale.com" ...
2022/02/27 19:45:28 bootstrapDNS("derp8b.tailscale.com", "2a03:b0c0:1:d0::ec1:e001") for "controlplane.tailscale.com" error: Get "https://derp8b.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp [2a03:b0c0:1:d0::ec1:e001]:443: connect: network is unreachable
2022/02/27 19:45:28 Received error: fetch control key: Get "https://controlplane.tailscale.com/key": dial tcp 52.29.8.43:443: connect: network is unreachable
2022/02/27 19:45:28 control: authRoutine: backoff: 15144 msec

Steps to reproduce

remove default route, place it on another adjacent table, and verify other services (like ping) can still reach the internet but tailscaled should be constantly unable to reach any control servers with the message “network is unreachable”.

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

kernel 4.19.152-al-linux-v10.2.0-v1.11.0.3921-f2e3fac, aarch64, unifi-os 1.11.0

Tailscale version

1.20.4

Bug report

BUG-c989981553e78e7a212129e04c3ed835b8c95a422272bd264c5ce12f3940c8a8-20220227200449Z-e441c8451363aa7d

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 19 (13 by maintainers)

Commits related to this issue

Most upvoted comments

Running the following script on my UDMP now. It’s first looking at the table used in rule 32766 and sets up an fwmark rule as mentioned above. Afterwards, it’s monitoring the message.log for WAN failover messages, adjusting the rule accordingly.

RULE_PRIORITY="5225"

function getDefaultTable() {
        /sbin/ip rule list priority 32766 | cut -d " " -f 4
}

function updateTailscaleRule() {
        /sbin/ip rule del priority $RULE_PRIORITY
        /sbin/ip rule add priority $RULE_PRIORITY from all fwmark 0x80000 lookup $1
}

echo Routing table with default route is $(getDefaultTable)
updateTailscaleRule $(getDefaultTable)

tail -Fn 0  /var/log/messages | while read line; do
        table=`echo $line | grep -e "ubios-udapi-server: wanFailover"`
        if [[ "$table" != ""  ]]
        then
                echo Detected WAN failover: Routing table with default route is $(getDefaultTable) now, adjusting rule
                updateTailscaleRule $(getDefaultTable)
        fi
done

Oh, it seems to be fixed in #14452. I can confirm it works by using (unstable) tailscale 1.79.130

Thank you @docbobo! My reply is a bit late, but I just setup your script on my UDM Pro today and it works perfectly! Perfect workaround for now until the issue is completely fixed.

Another update: apparently, tailscaled is adding the following rules:

5210:  from all fwmark 0x80000 lookup main 
5230:  from all fwmark 0x80000 lookup default 
5250:  from all fwmark 0x80000 unreachable
5270:  from all lookup 52 

Since UDM doesn’t have the default route in the default table but in 201 (and/or 202 in case if multiple WAN interfaces) that looked problematic to me. In my case, the following command solved the problem of not being able to connect to the control plane:

ip rule add from all fwmark 0x80000 lookup 201 priority 5229

Once this was executed, tailscaled started up correctly and I was able to ping the hosts and subnets that were added to table 52 - at least from the UDM itself. What didn’t yet work was pinging those systems from different machines.

Thanks for debugging! We don’t want to start using the ip command in general (we’ve been migrating away from it and other such tools and using netlink directly), but we can use netlink to get the default route.