tailscale: macOS: routing via exit-node or subnet-router bypasses routing table

What is the issue?

Hi there,

I opened this as a bug-report but this can also be a “feature” depending on how you look at it…

Long story short, I use a macOS device as a (middle-)gateway.

For example, in the following chart;

  • From client, request to 1.2.3.4/32 routed to internet (incorrectly)
  • Should’ve been routed to vpn. Since macOS is having specific entry in the route table.
  • Resulting packet transmission (leak?) to internet/subnet of the gateway.

Packet coming out from A destined to C leaves (leaks?) at B


flowchart LR
  entry-a[A] --> client[Client 100.65.33.44]
  client --> macos[macOS 100.64.11.22]
  macos --> internet[Internet 0.0.0.0/0]
  macos --> vpn[VPN 1.2.3.4/32]
  internet --> out-b[B]
  vpn --> out-c[C]

I suppose that could be the case because the VPN is another utun device. (OpenVPN)

PS: I have not tested the use-case on Linux (Linux as a middle-device) yet.

I tried enabling IP Forwarding via sudo sysctl -w net.inet.ip.forwarding=1

This may be related to the option --snat-subnet-routes in macOS…

Steps to reproduce

  1. Create a subnet-router or exit-node on macOS
  2. Connect macOS to another VPN
  3. Try to access a resource behind the VPN using another device than macOS through Tailscale.

Are there any recent changes that introduced the issue?

No response

OS

macOS

OS version

13.2.1

Tailscale version

1.38.1 (AppStore)

Other software

  • OpenVPN

Bug report

BUG-baed9af847873e680716aa6712227c7b3893b7bbed63f3be337f4f6b7d524204-20230320232923Z-bbf24661ad3d52b1

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 2
  • Comments: 18 (4 by maintainers)

Most upvoted comments

You may be interested in the (still experimental) https://tailscale.com/cap/bind-to-interface-by-route node attribute that you can add to your policy file.

That will cause the macOS client to take into account the routing table when deciding what interface to send them on, and presumably it should then choose the other VPN’s interface (see #5719 and #7065 for more details).

Can you give an example of how this capability be configured in the policy file json for a specific Mac host?

You may be interested in the (still experimental) https://tailscale.com/cap/bind-to-interface-by-route node attribute that you can add to your policy file.

That will cause the macOS client to take into account the routing table when deciding what interface to send them on, and presumably it should then choose the other VPN’s interface (see #5719 and #7065 for more details).

Furthermore :

  • MacOS App Store : 1.62.0 : has problem
  • standalone GUI app : 1.62.1 : also has problem 😦

Considering that AppStore/Standalone uses different extension ( https://tailscale.com/kb/1065/macos-variants ):

  • App Store : Network Extension
  • standalone : System Extension I just tried with the standalone variant. IT ALSO DOES NOT WORK when using TS_BIND_TO_INTERFACE_BY_ROUTE=y (the extension is also refusing to start).

Also I observed that the standalone variant has more pleasant behavior :

  • App Store : When “stopping” (tailscale down or disabling the GUI’s green button) : the IPNExtension process does not exit. It takes right-click-exit on the GUI menu bar to make the extension process exit.
  • standalone : When “stopping” (same 2 ways of doing it) : the [...].macsys.network-extension process also exit

The fact that the “extension process” exit “when stopping” is pleasant when trying to mangle up with configuration items in tailscaled-env.txt ; It takes more steps to apply modification to the “App Store” version : I had to quit/relaunch the GUI.

Also the doc of https://github.com/tailscale/tailscale/blob/main/envknob/envknob.go is MISSLEADING about the location of tailscaled-env.txt for the STANDALONE GUI version ;

  • documented : $USER/ + Library/Containers/$VARIANT/Data/tailscaled-env.txt
    • App Store : VARIANT = io.tailscale.ipn.macos.network-extension : yes !
    • standalone : VARIANT = io.tailscale.ipn.macsys : WRONG

What’s wrong (x2) about the STANDALONE location of tailscaled-env.txt :

  • It’s not really under $USER/Library/... but instead under root/Library/... (the standalone’s “extension process” runs as root, whereas the App Store run… I won’t trackback, but its cwd() was under $USER/....)
  • It’s not VARIANT = io.tailscale.ipn.macsys (which is even more missleading because it can be found under $USER/Library/Containers) but its in reality VARIANT = io.tailscale.ipn.macsys.network-extension which is found-able only under root/...

Of course [sudo] lsof -d cwd | grep PIC was used to find the cwd()'s.

Location of tailscaled-env.txt was HORRIBLE to handle. I just hardlinked the (now 5) places were it could exists. Like /etc/tailscale/tailscaled-env.txt + {/var/root,$USER}/Library/Containers/io.tailscale.ipn.*/Data/tailscaled-env.txt, and since they were hardlinked (not symlinked) due to plausible containerization, I didnt want to edit the ENV=VALUEwith an editor (to not un-hardlink) and had to usetruncate -s 0+echo xxx >> …` which behavior is without surprise.

Also when we quit the GUI (at least the stadlone one), and it previsouly didn’t successfully launched the “extension”, when we… empty the tailscaled-env.txt file, and relaunch the GUI, the GUI will falsely popup that “the network extension failed to start”, which is a truth only in the past, the GUI is reporting a knowledge of the past, it should better just not says something is does not know, and should instead report only truth of the present, which, if it did try to launch again the “extension”, it could have found that now it does. If I’m not clear, I will be repeating myself : at relaunch, the GUI popups if previously it could not launch the extension.

I’m having a bad time. Multiple hours trying to figures out when, how, to put the TS_ config variables on MacOS, with lots of undocumented/unexpected/surprising behavior of the daemon, missleading doc…

Maybe I will take the leap of trying the open source tailscale, to possibly have some usable outputs of what’s making the extension process or whatever not even starting.

When I select my mac as an exit node, I’m not able to access to internet nor the resources behind openvpn on my Linux machine. 😞

OK so I circled back to the App Store and standalone, and neither works as intented.

I re-circled back to the OSS which is the only one working as intented with TS_BIND_TO_INTERFACE_BY_ROUTE=true.

Thanks for reading my diary, I will now go sit in my room corner to cry and reflect about my life choices.

@spacelatte (OP) : You precised that you used the AppStore variant. You are now permitted to have regrets.

Hey there, you’re absolutely right about the VPNs not playing nice with each-other (obviously) 😃

This may be an expected behavior, just wanted to share… (also if people are trying to get a similar setup here)

Basically, the question is;

  • Should Tailscale respect the route-table of the host while being subnet-router or an exit node,
  • or
  • Should Tailscale bypass the routing-table of the host and send all traffic to the default gateway?

Regardless of the VPN, I can set following on the macOS subnet-router/exit-node;

route add "1.2.3.4/32" "192.168.1.99" -hopcount 1 -nostatic

Therefore, I expected packets to go out to C. However, due to some design decisions, it is fair to have B.


flowchart TD
  entry[A] --> client[Client]
  client --> macos[macOS GW]
  macos --> int[Internet]
  macos --> route[Route]
  int --> out-b[B]
  route --> out-c[C]

Sorry for the unusual problem & explanation 😃 I’m ok to close this if it’s a too much of an edge-case… 😃