tailscale: macOS: routing via exit-node or subnet-router bypasses routing table
What is the issue?
Hi there,
I opened this as a bug-report but this can also be a “feature” depending on how you look at it…
Long story short, I use a macOS device as a (middle-)gateway.
For example, in the following chart;
- From
client, request to 1.2.3.4/32 routed tointernet(incorrectly) - Should’ve been routed to
vpn. SincemacOSis having specific entry in the route table. - Resulting packet transmission (leak?) to internet/subnet of the gateway.
Packet coming out from
Adestined toCleaves (leaks?) atB
flowchart LR
entry-a[A] --> client[Client 100.65.33.44]
client --> macos[macOS 100.64.11.22]
macos --> internet[Internet 0.0.0.0/0]
macos --> vpn[VPN 1.2.3.4/32]
internet --> out-b[B]
vpn --> out-c[C]
I suppose that could be the case because the VPN is another utun device. (OpenVPN)
PS: I have not tested the use-case on Linux (Linux as a middle-device) yet.
I tried enabling IP Forwarding via sudo sysctl -w net.inet.ip.forwarding=1
This may be related to the option --snat-subnet-routes in macOS…
Steps to reproduce
- Create a subnet-router or exit-node on macOS
- Connect macOS to another VPN
- Try to access a resource behind the VPN using another device than macOS through Tailscale.
Are there any recent changes that introduced the issue?
No response
OS
macOS
OS version
13.2.1
Tailscale version
1.38.1 (AppStore)
Other software
- OpenVPN
Bug report
BUG-baed9af847873e680716aa6712227c7b3893b7bbed63f3be337f4f6b7d524204-20230320232923Z-bbf24661ad3d52b1
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 2
- Comments: 18 (4 by maintainers)
Can you give an example of how this capability be configured in the policy file json for a specific Mac host?
You may be interested in the (still experimental)
https://tailscale.com/cap/bind-to-interface-by-routenode attribute that you can add to your policy file.That will cause the macOS client to take into account the routing table when deciding what interface to send them on, and presumably it should then choose the other VPN’s interface (see #5719 and #7065 for more details).
Furthermore :
Considering that AppStore/Standalone uses different extension ( https://tailscale.com/kb/1065/macos-variants ):
Network ExtensionSystem ExtensionI just tried with thestandalonevariant. IT ALSO DOES NOT WORK when usingTS_BIND_TO_INTERFACE_BY_ROUTE=y(the extension is also refusing to start).Also I observed that the
standalonevariant has more pleasant behavior :tailscale downor disabling the GUI’s green button) : the IPNExtension process does not exit. It takes right-click-exit on the GUI menu bar to make the extension process exit.[...].macsys.network-extensionprocess also exitThe fact that the “extension process” exit “when stopping” is pleasant when trying to mangle up with configuration items in
tailscaled-env.txt; It takes more steps to apply modification to the “App Store” version : I had to quit/relaunch the GUI.Also the doc of https://github.com/tailscale/tailscale/blob/main/envknob/envknob.go is MISSLEADING about the location of
tailscaled-env.txtfor theSTANDALONEGUI version ;$USER/+Library/Containers/$VARIANT/Data/tailscaled-env.txtVARIANT=io.tailscale.ipn.macos.network-extension: yes !VARIANT=io.tailscale.ipn.macsys: WRONGWhat’s wrong (x2) about the STANDALONE location of
tailscaled-env.txt:$USER/Library/...but instead underroot/Library/...(the standalone’s “extension process” runs as root, whereas the App Store run… I won’t trackback, but its cwd() was under$USER/....)VARIANT=io.tailscale.ipn.macsys(which is even more missleading because it can be found under$USER/Library/Containers) but its in realityVARIANT=io.tailscale.ipn.macsys.network-extensionwhich is found-able only underroot/...Of course
[sudo] lsof -d cwd | grep PICwas used to find the cwd()'s.Location of
tailscaled-env.txtwas HORRIBLE to handle. I just hardlinked the (now 5) places were it could exists. Like/etc/tailscale/tailscaled-env.txt+ {/var/root,$USER}/Library/Containers/io.tailscale.ipn.*/Data/tailscaled-env.txt, and since they were hardlinked (not symlinked) due to plausible containerization, I didnt want to edit theENV=VALUEwith an editor (to not un-hardlink) and had to usetruncate -s 0+echo xxx >> …` which behavior is without surprise.Also when we quit the GUI (at least the stadlone one), and it previsouly didn’t successfully launched the “extension”, when we… empty the
tailscaled-env.txtfile, and relaunch the GUI, the GUI will falsely popup that “the network extension failed to start”, which is a truth only in the past, the GUI is reporting a knowledge of the past, it should better just not says something is does not know, and should instead report only truth of the present, which, if it did try to launch again the “extension”, it could have found that now it does. If I’m not clear, I will be repeating myself : at relaunch, the GUI popups if previously it could not launch the extension.I’m having a bad time. Multiple hours trying to figures out when, how, to put the
TS_config variables on MacOS, with lots of undocumented/unexpected/surprising behavior of the daemon, missleading doc…Maybe I will take the leap of trying the open source tailscale, to possibly have some usable outputs of what’s making the extension process or whatever not even starting.
When I select my mac as an exit node, I’m not able to access to internet nor the resources behind openvpn on my Linux machine. 😞
OK so I circled back to the
App Storeandstandalone, and neither works as intented.I re-circled back to the OSS which is the only one working as intented with
TS_BIND_TO_INTERFACE_BY_ROUTE=true.Thanks for reading my diary, I will now go sit in my room corner to cry and reflect about my life choices.
@spacelatte (OP) : You precised that you used the AppStore variant. You are now permitted to have regrets.
Hey there, you’re absolutely right about the VPNs not playing nice with each-other (obviously) 😃
This may be an expected behavior, just wanted to share… (also if people are trying to get a similar setup here)
Basically, the question is;
Regardless of the VPN, I can set following on the macOS subnet-router/exit-node;
route add "1.2.3.4/32" "192.168.1.99" -hopcount 1 -nostaticTherefore, I expected packets to go out to
C. However, due to some design decisions, it is fair to haveB.Sorry for the unusual problem & explanation 😃 I’m ok to close this if it’s a too much of an edge-case… 😃