tailscale: Fedora default `firewalld` rules block Tailscale forwarding (exit node, subnet routing)

What is the issue?

Hello Tailscale team,

I’m splitting out a new issue from #2654. I’m still experiencing the issue of a Tailscale exit node on Fedora not forwarding any traffic, not even pings (so not a DNS issue). Extending @bexelbie’s investigation in that issue got me to discover that Fedora’s default firewalld configuration places the tailscale0 interface in the default zone, which prohibits IP forwarding.

Steps to reproduce

  1. Turn forwarding on: net.ipv4.ip_forward = 1, net.ipv6.conf.all.forwarding = 1 in /etc/sysctl.conf
  2. tailscale up --advertise-exit-node
  3. ip a: the tailscale0 interface is 100.103.143.10 and the LAN interface is 192.168.12.183
  4. firewall-cmd --zone=FedoraWorkstation --list-interfaces: tailscale0 is in the FedoraWorkstation zone
  5. nft list ruleset: see below, observe that the hook forward chain in table inet firewalld has a higher priority than Tailscale’s chain and rejects the packet
  6. Enable the exit node in the Tailscale web UI
  7. On client, select the exit node
  8. ping 100.103.143.10, the exit node’s tailscale0 interface. Observe ping reply
  9. ping 192.168.0.25, the exit node’s LAN interface. Observe ping reply
  10. ping 1.1.1.1, on the Internet via the exit node. Observe reply from 100.103.143.10: Destination net unreachable
  11. On the exit node, firewall-cmd --zone=trusted --change-interface=tailscale0: moves tailscale0 to the trusted zone
  12. The firewalld reload eats Tailscale’s nftables tables completely, so do tailscale down and tailscale up --advertise-exit-node again
  13. nft list ruleset: see below, observe that in the firewalld chains, the forwarded packets to output interface tailscale0 are now accept, allowing them to fall through to Tailscale’s chains
  14. Wait ten seconds …
  15. From client, ping 1.1.1.1. Observe ping reply

Here are the relevant chains from the nftables ruleset, before moving tailscale0 to the trusted zone:

table ip filter {
	chain FORWARD {
		type filter hook forward priority filter; policy accept;
		counter packets 12369 bytes 7336248 jump ts-forward
	}

	chain ts-forward {
		iifname "tailscale0" counter packets 7114 bytes 6307854 meta mark set 0x40000 
		mark 0x40000 counter packets 7114 bytes 6307854 accept
		oifname "tailscale0" ip saddr 100.64.0.0/10 counter packets 0 bytes 0 drop
		oifname "tailscale0" counter packets 5255 bytes 1028394 accept
	}
}
table inet firewalld {
	chain filter_FORWARD {
		type filter hook forward priority filter + 10; policy accept;
		ct state { established, related } accept
		ct status dnat accept
		iifname "lo" accept
		ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } reject with icmpv6 addr-unreachable
		jump filter_FORWARD_ZONES
		ct state invalid drop
		reject with icmpx admin-prohibited
	}

	chain filter_FORWARD_POLICIES_pre {
	}

	chain filter_FORWARD_ZONES {
		iifname "tailscale0" goto filter_FWD_FedoraWorkstation
		iifname "wlp1s0" goto filter_FWD_FedoraWorkstation
		goto filter_FWD_FedoraWorkstation
	}

	chain filter_FORWARD_POLICIES_post {
	}

	chain filter_FWD_FedoraWorkstation {
		jump filter_FORWARD_POLICIES_pre
		jump filter_FWD_FedoraWorkstation_pre
		jump filter_FWD_FedoraWorkstation_log
		jump filter_FWD_FedoraWorkstation_deny
		jump filter_FWD_FedoraWorkstation_allow
		jump filter_FWD_FedoraWorkstation_post
		jump filter_FORWARD_POLICIES_post
		reject with icmpx admin-prohibited
	}

	chain filter_FWD_FedoraWorkstation_pre {
	}

	chain filter_FWD_FedoraWorkstation_log {
	}

	chain filter_FWD_FedoraWorkstation_deny {
	}

	chain filter_FWD_FedoraWorkstation_allow {
	}

	chain filter_FWD_FedoraWorkstation_post {
	}
}

And here’s the same after moving tailscale0 to the trusted zone (note the oifname "tailscale0" accept line):

table ip filter {
	chain FORWARD {
		type filter hook forward priority filter; policy accept;
		counter packets 12749 bytes 7385739 jump ts-forward
	}

	chain ts-forward {
		iifname "tailscale0" counter packets 7363 bytes 6335629 meta mark set 0x40000 
		mark 0x40000 counter packets 7363 bytes 6335629 accept
		oifname "tailscale0" ip saddr 100.64.0.0/10 counter packets 0 bytes 0 drop
		oifname "tailscale0" counter packets 5386 bytes 1050110 accept
	}
}
table inet firewalld {
	chain filter_FORWARD {
		type filter hook forward priority filter + 10; policy accept;
		ct state { established, related } accept
		ct status dnat accept
		iifname "lo" accept
		ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } reject with icmpv6 addr-unreachable
		jump filter_FORWARD_ZONES
		ct state invalid drop
		reject with icmpx admin-prohibited
	}

	chain filter_FORWARD_POLICIES_pre {
	}

	chain filter_FORWARD_ZONES {
		iifname "tailscale0" goto filter_FWD_trusted
		iifname "wlp1s0" goto filter_FWD_FedoraWorkstation
		goto filter_FWD_FedoraWorkstation
	}

	chain filter_FORWARD_POLICIES_post {
	}

	chain filter_FWD_FedoraWorkstation {
		jump filter_FORWARD_POLICIES_pre
		jump filter_FWD_FedoraWorkstation_pre
		jump filter_FWD_FedoraWorkstation_log
		jump filter_FWD_FedoraWorkstation_deny
		jump filter_FWD_FedoraWorkstation_allow
		jump filter_FWD_FedoraWorkstation_post
		jump filter_FORWARD_POLICIES_post
		reject with icmpx admin-prohibited
	}

	chain filter_FWD_FedoraWorkstation_pre {
	}

	chain filter_FWD_FedoraWorkstation_log {
	}

	chain filter_FWD_FedoraWorkstation_deny {
	}

	chain filter_FWD_FedoraWorkstation_allow {
	}

	chain filter_FWD_FedoraWorkstation_post {
	}

	chain filter_FWD_trusted {
		jump filter_FORWARD_POLICIES_pre
		jump filter_FWD_trusted_pre
		jump filter_FWD_trusted_log
		jump filter_FWD_trusted_deny
		jump filter_FWD_trusted_allow
		jump filter_FWD_trusted_post
		jump filter_FORWARD_POLICIES_post
		accept
	}

	chain filter_FWD_trusted_pre {
	}

	chain filter_FWD_trusted_log {
	}

	chain filter_FWD_trusted_deny {
	}

	chain filter_FWD_trusted_allow {
		oifname "tailscale0" accept
	}

	chain filter_FWD_trusted_post {
	}
}

Proposed solution

Add the Tailscale interface to the trusted firewalld zone, or advise Fedora users to do so manually.

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

Fedora Workstation 35

Tailscale version

1.24.2

Bug report

BUG-678e86f9848a0db601cdddf4a45c5646b7904f0ed4aae82f95707f4f172686df-20220508054813Z-4f5a6aa9ff539e00

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Reactions: 1
  • Comments: 15

Most upvoted comments

I do the following on CentOS/Fedora/EL

# firewall-cmd --add-masquerade --zone=public --permanent 
# firewall-cmd --add-interface=tailscale0 --zone=trusted --permanent
# firewall-cmd --reload

to make it work with firewalld

This is what worked for me - at some point the commands for setting up firewalld for routing changed.

I used https://forums.rockylinux.org/t/configure-firewalld-for-routing-traffic-between-internal-and-external/7384/2, and I’m running the machine as a router using these instructions anyway. so there might be some addtional steps involved

First, you’ll want to enable packet forwarding. Edit /etc/sysctl.conf and uncomment net.ipv4.ip_forward=1 and net.ipv6.conf.all.forwarding = 1 . run sysctl -p as root or sudo to activate these changes

I’m setting up firewalld to run tailscale as its own zone (so in theory you can set its own firewall rules, and in practice it dosen’t clobber my main network)

I’m assuming wan as the wan interface, and that its already set to the external zone, and able to connect out . If not you can set it as such with firewall-cmd --zone=external --add-interface=wan --permanent I’ve also set it to be the default zone.

Assuming those things- the following firewall-cmd commands should work.

firewall-cmd --permanent --new-zone=tailscale
firewall-cmd --zone=tailscale --add-interface=tailscale0  --permanent
firewall-cmd --new-policy tailscaleegress --permanent
firewall-cmd --policy tailscaleegress --add-ingress-zone=tailscale --permanent
firewall-cmd --policy tailscaleegress --add-egress-zone=external --permanent
firewall-cmd --policy tailscaleegress --set-target=ACCEPT --permanent

I could connect to the internet via my exit node after running this:

sudo firewall-cmd --permanent --add-interface=tailscale0 --zone=trusted
# sudo firewall-cmd --permanent --add-masquerade  # I don't know if this one is necessary
sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv6 masquerade'
sudo firewall-cmd --permanent --add-masquerade --zone public
sudo systemctl restart firewalld

im still having problems when following this, it does seem to be a nf_tables issues however after testing an older rocky instance

I actually experienced this issue working fine “by default” on Fedora Workstation, but not on OpenSUSE Tumbleweed

I put “by default” in quotes because I started this Fedora install with Fedora 34 and having been rolling forward since, so defaults could have changed but to my knowledge I have never touched the firewall settings. The OpenSUSE Tumbleweed install was very recent; I do not recall which install options I selected but I tend to stick to defaults for stuff I am not familiar with, like firewalls.

Note that I was not attempting exit node or subnet routing work, I was attempting to access a service across the tailnet

With Tailscale running on both machines: I was able to reach an echo server I was running on 0.0.0.0:8080 on Fedora, from the OpenSUSE machine I was not able to reach the same service running on the OpenSUSE machine, from any other machine

In all cases the service appeared in the Tailscale Services list

Fedora output which was already working:

sudo firewall-cmd --info-zone FedoraWorkstation
FedoraWorkstation (active)
  target: default
  icmp-block-inversion: no
  interfaces: vpn0 wlp0s20f3
  sources: 
  services: dhcpv6-client mdns samba-client ssh
  ports: 1025-65535/udp 1025-65535/tcp
  protocols: 
  forward: no
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

OpenSUSE output which was not working:

sudo firewall-cmd --info-zone public
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: wlp1s0
  sources: 
  services: dhcpv6-client ssh
  ports:
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

The only things differing were that Fedora had the ports open, and forward: no, while OpenSUSE had no ports open and forward: yes

Opening the UDP port on the OpenSUSE public zone did not fix it, I assume because I was running an HTTP server? Opening the TCP port did fix it

Having forward yes vs. no did not matter on OpenSUSE, though if I understand how that works, forward: no may have still blocked things if I wanted the requests to be let through to the docker firewall zone

Here’s the beginning of the nearly 10-year-old thread on Fedora mailing list with a bunch of extensive arguing about this policy: https://lists.fedoraproject.org/pipermail/devel/2014-December/205010.html

Adding some notes here:

sudo firewall-cmd --zone=internal --change-interface=tailscale0

(FYI I use the internal zone not ‘trusted’ but same idea)

and then:

sudo tailscale down && sudo tailscale up

is indeed the fix, and the extra aggravating part comes when you need to reboot – these commands have to be run every time.