tailscale: Incompatibility with Linux 6.11.{4,5} and 6.6.{57,58} due to `ip6tables` error

What is the issue?

I updated to Linux 6.11.4 (on NixOS) and now tailscale status reports the following:

# Health check:
#     - adding [-i tailscale0 -j MARK --set-mark 0x40000/0xff0000] in v6/filter/ts-forward: running [/nix/store/zpl4wlvc9a4ziq7b6ccrpxzn5mwc3frn-iptables-1.8.10/bin/ip6tables -t filter -A ts-forward -i tailscale0 -j MARK --set-mark 0x40000/0xff0000 --wait]: exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
ip6tables v1.8.10 (nf_tables): MARK: bad value for option "--set-mark", or out of range (0-4294967295).

Try `ip6tables -h' or 'ip6tables --help' for more information.

As a result, many things don’t work correctly, in particular MagicDNS.

Steps to reproduce

  1. Make a NixOS system with kernel 6.11.4 (boot.kernelPackages = pkgs.linuxPackages_latest at time of issue filing). I assume other distros with 6.11.4 would also exhibit the issue, but I haven’t checked myself.
  2. Install and configure tailscale (services.tailscale.enable = true, tailscale up etc)
  3. Run tailscale status

Observed:

# Health check:
#     - adding [-i tailscale0 -j MARK --set-mark 0x40000/0xff0000] in v6/filter/ts-forward: running [/nix/store/zpl4wlvc9a4ziq7b6ccrpxzn5mwc3frn-iptables-1.8.10/bin/ip6tables -t filter -A ts-forward -i tailscale0 -j MARK --set-mark 0x40000/0xff0000 --wait]: exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
ip6tables v1.8.10 (nf_tables): MARK: bad value for option "--set-mark", or out of range (0-4294967295).

Try `ip6tables -h' or 'ip6tables --help' for more information.

Expected: No health check warnings.

This was the case with 6.11.3.

Are there any recent changes that introduced the issue?

The changelog for kernel 6.11.4 lists the following commits to the netfilter code.

$ git log v6.11.3..v6.11.4 --oneline | grep netfilter
339dc6c7266c netfilter: fib: check correct rtable in vrf setups
4cdc55ec6222 netfilter: xtables: avoid NFPROTO_UNSPEC where needed
915717e0bb98 netfilter: br_netfilter: fix panic with metadata_dst skb
9f5c115077d3 netfilter: nf_reject: Fix build warning when CONFIG_BRIDGE_NETFILTER=n
a2c6c487ed9c netfilter: nf_nat: don't try nat source port reallocation for reverse dir clash

Of these revisions, netfilter: xtables: avoid NFPROTO_UNSPEC where needed has been confirmed to be the source of this issue.

OS

Linux

OS version

NixOS at revision 0fe3416c7f455a1e7ca6dec3c0b2d1e2cd30d4f6

Tailscale version

1.76.1

Other software

My full NixOS configuration is at https://github.com/philiptaron/flock.nix. There’s nothing fancy there, though; it’s mostly a stock NixOS configuration that has run Tailscale reliably until this kernel upgrade.

Bug report

BUG-a941a9d4060b13bb02ff199d2b96deac67ccf9b681a2dfc2f5ee5576fb062a46-20241019151759Z-9ca281bdd508a563

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 73
  • Comments: 115 (10 by maintainers)

Commits related to this issue

Most upvoted comments

just got this on our 24.04 tailscale subnet routers this fixed it for me

x@y:/etc/default$ sudo git diff HEAD tailscaled
diff --git a/default/tailscaled b/default/tailscaled
index e8384a4..facbfdf 100644
--- a/default/tailscaled
+++ b/default/tailscaled
@@ -6,3 +6,7 @@ PORT="41641"
 
 # Extra flags you might want to pass to tailscaled.
 FLAGS=""
+
+TS_DEBUG_FIREWALL_MODE=nftables

and restart tailscaled systemd service

Yes, the fix is queued for the next stable (also for the other stable series) so we can expect it to be fixed with 6.11.6, 6.6.59 etc.

Linux 6.11.5 and Linux 6.6.58, released today, do not contain the needed fix.

On NixOS, @K900 has cherry-picked the fix into the NixOS 6.11.5 – see the following PR:

This is caused by 0bfcb7b71e73 ("netfilter: xtables: avoid NFPROTO_UNSPEC where needed") and a fix is already in the works: https://lore.kernel.org/all/20241019-xtables-typos-v2-1-6b8b1735dc8e@0upti.me/

For now downgrading the kernel or patching it with the above should fix the issue, although I’d expect the issue to be fixed with the next stable kernel 😊

6.6.57 seems affected too

I installed the generic 6.11.0-21 kernel in Ubuntu 24.04 to replace the problematic 6.8.0-1025-aws in EC2: sudo apt install --install-recommends linux-generic-hwe-24.04 This works.

Today’s batch of stable kernels contains the fix.

As the issue is fixed upstream and released in Linux 6.11.6 and 6.6.59, I’ll close this.

@denisfalqueto check your kernel version, it’s already fixed in NixOS in kernels 6.6.58 and 6.11.5

on arch this is fixed with 6.11.4-arch2-1

Looks like Ubuntu has a fix in place for the Kernel. If you want it early, see here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2104282/comments/5

This is an issue with your kernel vendor and not tailscale (directly), did you already open a bug with ubuntu? This is the same thing that I have commented at an earlier time: https://github.com/tailscale/tailscale/issues/13863#issuecomment-2550948300

no issues on Ubuntu 22.04 6.8.0-1018-oracle but still have issue on Ubuntu 20.04 5.15.0-1072-oracle

Yeah please get in touch with your kernel vendors (ubuntu or oracle) instead of bumping this issue

Linux 6.11.5 and Linux 6.6.58, released today, do not contain the needed fix.

thanks, I was going to update it.

on arch this is fixed with 6.11.4-arch2-1

yeah, was gonna reply myself, but u r fast

Happened again in Ubuntu 24.04 6.8.0-56-generic #58-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 14 15:33:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

good news
unattended upgrades upgraded to AWS kernel 6.8.0-1027-aws on ubuntu 24.04 last night, fixes this issue
don’t need my hack to disable ipv6 anymore

$ uname -r
6.8.0-1027-aws

$ lsb_release -d
No LSB modules are available.
Description:	Ubuntu 24.04.2 LTS

$ tailscale status | grep -A50 -F 'Health check:'
# Health check:
#     - Some peers are advertising routes but --accept-routes is false

$ tailscale version
1.82.0
  tailscale commit: 6676b1261e51e0629553ca06b22e6631f8641100
  other commit: 3ec4bfb9c87718a3806a123585c825189cbceda4
  go version: go1.24.1

$ sysctl net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 0

Same issue applied by automatic update for Ubuntu 22.04 on Azure with kernel 6.8.0-1025-azure yesterday. Not using IPv6 so disabled it to mitigate the issue with sysctl net.ipv6.conf.all.disable_ipv6=1 for now.

Same error on kernel 6.8.0-57-generic 😢

This bit me today going from ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-arm64-server-20250305 AMI with the 6.8.0-1024-aws kernel to ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-arm64-server-20250327 with kernel 6.8.0-1025-aws.

So just a single build version increment.

I ended up having to try a different workaround.
Changing tailscaled to nftables mode seemed to mess up some legacy docker stuff that wants iptables.
Reverting to previous kernel version worked, but then I need to blocklist the bad kernel version.
Disabling IPv6 and restarting tailscaled avoided the issue, I don’t need it.
$ sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1

It looks like this bug has hit my Ubuntu 22.04 servers today. Kernel 5.15.0-128-generic

Oracle Cloud Ubuntu-based VMs 24.04 with latest 6.8.0-1023-oracle kernel also trigger the bug. Here is the fix:

sudo apt install --install-recommends linux-generic-hwe-24.04
sudo reboot

Your VM switched to the new Ubuntu kernel because GRUB automatically selects the highest version. Once booted, check:

uname -r
6.11.0-21-generic
sudo tailscale status

Should be no more health warning and exit node resume it’s function.

I have mailed members of the AWS kernel team (so far no response), and the GCE kernel lead today (we heard a report they also just released a “6.8” kernel with this backport).

Absolutely agree, @e12e. We’ll likely never know where it originated and it doesn’t exactly matter - at least to me. As long as we know what’s causing it and that it gets fixed.

We know exactly where it came from, and we attempted to coordinate with kernel maintainers the day that the first release went out the door. I mailed gregkh about it myself, so did distro kernel maintainers who are here helping.

Unfortunately it appears that some 6 months later the bad patch is turning up in distro releases 6.8 series kernels without the fix applied.

The fix is linked by here: https://github.com/tailscale/tailscale/issues/13863#issuecomment-2424752914 - https://lore.kernel.org/all/20241019-xtables-typos-v3-1-66dd2eaacf2f@0upti.me/

I can confirm that I also have the issue on Ubuntu Server 24.04 - after yesterdays update.

Can confirm that 6.11.6-300.fc41.x86_64 fixes the issue on our end.

At least for Fedora and probably Arch, the following drop-in file should resolve the issue for now.

/etc/modules-load.d/tailscale.conf

# Load needed kernel modules to get around Fedora BZ#2321325
xt_connmark
xt_multiport
xt_comment
xt_mark

Yeah, no dice here. Still getting the same error.

At least for Fedora and probably Arch, the following drop-in file should resolve the issue for now.

/etc/modules-load.d/tailscale.conf

# Load needed kernel modules to get around Fedora BZ#2321325
xt_connmark
xt_multiport
xt_comment
xt_mark

Happening the same for me in Arch Linux on same kernel. This made the route advertising not working properly

The arm kernels for AWS also hit the repos about 1.5g ago

1.5g ago? New time unit 😉

I updated to the kernel using the proposed pocket and can confirm the error is no longer happening

Is there an API where we can programmatically get the output of the tailscaled status healthchecks?

Being able to monitor this would have helped us to identify this issue much faster.

We use Grafana and Prometheus to monitor our Tailscale servers and it has a health stat you can check

Image

Also reported here: https://bugs.launchpad.net/ubuntu/+source/linux-meta-oracle-6.8/+bug/2104178

The temporary fix with set nftables working fine. TS_DEBUG_FIREWALL_MODE=nftables

Commands to downgrade for Ubuntu 24.04

sudo apt remove linux-image-6.8.0-56-generic
yes
no
sudo update-grub
sudo reboot

I’m only assuming there will be a proper fix - seems tailscale and/or kernel packages have made some wrong assumptions. This kind of breakage shouldn’t happen on a minor update.

Anyone else getting this on arch again with 6.11.8-arch1-2?

Thanks a lot. In F40 my upgrade to kernel 6.11.6-200.fc40.x86_64 fixed this also!

Can confirm, it works on arch with kernel 6.11.6-arch1-1

@Apple2007 It hasn’t been merged yet, hopefully it lands in the next one (as an end-user).

Still having the same issue on Fedora 41 Sever! This is also on kernel version 6.11.5.

# Health check:
#     - adding [-i tailscale0 -j MARK --set-mark 0x40000/0xff0000] in v6/filter/ts-forward: running [/usr/sbin/ip6tables -t filter -A ts-forward -i tailscale0 -j MARK --set-mark 0x40000/0xff0000 --wait]: exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
ip6tables v1.8.10 (nf_tables): MARK: bad value for option "--set-mark", or out of range (0-4294967295).

Try `ip6tables -h' or 'ip6tables --help' for more information.

Experienced on Fedora as well, reverted and this works now -

Linux Fedora-DellG15 6.11.4-201.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Oct 10 22:31:19 UTC 2024 x86_64 GNU/Linux

For Arch Linux users: rolling back from kernel 6.11.4 to 6.11.3 fixes the issue, e.g. via:

$ sudo pacman -U /var/cache/pacman/pkg/linux-6.11.3.arch1-1-x86_64.pkg.tar.zst /var/cache/pacman/pkg/linux-headers-6.11.3.arch1-1-x86_64.pkg.tar.zst

Closing this issue again, as the distros managed to catch up before we could find a workaround that wouldn’t cause regressions for others.

… but this is fixed with more recent versions of the Ubuntu kernel and/or tailscale:

$ uname -a
Linux xxxx 6.8.0-60-generic #63-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 15 19:04:15 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
$ tailscale --version
1.82.5
  tailscale commit: dec88625eafdcac4dfae8f592705919184ec4df7
  other commit: ec2eb973098fbcd878430fcda1496ca04b9b7328
  go version: go1.24.2
$ tailscale status | grep '^#' | wc -l
0

🎉

Is there an API where we can programmatically get the output of the tailscaled status healthchecks?

Being able to monitor this would have helped us to identify this issue much faster.

Yes, use metrics, specifically tailscaled_health_messages https://tailscale.com/kb/1482/client-metrics

The volume of users affected by this issue suggests we definitely need to revisit this to increase automation around addressing it some more.

just got this on our 24.04 tailscale subnet routers this fixed it for me

x@y:/etc/default$ sudo git diff HEAD tailscaled
diff --git a/default/tailscaled b/default/tailscaled
index e8384a4..facbfdf 100644
--- a/default/tailscaled
+++ b/default/tailscaled
@@ -6,3 +6,7 @@ PORT="41641"
 
 # Extra flags you might want to pass to tailscaled.
 FLAGS=""
+
+TS_DEBUG_FIREWALL_MODE=nftables

and restart tailscaled systemd service

This is the recommended workaround for now, thank you @thoughtextreme.

Thanks @thoughtextreme that seems to have worked for me. Much appreciated. I didn’t realise it was IPv6 causing the problems here.

This kind of breakage shouldn’t happen on a minor update.

Absolutely agree, @e12e. We’ll likely never know where it originated and it doesn’t exactly matter - at least to me. As long as we know what’s causing it and that it gets fixed.

Considering there hasn’t been a comment from Tailscale yet and this is a closed issue from last fall with a 3k item open queue, it seems doubtful they are aware yet. I’m not sure if @philiptaron or @raggi can re-open the issue, or if creating a new one would be better to get attention. As of now I didn’t see anything open.

just got this on our 24.04 tailscale subnet routers this fixed it for me

(…)

Thank you - we had this problem in some cloud vms that acts as exit nodes for us.

This messed up routing for our exit nodes, presumably triggered by a routine kernel update.


root@taily:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 24.04.2 LTS
Release:	24.04
Codename:	noble
root@taily:~# uname -a
Linux taily 6.8.0-56-generic #58-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 14 15:33:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
root@taily:~# tailscale version
1.80.3
  tailscale commit: b107adf2b69868e710d109a0a6992c00790b6e8e
  other commit: 9497b39a50d6db0e3e14d8856d22372c91b9953b
  go version: go1.23.5

Workaround helps - but for us that means putting it in cloud-init – which isn’t ideal – as we probably want to avoid this setting when this has a proper fix.

For Ubuntu a new kernel version is released which fixed the problem: 5.15.0-130.140

Thanks a lot! F41 and F40 Silverblue all fixed now!

working on nixos, 6.6.59

 $ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.59, NixOS, 24.11 (Vicuna), 24.11.20241102.7ffd9ae`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.24.9`
 - nixpkgs: `/nix/store/zq2axpgzd5kykk1v446rkffj3bxa2m2h-source`

At least for Fedora and probably Arch, the following drop-in file should resolve the issue for now.

/etc/modules-load.d/tailscale.conf

# Load needed kernel modules to get around Fedora BZ#2321325
xt_connmark
xt_multiport
xt_comment
xt_mark

Not work on fedora 41

@ccallahan Does that work? I think the module is loading, but it cannot be used correctly.

If I manually load those modules on Fedora, Tailscale starts working properly, so adding that drop-in should automate the process of it during boot.

@denisfalqueto The only platform that I know for certain it should work on is Fedora, any other distro I don’t know if it’ll help or not, though I suspect it’ll work on Arch as well based on personal experience.

@ccallahan Does that work? I think the module is loading, but it cannot be used correctly.

apt build-dep linux linux-image-unsigned-$(uname -r) apt-get install build-essential fakeroot linux-source bc kmod cpio flex libncurses5-dev libelf-dev libssl-dev dwarves bison apt-get install devscripts apt-get build-dep linux apt-get source linux chmod 777 linux_6.11.4-1.dsc chmod 777 linux_6.11.4.orig.tar.xz chmod 777 linux_6.11.4-1.debian.tar.xz rm -rf linux-6.11.4/ apt-get source linux’ cd linux-6.11.4/

apply patches from https://github.com/torvalds/linux/commit/306ed1728e8438caed30332e1ab46b28c25fe3d8

cp /lib/modules/6.11.4-amd64/build/.config . cp /lib/modules/6.11.4-amd64/build/Module.symvers . make kernelversion scripts prepare modules_prepare make modules M=net/netfilter -j22

after that u need simple manipulations with files (dont ask idk how debian distributes in compressed mode and couldn’t really figure out the wiki is lacking info on what they do and i cba figuring out) such as copying in /usr/lib/modules/6.11.4-amd64/kernel/net/netfilter/ and renaming .ko.xz into .ko.xz.backup for example, then running depmod -a, modprobe -r xt_mark and modprobe xt_mark (if any process is using it you have to terminate it, or, stop the service), downside obviously annoyment with /etc/modules (which most likely wont load your module (lack of signs and it doesnt expect the module to be in /net/netfilter but in root of kernel folder seems like) etc.) so after reboot the dependent services might not initialize in a correct way, so you have to restart.

tl;dr; just wait until they push the fix into upstream

someone can shorten the apt sides, also, in case if you somehow deleted a module from the /usr/lib/modules/…/net/netfilter/ and apt install --reinstall linux-image-6.11.4-amd64 does not help to restore files, try apt reinstall linux-image-6.11.4-amd64 instead xt_mark.ko.zip

I’m on NixOS and I had to downgrade to an 6.10.x kernel (pkgs.linuxPackages_6_10).

Thanks for the pointer, Christian (@christian-heusel). I applied the latest version of that patch and I can also confirm that it fixes this issue.