tailscale: Incompatibility with Linux 6.11.{4,5} and 6.6.{57,58} due to `ip6tables` error
What is the issue?
I updated to Linux 6.11.4 (on NixOS) and now tailscale status reports the following:
# Health check:
# - adding [-i tailscale0 -j MARK --set-mark 0x40000/0xff0000] in v6/filter/ts-forward: running [/nix/store/zpl4wlvc9a4ziq7b6ccrpxzn5mwc3frn-iptables-1.8.10/bin/ip6tables -t filter -A ts-forward -i tailscale0 -j MARK --set-mark 0x40000/0xff0000 --wait]: exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
ip6tables v1.8.10 (nf_tables): MARK: bad value for option "--set-mark", or out of range (0-4294967295).
Try `ip6tables -h' or 'ip6tables --help' for more information.
As a result, many things don’t work correctly, in particular MagicDNS.
Steps to reproduce
- Make a NixOS system with kernel 6.11.4 (
boot.kernelPackages = pkgs.linuxPackages_latestat time of issue filing). I assume other distros with 6.11.4 would also exhibit the issue, but I haven’t checked myself. - Install and configure tailscale (
services.tailscale.enable = true,tailscale upetc) - Run
tailscale status
Observed:
# Health check:
# - adding [-i tailscale0 -j MARK --set-mark 0x40000/0xff0000] in v6/filter/ts-forward: running [/nix/store/zpl4wlvc9a4ziq7b6ccrpxzn5mwc3frn-iptables-1.8.10/bin/ip6tables -t filter -A ts-forward -i tailscale0 -j MARK --set-mark 0x40000/0xff0000 --wait]: exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
ip6tables v1.8.10 (nf_tables): MARK: bad value for option "--set-mark", or out of range (0-4294967295).
Try `ip6tables -h' or 'ip6tables --help' for more information.
Expected: No health check warnings.
This was the case with 6.11.3.
Are there any recent changes that introduced the issue?
The changelog for kernel 6.11.4 lists the following commits to the netfilter code.
$ git log v6.11.3..v6.11.4 --oneline | grep netfilter
339dc6c7266c netfilter: fib: check correct rtable in vrf setups
4cdc55ec6222 netfilter: xtables: avoid NFPROTO_UNSPEC where needed
915717e0bb98 netfilter: br_netfilter: fix panic with metadata_dst skb
9f5c115077d3 netfilter: nf_reject: Fix build warning when CONFIG_BRIDGE_NETFILTER=n
a2c6c487ed9c netfilter: nf_nat: don't try nat source port reallocation for reverse dir clash
Of these revisions, netfilter: xtables: avoid NFPROTO_UNSPEC where needed has been confirmed to be the source of this issue.
OS
Linux
OS version
NixOS at revision 0fe3416c7f455a1e7ca6dec3c0b2d1e2cd30d4f6
Tailscale version
1.76.1
Other software
My full NixOS configuration is at https://github.com/philiptaron/flock.nix. There’s nothing fancy there, though; it’s mostly a stock NixOS configuration that has run Tailscale reliably until this kernel upgrade.
Bug report
BUG-a941a9d4060b13bb02ff199d2b96deac67ccf9b681a2dfc2f5ee5576fb062a46-20241019151759Z-9ca281bdd508a563
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 73
- Comments: 115 (10 by maintainers)
Links to this issue
Commits related to this issue
- fix(flake.lock): revert nixpkgs due to kernel bug reference: https://github.com/tailscale/tailscale/issues/13863 — committed to akosnad/nix-config by akosnad 2 years ago
- fix: update nixpkgs, fix https://github.com/tailscale/tailscale/issues/13863 — committed to ryan4yin/nixos-config-sbc by ryan4yin 2 years ago
- tailscale: Update to 1.76.1 **Summary** - [changelog](https://tailscale.com/changelog#2024-10-16-client) - [known issue](https://github.com/tailscale/tailscale/issues/13863) **Test Plan** - tailscal... — committed to nazar-stasiv/packages by nazar-stasiv 2 years ago
- tailscale: Update to 1.76.1 **Summary** - [changelog](https://tailscale.com/changelog#2024-10-16-client) - [known issue](https://github.com/tailscale/tailscale/issues/13863) **Test Plan** - tailscal... — committed to nazar-stasiv/packages by nazar-stasiv 2 years ago
- Pin to kernel 6.11.3 til tailscale/tailscale#13863 Waiting for kernel 6.11.6 — committed to NovaViper/NixConfig by NovaViper 2 years ago
- util/linuxfw: add improved health message for bad kernels Based on recent bug activity it appears a good number of major distros have just shipped Linux 6.8.0, a no longer maintained branch, with bac... — committed to tailscale/tailscale by raggi 2 years ago
- util/linuxfw: add improved health message for bad kernels Based on recent bug activity it appears a good number of major distros have just shipped Linux 6.8.0, a no longer maintained branch, with bac... — committed to tailscale/tailscale by raggi 2 years ago
- util/linuxfw: add improved health message for bad kernels Based on recent bug activity it appears a good number of major distros have just shipped Linux 6.8.0, a no longer maintained branch, with bac... — committed to tailscale/tailscale by raggi 2 years ago
just got this on our 24.04 tailscale subnet routers this fixed it for me
and restart tailscaled systemd service
Yes, the fix is queued for the next stable (also for the other stable series) so we can expect it to be fixed with 6.11.6, 6.6.59 etc.
Linux 6.11.5 and Linux 6.6.58, released today, do not contain the needed fix.
On NixOS, @K900 has cherry-picked the fix into the NixOS 6.11.5 – see the following PR:
This is caused by
0bfcb7b71e73 ("netfilter: xtables: avoid NFPROTO_UNSPEC where needed") and a fix is already in the works: https://lore.kernel.org/all/20241019-xtables-typos-v2-1-6b8b1735dc8e@0upti.me/For now downgrading the kernel or patching it with the above should fix the issue, although I’d expect the issue to be fixed with the next stable kernel 😊
6.6.57 seems affected too
I installed the generic 6.11.0-21 kernel in Ubuntu 24.04 to replace the problematic 6.8.0-1025-aws in EC2:
sudo apt install --install-recommends linux-generic-hwe-24.04This works.Today’s batch of stable kernels contains the fix.
As the issue is fixed upstream and released in Linux 6.11.6 and 6.6.59, I’ll close this.
@denisfalqueto check your kernel version, it’s already fixed in NixOS in kernels 6.6.58 and 6.11.5
on arch this is fixed with
6.11.4-arch2-1Tracked in Fedora Bugzilla at https://bugzilla.redhat.com/show_bug.cgi?id=2321325
Looks like Ubuntu has a fix in place for the Kernel. If you want it early, see here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2104282/comments/5
This is an issue with your kernel vendor and not tailscale (directly), did you already open a bug with ubuntu? This is the same thing that I have commented at an earlier time: https://github.com/tailscale/tailscale/issues/13863#issuecomment-2550948300
Yeah please get in touch with your kernel vendors (ubuntu or oracle) instead of bumping this issue
thanks, I was going to update it.
yeah, was gonna reply myself, but u r fast
Happened again in Ubuntu 24.04 6.8.0-56-generic #58-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 14 15:33:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
good news
unattended upgrades upgraded to AWS kernel 6.8.0-1027-aws on ubuntu 24.04 last night, fixes this issue
don’t need my hack to disable ipv6 anymore
Same issue applied by automatic update for Ubuntu 22.04 on Azure with
kernel 6.8.0-1025-azureyesterday. Not using IPv6 so disabled it to mitigate the issue withsysctl net.ipv6.conf.all.disable_ipv6=1for now.Same error on kernel
6.8.0-57-generic😢This bit me today going from
ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-arm64-server-20250305AMI with the6.8.0-1024-awskernel toubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-arm64-server-20250327with kernel6.8.0-1025-aws.So just a single build version increment.
I ended up having to try a different workaround.
Changing tailscaled to nftables mode seemed to mess up some legacy docker stuff that wants iptables.
Reverting to previous kernel version worked, but then I need to blocklist the bad kernel version.
Disabling IPv6 and restarting tailscaled avoided the issue, I don’t need it.
$ sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1It looks like this bug has hit my Ubuntu 22.04 servers today. Kernel 5.15.0-128-generic
Oracle Cloud Ubuntu-based VMs 24.04 with latest 6.8.0-1023-oracle kernel also trigger the bug. Here is the fix:
Your VM switched to the new Ubuntu kernel because GRUB automatically selects the highest version. Once booted, check:
Should be no more health warning and exit node resume it’s function.
I have mailed members of the AWS kernel team (so far no response), and the GCE kernel lead today (we heard a report they also just released a “6.8” kernel with this backport).
We know exactly where it came from, and we attempted to coordinate with kernel maintainers the day that the first release went out the door. I mailed gregkh about it myself, so did distro kernel maintainers who are here helping.
Unfortunately it appears that some 6 months later the bad patch is turning up in distro releases 6.8 series kernels without the fix applied.
The fix is linked by here: https://github.com/tailscale/tailscale/issues/13863#issuecomment-2424752914 - https://lore.kernel.org/all/20241019-xtables-typos-v3-1-66dd2eaacf2f@0upti.me/
I can confirm that I also have the issue on Ubuntu Server 24.04 - after yesterdays update.
Can confirm that
6.11.6-300.fc41.x86_64fixes the issue on our end.Yeah, no dice here. Still getting the same error.
At least for Fedora and probably Arch, the following drop-in file should resolve the issue for now.
/etc/modules-load.d/tailscale.conf
Happening the same for me in Arch Linux on same kernel. This made the route advertising not working properly
1.5g ago? New time unit 😉
I updated to the kernel using the proposed pocket and can confirm the error is no longer happening
We use Grafana and Prometheus to monitor our Tailscale servers and it has a health stat you can check
Also reported here: https://bugs.launchpad.net/ubuntu/+source/linux-meta-oracle-6.8/+bug/2104178
The temporary fix with set nftables working fine. TS_DEBUG_FIREWALL_MODE=nftables
Commands to downgrade for Ubuntu 24.04
I’m only assuming there will be a proper fix - seems tailscale and/or kernel packages have made some wrong assumptions. This kind of breakage shouldn’t happen on a minor update.
Anyone else getting this on arch again with
6.11.8-arch1-2?Thanks a lot. In F40 my upgrade to kernel 6.11.6-200.fc40.x86_64 fixed this also!
Can confirm, it works on arch with kernel 6.11.6-arch1-1
@Apple2007 It hasn’t been merged yet, hopefully it lands in the next one (as an end-user).
Still having the same issue on Fedora 41 Sever! This is also on kernel version 6.11.5.
Experienced on Fedora as well, reverted and this works now -
For Arch Linux users: rolling back from kernel 6.11.4 to 6.11.3 fixes the issue, e.g. via:
Closing this issue again, as the distros managed to catch up before we could find a workaround that wouldn’t cause regressions for others.
… but this is fixed with more recent versions of the Ubuntu kernel and/or tailscale:
🎉
Yes, use metrics, specifically tailscaled_health_messages https://tailscale.com/kb/1482/client-metrics
The volume of users affected by this issue suggests we definitely need to revisit this to increase automation around addressing it some more.
This is the recommended workaround for now, thank you @thoughtextreme.
Thanks @thoughtextreme that seems to have worked for me. Much appreciated. I didn’t realise it was IPv6 causing the problems here.
Absolutely agree, @e12e. We’ll likely never know where it originated and it doesn’t exactly matter - at least to me. As long as we know what’s causing it and that it gets fixed.
Considering there hasn’t been a comment from Tailscale yet and this is a closed issue from last fall with a 3k item open queue, it seems doubtful they are aware yet. I’m not sure if @philiptaron or @raggi can re-open the issue, or if creating a new one would be better to get attention. As of now I didn’t see anything open.
Thank you - we had this problem in some cloud vms that acts as exit nodes for us.
This messed up routing for our exit nodes, presumably triggered by a routine kernel update.
Workaround helps - but for us that means putting it in cloud-init – which isn’t ideal – as we probably want to avoid this setting when this has a proper fix.
For Ubuntu a new kernel version is released which fixed the problem: 5.15.0-130.140
Thanks a lot! F41 and F40 Silverblue all fixed now!
working on nixos, 6.6.59
Not work on fedora 41
If I manually load those modules on Fedora, Tailscale starts working properly, so adding that drop-in should automate the process of it during boot.
@denisfalqueto The only platform that I know for certain it should work on is Fedora, any other distro I don’t know if it’ll help or not, though I suspect it’ll work on Arch as well based on personal experience.
@ccallahan Does that work? I think the module is loading, but it cannot be used correctly.
apt build-dep linux linux-image-unsigned-$(uname -r)apt-get install build-essential fakeroot linux-source bc kmod cpio flex libncurses5-dev libelf-dev libssl-dev dwarves bisonapt-get install devscriptsapt-get build-dep linuxapt-get source linuxchmod 777 linux_6.11.4-1.dscchmod 777 linux_6.11.4.orig.tar.xzchmod 777 linux_6.11.4-1.debian.tar.xzrm -rf linux-6.11.4/apt-get source linux’cd linux-6.11.4/apply patches from https://github.com/torvalds/linux/commit/306ed1728e8438caed30332e1ab46b28c25fe3d8
cp /lib/modules/6.11.4-amd64/build/.config .cp /lib/modules/6.11.4-amd64/build/Module.symvers .make kernelversion scripts prepare modules_preparemake modules M=net/netfilter -j22after that u need simple manipulations with files (dont ask idk how debian distributes in compressed mode and couldn’t really figure out the wiki is lacking info on what they do and i cba figuring out) such as copying in
/usr/lib/modules/6.11.4-amd64/kernel/net/netfilter/and renaming.ko.xzinto.ko.xz.backupfor example, then runningdepmod -a,modprobe -r xt_markandmodprobe xt_mark(if any process is using it you have to terminate it, or, stop the service), downside obviously annoyment with /etc/modules (which most likely wont load your module (lack of signs and it doesnt expect the module to be in /net/netfilter but in root of kernel folder seems like) etc.) so after reboot the dependent services might not initialize in a correct way, so you have to restart.tl;dr; just wait until they push the fix into upstream
someone can shorten the apt sides, also, in case if you somehow deleted a module from the /usr/lib/modules/…/net/netfilter/ and apt install --reinstall linux-image-6.11.4-amd64 does not help to restore files, try
apt reinstall linux-image-6.11.4-amd64instead xt_mark.ko.zipI’m on NixOS and I had to downgrade to an 6.10.x kernel (pkgs.linuxPackages_6_10).
Thanks for the pointer, Christian (@christian-heusel). I applied the latest version of that patch and I can also confirm that it fixes this issue.