tailscale: Tailscale is slow: `wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument`

What is the issue?

Hello,

I have a problem. For a few days now, Tailscale has been extremely slow and the following message keeps appearing in the log:

Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: [RATELIMIT] format("wg: Failed to write packets to TUN device: %v")
Aug 05 23:46:09 ciel tailscaled[876]: [RATELIMIT] format("wg: Failed to write packets to TUN device: %v") (30 dropped)
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: [RATELIMIT] format("wg: Failed to write packets to TUN device: %v")

By slow I mean the following behavior (iperf3 is running on my tailscale exit node):

$ iperf3 --client vpn-lu.uaru-vimba.ts.net -R
Connecting to host vpn-lu.uaru-vimba.ts.net, port 5201
Reverse mode, remote host vpn-lu.uaru-vimba.ts.net is sending
[  5] local 100.97.92.115 port 44816 connected to 100.110.144.38 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   2.00-3.00   sec   128 KBytes  1.05 Mbits/sec                  
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   4.00-5.00   sec   128 KBytes  1.05 Mbits/sec                  
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   7.00-8.00   sec   128 KBytes  1.05 Mbits/sec                  
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   9.00-10.00  sec   128 KBytes  1.05 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.93  sec   609 KBytes   456 Kbits/sec  282             sender
[  5]   0.00-10.00  sec   512 KBytes   419 Kbits/sec                  receiver

iperf Done.

and for reference the other way around is works:

$ iperf3 --client vpn-lu.uaru-vimba.ts.net
Connecting to host vpn-lu.uaru-vimba.ts.net, port 5201
[  5] local 100.97.92.115 port 39542 connected to 100.110.144.38 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.50 MBytes  21.0 Mbits/sec    1   71.4 KBytes       
[  5]   1.00-2.00   sec  3.62 MBytes  30.4 Mbits/sec    0   99.3 KBytes       
[  5]   2.00-3.00   sec  3.88 MBytes  32.5 Mbits/sec    1   87.2 KBytes       
[  5]   3.00-4.00   sec  3.75 MBytes  31.5 Mbits/sec    1    102 KBytes       
[  5]   4.00-5.00   sec  3.88 MBytes  32.5 Mbits/sec    0    104 KBytes       
[  5]   5.00-6.00   sec  3.75 MBytes  31.5 Mbits/sec    1   93.2 KBytes       
[  5]   6.00-7.00   sec  4.00 MBytes  33.6 Mbits/sec    1   83.6 KBytes       
[  5]   7.00-8.00   sec  3.88 MBytes  32.5 Mbits/sec    0    111 KBytes       
[  5]   8.00-9.00   sec  3.62 MBytes  30.4 Mbits/sec    1   96.9 KBytes       
[  5]   9.00-10.00  sec  3.88 MBytes  32.5 Mbits/sec    1   84.8 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  36.8 MBytes  30.8 Mbits/sec    7             sender
[  5]   0.00-10.93  sec  36.5 MBytes  28.0 Mbits/sec                  receiver

iperf Done.

Steps to reproduce

No idea. I undid all the changes I made to tailscale (changing the systemd unit and disabling the support logs) for testing. It did not help.

OS

Arch Linux

OS version

$ uname -a
Linux ciel 6.6.44-2-lts #1 SMP PREEMPT_DYNAMIC Sat, 03 Aug 2024 18:10:33 +0000 x86_64 GNU/Linux

Tailscale version

$ tailscale version
1.70.0
  tailscale commit: 0e0a212418fbf8243cb3f06634367b61e81ea9db-dirty
  go version: go1.22.5

Bug report

$ tailscale bugreport
BUG-c75d50b30c42938e61b3d6dfd8683ea7f95df6373200ac4dafdee5700dbabca1-20240805234440Z-89d5c5a39dee5ae1

Additional information

Installed via pacman (tailscale package).

I also run a few other things like cjdns, Yggdrasil, libvirt and docker (-> tailscale is not the only tun interface) on the computer:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp1s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 74:5d:22:f4:8d:6b brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7e:1b:d9:a1:a2:d7 brd ff:ff:ff:ff:ff:ff permaddr 04:7b:cb:c7:41:10
    inet 192.168.41.[redacted]/24 brd 192.168.41.255 scope global dynamic noprefixroute wlp2s0
       valid_lft 42776sec preferred_lft 42776sec
    inet6 2a02:3100:[redacted]/64 scope global dynamic noprefixroute 
       valid_lft 232740sec preferred_lft 146340sec
    inet6 fd5f:28a6:[redacted]/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::[redacted]/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1304 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet6 fcf6:773c:[redacted]/8 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::8f97:eaaa:dd2:ee16/64 scope link stable-privacy proto kernel_ll 
       valid_lft forever preferred_lft forever
5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:b2:fa:7f brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.1/24 brd 192.168.100.255 scope global virbr0
       valid_lft forever preferred_lft forever
    inet6 fd6c:ffa:eb7a:74e2::1/64 scope global tentative 
       valid_lft forever preferred_lft forever
6: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 100.97.92.115/32 scope global tailscale0
       valid_lft forever preferred_lft forever
    inet6 fd7a:115c:a1e0::8e21:5c73/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::7986:2c09:c665:1b26/64 scope link stable-privacy proto kernel_ll 
       valid_lft forever preferred_lft forever
7: tun1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 65535 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet6 200:[redacted]/7 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::b56c:e008:2405:5b50/64 scope link stable-privacy proto kernel_ll 
       valid_lft forever preferred_lft forever
9: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:70:f9:2d:b1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.128.1/24 brd 192.168.128.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 dd64:6f63:6b65:72::1/80 scope global tentative 
       valid_lft forever preferred_lft forever

strace (when doing iperf3):

[pid 15854] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\3@\0@\6\260\213dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\4@\0@\6\260\212dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\5@\0@\6\260\211dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\6@\0@\6\260\210dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15826] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\7@\0@\6\260\207dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15864] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\10@\0@\6\260\206dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15826] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\t@\0@\6\253\255dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15826] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\v@\0@\6\253\253dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15826] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\r@\0@\6\253\251dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15826] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\17@\0@\6\253\247dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15864] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\21@\0@\6\260}dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\22@\0@\6\253\244dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15864] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\24@\0@\6\253\242dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15864] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\26@\0@\6\260xdn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\27@\0@\6\260wdn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\30@\0@\6\260vdn\220&da\\s\24Q"..., 1290) = 1290
[pid 15826] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\31@\0@\6\260udn\220&da\\s\24Q"..., 1290) = 1290
[pid 15854] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\32@\0@\6\260tdn\220&da\\s\24Q"..., 1290) = 1290
[pid 15854] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\33@\0@\6\260sdn\220&da\\s\24Q"..., 1290) = 1290

If needed, I can also send the output of sudo strace -f -o strace.log tailscaled -verbose 1 > tailscale.log 2>&1 privately.

Another observation that might be interesting: I have a second laptop with pretty much the same setup (Arch Linux + Tailscale). I hadn’t updated it for a few days / weeks, so kernel 6.6.35-2-lts was still installed. Tailscale worked there. I have now updated it to kernel 6.6.44-2-lts and since then the same problem occurs.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 37 (16 by maintainers)

Commits related to this issue

Most upvoted comments

https://github.com/jwhited/tun-einval-repro contains a simplified reproduction of the issue. This writes a GSO_TCPv4 packet to a TUN device w/GSO=1240 and 2 equal length segments.

6.10.2

jwhited@i5-12400-1:~$ uname -a && sudo ./tun-einval-repro
Linux i5-12400-1 6.10.2-061002-generic #202407271100 SMP PREEMPT_DYNAMIC Sat Jul 27 11:37:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
2024/08/07 18:10:24 successfully wrote packet

6.10.3

jwhited@i5-12400-2:~$ uname -a && sudo ./tun-einval-repro
Linux i5-12400-2 6.10.3-061003-generic #202408030740 SMP PREEMPT_DYNAMIC Sat Aug  3 07:53:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
2024/08/07 18:10:17 error writing packet to TUN: write /dev/net/tun: invalid argument

6.11-rc2

jwhited@i5-12400-1:~$ uname -a && sudo ./tun-einval-repro
Linux i5-12400-1 6.11.0-061100rc2-generic #202408042216 SMP PREEMPT_DYNAMIC Sun Aug  4 22:30:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
2024/08/07 18:23:48 successfully wrote packet
% git log --pretty=oneline v6.10.3...v6.10.2 | grep 'net: missing check virtio'
e9164903b8b303c34723177b02fe91e49e3c4cd7 net: missing check virtio
% git log --pretty=oneline v6.10.3...v6.10.2 | grep 'net: drop bad gso csum_start and offset in virtio_net_hdr'
%
% git log --pretty=oneline v6.11-rc2...v6.10.3 | grep 'net: drop bad gso csum_start and offset in virtio_net_hdr'
89add40066f9ed9abe5f7f886fe5789ff7e0c50e net: drop bad gso csum_start and offset in virtio_net_hdr
%

It looks like 6.10.5 patches this?

https://lwn.net/Articles/985670/

Willem de Bruijn (1): net: drop bad gso csum_start and offset in virtio_net_hdr

Fedora Testing Update tree: sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-cb22d42b54

Closing this issue as the client workaround has been present since Tailscale v1.74, and Linux has landed the fix across 6.6, 6.1, and 5.15 LTS releases.

I seems to work (-> no longer slow, no error messages in log anymore) on the latest linux-lts (6.6.46-1-lts).

So from my side the issue could be closed. Thanks for the quick processing of the issue!

$ uname -a
Linux ciel 6.6.44-3.1-lts #1 SMP PREEMPT_DYNAMIC Tue, 06 Aug 2024 23:09:53 +0000 x86_64 GNU/Linux
$ sudo ping -f -c1000 vpn-lu.uaru-vimba.ts.net
PING vpn-lu.uaru-vimba.ts.net (100.110.144.38) 56(84) bytes of data.
           
--- vpn-lu.uaru-vimba.ts.net ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 18152ms
rtt min/avg/max/mdev = 19.863/37.090/185.793/36.109 ms, pipe 11, ipg/ewma 18.170/21.747 ms
$ sudo ping -f -c1000 meyrin.uaru-vimba.ts.net
PING meyrin.uaru-vimba.ts.net (100.69.197.8) 56(84) bytes of data.
    
--- meyrin.uaru-vimba.ts.net ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 2686ms
rtt min/avg/max/mdev = 1.781/2.715/55.687/2.309 ms, pipe 4, ipg/ewma 2.688/3.092 ms
$ sudo ping -f -c1000 aurora.uaru-vimba.ts.net
PING aurora.uaru-vimba.ts.net (100.104.102.72) 56(84) bytes of data.
                                           
--- aurora.uaru-vimba.ts.net ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 15643ms
rtt min/avg/max/mdev = 137.426/149.168/710.629/60.473 ms, pipe 43, ipg/ewma 15.659/139.399 ms
$ sudo ping -f -c1000 herzstein.uaru-vimba.ts.net
PING herzstein.uaru-vimba.ts.net (100.72.64.53) 56(84) bytes of data.
                                                                 
--- herzstein.uaru-vimba.ts.net ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 17023ms
rtt min/avg/max/mdev = 256.965/285.424/1077.871/103.334 ms, pipe 65, ipg/ewma 17.039/264.947 ms

Mhh, I don’t see any increased packet loss now (after downloading the fixed kernel).

I have now also poked the stable team to apply this to the stable trees: https://lore.kernel.org/all/146d2c9f-f2c3-4891-ac48-a3e50c863530@heusel.eu/

The issue is though that the fix does not apply cleanly to the 6.1 series, so if someone wants to provide a resolution for that this would be cool, I can take care of the submission then if that’s of any help:

Click to see merge conflict
diff --cc include/linux/virtio_net.h
index 29b19d0a324c,6c395a2600e8..000000000000
--- a/include/linux/virtio_net.h
+++ b/include/linux/virtio_net.h
@@@ -155,9 -156,27 +144,42 @@@ retry
  		unsigned int nh_off = p_off;
  		struct skb_shared_info *shinfo = skb_shinfo(skb);
  
 -		switch (gso_type & ~SKB_GSO_TCP_ECN) {
 -		case SKB_GSO_UDP:
 -			/* UFO may not include transport header in gso_size. */
 +		/* UFO may not include transport header in gso_size. */
 +		if (gso_type & SKB_GSO_UDP)
  			nh_off -= thlen;
++<<<<<<< HEAD
++||||||| parent of 89add40066f9 (net: drop bad gso csum_start and offset in virtio_net_hdr)
++			break;
++		case SKB_GSO_UDP_L4:
++			if (!(hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM))
++				return -EINVAL;
++			if (skb->csum_offset != offsetof(struct udphdr, check))
++				return -EINVAL;
++			if (skb->len - p_off > gso_size * UDP_MAX_SEGMENTS)
++				return -EINVAL;
++			if (gso_type != SKB_GSO_UDP_L4)
++				return -EINVAL;
++			break;
++		}
++=======
+ 			break;
+ 		case SKB_GSO_UDP_L4:
+ 			if (!(hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM))
+ 				return -EINVAL;
+ 			if (skb->csum_offset != offsetof(struct udphdr, check))
+ 				return -EINVAL;
+ 			if (skb->len - p_off > gso_size * UDP_MAX_SEGMENTS)
+ 				return -EINVAL;
+ 			if (gso_type != SKB_GSO_UDP_L4)
+ 				return -EINVAL;
+ 			break;
+ 		case SKB_GSO_TCPV4:
+ 		case SKB_GSO_TCPV6:
+ 			if (skb->csum_offset != offsetof(struct tcphdr, check))
+ 				return -EINVAL;
+ 			break;
+ 		}
++>>>>>>> 89add40066f9 (net: drop bad gso csum_start and offset in virtio_net_hdr)
  
  		/* Kernel has a special handling for GSO_BY_FRAGS. */
  		if (gso_size == GSO_BY_FRAGS)
diff --cc net/ipv4/udp_offload.c
index 794ea24292f6,bc8a9da750fe..000000000000
--- a/net/ipv4/udp_offload.c
+++ b/net/ipv4/udp_offload.c
@@@ -280,6 -278,20 +280,34 @@@ struct sk_buff *__udp_gso_segment(struc
  	if (gso_skb->len <= sizeof(*uh) + mss)
  		return ERR_PTR(-EINVAL);
  
++<<<<<<< HEAD
++||||||| parent of 89add40066f9 (net: drop bad gso csum_start and offset in virtio_net_hdr)
++	if (skb_gso_ok(gso_skb, features | NETIF_F_GSO_ROBUST)) {
++		/* Packet is from an untrusted source, reset gso_segs. */
++		skb_shinfo(gso_skb)->gso_segs = DIV_ROUND_UP(gso_skb->len - sizeof(*uh),
++							     mss);
++		return NULL;
++	}
++
++	if (skb_shinfo(gso_skb)->gso_type & SKB_GSO_FRAGLIST)
++		return __udp_gso_segment_list(gso_skb, features, is_ipv6);
++
++=======
+ 	if (unlikely(skb_checksum_start(gso_skb) !=
+ 		     skb_transport_header(gso_skb)))
+ 		return ERR_PTR(-EINVAL);
+ 
+ 	if (skb_gso_ok(gso_skb, features | NETIF_F_GSO_ROBUST)) {
+ 		/* Packet is from an untrusted source, reset gso_segs. */
+ 		skb_shinfo(gso_skb)->gso_segs = DIV_ROUND_UP(gso_skb->len - sizeof(*uh),
+ 							     mss);
+ 		return NULL;
+ 	}
+ 
+ 	if (skb_shinfo(gso_skb)->gso_type & SKB_GSO_FRAGLIST)
+ 		return __udp_gso_segment_list(gso_skb, features, is_ipv6);
+ 
++>>>>>>> 89add40066f9 (net: drop bad gso csum_start and offset in virtio_net_hdr)
  	skb_pull(gso_skb, sizeof(*uh));
  
  	/* clear destructor to avoid skb_segment assigning it to tail */

Also if you want to test the kernel side of things here are the linux and the linux-lts packages with gregkh/linux@89add40 applied (which fixes the problem according to #13041 (comment)).

linux:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-6.10.3.arch1-2.1-x86_64.pkg.tar.zst

linux-lts:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-lts-6.6.44-3.1-x86_64.pkg.tar.zst

They both fix the problem for me.

Also if you want to test the kernel side of things here are the linux and the linux-lts packages with https://github.com/gregkh/linux/commit/89add40066f9ed9abe5f7f886fe5789ff7e0c50e applied (which fixes the problem according to https://github.com/tailscale/tailscale/issues/13041#issuecomment-2272249663).

linux:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-6.10.3.arch1-2.1-x86_64.pkg.tar.zst

linux-lts:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-lts-6.6.44-3.1-x86_64.pkg.tar.zst

I was able to partially reproduce this, specifically:

  • Using arch LTS Linux archlinux 6.6.44-2-lts #1 SMP PREEMPT_DYNAMIC Sat, 03 Aug 2024 18:10:33 +0000 x86_64 GNU/Linux
  • Observed iperf3 resulting in wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
  • Observed in dmesg: [ 96.578967] ens18: bad gso: type: 4, size: 1348

This appears to be scoped to IPv6.

I was not however able to reproduce this occurring persistently, in my test setup the issue certainly occurred, but it occurred for a short time and then throughput recovered, and the error stopped being logged, both on the tailscaled side and kernel side.

I was not able to reproduce on Linux archlinux 6.10.3-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 04 Aug 2024 05:11:32 +0000 x86_64 GNU/Linux.

We will need to dig into what patches have been ported into LTS, there have been some activities in virtio header handling which we’re initially now concerned about, but upstream has already worked on improvements there, it’s entirely possible LTS brought in a bug that needs to be addressed with further backports.