tailscale: Tailscale is slow: `wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument`
What is the issue?
Hello,
I have a problem. For a few days now, Tailscale has been extremely slow and the following message keeps appearing in the log:
Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: [RATELIMIT] format("wg: Failed to write packets to TUN device: %v")
Aug 05 23:46:09 ciel tailscaled[876]: [RATELIMIT] format("wg: Failed to write packets to TUN device: %v") (30 dropped)
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: [RATELIMIT] format("wg: Failed to write packets to TUN device: %v")
By slow I mean the following behavior (iperf3 is running on my tailscale exit node):
$ iperf3 --client vpn-lu.uaru-vimba.ts.net -R
Connecting to host vpn-lu.uaru-vimba.ts.net, port 5201
Reverse mode, remote host vpn-lu.uaru-vimba.ts.net is sending
[ 5] local 100.97.92.115 port 44816 connected to 100.110.144.38 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 2.00-3.00 sec 128 KBytes 1.05 Mbits/sec
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 4.00-5.00 sec 128 KBytes 1.05 Mbits/sec
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 7.00-8.00 sec 128 KBytes 1.05 Mbits/sec
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.93 sec 609 KBytes 456 Kbits/sec 282 sender
[ 5] 0.00-10.00 sec 512 KBytes 419 Kbits/sec receiver
iperf Done.
and for reference the other way around is works:
$ iperf3 --client vpn-lu.uaru-vimba.ts.net
Connecting to host vpn-lu.uaru-vimba.ts.net, port 5201
[ 5] local 100.97.92.115 port 39542 connected to 100.110.144.38 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.50 MBytes 21.0 Mbits/sec 1 71.4 KBytes
[ 5] 1.00-2.00 sec 3.62 MBytes 30.4 Mbits/sec 0 99.3 KBytes
[ 5] 2.00-3.00 sec 3.88 MBytes 32.5 Mbits/sec 1 87.2 KBytes
[ 5] 3.00-4.00 sec 3.75 MBytes 31.5 Mbits/sec 1 102 KBytes
[ 5] 4.00-5.00 sec 3.88 MBytes 32.5 Mbits/sec 0 104 KBytes
[ 5] 5.00-6.00 sec 3.75 MBytes 31.5 Mbits/sec 1 93.2 KBytes
[ 5] 6.00-7.00 sec 4.00 MBytes 33.6 Mbits/sec 1 83.6 KBytes
[ 5] 7.00-8.00 sec 3.88 MBytes 32.5 Mbits/sec 0 111 KBytes
[ 5] 8.00-9.00 sec 3.62 MBytes 30.4 Mbits/sec 1 96.9 KBytes
[ 5] 9.00-10.00 sec 3.88 MBytes 32.5 Mbits/sec 1 84.8 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 36.8 MBytes 30.8 Mbits/sec 7 sender
[ 5] 0.00-10.93 sec 36.5 MBytes 28.0 Mbits/sec receiver
iperf Done.
Steps to reproduce
No idea. I undid all the changes I made to tailscale (changing the systemd unit and disabling the support logs) for testing. It did not help.
OS
Arch Linux
OS version
$ uname -a
Linux ciel 6.6.44-2-lts #1 SMP PREEMPT_DYNAMIC Sat, 03 Aug 2024 18:10:33 +0000 x86_64 GNU/Linux
Tailscale version
$ tailscale version
1.70.0
tailscale commit: 0e0a212418fbf8243cb3f06634367b61e81ea9db-dirty
go version: go1.22.5
Bug report
$ tailscale bugreport
BUG-c75d50b30c42938e61b3d6dfd8683ea7f95df6373200ac4dafdee5700dbabca1-20240805234440Z-89d5c5a39dee5ae1
Additional information
Installed via pacman (tailscale package).
I also run a few other things like cjdns, Yggdrasil, libvirt and docker (-> tailscale is not the only tun interface) on the computer:
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp1s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
link/ether 74:5d:22:f4:8d:6b brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 7e:1b:d9:a1:a2:d7 brd ff:ff:ff:ff:ff:ff permaddr 04:7b:cb:c7:41:10
inet 192.168.41.[redacted]/24 brd 192.168.41.255 scope global dynamic noprefixroute wlp2s0
valid_lft 42776sec preferred_lft 42776sec
inet6 2a02:3100:[redacted]/64 scope global dynamic noprefixroute
valid_lft 232740sec preferred_lft 146340sec
inet6 fd5f:28a6:[redacted]/64 scope global noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::[redacted]/64 scope link noprefixroute
valid_lft forever preferred_lft forever
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1304 qdisc fq_codel state UNKNOWN group default qlen 500
link/none
inet6 fcf6:773c:[redacted]/8 scope global
valid_lft forever preferred_lft forever
inet6 fe80::8f97:eaaa:dd2:ee16/64 scope link stable-privacy proto kernel_ll
valid_lft forever preferred_lft forever
5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:b2:fa:7f brd ff:ff:ff:ff:ff:ff
inet 192.168.100.1/24 brd 192.168.100.255 scope global virbr0
valid_lft forever preferred_lft forever
inet6 fd6c:ffa:eb7a:74e2::1/64 scope global tentative
valid_lft forever preferred_lft forever
6: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
link/none
inet 100.97.92.115/32 scope global tailscale0
valid_lft forever preferred_lft forever
inet6 fd7a:115c:a1e0::8e21:5c73/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::7986:2c09:c665:1b26/64 scope link stable-privacy proto kernel_ll
valid_lft forever preferred_lft forever
7: tun1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 65535 qdisc fq_codel state UNKNOWN group default qlen 500
link/none
inet6 200:[redacted]/7 scope global
valid_lft forever preferred_lft forever
inet6 fe80::b56c:e008:2405:5b50/64 scope link stable-privacy proto kernel_ll
valid_lft forever preferred_lft forever
9: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:70:f9:2d:b1 brd ff:ff:ff:ff:ff:ff
inet 192.168.128.1/24 brd 192.168.128.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 dd64:6f63:6b65:72::1/80 scope global tentative
valid_lft forever preferred_lft forever
strace (when doing iperf3):
[pid 15854] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\3@\0@\6\260\213dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\4@\0@\6\260\212dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\5@\0@\6\260\211dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\6@\0@\6\260\210dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15826] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\7@\0@\6\260\207dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15864] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\10@\0@\6\260\206dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15826] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\t@\0@\6\253\255dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15826] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\v@\0@\6\253\253dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15826] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\r@\0@\6\253\251dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15826] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\17@\0@\6\253\247dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15864] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\21@\0@\6\260}dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\22@\0@\6\253\244dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15864] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\24@\0@\6\253\242dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15864] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\26@\0@\6\260xdn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\27@\0@\6\260wdn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\30@\0@\6\260vdn\220&da\\s\24Q"..., 1290) = 1290
[pid 15826] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\31@\0@\6\260udn\220&da\\s\24Q"..., 1290) = 1290
[pid 15854] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\32@\0@\6\260tdn\220&da\\s\24Q"..., 1290) = 1290
[pid 15854] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\33@\0@\6\260sdn\220&da\\s\24Q"..., 1290) = 1290
If needed, I can also send the output of sudo strace -f -o strace.log tailscaled -verbose 1 > tailscale.log 2>&1 privately.
Another observation that might be interesting: I have a second laptop with pretty much the same setup (Arch Linux + Tailscale). I hadn’t updated it for a few days / weeks, so kernel 6.6.35-2-lts was still installed. Tailscale worked there. I have now updated it to kernel 6.6.44-2-lts and since then the same problem occurs.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 37 (16 by maintainers)
Commits related to this issue
- go.mod: bump wireguard-go to 6.6.44 mitigation test Linux 6.6.44 introduced a regression in GSO handling that spuriously returns EINVAL for tail packets shorter than gso_size. The referenced wireguar... — committed to tailscale/tailscale by raggi 2 years ago
- net/tstun: probe TCP GRO Disable TCP & UDP GRO if the probe fails. torvalds/linux@e269d79 broke virtio_net TCP & UDP GRO causing GRO writes to return EINVAL. The bug was then resolved later in torva... — committed to tailscale/tailscale by jwhited 2 years ago
- net/tstun: probe TCP GRO Disable TCP & UDP GRO if the probe fails. torvalds/linux@e269d79c7d35aa3808b1f3c1737d63dab504ddc8 broke virtio_net TCP & UDP GRO causing GRO writes to return EINVAL. The bug... — committed to tailscale/tailscale by jwhited 2 years ago
- tun: add method for disabling TCP GRO on Linux torvalds/linux@e269d79c7d35aa3808b1f3c1737d63dab504ddc8 broke virtio_net TCP & UDP GRO causing GRO writes to return EINVAL. The bug was then resolved la... — committed to tailscale/wireguard-go by jwhited 2 years ago
- net/tstun: probe TCP GRO Disable TCP & UDP GRO if the probe fails. torvalds/linux@e269d79c7d35aa3808b1f3c1737d63dab504ddc8 broke virtio_net TCP & UDP GRO causing GRO writes to return EINVAL. The bug... — committed to tailscale/tailscale by jwhited 2 years ago
- tun: add method for disabling TCP GRO on Linux torvalds/linux@e269d79c7d35aa3808b1f3c1737d63dab504ddc8 broke virtio_net TCP & UDP GRO causing GRO writes to return EINVAL. The bug was then resolved la... — committed to tailscale/wireguard-go by jwhited 2 years ago
- tun: add method for disabling TCP GRO on Linux torvalds/linux@e269d79c7d35aa3808b1f3c1737d63dab504ddc8 broke virtio_net TCP & UDP GRO causing GRO writes to return EINVAL. The bug was then resolved la... — committed to tailscale/wireguard-go by jwhited 2 years ago
- net/tstun: probe TCP GRO Disable TCP & UDP GRO if the probe fails. torvalds/linux@e269d79c7d35aa3808b1f3c1737d63dab504ddc8 broke virtio_net TCP & UDP GRO causing GRO writes to return EINVAL. The bug... — committed to tailscale/tailscale by jwhited 2 years ago
- net/tstun: probe TCP GRO Disable TCP & UDP GRO if the probe fails. torvalds/linux@e269d79c7d35aa3808b1f3c1737d63dab504ddc8 broke virtio_net TCP & UDP GRO causing GRO writes to return EINVAL. The bug... — committed to tailscale/tailscale by jwhited 2 years ago
- net/tstun: probe TCP GRO Disable TCP & UDP GRO if the probe fails. torvalds/linux@e269d79c7d35aa3808b1f3c1737d63dab504ddc8 broke virtio_net TCP & UDP GRO causing GRO writes to return EINVAL. The bug... — committed to tailscale/tailscale by jwhited 2 years ago
- net/tstun: probe TCP GRO (#13376) Disable TCP & UDP GRO if the probe fails. torvalds/linux@e269d79c7d35aa3808b1f3c1737d63dab504ddc8 broke virtio_net TCP & UDP GRO causing GRO writes to return EIN... — committed to tailscale/tailscale by jwhited 2 years ago
- tun: add method for disabling TCP GRO on Linux torvalds/linux@e269d79c7d35aa3808b1f3c1737d63dab504ddc8 broke virtio_net TCP & UDP GRO causing GRO writes to return EINVAL. The bug was then resolved la... — committed to getlantern/wireguard-go by jwhited 2 years ago
https://github.com/jwhited/tun-einval-repro contains a simplified reproduction of the issue. This writes a GSO_TCPv4 packet to a TUN device w/GSO=1240 and 2 equal length segments.
6.10.2
6.10.3
6.11-rc2
It looks like 6.10.5 patches this?
https://lwn.net/Articles/985670/
Fedora Testing Update tree:
sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-cb22d42b54Closing this issue as the client workaround has been present since Tailscale v1.74, and Linux has landed the fix across 6.6, 6.1, and 5.15 LTS releases.
I seems to work (-> no longer slow, no error messages in log anymore) on the latest
linux-lts(6.6.46-1-lts).So from my side the issue could be closed. Thanks for the quick processing of the issue!
The patch made it to the stable queues:
Note that this still misses the 6.1 series (see https://github.com/tailscale/tailscale/issues/13041#issuecomment-2274132169 for the relevant conflict)
Mhh, I don’t see any increased packet loss now (after downloading the fixed kernel).
I have now also poked the stable team to apply this to the stable trees: https://lore.kernel.org/all/146d2c9f-f2c3-4891-ac48-a3e50c863530@heusel.eu/
The issue is though that the fix does not apply cleanly to the 6.1 series, so if someone wants to provide a resolution for that this would be cool, I can take care of the submission then if that’s of any help:
Click to see merge conflict
They both fix the problem for me.
Also if you want to test the kernel side of things here are the
linuxand thelinux-ltspackages with https://github.com/gregkh/linux/commit/89add40066f9ed9abe5f7f886fe5789ff7e0c50e applied (which fixes the problem according to https://github.com/tailscale/tailscale/issues/13041#issuecomment-2272249663).linux:linux-lts:After some digging, so far it looks likely that the culprit is https://github.com/gregkh/linux/commit/e269d79c7d35aa3808b1f3c1737d63dab504ddc8 and the fix is https://github.com/gregkh/linux/commit/89add40066f9ed9abe5f7f886fe5789ff7e0c50e
Next up, verify this.
I was able to partially reproduce this, specifically:
Linux archlinux 6.6.44-2-lts #1 SMP PREEMPT_DYNAMIC Sat, 03 Aug 2024 18:10:33 +0000 x86_64 GNU/Linuxwg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument[ 96.578967] ens18: bad gso: type: 4, size: 1348This appears to be scoped to IPv6.
I was not however able to reproduce this occurring persistently, in my test setup the issue certainly occurred, but it occurred for a short time and then throughput recovered, and the error stopped being logged, both on the tailscaled side and kernel side.
I was not able to reproduce on
Linux archlinux 6.10.3-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 04 Aug 2024 05:11:32 +0000 x86_64 GNU/Linux.We will need to dig into what patches have been ported into LTS, there have been some activities in virtio header handling which we’re initially now concerned about, but upstream has already worked on improvements there, it’s entirely possible LTS brought in a bug that needs to be addressed with further backports.