tailscale: k8s operator / ingress slow (low bandwidth)
What is the issue?
I have a docker registry running in a bare metal k8s cluster.
I’m using tailscale ingress to expose the service and tailscale operator to expose the kubernetes API.
tailscale ping shows a direct connection (no DERP).
Measured bandwidth pushing to images the registry (averaging over the push of multi-GB images):
- From a pod inside the cluster (no Tailscale involved): 630Mb/s
- From outside the cluster, on the same physical LAN, through tailscale ingress: 99Mb/s
- From outside the cluster, on the same physical LAN, through
kubectl port-forwardthrough tailscale operator: 181Mb/s - From outside the cluster, on the same physical LAN, through
kubectl port-forward, + SSH port-forward from one of the control plane nodes to the machine doing the push: 227Mb/s
All of the machines involved have at least 8 fairly modern CPUs and are connected on the same physical gigabit LAN.
Ideally ingress speed would be the same as within-cluster speed, but at least matching the kubectl port-forward + SSH port-forward would be a huge improvement.
Steps to reproduce
Here’s my ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: docker-registry-ingress
namespace: docker-registry
spec:
ingressClassName: tailscale
defaultBackend:
service:
name: docker-registry
port:
number: 80
tls:
- hosts:
- "berkeley-docker"
Are there any recent changes that introduced the issue?
No response
OS
Linux
OS version
Ubuntu 22.04
Tailscale version
1.66.3-1
Other software
ingress and operator is running 1.66.3-1.
Client is running 1.64.0 (though slowness was observed informally on other clients running 1.66)
Using kubernetes on bare metal with Calico CNI.
Bug report
BUG-cbaf45dad2d41a5f749df193623e240a74ae977c7b084b2a84f465346bdb69cb-20240607041559Z-5b058be542fb1cc0
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 7
- Comments: 52 (19 by maintainers)
Commits related to this issue
- wgengine/netstack: block link writes when full rather than drop Originally identified by Coder and documented in their blog post, this implementation differs slightly as our link endpoint was introdu... — committed to tailscale/tailscale by raggi 2 years ago
- go.mod,wgengine/netstack: use cubic congestion control Cubic performs better than Reno in higher BDP scenarios, and enables the use of the hystart++ implementation contributed by Coder. This improves... — committed to tailscale/tailscale by raggi 2 years ago
- go.mod,wgengine/netstack: use cubic congestion control, bump gvisor Cubic performs better than Reno in higher BDP scenarios, and enables the use of the hystart++ implementation contributed by Coder. ... — committed to tailscale/tailscale by raggi 2 years ago
- wgengine/netstack: block link writes when full rather than drop Originally identified by Coder and documented in their blog post, this implementation differs slightly as our link endpoint was introdu... — committed to tailscale/tailscale by raggi 2 years ago
- wgengine/netstack: block link writes when full rather than drop Originally identified by Coder and documented in their blog post, this implementation differs slightly as our link endpoint was introdu... — committed to tailscale/tailscale by raggi 2 years ago
- wgengine/netstack: block link writes when full rather than drop Originally identified by Coder and documented in their blog post, this implementation differs slightly as our link endpoint was introdu... — committed to tailscale/tailscale by raggi 2 years ago
- go.mod,wgengine/netstack: use cubic congestion control, bump gvisor Cubic performs better than Reno in higher BDP scenarios, and enables the use of the hystart++ implementation contributed by Coder. ... — committed to tailscale/tailscale by raggi 2 years ago
- wgengine/netstack: block link writes when full rather than drop Originally identified by Coder and documented in their blog post, this implementation differs slightly as our link endpoint was introdu... — committed to ninech/tailscale by raggi 2 years ago
- go.mod,wgengine/netstack: use cubic congestion control, bump gvisor Cubic performs better than Reno in higher BDP scenarios, and enables the use of the hystart++ implementation contributed by Coder. ... — committed to ninech/tailscale by raggi 2 years ago
Hi @irbekrm, I just tried the latest unstable (v1.81.140) and I’m seeing roughly the same performance as I was before (~200kb/s).
I also don’t see any noticeable improvement in performance either. I am stuck at ~140Kbit/s.
So what seems to be happening is that since 1.17 Cilium is autodetecting MTU by looking at all interfaces on the host. https://github.com/cilium/cilium/pull/34314
The
tailscale0interface has MTU 1280, so when Cilium sees that, it changes the MTU of all container and Cilium interfaces to 1280.In my case, I have a cluster that with 8896 MTU on default host interface.
Without Tailscale on host Cilium sets the container and other Cilium interfaces’ MTUs to 8896:
root@irbe-cilium-worker-1:/home/irbe# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8896 qdisc mq state UP group default qlen 1000 link/ether 42:01:0a:f0:00:15 brd ff:ff:ff:ff:ff:ff altname enp0s4 inet 10.240.0.21/32 metric 100 scope global dynamic ens4 valid_lft 1879sec preferred_lft 1879sec inet6 fe80::4001:aff:fef0:15/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:b6:1d:a0:6b brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 4: cilium_net@cilium_host: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 8896 qdisc noqueue state UP group default qlen 1000 link/ether 92:cc:aa:86:a1:b9 brd ff:ff:ff:ff:ff:ff inet6 fe80::90cc:aaff:fe86:a1b9/64 scope link valid_lft forever preferred_lft forever 5: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 8896 qdisc noqueue state UP group default qlen 1000 link/ether d6:9e:f0:1b:03:07 brd ff:ff:ff:ff:ff:ff inet 10.244.1.125/32 scope global cilium_host valid_lft forever preferred_lft forever inet6 fe80::d49e:f0ff:fe1b:307/64 scope link valid_lft forever preferred_lft forever 94: cilium_wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 8816 qdisc noqueue state UNKNOWN group default qlen 1000 link/none inet6 fe80::1bb5:2e6c:6770:f717/64 scope link stable-privacy valid_lft forever preferred_lft forever 95: cilium_vxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8896 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 2a:46:9c:ff:a8:d8 brd ff:ff:ff:ff:ff:ff inet6 fe80::2846:9cff:feff:a8d8/64 scope link valid_lft forever preferred_lft forever 123: lxc798785db2ffe@if122: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8896 qdisc noqueue state UP group default qlen 1000 link/ether 6e:28:ef:d0:9e:ff brd ff:ff:ff:ff:ff:ff link-netns cni-45b538cd-5c76-d3b4-8d13-e469cdd3d6e1 inet6 fe80::6c28:efff:fed0:9eff/64 scope link valid_lft forever preferred_lft forever ...When I install Tailscale in tun mode on host, it changes to 1280:
root@irbe-cilium-worker-1:/home/irbe# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8896 qdisc mq state UP group default qlen 1000 link/ether 42:01:0a:f0:00:15 brd ff:ff:ff:ff:ff:ff altname enp0s4 inet 10.240.0.21/32 metric 100 scope global dynamic ens4 valid_lft 3554sec preferred_lft 3554sec inet6 fe80::4001:aff:fef0:15/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:b6:1d:a0:6b brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 4: cilium_net@cilium_host: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 8896 qdisc noqueue state UP group default qlen 1000 link/ether 92:cc:aa:86:a1:b9 brd ff:ff:ff:ff:ff:ff inet6 fe80::90cc:aaff:fe86:a1b9/64 scope link valid_lft forever preferred_lft forever 5: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 8896 qdisc noqueue state UP group default qlen 1000 link/ether d6:9e:f0:1b:03:07 brd ff:ff:ff:ff:ff:ff inet 10.244.1.125/32 scope global cilium_host valid_lft forever preferred_lft forever inet6 fe80::d49e:f0ff:fe1b:307/64 scope link valid_lft forever preferred_lft forever94: cilium_wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 8816 qdisc noqueue state UNKNOWN group default qlen 1000 link/none inet6 fe80::1bb5:2e6c:6770:f717/64 scope link stable-privacy valid_lft forever preferred_lft forever 95: cilium_vxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8896 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 2a:46:9c:ff:a8:d8 brd ff:ff:ff:ff:ff:ff inet6 fe80::2846:9cff:feff:a8d8/64 scope link valid_lft forever preferred_lft forever 123: lxc798785db2ffe@if122: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8896 qdisc noqueue state UP group default qlen 1000 link/ether 6e:28:ef:d0:9e:ff brd ff:ff:ff:ff:ff:ff link-netns cni-45b538cd-5c76-d3b4-8d13-e469cdd3d6e1 inet6 fe80::6c28:efff:fed0:9eff/64 scope link valid_lft forever preferred_lft forever 167: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500 link/none inet6 fe80::dbdf:3a6b:2f55:2d3/64 scope link stable-privacy valid_lft forever preferred_lft forever …
I think the problem is the mismatch between the host’s primary interface’s MTU (8896) and that of the container interfaces. I am not sure yet why it affects UDP traffic in particular (i.e. direct connections between tailnet peers, but I have also observed that this setup causes bandwidth drop for non-Tailscale UDP traffic into the cluster).
I assume that other folks in this thread end up with a similar setup, with mismatching MTU on primary and container interfaces.
I think in this case it would make sense for Cilium to ignore the MTU on
tailscale0interface, but I can’t see an option to configure that. So alternatively, if folks know what is the desired MTU (probably that of the primary interface, in my case 8896) they could set it for Cilium using the--mtuflag/mtuconfig option, see 1.17 upgrade notes. I see that fixing the issue for my setup.I’ve not yet tested if Calico (as per the original issue description) does the same, but seems likely.
@irbekrm I can confirm that
devicesfixes my issue! (MTU did too, but this seems to be a better solution) It looks like the default behavior is to detect all non-local interfaces and use the lowest MTU that gets found, so it makes sense why tailscaled with an MTU of 1280 would cause issues.I’d also be curious if anybody has a reason to monitor
tailscale0. I don’t, and I suspect most people are just using the default value…We chatted about this issue to Cilium folks, who recommended using
--devicesflag in Cilium to explicitly specify which devices Cilium should monitor and I’ve verified that this works (after setting it to my node’s primary interface, Cilium no longer usedtailscale0’s MTU to determine container MTU).We are probably going to document this as the recommended solution for this issue on Cilium - interested to hear if anyone has reasons for wanting to have
tailscale0interface monitored by Cilium.⚠ Edit: we do not recommend manually modifying
tailscale0interface as a permanent solution to this issue.It seems like bumping mtu on the
tailscale0interface on the host helps, at least for me:I see bandwidth go up from .42 Mbps to bandwidth that matches what I get with port forwarding in this setup.
Still need to figure out what is happening though.
Well this is interesting… I checked the version of cilium and it was running 1.13.4 which is quite old. On a whim I decided to see what would happen if I upgraded it. After bumping it to the next minor version (1.14.19), all of the TS instances became completely unreachable. So I did a restart of all of the statefulsets. After coming back up I’m now seeing vastly improved performance over TS:
I can’t say whether it was the upgrade itself that fixed it (a commenter above was seeing issues with cilium 1.16.x), or if it had to do with the Cilium daemonset restarting followed by needing to restart the TS instances. Since I was seeing it on the
perfinstance which was net-new I can’t imagine it has to do with restarting TS alone. I’ll continue to monitor and can report back if the issue resurfaces.To answer your other questions:
socketLB.hostNamespaceOnlyis unset)@irbekrm I gave
v1.71.137a try, and there are noticeable performance improvements! Though, in my case, it still goes through a relayer.Yes (as noted in the initial issue description). The first few pongs are via DERP but it reliably connects directly after a while:
I can also confirm that running
ip link set dev tailscale0 mtu 1420on the HOST machine (that is also running tailscale “native”) did indeed fix the issue for the slow speed through the exit nodig running as a container.To be clear, the following scenario’s:
Speedtest via exit node running on K8s (K3s + Cilium 1.17.3 with kube proxy replacement) with MTU of tailscale0 interface set to the default 1280 on HOST
2.68 mbps / 380.20 mbps(why is only download impacted)Speedtest via exit node running on K8s (K3s + Cilium 1.17.3 with kube proxy replacement) with MTU of tailscale0 interface set to the fixed (manual) 1420 on HOST
583.39 mbps / 474.05 mbpsSpeedtest via native exit node (tailscale running on host itself) with MTU of tailscale0 interface set to 1280
730.07 mbps / 474.05 mbps(note the minimal difference MTU makes when running native)Speedtest via native exit node (tailscale running on host itself) with MTU of tailscale0 interface set to 1420
724.19 mbps / 478.23 mbpsNative speedtest without exit node
1044.08 mbps / 1025.85 mbpsI run a rather “old” Intel Atom® Processor C3758 server board (low power) and it is getting hammered when using it as an exit node. So the significant lower speeds versus native I chalk up to my aging hardware.
@irbekrm
This fixed it for me. Yay! I can access my services again. Thank you.
i am having the same issue on a new talos 1.9.5 + cilium 1.17.3 install with kube-operator replacement and socket load balancer bypass mode
i am not able to stream 1080p video through tailscale operator + cilium 1.17.3 no problem through tailscale operator + cilium 1.15.x or traefik + cilium 1.17.3
@irbekrm Sorry for the late reply. I do run Tailscale on each node. I’ll disable it for now to do some speedtests, but long-term I’d like to keep Tailscale running on the host in case the Kubernetes API goes down or anything while I’m not at home.
I guess I could look into running Tailscale in userspace mode on the host, since I only use it for remote SSH access.
@irbekrm I do run Tailscale on the node. The use case is I have a docker registry (harbor) running in my Tailnet which is hosting images that are used in the cluster. In order for the kubelet to pull the images, the host needs to be able to access the registry. I wonder if it would be possible to hack together a proxy in the cluster and expose it on a nodeport or something 🤔
I do run tailscale on my nodes, but, still, the issue only occurs with Cilium and only with the operator, not with the host bandwidth
I can reproduce this exact low bandwidth if I also run Tailscale in tun mode on the node (or in Pod in host network namespace) on the same node as the Ingress proxy. I’ve not been able to reproduce this behaviour with any other setup.
@gabe565 @jscaltreto is it possible that you folks also run Tailscale on nodes or on host network?
I don’t think kube-proxy replacement is the issue. I use Cilium v1.17.3 on k3s v1.32.3+k3s1 without kube-proxy replacement and I’m seeing similar results.
Perf over Tailscale
Perf over NodePort
Edit: Here’s a link to my Cilium config if that’s helpful.
I’m seeing the same issue; my setup is TS operator + Cilium on Talos Linux.
@irbekrm I ran the perf test as suggested in the gist and the difference between TS and the nodeport are stark.
The results when using TS (the ping suggests it’s not using DERP):
Versus using the nodeport:
This is my cilium just before I ripped it out, also 1.17.2
I am running tailscale using a service, with
loadBalancerClass: tailscaleand using nginx as the ingresscilium config was pretty vanilla:
and I was running version
1.17.2of the cilium/cilium helm chart.All of my configs are open source. I use the operator + ingress
Here’s one that’s funneled: https://github.com/myoung34/homelab/blob/main/k8s/prod/hass/tailscale.yaml Here’s a standard one: https://github.com/myoung34/homelab/blob/main/k8s/prod/esphome/tailscale.yaml
I ran into this and it always averaged about 1-200kbps
I just removed cilium and its now super fast. I don’t know enough about CNI , but I can confirm this is 100% cilium related.
We are also running into this issue. Below are details about our environment. As it stands now the tailscale ingress and service annotations are not usable for us. Im willing to provide more information, but iam unsure on how to debug this.
Talos Linux v1.8.2 Cilium 1.16.4 tailscale k8s operator v1.78.3 tailscale talos linux extension 1.72.1
Cilum values
I’m seeing significantly lower speeds with the ingress using Cilium CNI than with port-forward as well. I am seeing a direct connection, and interestingly the slowness seems asymetric:
Thanks for confirming this!
That would be a separate issue. It could be because of CNI has enabled source port randomization for conntrack, see i.e https://github.com/tailscale/tailscale/issues/11427. If it appears not to be because of source port randomization and it seems like you should be gettting direct connections https://tailscale.com/kb/1257/connection-types could you open another issue for that?
FWIW I’m seeing something similar, using k3s/Service annotation/TS Operator accessing the typical k8s dashboard. If I
kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443and access it*, it loads the page quite quick, including the 2.2MB packed JS:However, if I instead expose the k8s dashboard and access it through the generated machine IP or the FQDN, it slows to a crawl, serving the same 2.2MB of JS in >18s (!)
I hope I’m not conflating
IngressandServicein this bug; happy to open another if necessary. Let me know if there’s anything else that would be helpful from my end.* Note: I’ve got my kubectl config set up to access the cluster over Tailscale, so AFAICT the port-forward is also going over TS’s network.
EDIT
Another data point: if I’m on the same host
Athe pod is running on and Icurl -k https://k8s-dashboard/en.main.113d11c2245934fd.jsit completes in a second or so. Running the samecurlcommand on a different machineBthan the pod (machineA) slows it down by a factor of >10x, yetport-forwardonBis just as fast if not faster than the same-host scenario.I didn’t write “kind” anywhere in my configs, I just put an annotation on my service. Here’s the service and the statefulset: https://gist.github.com/garymm/8bbdced650da81a42676000dc5b6b4ab
I don’t see any reference to “ingress”.
Calico. I’m using the defaults in kubesrpay (not sure what configuration is relevant), except I set this:
FELIX_FEATUREDETECTOVERRIDE: MASQFullyRandom=falseBecause I read that may help with Tailscale here: https://github.com/tailscale/tailscale/issues/11427#issuecomment-2021287969.
For 2 and 3, I added the output of
ethtool -kto the gist I linked to above.