tailscale: k8s operator / ingress slow (low bandwidth)

What is the issue?

I have a docker registry running in a bare metal k8s cluster. I’m using tailscale ingress to expose the service and tailscale operator to expose the kubernetes API. tailscale ping shows a direct connection (no DERP).

Measured bandwidth pushing to images the registry (averaging over the push of multi-GB images):

  • From a pod inside the cluster (no Tailscale involved): 630Mb/s
  • From outside the cluster, on the same physical LAN, through tailscale ingress: 99Mb/s
  • From outside the cluster, on the same physical LAN, through kubectl port-forward through tailscale operator: 181Mb/s
  • From outside the cluster, on the same physical LAN, through kubectl port-forward, + SSH port-forward from one of the control plane nodes to the machine doing the push: 227Mb/s

All of the machines involved have at least 8 fairly modern CPUs and are connected on the same physical gigabit LAN. Ideally ingress speed would be the same as within-cluster speed, but at least matching the kubectl port-forward + SSH port-forward would be a huge improvement.

Steps to reproduce

Here’s my ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: docker-registry-ingress
  namespace: docker-registry
spec:
  ingressClassName: tailscale
  defaultBackend:
    service:
      name: docker-registry
      port:
        number: 80
  tls:
    - hosts:
        - "berkeley-docker"

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

Ubuntu 22.04

Tailscale version

1.66.3-1

Other software

ingress and operator is running 1.66.3-1. Client is running 1.64.0 (though slowness was observed informally on other clients running 1.66)

Using kubernetes on bare metal with Calico CNI.

Bug report

BUG-cbaf45dad2d41a5f749df193623e240a74ae977c7b084b2a84f465346bdb69cb-20240607041559Z-5b058be542fb1cc0

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 7
  • Comments: 52 (19 by maintainers)

Commits related to this issue

Most upvoted comments

Hi @irbekrm, I just tried the latest unstable (v1.81.140) and I’m seeing roughly the same performance as I was before (~200kb/s).

I also don’t see any noticeable improvement in performance either. I am stuck at ~140Kbit/s.

So what seems to be happening is that since 1.17 Cilium is autodetecting MTU by looking at all interfaces on the host. https://github.com/cilium/cilium/pull/34314

The tailscale0 interface has MTU 1280, so when Cilium sees that, it changes the MTU of all container and Cilium interfaces to 1280.

In my case, I have a cluster that with 8896 MTU on default host interface.

Without Tailscale on host Cilium sets the container and other Cilium interfaces’ MTUs to 8896:

root@irbe-cilium-worker-1:/home/irbe# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8896 qdisc mq state UP group default qlen 1000 link/ether 42:01:0a:f0:00:15 brd ff:ff:ff:ff:ff:ff altname enp0s4 inet 10.240.0.21/32 metric 100 scope global dynamic ens4 valid_lft 1879sec preferred_lft 1879sec inet6 fe80::4001:aff:fef0:15/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:b6:1d:a0:6b brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 4: cilium_net@cilium_host: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 8896 qdisc noqueue state UP group default qlen 1000 link/ether 92:cc:aa:86:a1:b9 brd ff:ff:ff:ff:ff:ff inet6 fe80::90cc:aaff:fe86:a1b9/64 scope link valid_lft forever preferred_lft forever 5: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 8896 qdisc noqueue state UP group default qlen 1000 link/ether d6:9e:f0:1b:03:07 brd ff:ff:ff:ff:ff:ff inet 10.244.1.125/32 scope global cilium_host valid_lft forever preferred_lft forever inet6 fe80::d49e:f0ff:fe1b:307/64 scope link valid_lft forever preferred_lft forever 94: cilium_wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 8816 qdisc noqueue state UNKNOWN group default qlen 1000 link/none inet6 fe80::1bb5:2e6c:6770:f717/64 scope link stable-privacy valid_lft forever preferred_lft forever 95: cilium_vxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8896 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 2a:46:9c:ff:a8:d8 brd ff:ff:ff:ff:ff:ff inet6 fe80::2846:9cff:feff:a8d8/64 scope link valid_lft forever preferred_lft forever 123: lxc798785db2ffe@if122: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8896 qdisc noqueue state UP group default qlen 1000 link/ether 6e:28:ef:d0:9e:ff brd ff:ff:ff:ff:ff:ff link-netns cni-45b538cd-5c76-d3b4-8d13-e469cdd3d6e1 inet6 fe80::6c28:efff:fed0:9eff/64 scope link valid_lft forever preferred_lft forever ...

When I install Tailscale in tun mode on host, it changes to 1280:

root@irbe-cilium-worker-1:/home/irbe# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8896 qdisc mq state UP group default qlen 1000 link/ether 42:01:0a:f0:00:15 brd ff:ff:ff:ff:ff:ff altname enp0s4 inet 10.240.0.21/32 metric 100 scope global dynamic ens4 valid_lft 3554sec preferred_lft 3554sec inet6 fe80::4001:aff:fef0:15/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:b6:1d:a0:6b brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 4: cilium_net@cilium_host: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 8896 qdisc noqueue state UP group default qlen 1000 link/ether 92:cc:aa:86:a1:b9 brd ff:ff:ff:ff:ff:ff inet6 fe80::90cc:aaff:fe86:a1b9/64 scope link valid_lft forever preferred_lft forever 5: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 8896 qdisc noqueue state UP group default qlen 1000 link/ether d6:9e:f0:1b:03:07 brd ff:ff:ff:ff:ff:ff inet 10.244.1.125/32 scope global cilium_host valid_lft forever preferred_lft forever inet6 fe80::d49e:f0ff:fe1b:307/64 scope link valid_lft forever preferred_lft forever

94: cilium_wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 8816 qdisc noqueue state UNKNOWN group default qlen 1000 link/none inet6 fe80::1bb5:2e6c:6770:f717/64 scope link stable-privacy valid_lft forever preferred_lft forever 95: cilium_vxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8896 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 2a:46:9c:ff:a8:d8 brd ff:ff:ff:ff:ff:ff inet6 fe80::2846:9cff:feff:a8d8/64 scope link valid_lft forever preferred_lft forever 123: lxc798785db2ffe@if122: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8896 qdisc noqueue state UP group default qlen 1000 link/ether 6e:28:ef:d0:9e:ff brd ff:ff:ff:ff:ff:ff link-netns cni-45b538cd-5c76-d3b4-8d13-e469cdd3d6e1 inet6 fe80::6c28:efff:fed0:9eff/64 scope link valid_lft forever preferred_lft forever 167: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500 link/none inet6 fe80::dbdf:3a6b:2f55:2d3/64 scope link stable-privacy valid_lft forever preferred_lft forever …

I think the problem is the mismatch between the host’s primary interface’s MTU (8896) and that of the container interfaces. I am not sure yet why it affects UDP traffic in particular (i.e. direct connections between tailnet peers, but I have also observed that this setup causes bandwidth drop for non-Tailscale UDP traffic into the cluster).

I assume that other folks in this thread end up with a similar setup, with mismatching MTU on primary and container interfaces.

I think in this case it would make sense for Cilium to ignore the MTU on tailscale0 interface, but I can’t see an option to configure that. So alternatively, if folks know what is the desired MTU (probably that of the primary interface, in my case 8896) they could set it for Cilium using the --mtu flag/mtu config option, see 1.17 upgrade notes. I see that fixing the issue for my setup.

I’ve not yet tested if Calico (as per the original issue description) does the same, but seems likely.

@irbekrm I can confirm that devices fixes my issue! (MTU did too, but this seems to be a better solution) It looks like the default behavior is to detect all non-local interfaces and use the lowest MTU that gets found, so it makes sense why tailscaled with an MTU of 1280 would cause issues.

I’d also be curious if anybody has a reason to monitor tailscale0. I don’t, and I suspect most people are just using the default value…

So alternatively, if folks know what is the desired MTU (probably that of the primary interface, in my case 8896) they could set it for Cilium using the --mtu flag/mtu config option, see 1.17 upgrade notes. I see that fixing the issue for my setup.

We chatted about this issue to Cilium folks, who recommended using --devices flag in Cilium to explicitly specify which devices Cilium should monitor and I’ve verified that this works (after setting it to my node’s primary interface, Cilium no longer used tailscale0’s MTU to determine container MTU).

We are probably going to document this as the recommended solution for this issue on Cilium - interested to hear if anyone has reasons for wanting to have tailscale0 interface monitored by Cilium.

⚠ Edit: we do not recommend manually modifying tailscale0 interface as a permanent solution to this issue.

It seems like bumping mtu on the tailscale0 interface on the host helps, at least for me:

sudo ip link set dev tailscale0 mtu 1420

I see bandwidth go up from .42 Mbps to bandwidth that matches what I get with port forwarding in this setup.

Still need to figure out what is happening though.

Thanks a lot @jscaltreto this really helps. We’ll try to get to it next week and repro again- this is quite stark indeed. Could you verify

* whether this is Cilium [in kube proxy replacement mode](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/)

* Cilium version

* if you have the [socket load balancer](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#socket-loadbalancer-bypass-in-pod-namespace) enabled

Well this is interesting… I checked the version of cilium and it was running 1.13.4 which is quite old. On a whim I decided to see what would happen if I upgraded it. After bumping it to the next minor version (1.14.19), all of the TS instances became completely unreachable. So I did a restart of all of the statefulsets. After coming back up I’m now seeing vastly improved performance over TS:

Test 1 Results:
  Data: 1048576 bytes
  Server time: 2.243375ms
  Client transfer time: 0.0225 seconds
  Client measured bandwidth: 372.82 Mbps

I can’t say whether it was the upgrade itself that fixed it (a commenter above was seeing issues with cilium 1.16.x), or if it had to do with the Cilium daemonset restarting followed by needing to restart the TS instances. Since I was seeing it on the perf instance which was net-new I can’t imagine it has to do with restarting TS alone. I’ll continue to monitor and can report back if the issue resurfaces.

To answer your other questions:

  • Yes, Cilium is in kube-proxy replacement mode
  • I have not enabled the socket load balancer bypass mode (socketLB.hostNamespaceOnly is unset)

@irbekrm I gave v1.71.137 a try, and there are noticeable performance improvements! Though, in my case, it still goes through a relayer.

Yes (as noted in the initial issue description). The first few pongs are via DERP but it reliably connects directly after a while:

$ tailscale ping berkeley-docker
pong from berkeley-docker (100.75.177.66) via DERP(sfo) in 8ms
pong from berkeley-docker (100.75.177.66) via DERP(sfo) in 9ms
pong from berkeley-docker (100.75.177.66) via DERP(sfo) in 8ms
pong from berkeley-docker (100.75.177.66) via 192.168.2.147:1678 in 1ms
$ tailscale ping berkeley-tailscale-operator
pong from berkeley-tailscale-operator (100.76.36.95) via DERP(sfo) in 8ms
pong from berkeley-tailscale-operator (100.76.36.95) via DERP(sfo) in 9ms
pong from berkeley-tailscale-operator (100.76.36.95) via DERP(sfo) in 9ms
pong from berkeley-tailscale-operator (100.76.36.95) via 192.168.2.147:5718 in 1ms

I can also confirm that running ip link set dev tailscale0 mtu 1420 on the HOST machine (that is also running tailscale “native”) did indeed fix the issue for the slow speed through the exit nodig running as a container.

To be clear, the following scenario’s:

Speedtest via exit node running on K8s (K3s + Cilium 1.17.3 with kube proxy replacement) with MTU of tailscale0 interface set to the default 1280 on HOST 2.68 mbps / 380.20 mbps (why is only download impacted)

Speedtest via exit node running on K8s (K3s + Cilium 1.17.3 with kube proxy replacement) with MTU of tailscale0 interface set to the fixed (manual) 1420 on HOST 583.39 mbps / 474.05 mbps

Speedtest via native exit node (tailscale running on host itself) with MTU of tailscale0 interface set to 1280 730.07 mbps / 474.05 mbps (note the minimal difference MTU makes when running native)

Speedtest via native exit node (tailscale running on host itself) with MTU of tailscale0 interface set to 1420 724.19 mbps / 478.23 mbps

Native speedtest without exit node 1044.08 mbps / 1025.85 mbps

I run a rather “old” Intel Atom® Processor C3758 server board (low power) and it is getting hammered when using it as an exit node. So the significant lower speeds versus native I chalk up to my aging hardware. Image

@irbekrm

It seems like bumping mtu on the tailscale0 interface on the host helps, at least for me:

sudo ip link set dev tailscale0 mtu 1420

I see bandwidth go up from .42 Mbps to bandwidth that matches what I get with port forwarding in this setup.

Still need to figure out what is happening though.

This fixed it for me. Yay! I can access my services again. Thank you.

i am having the same issue on a new talos 1.9.5 + cilium 1.17.3 install with kube-operator replacement and socket load balancer bypass mode

helm install \
    cilium \
    cilium/cilium \
    --version 1.17.3 \
    --namespace kube-system \
    --set ipam.mode=kubernetes \
    --set kubeProxyReplacement=true \
    --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
    --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
    --set cgroup.autoMount.enabled=false \
    --set cgroup.hostRoot=/sys/fs/cgroup \
    --set k8sServiceHost=localhost \
    --set k8sServicePort=7445 \
    --set socketLB.hostNamespaceOnly=true \
    --set hubble.relay.enabled=true \
    --set hubble.ui.enabled=true

i am not able to stream 1080p video through tailscale operator + cilium 1.17.3 no problem through tailscale operator + cilium 1.15.x or traefik + cilium 1.17.3

@irbekrm Sorry for the late reply. I do run Tailscale on each node. I’ll disable it for now to do some speedtests, but long-term I’d like to keep Tailscale running on the host in case the Kubernetes API goes down or anything while I’m not at home.

I guess I could look into running Tailscale in userspace mode on the host, since I only use it for remote SSH access.

@irbekrm I do run Tailscale on the node. The use case is I have a docker registry (harbor) running in my Tailnet which is hosting images that are used in the cluster. In order for the kubelet to pull the images, the host needs to be able to access the registry. I wonder if it would be possible to hack together a proxy in the cluster and expose it on a nodeport or something 🤔

I do run tailscale on my nodes, but, still, the issue only occurs with Cilium and only with the operator, not with the host bandwidth

Client measured bandwidth: .42 Mbps

I can reproduce this exact low bandwidth if I also run Tailscale in tun mode on the node (or in Pod in host network namespace) on the same node as the Ingress proxy. I’ve not been able to reproduce this behaviour with any other setup.

@gabe565 @jscaltreto is it possible that you folks also run Tailscale on nodes or on host network?

I don’t think kube-proxy replacement is the issue. I use Cilium v1.17.3 on k3s v1.32.3+k3s1 without kube-proxy replacement and I’m seeing similar results.

Perf over Tailscale
verifying connectivity before test..
pong from perf (100.114.133.117) via 192.168.1.245:45363 in 4ms
Running test 1 of 5...
Test 1 Results:
  Data: 1048576 bytes
  Server time: 5.968704ms
  Client transfer time: 19.5387 seconds
  Client measured bandwidth: .42 Mbps
------------------------
Running test 2 of 5...
Test 2 Results:
  Data: 1048576 bytes
  Server time: 6.047985ms
  Client transfer time: 19.5588 seconds
  Client measured bandwidth: .42 Mbps
------------------------
Running test 3 of 5...
Test 3 Results:
  Data: 1048576 bytes
  Server time: 11.574515ms
  Client transfer time: 19.5565 seconds
  Client measured bandwidth: .42 Mbps
------------------------
Running test 4 of 5...
Test 4 Results:
  Data: 1048576 bytes
  Server time: 6.91318ms
  Client transfer time: 19.5620 seconds
  Client measured bandwidth: .42 Mbps
------------------------
Running test 5 of 5...
Test 5 Results:
  Data: 1048576 bytes
  Server time: 8.630412ms
  Client transfer time: 19.2622 seconds
  Client measured bandwidth: .43 Mbps
------------------------
verifying connectivity after test..
pong from perf (100.114.133.117) via 192.168.1.245:45363 in 23ms
Average Client Bandwidth: .42 Mbps
Perf over NodePort
verifying connectivity before test..
2025/04/19 22:39:27 error looking up IP of "192.168.1.250:31453": lookup 192.168.1.250:31453: no such host
Running test 1 of 5...
Test 1 Results:
  Data: 1048576 bytes
  Server time: 15.922535ms
  Client transfer time: 0.0360 seconds
  Client measured bandwidth: 233.01 Mbps
------------------------
Running test 2 of 5...
Test 2 Results:
  Data: 1048576 bytes
  Server time: 17.116692ms
  Client transfer time: 0.0498 seconds
  Client measured bandwidth: 168.44 Mbps
------------------------
Running test 3 of 5...
Test 3 Results:
  Data: 1048576 bytes
  Server time: 17.839095ms
  Client transfer time: 0.0676 seconds
  Client measured bandwidth: 124.09 Mbps
------------------------
Running test 4 of 5...
Test 4 Results:
  Data: 1048576 bytes
  Server time: 18.544743ms
  Client transfer time: 0.0403 seconds
  Client measured bandwidth: 208.15 Mbps
------------------------
Running test 5 of 5...
Test 5 Results:
  Data: 1048576 bytes
  Server time: 16.475939ms
  Client transfer time: 0.0348 seconds
  Client measured bandwidth: 241.05 Mbps
------------------------
verifying connectivity after test..
2025/04/19 22:39:39 error looking up IP of "192.168.1.250:31453": lookup 192.168.1.250:31453: no such host
Average Client Bandwidth: 194.94 Mbps

Edit: Here’s a link to my Cilium config if that’s helpful.

I’m seeing the same issue; my setup is TS operator + Cilium on Talos Linux.

@irbekrm I ran the perf test as suggested in the gist and the difference between TS and the nodeport are stark.

The results when using TS (the ping suggests it’s not using DERP):

verifying connectivity before test..
pong from perf (100.119.217.40) via 192.168.10.172:49187 in 1ms
Running test 1 of 5...
Test 1 Results:
  Data: 1048576 bytes
  Server time: 6.818311ms
  Client transfer time: 19.5885 seconds
  Client measured bandwidth: .42 Mbps
------------------------
Running test 2 of 5...
Test 2 Results:
  Data: 1048576 bytes
  Server time: 16.901888ms
  Client transfer time: 19.5894 seconds
  Client measured bandwidth: .42 Mbps
------------------------
Running test 3 of 5...
Test 3 Results:
  Data: 1048576 bytes
  Server time: 9.998883ms
  Client transfer time: 19.5879 seconds
  Client measured bandwidth: .42 Mbps
------------------------
Running test 4 of 5...
Test 4 Results:
  Data: 1048576 bytes
  Server time: 8.08288ms
  Client transfer time: 19.5894 seconds
  Client measured bandwidth: .42 Mbps
------------------------
Running test 5 of 5...
Test 5 Results:
  Data: 1048576 bytes
  Server time: 7.414319ms
  Client transfer time: 19.5879 seconds
  Client measured bandwidth: .42 Mbps
------------------------
verifying connectivity after test..
pong from perf (100.119.217.40) via 192.168.10.172:49187 in 1ms
Average Client Bandwidth: .42 Mbps

Versus using the nodeport:

Running test 1 of 5...
Test 1 Results:
  Data: 1048576 bytes
  Server time: 24.086309ms
  Client transfer time: 0.0893 seconds
  Client measured bandwidth: 93.93 Mbps
------------------------
Running test 2 of 5...
Test 2 Results:
  Data: 1048576 bytes
  Server time: 17.096499ms
  Client transfer time: 0.0306 seconds
  Client measured bandwidth: 274.13 Mbps
------------------------
Running test 3 of 5...
Test 3 Results:
  Data: 1048576 bytes
  Server time: 11.34487ms
  Client transfer time: 0.0364 seconds
  Client measured bandwidth: 230.45 Mbps
------------------------
Running test 4 of 5...
Test 4 Results:
  Data: 1048576 bytes
  Server time: 19.207713ms
  Client transfer time: 0.0277 seconds
  Client measured bandwidth: 302.83 Mbps
------------------------
Running test 5 of 5...
Test 5 Results:
  Data: 1048576 bytes
  Server time: 19.255411ms
  Client transfer time: 0.0239 seconds
  Client measured bandwidth: 350.98 Mbps
------------------------
Average Client Bandwidth: 250.46 Mbps

I am running tailscale using a service, with loadBalancerClass: tailscale and using nginx as the ingress

cilium config was pretty vanilla:

ipam:
  operator:
    clusterPoolIPv4PodCIDRList: ["10.1.0.0/16"]

monitor:
  enabled: true

prometheus:
  enabled: true
  serviceMonitor:
    enabled: true
    labels:
      release: prom_operator
dashboards:
  enabled: true
  label: grafana_dashboard

hubble:
  tls:
    auto:
      method: certmanager
      certManagerIssuerRef:
        group: cert-manager.io
        kind: ClusterIssuer
        name: internal

envoy:
  enabled: false

egressGateway:
  enabled: true

bpf:
  masquerade: true
  hostLegacyRouting: true

kubeProxyReplacement: true

tls:
  auto:
    method: certmanager

and I was running version 1.17.2 of the cilium/cilium helm chart.

All of my configs are open source. I use the operator + ingress

Here’s one that’s funneled: https://github.com/myoung34/homelab/blob/main/k8s/prod/hass/tailscale.yaml Here’s a standard one: https://github.com/myoung34/homelab/blob/main/k8s/prod/esphome/tailscale.yaml

I ran into this and it always averaged about 1-200kbps

I just removed cilium and its now super fast. I don’t know enough about CNI , but I can confirm this is 100% cilium related.

We are also running into this issue. Below are details about our environment. As it stands now the tailscale ingress and service annotations are not usable for us. Im willing to provide more information, but iam unsure on how to debug this.

Talos Linux v1.8.2 Cilium 1.16.4 tailscale k8s operator v1.78.3 tailscale talos linux extension 1.72.1

Cilum values

ipam:
  mode: kubernetes
kubeProxyReplacement: true
securityContext:
  capabilities:
    ciliumAgent:
      - CHOWN
      - KILL
      - NET_ADMIN
      - NET_RAW
      - IPC_LOCK
      - SYS_ADMIN
      - SYS_RESOURCE
      - DAC_OVERRIDE
      - FOWNER
      - SETGID
      - SETUID
    cleanCiliumState:
      - NET_ADMIN
      - SYS_ADMIN
      - SYS_RESOURCE
cgroup:
  autoMount:
    enabled: false
  hostRoot: /sys/fs/cgroup
k8sServiceHost: localhost
k8sServicePort: 7445
l2announcements:
  enabled: true
k8sClientRateLimit:
  qps: 32
  burst: 50
devices: enp1s0
socketLB:
  hostNamespaceOnly: true
bpf:
  lbExternalClusterIP: true
hubble:
  ui:
    enabled: true
  relay:
    enabled: true

I’m seeing significantly lower speeds with the ingress using Cilium CNI than with port-forward as well. I am seeing a direct connection, and interestingly the slowness seems asymetric: image

there are noticeable performance improvements

Thanks for confirming this!

Though, in my case, it still goes through a relayer.

That would be a separate issue. It could be because of CNI has enabled source port randomization for conntrack, see i.e https://github.com/tailscale/tailscale/issues/11427. If it appears not to be because of source port randomization and it seems like you should be gettting direct connections https://tailscale.com/kb/1257/connection-types could you open another issue for that?

FWIW I’m seeing something similar, using k3s/Service annotation/TS Operator accessing the typical k8s dashboard. If I kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443 and access it*, it loads the page quite quick, including the 2.2MB packed JS:

image

However, if I instead expose the k8s dashboard and access it through the generated machine IP or the FQDN, it slows to a crawl, serving the same 2.2MB of JS in >18s (!)

image

I hope I’m not conflating Ingress and Service in this bug; happy to open another if necessary. Let me know if there’s anything else that would be helpful from my end.

* Note: I’ve got my kubectl config set up to access the cluster over Tailscale, so AFAICT the port-forward is also going over TS’s network.

EDIT

Another data point: if I’m on the same host A the pod is running on and I curl -k https://k8s-dashboard/en.main.113d11c2245934fd.js it completes in a second or so. Running the same curl command on a different machine B than the pod (machine A) slows it down by a factor of >10x, yet port-forward on B is just as fast if not faster than the same-host scenario.

was Tailscale configured as a kind: Ingress or kind: Service?

I didn’t write “kind” anywhere in my configs, I just put an annotation on my service. Here’s the service and the statefulset: https://gist.github.com/garymm/8bbdced650da81a42676000dc5b6b4ab

I don’t see any reference to “ingress”.

What CNI plugin is in use and what is its configuration?

Calico. I’m using the defaults in kubesrpay (not sure what configuration is relevant), except I set this: FELIX_FEATUREDETECTOVERRIDE: MASQFullyRandom=false

Because I read that may help with Tailscale here: https://github.com/tailscale/tailscale/issues/11427#issuecomment-2021287969.

For 2 and 3, I added the output of ethtool -k to the gist I linked to above.