tailscale: Can't establish direct connection to a Kubernetes pod despite having a direct connection to its node (Flannel CNI)

What is the issue?

I have configured the Tailscale Kubernetes operator on my cluster, so that I could expose my services to the tailnet. It works, and I can get access, however I am unable to establish direct connection with the tailnet node that’s responsible for my exposed K8s pod. The node running said pod however can establish a direct connection without any issues.

tailscale status output (with irrelevant nodes and public IP addresses omitted):

[liquid@liquid-desktop ~]$ tailscale status
# control plane node
100.64.193.4    control01            tagged-devices linux   active; direct <home connection ip>:14309, tx 652432 rx 6276240
# subnet router I use for direct access
100.116.6.41    homeassistant        <username>@ linux   active; offers exit node; direct <home connection ip>:51069, tx 1644892 rx 24845572
# exposed Traefik pod, connects over the waw relay despite it's k8s node having a direct connection
100.83.180.32   network-rp           tagged-devices linux   active; relay "waw", tx 430720 rx 10129968
# worker node
100.76.232.98   node01               tagged-devices linux   active; direct <home connection ip>:49077, tx 26756 rx 43212
100.127.193.3   tailscale-operator   tagged-devices linux   -

Steps to reproduce

  1. Install the Kubernetes operator on a new K3s cluster.
  2. Expose a pod to the tailnet.
  3. Connect to said pod over Tailscale.

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

Latest Fedora CoreOS on the k8s nodes, Arch Linux on the desktop

Tailscale version

1.62.0 on both desktop and the k8s node

Other software

Standard K3s installation with (default) Flannel networking on the cluster

Bug report

BUG-451fb6ee26ceb7c75ed1267cd48d26d4a43de5259c2e6db1ff0cb06353ea0d8e-20240315170435Z-3c58dcacadf8a4b7

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 18 (4 by maintainers)

Most upvoted comments

Hi @LiquidPL thank you very much for the confirmation that you now get direct connections with Calico.

Should I close this issue now? My issue is fixed but I suppose it could also track any changes to the docs.

I think we could leave this open- as you say it would be great to update docs and I would also like to reach out to Flannel folks and see if they might be willing to make the port randomization for external connections configurable.

Small update, recently I found out that Tailscale is able to make direct connections when using K3s without custom CNI. This might be due to upgrade of Tailscale version. However, this could also be due to changes I made in underlying networking as I am now using vxlan as a backend instead of wireguard-native combined with setting bind-address, node-ip and flannel-iface for each node.

Hi @LiquidPL thank you very much for the confirmation that you now get direct connections with Calico.

Should I close this issue now? My issue is fixed but I suppose it could also track any changes to the docs.

I think we could leave this open- as you say it would be great to update docs and I would also like to reach out to Flannel folks and see if they might be willing to make the port randomization for external connections configurable.

Did you reach out to Flannel folks about this? I am definitely interested as well!

I’m using calico, can you specify exactly what you did to solve it?

You need to apply a manifest like this:

apiVersion: projectcalico.org/v3
kind: FelixConfiguration
metadata:
  name: default
spec:
  # this is necessary to allow direct Tailscale connections to the cluster
  # see https://github.com/tailscale/tailscale/issues/11427#issuecomment-2021287969
  featureDetectOverride: 'MASQFullyRandom=false'

I had this same issue. Switching the CNI to Cillium solved it for me.

Small update: I have switched to Calico for networking on my cluster, since it has a way to disable source port randomization, by configuring Felix (Calico’s node agent) to override iptables feature detection and make it think the randomization is not supported.

The relevant bits of documentation can be found here: https://docs.tigera.io/calico/latest/reference/felix/configuration. The relevant config key is referenced by some variation of featureDetectOverride, based on the method it’s configured (env vars, config file, the Calico k8s operator, etc.), and the specific feature flag is MASQFullyRandom.

With this set, I can now directly connect to my pods, and iptables only reports a single rule with --random-fully, presumably added by Kubernetes itself:

root@control01:~# iptables-save | grep random-fully
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully

I’m guessing if someone removed the rules created by Flannel while also preventing it from adding them back, it would have the same effect.

Should I close this issue now? My issue is fixed but I suppose it could also track any changes to the docs.

how to properly quantify how hard its NAT is

If you take a look at your client in Tailscale Machines panel, there is a field named varies, if that’s set to true, you’re behind what we consider hard nat. However, looking at your debug logs it actually doesn’t seem to be the case