tailscale: Can't establish direct connection to a Kubernetes pod despite having a direct connection to its node (Flannel CNI)
What is the issue?
I have configured the Tailscale Kubernetes operator on my cluster, so that I could expose my services to the tailnet. It works, and I can get access, however I am unable to establish direct connection with the tailnet node that’s responsible for my exposed K8s pod. The node running said pod however can establish a direct connection without any issues.
tailscale status output (with irrelevant nodes and public IP addresses omitted):
[liquid@liquid-desktop ~]$ tailscale status
# control plane node
100.64.193.4 control01 tagged-devices linux active; direct <home connection ip>:14309, tx 652432 rx 6276240
# subnet router I use for direct access
100.116.6.41 homeassistant <username>@ linux active; offers exit node; direct <home connection ip>:51069, tx 1644892 rx 24845572
# exposed Traefik pod, connects over the waw relay despite it's k8s node having a direct connection
100.83.180.32 network-rp tagged-devices linux active; relay "waw", tx 430720 rx 10129968
# worker node
100.76.232.98 node01 tagged-devices linux active; direct <home connection ip>:49077, tx 26756 rx 43212
100.127.193.3 tailscale-operator tagged-devices linux -
Steps to reproduce
- Install the Kubernetes operator on a new K3s cluster.
- Expose a pod to the tailnet.
- Connect to said pod over Tailscale.
Are there any recent changes that introduced the issue?
No response
OS
Linux
OS version
Latest Fedora CoreOS on the k8s nodes, Arch Linux on the desktop
Tailscale version
1.62.0 on both desktop and the k8s node
Other software
Standard K3s installation with (default) Flannel networking on the cluster
Bug report
BUG-451fb6ee26ceb7c75ed1267cd48d26d4a43de5259c2e6db1ff0cb06353ea0d8e-20240315170435Z-3c58dcacadf8a4b7
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 18 (4 by maintainers)
Hi @LiquidPL thank you very much for the confirmation that you now get direct connections with Calico.
I think we could leave this open- as you say it would be great to update docs and I would also like to reach out to Flannel folks and see if they might be willing to make the port randomization for external connections configurable.
Small update, recently I found out that Tailscale is able to make direct connections when using K3s without custom CNI. This might be due to upgrade of Tailscale version. However, this could also be due to changes I made in underlying networking as I am now using vxlan as a backend instead of wireguard-native combined with setting
bind-address, node-ipandflannel-ifacefor each node.Did you reach out to Flannel folks about this? I am definitely interested as well!
You need to apply a manifest like this:
I had this same issue. Switching the CNI to Cillium solved it for me.
Small update: I have switched to Calico for networking on my cluster, since it has a way to disable source port randomization, by configuring Felix (Calico’s node agent) to override iptables feature detection and make it think the randomization is not supported.
The relevant bits of documentation can be found here: https://docs.tigera.io/calico/latest/reference/felix/configuration. The relevant config key is referenced by some variation of
featureDetectOverride, based on the method it’s configured (env vars, config file, the Calico k8s operator, etc.), and the specific feature flag isMASQFullyRandom.With this set, I can now directly connect to my pods, and iptables only reports a single rule with
--random-fully, presumably added by Kubernetes itself:I’m guessing if someone removed the rules created by Flannel while also preventing it from adding them back, it would have the same effect.
Should I close this issue now? My issue is fixed but I suppose it could also track any changes to the docs.
If you take a look at your client in Tailscale Machines panel, there is a field named
varies, if that’s set to true, you’re behind what we consider hard nat. However, looking at your debug logs it actually doesn’t seem to be the case