tailscale: Kubernetes operator: Proxy pod not started with --accept-routes causing dropped traffic from subnet-router subnet device
What is the issue?
Topology:
10.1.1.x <> Tailscale with subnet route 10.1.1.0/24 <> Tailscale Kubernetes Operator <> Tailscale Kubernetes Operator Proxy Pod
Problem
When deploying with Kubernetes operator, and trying to access from an IP which sits in a remote subnet handled by a remote subnet router no connectivity can be established.
Troubleshooting
❯ curl 100.96.101.35 (Container Talent IP)
... times out
The log of the proxy container also shows Some peers are advertising routes but --accept-routes is false
2023/12/23 17:58:54 Switching ipn state Starting -> Running (WantRunning=true, nm=true)
2023/12/23 17:58:54 derphttp.Client.Connect: connecting to derp-18 (par)
2023/12/23 17:58:54 magicsock: endpoints changed: 31.19.220.235:59292 (stun), 10.1.0.93:46271 (local)
2023/12/23 17:58:54 control: NetInfo: NetInfo{varies=true hairpin=false ipv6=false ipv6os=false udp=true icmpv4=false derp=#18 portmap= link="" firewallmode="nft"}
Some peers are advertising routes but --accept-routes is false
Fixing connectivity:
After connecting to the proxy pd ts-*** and issuing tailscale up --accept-routes --reset connectivity starts to work as expected.
❯ curl 100.96.101.35
Hostname: whoami-56647559d7-wrgx5
IP: 127.0.0.1
IP: 10.1.0.90
RemoteAddr: 10.1.0.89:55287
GET / HTTP/1.1
Host: 100.96.101.35
User-Agent: curl/8.4.0
Accept: */*
Suggested fix
Operator needs to start tailscale proxy pods with TAILSCALE_ARGS: ...--accept-routes or needs to make TAILSCALE_ARGS configurable through the helm chart.
Steps to reproduce
10.1.1.100 <> Tailscale with subnet route 10.1.1.0/24 <> Kubernetes Operator <> Kubernetes Operator Proxy Pod
Operator Install:
helm upgrade \
--install \
tailscale-operator \
tailscale/tailscale-operator \
--namespace=tailscale \
--create-namespace \
--set-string oauth.clientId=kqqVCR3CNTRL \
--set-string oauth.clientSecret=tskey-client-kqqVCR3CNTRL-*** \
--set-string apiServerProxyConfig.mode=true \
--wait
Test container:
helm upgrade --install whoami --set-string service.annotations."tailscale\.com\/expose"=true \
cowboysysop/whoami --create-namespace -n whoami --wait
Are there any recent changes that introduced the issue?
No response
OS
Linux
OS version
kubernetes-operator
Tailscale version
v1.56.1
Other software
No response
Bug report
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 17 (7 by maintainers)
Commits related to this issue
- cmd/k8s-operator,k8s-operator: allow to optionally configure proxies to --accept-routes A new ProxyClass.tailscaledConfig.acceptRoutes field (defaults to false) can be used to configure proxies creat... — committed to tailscale/tailscale by irbekrm 3 years ago
- cmd/k8s-operator,k8s-operator: allow to optionally configure proxies to --accept-routes A new ProxyClass.tailscaledConfig.acceptRoutes field (defaults to false) can be used to configure proxies creat... — committed to tailscale/tailscale by irbekrm 3 years ago
- cmd/k8s-operator,k8s-operator: allow to optionally configure proxies to --accept-routes A new ProxyClass.tailscaledConfig.acceptRoutes field (defaults to false) can be used to configure proxies creat... — committed to tailscale/tailscale by irbekrm 3 years ago
- cmd/k8s-operator,k8s-operator: allow to optionally configure proxies to --accept-routes A new ProxyClass.tailscaledConfig.acceptRoutes field (defaults to false) can be used to configure proxies creat... — committed to tailscale/tailscale by irbekrm 3 years ago
- cmd/k8s-operator,k8s-operator: allow to optionally configure proxies to --accept-routes A new ProxyClass.tailscaledConfig.acceptRoutes field (defaults to false) can be used to configure proxies creat... — committed to tailscale/tailscale by irbekrm 3 years ago
- cmd/k8s-operator,k8s-operator: allow proxies accept advertized routes. Add a new .spec.tailscale.acceptRoutes field to ProxyClass, that can be optionally set to true for the proxies to accept routes ... — committed to tailscale/tailscale by irbekrm 2 years ago
- cmd/k8s-operator,k8s-operator: allow proxies accept advertized routes. Add a new .spec.tailscale.acceptRoutes field to ProxyClass, that can be optionally set to true for the proxies to accept routes ... — committed to tailscale/tailscale by irbekrm 2 years ago
- cmd/k8s-operator,k8s-operator: allow proxies accept advertized routes. (#12388) Add a new .spec.tailscale.acceptRoutes field to ProxyClass, that can be optionally set to true for the proxies to acc... — committed to tailscale/tailscale by irbekrm 2 years ago
- cmd/k8s-operator,k8s-operator: allow proxies accept advertized routes. (#12388) Add a new .spec.tailscale.acceptRoutes field to ProxyClass, that can be optionally set to true for the proxies to acc... — committed to Ckid-Home/tailscale by irbekrm 2 years ago
- cmd/k8s-operator,k8s-operator: allow proxies accept advertized routes. (#12388) Add a new .spec.tailscale.acceptRoutes field to ProxyClass, that can be optionally set to true for the proxies to acc... — committed to Asutorufa/tailscale by irbekrm 2 years ago
Just a note that we are planning to do this. It is a relatively small piece of work, but it is blocked on https://github.com/tailscale/tailscale/issues/10869 as otherwise there is currently no way to reconfigure the
--accept-routesvalue once a proxy device has been created. #10869 is high on our priority list, so I’d expect us to be able to get this done soon-ish.No, I don’t think there is a way yet to set --accept-routes on the proxy in any way through the current manifests or CRDs.
#10724 just added other functionality in regards to exit nodes and proxy subnet router.
@irbekrm mentioned that creating a possibility to set “accept-routes” for proxy nodes is on the todo list though.
This would be super useful to enable the below routing as a kubernetes service!
Kubernetes --> Tailscale --> Machine with Route to
192.168.0.0/24–> Local Device on192.168.0.0/24.Yes, correct the client sits on
10.1.1.x. This machine has a static default route to the first Tailgate node ˙Tailscale linux instance with subnet route 10.1.1.0/24˙.This would perfectly address my use case! I just would like ensure that the proxy node will have the remote subnet routes available so the traffic finds it path back to the client.
best regards, Ollie.
It won’t come out as part of 1.60 release as I was initially hoping, because we prioritized proxy cluster resource customization and that took longer than expected. So hopefully for the next release. Thanks for your patience. Generally we are also happy to review folks’ PRs, but this is blocked on #10869 which will require some re-designing of the tailscale image.