tailscale: Kubernetes operator: Proxy pod not started with --accept-routes causing dropped traffic from subnet-router subnet device

What is the issue?

Topology:

10.1.1.x <> Tailscale with subnet route 10.1.1.0/24 <> Tailscale Kubernetes Operator <> Tailscale Kubernetes Operator Proxy Pod

Problem

When deploying with Kubernetes operator, and trying to access from an IP which sits in a remote subnet handled by a remote subnet router no connectivity can be established.

Troubleshooting

❯ curl 100.96.101.35 (Container Talent IP)

... times out

The log of the proxy container also shows Some peers are advertising routes but --accept-routes is false

2023/12/23 17:58:54 Switching ipn state Starting -> Running (WantRunning=true, nm=true)
2023/12/23 17:58:54 derphttp.Client.Connect: connecting to derp-18 (par)
2023/12/23 17:58:54 magicsock: endpoints changed: 31.19.220.235:59292 (stun), 10.1.0.93:46271 (local)
2023/12/23 17:58:54 control: NetInfo: NetInfo{varies=true hairpin=false ipv6=false ipv6os=false udp=true icmpv4=false derp=#18 portmap= link="" firewallmode="nft"}
Some peers are advertising routes but --accept-routes is false

Fixing connectivity:

After connecting to the proxy pd ts-*** and issuing tailscale up --accept-routes --reset connectivity starts to work as expected.

❯ curl 100.96.101.35
Hostname: whoami-56647559d7-wrgx5
IP: 127.0.0.1
IP: 10.1.0.90
RemoteAddr: 10.1.0.89:55287
GET / HTTP/1.1
Host: 100.96.101.35
User-Agent: curl/8.4.0
Accept: */*

Suggested fix

Operator needs to start tailscale proxy pods with TAILSCALE_ARGS: ...--accept-routes or needs to make TAILSCALE_ARGS configurable through the helm chart.

Steps to reproduce

10.1.1.100 <> Tailscale with subnet route 10.1.1.0/24 <> Kubernetes Operator <> Kubernetes Operator Proxy Pod

Operator Install:

helm upgrade \
  --install \
  tailscale-operator \
  tailscale/tailscale-operator \
  --namespace=tailscale \
  --create-namespace \
  --set-string oauth.clientId=kqqVCR3CNTRL \
  --set-string oauth.clientSecret=tskey-client-kqqVCR3CNTRL-***  \
  --set-string apiServerProxyConfig.mode=true \
  --wait

Test container:

helm upgrade --install whoami --set-string service.annotations."tailscale\.com\/expose"=true \
   cowboysysop/whoami --create-namespace -n whoami --wait

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

kubernetes-operator

Tailscale version

v1.56.1

Other software

No response

Bug report

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 17 (7 by maintainers)

Commits related to this issue

Most upvoted comments

Just a note that we are planning to do this. It is a relatively small piece of work, but it is blocked on https://github.com/tailscale/tailscale/issues/10869 as otherwise there is currently no way to reconfigure the --accept-routes value once a proxy device has been created. #10869 is high on our priority list, so I’d expect us to be able to get this done soon-ish.

hi @cubinet-code, did you figure out to solve this? I am looking into a similar issue.

No, I don’t think there is a way yet to set --accept-routes on the proxy in any way through the current manifests or CRDs.

#10724 just added other functionality in regards to exit nodes and proxy subnet router.

@irbekrm mentioned that creating a possibility to set “accept-routes” for proxy nodes is on the todo list though.

This would be super useful to enable the below routing as a kubernetes service!

Kubernetes --> Tailscale --> Machine with Route to 192.168.0.0/24–> Local Device on 192.168.0.0/24.

Thanks for the extra info!

❯ curl 100.96.101.35

Maybe I am missing something- do you run this from the 10.1.1.x client within the exposed subnet? If so, do you have Tailscale installed on the client device or how is it able to discover 100.96.101.35?

Yes, correct the client sits on 10.1.1.x. This machine has a static default route to the first Tailgate node ˙Tailscale linux instance with subnet route 10.1.1.0/24˙.

We could either introduce another Service/Ingress annotation (tailscale.com/accept-routes) or plug this into the generic proxy configuration work that we have in progress

This would perfectly address my use case! I just would like ensure that the proxy node will have the remote subnet routes available so the traffic finds it path back to the client.

best regards, Ollie.

What’s the status on this? @irbekrm

It won’t come out as part of 1.60 release as I was initially hoping, because we prioritized proxy cluster resource customization and that took longer than expected. So hopefully for the next release. Thanks for your patience. Generally we are also happy to review folks’ PRs, but this is blocked on #10869 which will require some re-designing of the tailscale image.