tailscale: k8s operator leaving expired authkey in proxy Secrets

What is the issue?

Expose a k8s service on the tailnet, eg:

apiVersion: v1
kind: Service
metadata:
  name: octobox-server-tailscale
  labels:
    app: octobox
  annotations:
    tailscale.com/hostname: octobox
    tailscale.com/tags: tag:k8s-octobox
spec:
  type: LoadBalancer
  loadBalancerClass: tailscale
  ports:
    - name: http
      port: 443
      targetPort: 3001
  selector:
    app: octobox

The tailscale proxy statefulset and secret are created and the service becomes available on the tailnet.

After some time (not sure exactly… as low as a few days) the statefulset pod is restarted due to normal k8s operations, ie: node ttl, node drains/upgrades, node failures, etc). The new pod is unable to join the tailnet. Error in the statefulset proxy pod logs:

2024/09/12 17:05:17 control: RegisterReq: got response; nodeKeyExpired=false, machineAuthorized=false; authURL=false
2024/09/12 17:05:17 Received error: invalid key: API key does not exist
2024/09/12 17:05:17 health(warnable=login-state): error: You are logged out. The last login error was: invalid key: API key does not exist
2024/09/12 17:05:17 control: doLogin(regen=false, hasUrl=false)
2024/09/12 17:05:17 control: RegisterReq: onode= node=[U7Pe6] fup=false nks=false

For an unknown reason this is only impacting new services that we have added/annotated in the last few months. Unfortunately I can’t give a more specific timeframe.

The only thing different I see between the older services which are able to restart fine and new services which cannot restart is that the newer services have an Authkey field in their tailscale config k8s Secret objects, eg:

GOOD:

cap-95.hujson={"Version":"alpha0","Locked":false,"Hostname":"my-service","acceptDNS":false,"acceptRoutes":false,"NoStatefulFiltering":false}
tailscaled={"Version":"alpha0","Locked":false,"Hostname":"my-service","acceptDNS":false,"acceptRoutes":false}

BAD:

cap-95.hujson={"Version":"alpha0","Locked":false,"AuthKey":"tskey-auth-REDACTED","Hostname":"my-bad-serivce","acceptDNS":false,"acceptRoutes":false,"NoStatefulFiltering":false}
tailscaled={"Version":"alpha0","Locked":false,"AuthKey":"tskey-auth-REDACTED","Hostname":"my-bad-service","acceptDNS":false,"acceptRoutes":false}

What I suspect is happening is:

  1. ts operator creates new STS + Secret k8s objects, including creating a fresh Authkey that is stored in the secret
  2. ts proxy pod starts up, uses authkey to join the tailnet
  3. authkey expires
  4. ts proxy pod restarts, sees the Authkey in the config secret, attempts to use it and fails to auth because it is expired.

I have found several ways to work around those. None of them ideal since it requires a human to manually fix services every week or two:

  1. scale operator down to 0; delete sts + secret for the broken service in the tailscale namespace: kubectl delete sts,pod,svc,secret -ltailscale.com/parent-resource=my-service -n tailscale; scale operator back to 1. Operator creates new STS and secret. New proxy pod starts up and joins tailnet. (ALSO need to manually delete the machine from tailscale console otherwise the new pod comes up with -1 appended to its tailnet hostname)
  2. or, kubectl edit the broken proxies k8s secret and remove the cap-95.hujson and tailscaled keys. Delete/restart the proxy pod

My suspicion is that something should be deleting the Authkey from the secret after first auth. It does not appear that the operator tries to do this. I can see in cmd/k8s-operator/sts.go where it attempts to generate a new authkey only if one is not set in the Secret yet. And if one is set, it preserves it when it ‘updates’ the Secret.

I see a deleteAuthKey() func in cmd/containerboot that might be called by the pod to patch its own Secret if TS_AUTH_ONCE is set but the operator does not set this on the STS pods it creates nor do I see a way to configure the operator to set this or any other env vars on the STS pods it creates.

Looking for any guidance on how to resolve this situation. thank you

OS

Linux

OS version

ubuntu 20.04 EKS ami’s

Tailscale version

1.72.1

Other software

No response

Bug report

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 4
  • Comments: 17 (13 by maintainers)

Commits related to this issue

Most upvoted comments

We will cut a new stable release next week that will have this change - will you folks be able to wait for that?

No worries. Let it simmer for a while 😊 thanks

Hi @irbekrm, I deployed k8s-operator/unstable-v1.75.48 to Dev and checked the services currently failing were not recovered. Is this expected?

@GMartinez-Sisti could you produce a bugreport for either of those/show us the logs?

Also, would you be able to verify that the auth key has been deleted? (You could run something like kubectl get secret -n tailscale <secret-name> -ojson | jq -r '.data["cap-95.hujson"]' | base64 -d to check that)

It worked! We had a couple of issues on our operators that I missed after upgrading to the unstable version. We were on 1.72.1 so we were missing the new RBAC permissions and the new recorders CRD from 1.74.1. After fixing it the failing agents restarted and are now online.

I can also confirm that the key AuthKey disappeared from the secrets. 🚀

Thank you for the help @irbekrm. Let me know if you need more tests.

@irbekrm i was not able to test the PR. @GMartinez-Sisti’s test gives me enough confidence though, and we know how to manually fix services at the moment. i’ll wait on the release. ty very much for persisting on this issue!!

Awesome, thanks for confirming that!

Let me know if you need more tests.

There aren’t any edge cases I can think of at the moment.

We will cut a new stable release next week that will have this change - will you folks be able to wait for that?

Thanks a lot @GMartinez-Sisti, this is appreciated.

I’ll send this information to Remy.

In your case, I know which tailnet you are, this was more aimed at other folks who might see this (I saw a few thumbs up on this issue now)

We are having the exact same issue. This only started to happen after updating to the latest k8s-operator:v1.72.1, however since we updated from a very old version incrementally in a couple of days I can’t tell for sure which one introduced the issue.

The steps provided in the issue description are the same we perform to fix this. Since this requires manually intervention, engineers are starting to lose the trust on tailscale. We have 100+ clusters and almost half of them will have a random tailscale agent crashed with this issue.

what is the expected sequence of events during startup/shutdown with regards to the initial authkey and whatever is supposed to happen on successive startups?

Sorry, it’s hard for me to describe what the flow is because large part of it is control plane side.

In this case, what matters is that:

Basically what should happen on startup flow for a tagged node (and, no matter whether the auth key is set/expired):

  1. a node comes up and attempts to log in here-ish
  2. crucially, expired will be set to false here because expiry should be 0 as it’s a tagged node
  3. because it is not expired, node key is NOT regenerated (which means that re-auth is not needed)

Whilst for startup flow for a node that is not tagged and is expired/needs re-auth regen will be set to true at step 3 which means that node key will be regenerated, device attempts to register with control plane with a new node key, which would mean that it needs to re-auth and then fail because the auth key is expired.

If you happen to have a node that is currently in this state, it would be good to get a handle, so we can take a look at it from our side. I still haven’t been able to reproduce this with operator proxies. I’m also curious if there are any tailnet settings that would affect device auth that you might have changed recently, i.e device approval etc?

@irbekrm here is one from a pod that i’ve already “fixed” by removing the authkey manually: BUG-c0c90dd4ff29cf28a3c1520b3b2d28cda5848c99f0d82e64fdd273cdba34d305-20240912195424Z-015bba2a8316fa14

and this is from another proxy pod that has authkey in its config secret and i expect will fail to join the tailnet on the next pod restart: BUG-5bdc43024899d167c2d458f0f743b9233c194db16867eb07475b4a1dc9e3797b-20240912202228Z-a56be7307e438841