tailscale: k8s operator leaving expired authkey in proxy Secrets
What is the issue?
Expose a k8s service on the tailnet, eg:
apiVersion: v1
kind: Service
metadata:
name: octobox-server-tailscale
labels:
app: octobox
annotations:
tailscale.com/hostname: octobox
tailscale.com/tags: tag:k8s-octobox
spec:
type: LoadBalancer
loadBalancerClass: tailscale
ports:
- name: http
port: 443
targetPort: 3001
selector:
app: octobox
The tailscale proxy statefulset and secret are created and the service becomes available on the tailnet.
After some time (not sure exactly… as low as a few days) the statefulset pod is restarted due to normal k8s operations, ie: node ttl, node drains/upgrades, node failures, etc). The new pod is unable to join the tailnet. Error in the statefulset proxy pod logs:
2024/09/12 17:05:17 control: RegisterReq: got response; nodeKeyExpired=false, machineAuthorized=false; authURL=false
2024/09/12 17:05:17 Received error: invalid key: API key does not exist
2024/09/12 17:05:17 health(warnable=login-state): error: You are logged out. The last login error was: invalid key: API key does not exist
2024/09/12 17:05:17 control: doLogin(regen=false, hasUrl=false)
2024/09/12 17:05:17 control: RegisterReq: onode= node=[U7Pe6] fup=false nks=false
For an unknown reason this is only impacting new services that we have added/annotated in the last few months. Unfortunately I can’t give a more specific timeframe.
The only thing different I see between the older services which are able to restart fine and new services which cannot restart is that the newer services have an Authkey field in their tailscale config k8s Secret objects, eg:
GOOD:
cap-95.hujson={"Version":"alpha0","Locked":false,"Hostname":"my-service","acceptDNS":false,"acceptRoutes":false,"NoStatefulFiltering":false}
tailscaled={"Version":"alpha0","Locked":false,"Hostname":"my-service","acceptDNS":false,"acceptRoutes":false}
BAD:
cap-95.hujson={"Version":"alpha0","Locked":false,"AuthKey":"tskey-auth-REDACTED","Hostname":"my-bad-serivce","acceptDNS":false,"acceptRoutes":false,"NoStatefulFiltering":false}
tailscaled={"Version":"alpha0","Locked":false,"AuthKey":"tskey-auth-REDACTED","Hostname":"my-bad-service","acceptDNS":false,"acceptRoutes":false}
What I suspect is happening is:
- ts operator creates new STS + Secret k8s objects, including creating a fresh Authkey that is stored in the secret
- ts proxy pod starts up, uses authkey to join the tailnet
- authkey expires
- ts proxy pod restarts, sees the
Authkeyin the config secret, attempts to use it and fails to auth because it is expired.
I have found several ways to work around those. None of them ideal since it requires a human to manually fix services every week or two:
- scale operator down to 0; delete sts + secret for the broken service in the tailscale namespace:
kubectl delete sts,pod,svc,secret -ltailscale.com/parent-resource=my-service -n tailscale; scale operator back to 1. Operator creates new STS and secret. New proxy pod starts up and joins tailnet. (ALSO need to manually delete the machine from tailscale console otherwise the new pod comes up with-1appended to its tailnet hostname) - or,
kubectl editthe broken proxies k8s secret and remove thecap-95.hujsonandtailscaledkeys. Delete/restart the proxy pod
My suspicion is that something should be deleting the Authkey from the secret after first auth. It does not appear that the operator tries to do this. I can see in cmd/k8s-operator/sts.go where it attempts to generate a new authkey only if one is not set in the Secret yet. And if one is set, it preserves it when it ‘updates’ the Secret.
I see a deleteAuthKey() func in cmd/containerboot that might be called by the pod to patch its own Secret if TS_AUTH_ONCE is set but the operator does not set this on the STS pods it creates nor do I see a way to configure the operator to set this or any other env vars on the STS pods it creates.
Looking for any guidance on how to resolve this situation. thank you
OS
Linux
OS version
ubuntu 20.04 EKS ami’s
Tailscale version
1.72.1
Other software
No response
Bug report
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 4
- Comments: 17 (13 by maintainers)
Commits related to this issue
- cmd/k8s-operator: remove auth key once proxy has logged in The operator creates a non-reusable auth key for each of the cluster proxies that it creates and puts in the tailscaled configfile mounted t... — committed to tailscale/tailscale by irbekrm 2 years ago
- cmd/k8s-operator: remove auth key once proxy has logged in The operator creates a non-reusable auth key for each of the cluster proxies that it creates and puts in the tailscaled configfile mounted t... — committed to tailscale/tailscale by irbekrm 2 years ago
- cmd/k8s-operator: remove auth key once proxy has logged in (#13612) The operator creates a non-reusable auth key for each of the cluster proxies that it creates and puts in the tailscaled configfil... — committed to tailscale/tailscale by irbekrm 2 years ago
No worries. Let it simmer for a while 😊 thanks
It worked! We had a couple of issues on our operators that I missed after upgrading to the unstable version. We were on
1.72.1so we were missing the new RBAC permissions and the new recorders CRD from1.74.1. After fixing it the failing agents restarted and are now online.I can also confirm that the key
AuthKeydisappeared from the secrets. 🚀Thank you for the help @irbekrm. Let me know if you need more tests.
@irbekrm i was not able to test the PR. @GMartinez-Sisti’s test gives me enough confidence though, and we know how to manually fix services at the moment. i’ll wait on the release. ty very much for persisting on this issue!!
Awesome, thanks for confirming that!
There aren’t any edge cases I can think of at the moment.
We will cut a new stable release next week that will have this change - will you folks be able to wait for that?
Thanks a lot @GMartinez-Sisti, this is appreciated.
In your case, I know which tailnet you are, this was more aimed at other folks who might see this (I saw a few thumbs up on this issue now)
We are having the exact same issue. This only started to happen after updating to the latest
k8s-operator:v1.72.1, however since we updated from a very old version incrementally in a couple of days I can’t tell for sure which one introduced the issue.The steps provided in the issue description are the same we perform to fix this. Since this requires manually intervention, engineers are starting to lose the trust on tailscale. We have 100+ clusters and almost half of them will have a random tailscale agent crashed with this issue.
Sorry, it’s hard for me to describe what the flow is because large part of it is control plane side.
In this case, what matters is that:
Basically what should happen on startup flow for a tagged node (and, no matter whether the auth key is set/expired):
Whilst for startup flow for a node that is not tagged and is expired/needs re-auth regen will be set to true at step 3 which means that node key will be regenerated, device attempts to register with control plane with a new node key, which would mean that it needs to re-auth and then fail because the auth key is expired.
If you happen to have a node that is currently in this state, it would be good to get a handle, so we can take a look at it from our side. I still haven’t been able to reproduce this with operator proxies. I’m also curious if there are any tailnet settings that would affect device auth that you might have changed recently, i.e device approval etc?
@irbekrm here is one from a pod that i’ve already “fixed” by removing the authkey manually:
BUG-c0c90dd4ff29cf28a3c1520b3b2d28cda5848c99f0d82e64fdd273cdba34d305-20240912195424Z-015bba2a8316fa14and this is from another proxy pod that has authkey in its config secret and i expect will fail to join the tailnet on the next pod restart:
BUG-5bdc43024899d167c2d458f0f743b9233c194db16867eb07475b4a1dc9e3797b-20240912202228Z-a56be7307e438841