kubernetes: Can not authenticate with service-account default token
I have a pod that uses the default service account token in the pod to speak to the API server. However, sometimes this token cannot be used to authenticate with the API server and the logs give this error:
E0302 12:32:50.114892 7 handlers.go:37] Unable to authenticate the request due to an error: crypto/rsa: verification error
The same thing happens when manually reading the token from the service account secret with kubectl and using the token to curl the API server using the token.
It appears that the token signature is invalid and that the token is either bad or the API server has changed its signing key and not updated the token.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 2
- Comments: 23 (12 by maintainers)
To get the default toker secret execute: sudo kubectl get secrets --all-namespaces the result will be something like this: NAMESPACE NAME TYPE DATA AGE calico-system default-token-jon84 kubernetes.io/service-account-token 3 5m default default-token-cqe56 kubernetes.io/service-account-token 3 4m kube-system default-token-6uce0 kubernetes.io/service-account-token 3 4m
To delete for example the kube-system namespace to regenerate, execute: sudo kubectl delete secret --namespace=kube-system default-token-6uce0
It ins’t necessary to restart kubelet service, the token will be create.
That’s what I don’t get. Why not? Everything else uses certs and validates using a CA. This even has the (api) infra in place with
--service-account-private-key-fileand--service-account-key-filewhich accept RSA private key and private key or signed cert.Why does the controller not follow the same pattern and instead of issuing a token, issue a cert, and have the API server validate it with a CA?
@EamonZhang
--service-account-private-key-fileprovided to the controller manager is used to sign service account tokens. The corresponding public key must be provided to the api server with--service-account-key-file, which uses it to verify tokens.As a convenience, you can provide a private key to both, and the public key portion of it will be used by the api server to verify token signatures.
As a further convenience, the api server’s private key for it’s serving certificate is used to verify service account tokens if you don’t specify
--service-account-key-file--tls-cert-fileand--tls-private-key-fileare used to provide the serving cert and key to the api server. If you don’t specify these, the api server will make a self-signed cert/key-pair and store it at apiserver.crt/apiserver.keyDid you try to delete the pod? It could still have the old token mounted maybe
Update: deleting the default token secret and letting Kubernetes create a new secret solved the problem. Still, it is a bit concerning that Kubernetes can allow invalid token secrets to stick around.
@erimatnor, how did you delete the default token secret?