kubernetes: API server can return "Unauthorized" responses immediately after startup
/kind bug
What happened: When the Kubernetes API server starts up, requests return a 401 Unauthorized error for a few seconds after the API server starts. After about 5 seconds or so, requests start working again. In the API server logs, when this happens, we see this error:
E0703 21:19:13.231705 1 authentication.go:62] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, oidc: authenticator not initialized]]
I think this is because of this commit:
https://github.com/kubernetes/apiserver/commit/64c5ecc45e9252a46103abec2ddfe811ac8462bc
What you expected to happen: I expected the API server to not return any responses until it’s ready to authenticate requests. As it currently stands, every client of the API server needs to catch a 401 error and basically treat it the same as a “connection refused” error.
How to reproduce it (as minimally and precisely as possible): Make requests to the Kube API server in a loop like:
while true; do sleep 0.5; date; curl -i -X GET --cacert ca.pem --cert cert.pem --key key.pem https://<IP address>/api/v1/namespaces/default/pods; done
Then restart the Kube API server.
Environment: 1.11.0-beta.0
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 23 (9 by maintainers)
This should be fixed in v1.21 via #97693
/close
@wsong I believe the issue is due to how the OIDC authenticator within the apiserver is initialized; it seems to wait 10 seconds before initializing its internal verifier. Reference: plugin/pkg/authenticator/token/oidc/oidc.go
I imagine the fix would be to call
wait.PollImmediatelyUntilrather thanwait.PollUtil. I’d be happy to submit a PR.