kubernetes: API server can return "Unauthorized" responses immediately after startup

/kind bug

What happened: When the Kubernetes API server starts up, requests return a 401 Unauthorized error for a few seconds after the API server starts. After about 5 seconds or so, requests start working again. In the API server logs, when this happens, we see this error:

E0703 21:19:13.231705       1 authentication.go:62] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, oidc: authenticator not initialized]]

I think this is because of this commit:

https://github.com/kubernetes/apiserver/commit/64c5ecc45e9252a46103abec2ddfe811ac8462bc

What you expected to happen: I expected the API server to not return any responses until it’s ready to authenticate requests. As it currently stands, every client of the API server needs to catch a 401 error and basically treat it the same as a “connection refused” error.

How to reproduce it (as minimally and precisely as possible): Make requests to the Kube API server in a loop like:

while true; do sleep 0.5; date; curl -i -X GET --cacert ca.pem --cert cert.pem --key key.pem https://<IP address>/api/v1/namespaces/default/pods; done

Then restart the Kube API server.

Environment: 1.11.0-beta.0

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 23 (9 by maintainers)

Most upvoted comments

This should be fixed in v1.21 via #97693

/close

@wsong I believe the issue is due to how the OIDC authenticator within the apiserver is initialized; it seems to wait 10 seconds before initializing its internal verifier. Reference: plugin/pkg/authenticator/token/oidc/oidc.go

I imagine the fix would be to call wait.PollImmediatelyUntil rather than wait.PollUtil. I’d be happy to submit a PR.