kubernetes: Potential inconsistent behaviour with apiserver apiextensions (custom resources) and 404s

What happened?

I’m working on a system the extends the kubernetes api server. This system consists of a kubernetes cluster, with several custom resource definitions and custom resources of each kind. These resources are consumed by in-cluster and external clients. The difference between the two types of clients is that the in-cluster client uses a cached client for reads, while external clients use the REST API for reads as well as writes.

Including the kubernetes components and other components that are deployed, the system has a few dozen in-cluster clients (including kubelets). The number of external clients is > 5000.

Recently, there was an issue with all the k8s-apiservers of the cluster restarting at the same time. The in-cluster clients failed during this time window, eventually resulting in container restarts. And this was expected.

The potential inconsistent behaviour is in relation to the external clients. Per system logs (apiserver audit logs), around the time of the incident, the following two behaviours were observed

• external clients receiving a 5xx response, (expected)
• external clients receiving 404s for resources that exist.

Most of the external clients got 5xx’s, but there were four instances where the response was a 404. Even in these cases (404s), the clients received 200 responses within ~30 seconds of receiving a 404.

During the incident window, the restarts of the k8s apiservers were observed at different times over a ~30s window. No delay was observed in the container restarts, and each apiserver pod was back up and receiving traffic within 30-40 seconds (or sooner).

• per the audit logs, the same external client received, 200, 404, 200, 404, 200, 200, 200, … (at ~30s intervals).
• the audit logs are missing any information regarding the apiserver instance that emitted the log.
• it’s likely that the different 404s were seen by the same external client (for the same resource), for responses from two different apiserver instances

What did you expect to happen?

All external clients should’ve received 5xx response codes if apiserver was down. Otherwise, the response should have been a 200, (because the resource the client was requesting was known to exist in etcd).

How can we reproduce it (as minimally and precisely as possible)?

• create k8s cluster.
• add several custom resource definitions, and several resources of each kind
• (may or may not be relevant: etcd size is approximately 4 GB)
• run a few clients that use the k8s REST API to query for different resources that were created earlier
• (overall rate of query ^ should be high, so that the intermittent inconsistency is observed).
• shut down all api servers.
• (clients continue: start seeing 5xx)
• restart the api servers.
• watch for 404s in client logs
• since clients are only requesting resources that are already created, 404 is not an expected response.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:33:49Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.10", GitCommit:"a7a32748b5c60445c4c7ee904caf01b91f2dbb71", GitTreeState:"clean", BuildDate:"2022-02-16T11:18:16Z", GoVersion:"go1.16.14", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

OS version

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ uname -a
Linux cp-vmss000001 5.15.0-1019-azure #24~20.04.1-Ubuntu SMP Tue Aug 23 15:52:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)