kubernetes: Ephemeral container fails to be created for Static Pod

What happened?

I created an ephemeral container for a Static Pod using the kubectl debug command. The command succeed.

❯ kubectl debug --image=busybox --share-processes=true --target=kube-apiserver -n kube-system kube-apiserver-kind-control-plane -- sleep infinity
Targeting container "kube-apiserver". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-vgkn4.

The API server accepted the ephemeral container:

❯ kubectl -n kube-system get pod kube-apiserver-kind-control-plane --output=jsonpath='{.spec.ephemeralContainers}'
[{"command":["sleep","infinity"],"image":"busybox","imagePullPolicy":"Always","name":"debugger-vgkn4","resources":{},"targetContainerName":"kube-apiserver","terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"}]

However, the container is not created.

Its status is also not available:

❯ kubectl -n kube-system get pod kube-apiserver-kind-control-plane --output=jsonpath='{.status.ephemeralContainerStatuses}'

What did you expect to happen?

I expected one of these:

(a) the API server rejects the request to create an ephemeral container for a Static Pod (b) kubelet creates the ephemeral container, and update the Static Pod’s status sub-resource

How can we reproduce it (as minimally and precisely as possible)?

Using a recent version of docker engine, kind v0.17 or newer, and kubectl v1.25.0 or newer, create a cluster and create an ephemeral container running sleep infinity. Since the ephemeral container is not running, the output of grep is 0.

kind create cluster
kubectl debug --image=busybox --share-processes=true --target=kube-apiserver -n kube-system kube-apiserver-kind-control-plane -- sleep infinity
docker exec -it kind-control-plane ps -axw | grep --count "sleep infinity"

Anything else we need to know?

The Ephemeral Containers KEP does not address Static Pods.

Kubernetes version

$ kubectl version --short
Client Version: v1.25.3
Kustomize Version: v4.5.7
Server Version: v1.25.3

Cloud provider

None

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

❯ docker exec -it kind-control-plane containerd --version containerd github.com/containerd/containerd v1.6.9 1c90a442489720eec95342e1789ee8a5e1b9536f

Related plugins (CNI, CSI, …) and versions (if applicable)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 24 (24 by maintainers)

Commits related to this issue

Most upvoted comments

I’ll take it, per the discussion here: add a check in ValidatePodEphemeralContainersUpdate to reject the attempt to add an ephemeral container if the request is aimed at a static pod. Also include a test case.

Also some additional documantation comments from this ticket and from kubernetes/website#37993

We should not allow ephemeral containers for static pods… the API content is not authoritative for static pods, the manifests on disk are, so I would expect the kubelet to ignore API spec content and the API to disallow adding ephemeral containers to static pods

Does the API server actually know that mirror pods are mirror pods? I wonder how you can tell.

It sounds like a documentation update would be appropriate here. I think the Static Pods page would be a good place for it. I’ll get a PR going.

Thanks @dlipovetsky

I do see a difference in the test results, but I still can’t exec into the ephemeral container. I’ll run some more tests. Thanks

Here is the pod spec showing something for the ephemeral container, but i can’t exec into it anyway:

bmcqueen-mn1:~ bmcqueen$ kubectl -n kube-system get pod kube-apiserver-kind-control-plane --output=jsonpath='{.spec.ephemeralContainers}' | jq .
[
  {
    "command": [
      "sleep",
      "infinity"
    ],
    "image": "busybox",
    "imagePullPolicy": "Always",
    "name": "debugger-j5x5g",
    "resources": {},
    "targetContainerName": "kube-apiserver",
    "terminationMessagePath": "/dev/termination-log",
    "terminationMessagePolicy": "File"
  }
]
bmcqueen-mn1:~ bmcqueen$ kubectl -n kube-system exec -it kube-apiserver-kind-control-plane -c debugger-j5x5g -- sh
error: unable to upgrade connection: container not found ("debugger-j5x5g")
bmcqueen-mn1:~ bmcqueen$