kubernetes: Ephemeral container fails to be created for Static Pod
What happened?
I created an ephemeral container for a Static Pod using the kubectl debug
command. The command succeed.
❯ kubectl debug --image=busybox --share-processes=true --target=kube-apiserver -n kube-system kube-apiserver-kind-control-plane -- sleep infinity
Targeting container "kube-apiserver". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-vgkn4.
The API server accepted the ephemeral container:
❯ kubectl -n kube-system get pod kube-apiserver-kind-control-plane --output=jsonpath='{.spec.ephemeralContainers}'
[{"command":["sleep","infinity"],"image":"busybox","imagePullPolicy":"Always","name":"debugger-vgkn4","resources":{},"targetContainerName":"kube-apiserver","terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"}]
However, the container is not created.
Its status is also not available:
❯ kubectl -n kube-system get pod kube-apiserver-kind-control-plane --output=jsonpath='{.status.ephemeralContainerStatuses}'
What did you expect to happen?
I expected one of these:
(a) the API server rejects the request to create an ephemeral container for a Static Pod (b) kubelet creates the ephemeral container, and update the Static Pod’s status sub-resource
How can we reproduce it (as minimally and precisely as possible)?
Using a recent version of docker engine, kind v0.17 or newer, and kubectl v1.25.0 or newer, create a cluster and create an ephemeral container running sleep infinity
. Since the ephemeral container is not running, the output of grep
is 0.
kind create cluster
kubectl debug --image=busybox --share-processes=true --target=kube-apiserver -n kube-system kube-apiserver-kind-control-plane -- sleep infinity
docker exec -it kind-control-plane ps -axw | grep --count "sleep infinity"
Anything else we need to know?
The Ephemeral Containers KEP does not address Static Pods.
Kubernetes version
$ kubectl version --short
Client Version: v1.25.3
Kustomize Version: v4.5.7
Server Version: v1.25.3
Cloud provider
None
OS version
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
Install tools
Container runtime (CRI) and version (if applicable)
❯ docker exec -it kind-control-plane containerd --version
containerd github.com/containerd/containerd v1.6.9 1c90a442489720eec95342e1789ee8a5e1b9536f
Related plugins (CNI, CSI, …) and versions (if applicable)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 24 (24 by maintainers)
Commits related to this issue
- updated docs regarding ephemeral pods not being supported by static pods kubernetes/kubernetes#113935 — committed to xmcqueen/website by xmcqueen 2 years ago
- updated docs regarding ephemeral pods not being supported by static pods kubernetes/kubernetes#113935 — committed to nilekhc/website by xmcqueen 2 years ago
- updated docs regarding ephemeral pods not being supported by static pods kubernetes/kubernetes#113935 — committed to ndixita/website by xmcqueen 2 years ago
- updated docs regarding ephemeral pods not being supported by static pods kubernetes/kubernetes#113935 — committed to ndixita/website by xmcqueen 2 years ago
I’ll take it, per the discussion here: add a check in
ValidatePodEphemeralContainersUpdate
to reject the attempt to add an ephemeral container if the request is aimed at a static pod. Also include a test case.Also some additional documantation comments from this ticket and from kubernetes/website#37993
We should not allow ephemeral containers for static pods… the API content is not authoritative for static pods, the manifests on disk are, so I would expect the kubelet to ignore API spec content and the API to disallow adding ephemeral containers to static pods
Does the API server actually know that mirror pods are mirror pods? I wonder how you can tell.
It sounds like a documentation update would be appropriate here. I think the Static Pods page would be a good place for it. I’ll get a PR going.
Thanks @dlipovetsky
I do see a difference in the test results, but I still can’t exec into the ephemeral container. I’ll run some more tests. Thanks
Here is the pod spec showing something for the ephemeral container, but i can’t exec into it anyway: