kubernetes: kubectl describe shows SandboxChanged which is not a helpful user message
I see the following kubectl describe message on a pod
Normal SandboxChanged 3m (x8 over 3m) kubelet, hostname.net Pod sandbox changed, it will be killed and re-created.
Further digging into the kubelet logs, i see the following
Error response from daemon: {"message":"no available IPv4 addresses on this network's address pools: bridge (872bba0c06bf6f65e6185240b5b6
It will be so much more awesome if we can actually display this helpful message in the events for that pod. SandBoxChanged to a cluster administrator has no meaning
kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.14", GitCommit:"d1303b003001cf2b5b8cec099383125096b3dac0", GitTreeState:"clean", BuildDate:"2018-03-12T16:24:22Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.14", GitCommit:"d1303b003001cf2b5b8cec099383125096b3dac0", GitTreeState:"clean", BuildDate:"2018-03-12T16:16:03Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 6
- Comments: 26 (6 by maintainers)
We just hit this case, which makes it hard to diagnose what the actual problem with the Pod/Container was.
Can this ticket be re-opened?
In our case, we reviewed the kubelet logs on the node, which indicated that the pod’s entrypoint process was failing. Then we had to look more broadly at all the system logs on the node to diagnose that the OS was killing the process because of memory pressure. It turned out that our RAM resource limits were set too low. It would have been very helpful if that root cause would have been floated up to the cluster events.
@Davidrjx i am not concerned about the specific problem in hand. We know why we are seeing the network issue which is basically because we have only limited ip per host. What i am suggesting here if instead of displaying
SandboxChangedevent for the pod, we can actually show the real error coming out from the docker which would be super helpful. SandboxChanged doesnt mean anything to the cluster operatorfor some reason it seems like 90% of the messages on this project are the bots telling us about how the question is stale.