kubernetes: Kubelet CNI nsenter failure

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: Bare metal
OS (e.g. from /etc/os-release): Ubuntu precise (12.04.4 LTS)
Kernel (e.g. uname -a): 3.13.0-55-generic
Install tools: N/A
Others: N/A

What happened:

During an otherwise-normal rollingupdate for a replication controller, we see this error in the logs:

E0307 21:43:51.235958    3650 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'foo-r3rg5' - Unexpected command output nsenter: cannot open /proc/5875/ns/net: No such file or directory\n with error: exit status 1

What you expected to happen:

Expected that successful rollingupdate wouldn’t generate error-level logs without an error-level problem – trying to determine what the error-level problem is.

How to reproduce it (as minimally and precisely as possible):

During our prod upgrade of various services, this occurred during 3 out of 70 rollingupdates.

Anything else we need to know:

Perhaps this is the same general issue as #25281? But I couldn’t find any references to the “Unexpected command output nsenter” variation seen above, thus filing separately.

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 27 (16 by maintainers)

Commits related to this issue

Merge pull request #46823 from dcbw/fix-up-runtime-GetNetNS2 Automatic merge from submit-queue (batch tested with PRs 46441, 43987, 46921, 46823, 47276) kubelet/network: report but tolerate errors r... — committed to kubernetes/kubernetes by deleted user 7 years ago

Most upvoted comments

I’m also getting this every now and then, and when it starts happening, the only workaround seems to be tearing down the cluster and bringing it back up. At least in my case it seems to be related to resource limits. I’m trying to run a pod both with and without a resource limit - every time I specify a memory limit, the pod is stuck at ContainerCreating, and every time I comment them out it starts Running without a hitch.

Here’s some logs from the kubelet trying to run the resource limited pods. I’m running with a kubeadm install + flannel.

EDIT: kubeadm, kubectl version 1.6.4

Jun 13 14:43:52 my.server.com kubelet[25154]: with error: exit status 1
Jun 13 14:43:52 my.server.com kubelet[25154]: W0613 14:43:52.344639   25154 docker_sandbox.go:263] NetworkPlugin cni failed on the status hook for pod "scala-test-bqr59_grader": Unexpected command output nsenter: cannot open : No such file or directory
Jun 13 14:43:52 my.server.com kubelet[25154]: with error: exit status 1
Jun 13 14:43:52 my.server.com kubelet[25154]: W0613 14:43:52.365529   25154 docker_sandbox.go:263] NetworkPlugin cni failed on the status hook for pod "scala-test-bqr59_grader": Unexpected command output nsenter: cannot open : No such file or directory
Jun 13 14:43:52 my.server.com kubelet[25154]: with error: exit status 1
Jun 13 14:43:52 my.server.com kubelet[25154]: W0613 14:43:52.387687   25154 docker_sandbox.go:263] NetworkPlugin cni failed on the status hook for pod "scala-test-bqr59_grader": Unexpected command output nsenter: cannot open : No such file or directory
Jun 13 14:43:52 my.server.com kubelet[25154]: with error: exit status 1
Jun 13 14:43:52 my.server.com kubelet[25154]: W0613 14:43:52.914538   25154 container.go:352] Failed to create summary reader for "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5940c2_502d_11e7_9b32_005056940720.slice/docker-875a65dd3e3c4e1e6919ce34e4d125699f5e64db8cdd804626a6eac38802b640.scope": none of the resources are being tracked.
Jun 13 14:43:52 my.server.com kubelet[25154]: E0613 14:43:52.915062   25154 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "scala-test-bqr59": Error response from daemon: {"message":"invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:286: decoding sync type from init pipe caused \\\\\\\"read parent: connection reset by peer\\\\\\\"\\\"\\n\""}
Jun 13 14:43:52 my.server.com kubelet[25154]: E0613 14:43:52.915133   25154 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "scala-test-bqr59_grader(6d5940c2-502d-11e7-9b32-005056940720)" failed: rpc error: code = 2 desc = failed to start sandbox container for pod "scala-test-bqr59": Error response from daemon: {"message":"invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:286: decoding sync type from init pipe caused \\\\\\\"read parent: connection reset by peer\\\\\\\"\\\"\\n\""}

murhum1 on Jun 13, 2017

https://github.com/kubernetes/kubernetes/pull/43879 may help this, though it has been reverted and needs some fixups.

dcbw on Jun 1, 2017