kubernetes: kubelet failed to teardown pod network

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened: I get an Error log when deleting a pod:

I0912 17:24:28.106047   15221 plugins.go:405] Calling network plugin cni to tear down pod "choury-2_default"
I0912 17:24:28.106566   15221 cni.go:326] Got netns path /proc/16933/ns/net
I0912 17:24:28.106570   15221 cni.go:327] Using netns path default
I0912 17:24:28.106648   15221 cni.go:316] About to del CNI network containernet (type=flannel)
I0912 17:24:28.346771   15221 manager.go:989] Destroyed container: "/docker/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145" (aliases: [k8s_POD_choury-2_default_0bbb310e-979c-11e7-a60b-246e962f8920_0 69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145], namespace: "docker")
I0912 17:24:28.346800   15221 handler.go:325] Added event &{/docker/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145 2017-09-12 17:24:28.34679296 +0800 CST containerDeletion {<nil>}}
I0912 17:24:28.843224   15221 generic.go:146] GenericPLEG: 0bbb310e-979c-11e7-a60b-246e962f8920/63e008b581037b38b0bd11bb9f11fc01bdb2e044b10b7441719ba8acb4fa7476: running -> exited
I0912 17:24:28.843232   15221 generic.go:146] GenericPLEG: 0bbb310e-979c-11e7-a60b-246e962f8920/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: running -> exited
I0912 17:24:28.843691   15221 kuberuntime_manager.go:809] getSandboxIDByPodUID got sandbox IDs ["69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145"] for pod "choury-2_default(0bbb310e-979c-11e7-a60b-246e962f8920)"
I0912 17:24:28.846665   15221 generic.go:345] PLEG: Write status for choury-2/default: &container.PodStatus{ID:"0bbb310e-979c-11e7-a60b-246e962f8920", Name:"choury-2", Namespace:"default", IP:"", ContainerStatuses:[]*container.ContainerStatus{(*container.ContainerStatus)(0xc4210f2540)}, SandboxStatuses:[]*runtime.PodSandboxStatus{(*runtime.PodSandboxStatus)(0xc4214b1680)}} (err: <nil>)
I0912 17:24:28.846698   15221 kubelet.go:1849] SyncLoop (PLEG): "choury-2_default(0bbb310e-979c-11e7-a60b-246e962f8920)", event: &pleg.PodLifecycleEvent{ID:"0bbb310e-979c-11e7-a60b-246e962f8920", Type:"ContainerDied", Data:"63e008b581037b38b0bd11bb9f11fc01bdb2e044b10b7441719ba8acb4fa7476"}
I0912 17:24:28.846720   15221 kubelet.go:1849] SyncLoop (PLEG): "choury-2_default(0bbb310e-979c-11e7-a60b-246e962f8920)", event: &pleg.PodLifecycleEvent{ID:"0bbb310e-979c-11e7-a60b-246e962f8920", Type:"ContainerDied", Data:"69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145"}
W0912 17:24:28.846732   15221 pod_container_deletor.go:77] Container "69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145" not found in pod's containers
I0912 17:24:28.846747   15221 kubelet_pods.go:1284] Generating status for "choury-2_default(0bbb310e-979c-11e7-a60b-246e962f8920)"
I0912 17:24:28.847422   15221 plugins.go:405] Calling network plugin cni to tear down pod "choury-2_default"
W0912 17:24:28.847895   15221 cni.go:265] CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145"
I0912 17:24:28.847902   15221 cni.go:326] Got netns path
I0912 17:24:28.847904   15221 cni.go:327] Using netns path default
I0912 17:24:28.847951   15221 cni.go:316] About to del CNI network containernet (type=flannel)
E0912 17:24:28.850957   15221 cni.go:319] Error deleting network: open /var/lib/cni/flannel/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: no such file or directory
E0912 17:24:28.851430   15221 remote_runtime.go:115] StopPodSandbox "69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "choury-2_default" network: open /var/lib/cni/flannel/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: no such file or directory
E0912 17:24:28.851477   15221 kuberuntime_manager.go:756] Failed to stop sandbox {"docker" "69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145"}
E0912 17:24:28.851514   15221 kubelet.go:1510] error killing pod: failed to "KillPodSandbox" for "0bbb310e-979c-11e7-a60b-246e962f8920" with KillPodSandboxError: "rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod \"choury-2_default\" network: open /var/lib/cni/flannel/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: no such file or directory"
E0912 17:24:28.851571   15221 pod_workers.go:182] Error syncing pod 0bbb310e-979c-11e7-a60b-246e962f8920 ("choury-2_default(0bbb310e-979c-11e7-a60b-246e962f8920)"), skipping: error killing pod: failed to "KillPodSandbox" for "0bbb310e-979c-11e7-a60b-246e962f8920" with KillPodSandboxError: "rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod \"choury-2_default\" network: open /var/lib/cni/flannel/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: no such file or directory"
I0912 17:24:28.851666   15221 server.go:226] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"choury-2", UID:"0bbb310e-979c-11e7-a60b-246e962f8920", APIVersion:"v1", ResourceVersion:"14649589", FieldPath:""}): type: 'Warning' reason: 'FailedSync' Error syncing pod

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?: It seems kubelet does not think that the first deletion is successful ?

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-beta.1.83+586802fc9d9401-dirty", GitCommit:"586802fc9d940148e9655fc1a934adb122c61d8f", GitTreeState:"dirty", BuildDate:"2017-09-12T07:52:49Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-beta.1.83+586802fc9d9401-dirty", GitCommit:"586802fc9d940148e9655fc1a934adb122c61d8f", GitTreeState:"dirty", BuildDate:"2017-09-12T07:52:49Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration**: docker version
Client:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6
 Git commit:   dc67f83
 Built:        Thu Nov 24 16:50:12 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6
 Git commit:   daf0db7
 Built:        Mon Oct 31 15:46:15 2016
 OS/Arch:      linux/amd64
  • OS (e.g. from /etc/os-release): centos 6.4
  • Kernel (e.g. uname -a): linux 3.10.104
  • Install tools:
  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 16 (16 by maintainers)

Most upvoted comments

We should move this off of the 1.8 milestone and close.