kubernetes: kubelet failed to teardown pod network
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug
What happened: I get an Error log when deleting a pod:
I0912 17:24:28.106047 15221 plugins.go:405] Calling network plugin cni to tear down pod "choury-2_default"
I0912 17:24:28.106566 15221 cni.go:326] Got netns path /proc/16933/ns/net
I0912 17:24:28.106570 15221 cni.go:327] Using netns path default
I0912 17:24:28.106648 15221 cni.go:316] About to del CNI network containernet (type=flannel)
I0912 17:24:28.346771 15221 manager.go:989] Destroyed container: "/docker/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145" (aliases: [k8s_POD_choury-2_default_0bbb310e-979c-11e7-a60b-246e962f8920_0 69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145], namespace: "docker")
I0912 17:24:28.346800 15221 handler.go:325] Added event &{/docker/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145 2017-09-12 17:24:28.34679296 +0800 CST containerDeletion {<nil>}}
I0912 17:24:28.843224 15221 generic.go:146] GenericPLEG: 0bbb310e-979c-11e7-a60b-246e962f8920/63e008b581037b38b0bd11bb9f11fc01bdb2e044b10b7441719ba8acb4fa7476: running -> exited
I0912 17:24:28.843232 15221 generic.go:146] GenericPLEG: 0bbb310e-979c-11e7-a60b-246e962f8920/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: running -> exited
I0912 17:24:28.843691 15221 kuberuntime_manager.go:809] getSandboxIDByPodUID got sandbox IDs ["69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145"] for pod "choury-2_default(0bbb310e-979c-11e7-a60b-246e962f8920)"
I0912 17:24:28.846665 15221 generic.go:345] PLEG: Write status for choury-2/default: &container.PodStatus{ID:"0bbb310e-979c-11e7-a60b-246e962f8920", Name:"choury-2", Namespace:"default", IP:"", ContainerStatuses:[]*container.ContainerStatus{(*container.ContainerStatus)(0xc4210f2540)}, SandboxStatuses:[]*runtime.PodSandboxStatus{(*runtime.PodSandboxStatus)(0xc4214b1680)}} (err: <nil>)
I0912 17:24:28.846698 15221 kubelet.go:1849] SyncLoop (PLEG): "choury-2_default(0bbb310e-979c-11e7-a60b-246e962f8920)", event: &pleg.PodLifecycleEvent{ID:"0bbb310e-979c-11e7-a60b-246e962f8920", Type:"ContainerDied", Data:"63e008b581037b38b0bd11bb9f11fc01bdb2e044b10b7441719ba8acb4fa7476"}
I0912 17:24:28.846720 15221 kubelet.go:1849] SyncLoop (PLEG): "choury-2_default(0bbb310e-979c-11e7-a60b-246e962f8920)", event: &pleg.PodLifecycleEvent{ID:"0bbb310e-979c-11e7-a60b-246e962f8920", Type:"ContainerDied", Data:"69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145"}
W0912 17:24:28.846732 15221 pod_container_deletor.go:77] Container "69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145" not found in pod's containers
I0912 17:24:28.846747 15221 kubelet_pods.go:1284] Generating status for "choury-2_default(0bbb310e-979c-11e7-a60b-246e962f8920)"
I0912 17:24:28.847422 15221 plugins.go:405] Calling network plugin cni to tear down pod "choury-2_default"
W0912 17:24:28.847895 15221 cni.go:265] CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145"
I0912 17:24:28.847902 15221 cni.go:326] Got netns path
I0912 17:24:28.847904 15221 cni.go:327] Using netns path default
I0912 17:24:28.847951 15221 cni.go:316] About to del CNI network containernet (type=flannel)
E0912 17:24:28.850957 15221 cni.go:319] Error deleting network: open /var/lib/cni/flannel/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: no such file or directory
E0912 17:24:28.851430 15221 remote_runtime.go:115] StopPodSandbox "69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "choury-2_default" network: open /var/lib/cni/flannel/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: no such file or directory
E0912 17:24:28.851477 15221 kuberuntime_manager.go:756] Failed to stop sandbox {"docker" "69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145"}
E0912 17:24:28.851514 15221 kubelet.go:1510] error killing pod: failed to "KillPodSandbox" for "0bbb310e-979c-11e7-a60b-246e962f8920" with KillPodSandboxError: "rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod \"choury-2_default\" network: open /var/lib/cni/flannel/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: no such file or directory"
E0912 17:24:28.851571 15221 pod_workers.go:182] Error syncing pod 0bbb310e-979c-11e7-a60b-246e962f8920 ("choury-2_default(0bbb310e-979c-11e7-a60b-246e962f8920)"), skipping: error killing pod: failed to "KillPodSandbox" for "0bbb310e-979c-11e7-a60b-246e962f8920" with KillPodSandboxError: "rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod \"choury-2_default\" network: open /var/lib/cni/flannel/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: no such file or directory"
I0912 17:24:28.851666 15221 server.go:226] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"choury-2", UID:"0bbb310e-979c-11e7-a60b-246e962f8920", APIVersion:"v1", ResourceVersion:"14649589", FieldPath:""}): type: 'Warning' reason: 'FailedSync' Error syncing pod
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?: It seems kubelet does not think that the first deletion is successful ?
Environment:
- Kubernetes version (use
kubectl version
):
Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-beta.1.83+586802fc9d9401-dirty", GitCommit:"586802fc9d940148e9655fc1a934adb122c61d8f", GitTreeState:"dirty", BuildDate:"2017-09-12T07:52:49Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-beta.1.83+586802fc9d9401-dirty", GitCommit:"586802fc9d940148e9655fc1a934adb122c61d8f", GitTreeState:"dirty", BuildDate:"2017-09-12T07:52:49Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration**: docker version
Client:
Version: 1.12.5
API version: 1.24
Go version: go1.6
Git commit: dc67f83
Built: Thu Nov 24 16:50:12 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.3
API version: 1.24
Go version: go1.6
Git commit: daf0db7
Built: Mon Oct 31 15:46:15 2016
OS/Arch: linux/amd64
- OS (e.g. from /etc/os-release): centos 6.4
- Kernel (e.g.
uname -a
): linux 3.10.104 - Install tools:
- Others:
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 16 (16 by maintainers)
We should move this off of the 1.8 milestone and close.