containerd: Creating task failed with "failed to find pid"
Description
Sometimes (about 1 in 100 pods) containerd fails to create task and logs error failed to mountat: failed to find pid=12345 process: no child processes: unknown
. Usually the problem goes away when re-creating the pod.
Not able to reproduce but the error has been observed in one of our production clusters.
Output of containerd --version
:
containerd github.com/containerd/containerd v1.2.5 bb71b10fd8f58240ca47fbb579b9d1028eea7c84
Extra info OS: CoreOS 2247.5.0 Kernel: Linux 4.19.78-coreos
Error log:
Feb 10 19:43:42 ip-10-28-14-134.ec2.internal containerd[2138]: time="2020-02-10T19:43:42.282543411Z" level=error msg="Failed to pipe stderr of container "db585f7a4673817c358f9291e200076691d9fb9fe8cee5ae1efb430751adc3fc"" error="reading from a closed fifo"
Feb 10 19:43:42 ip-10-28-14-134.ec2.internal containerd[2138]: time="2020-02-10T19:43:42.284857193Z" level=error msg="Failed to pipe stdout of container "db585f7a4673817c358f9291e200076691d9fb9fe8cee5ae1efb430751adc3fc"" error="reading from a closed fifo"
Feb 10 19:43:42 ip-10-28-14-134.ec2.internal containerd[2138]: time="2020-02-10T19:43:42.291562435Z" level=error msg="StartContainer for "db585f7a4673817c358f9291e200076691d9fb9fe8cee5ae1efb430751adc3fc" failed" error="failed to create containerd task: failed to mount rootfs component &{overlay overlay [workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/829/work upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/829/fs lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/630/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/604/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/603/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/602/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/601/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/600/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/599/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/598/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/597/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/581/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/579/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/578/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/555/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/554/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/553/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/552/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/551/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/550/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/549/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/548/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/547/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/546/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/545/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/544/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/543/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/542/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/540/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/539/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/537/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/536/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/535/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/534/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/533/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/532/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/531/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/530/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/529/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/528/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/527/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/526/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/525/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/524/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/523/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/522/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/521/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/520/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/519/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/518/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/517/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/516/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/515/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/514/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/513/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/512/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/511/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/510/fs]}: failed to mountat: failed to find pid=29186 process: no child processes: unknown"
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 17 (17 by maintainers)
Commits related to this issue
- sys/mount_linux: use pipe for communicating mount result forkAndMountat forks a process to chdir then mount layers. Signals are blocked (using runtime_beforeFork) during fork. There is a race condit... — committed to atlassian-forks/containerd by hligit 4 years ago
- sys/mount_linux: use pipe for communicating mount result forkAndMountat forks a process to chdir then mount layers. Signals are blocked (using runtime_beforeFork) during fork. There is a race condit... — committed to thaJeztah/containerd by hligit 4 years ago
- sys/mount_linux: use pipe for communicating mount result forkAndMountat forks a process to chdir then mount layers. Signals are blocked (using runtime_beforeFork) during fork. There is a race condit... — committed to thaJeztah/containerd by hligit 4 years ago
- sys/mount_linux: use pipe for communicating mount result forkAndMountat forks a process to chdir then mount layers. Signals are blocked (using runtime_beforeFork) during fork. There is a race condit... — committed to fahedouch/containerd by hligit 4 years ago
- sys/mount_linux: use pipe for communicating mount result forkAndMountat forks a process to chdir then mount layers. Signals are blocked (using runtime_beforeFork) during fork. There is a race condit... — committed to tussennet/containerd by hligit 4 years ago
@hligit finally close it. Thanks for your patience
Thanks for the link, @fuweid ! Interesting that pipe was your initial implementation. I guess child subreaper wasn’t in place when you were working on the feature.
I’ll raise a PR tomorrow.
The trace in my last comment was captured last week and it might be confusing. Just reproduced the issue again and with signal traces enabled. SIGCHLD of pid 6979 was delivered to the shim(6969) and wait4-ed by 6978(reaper?).
The only other
unix.Wait4
is in package sys.reaper.Full strace log
SIGCHLD is not blocked by other threads, which may have processed it. Especially since
runtime.LockOSThread
is executed beforeforkAndMountat
, the reaper is guaranteed to be executed in another thread.This is demonstrated in the strace log below.
And no zombie has been observed.
Full strace log
The theory is since signals are blocked during forking, the child process might finish before the parent unblocking signals, signal SIGCHLD would not be delivered. After that, parent wait4(child) would fail because the child process was already reaped.
After applying below change, I managed to reproduce the issue easily.
I think a pipe may better suited for communicating status from child to parent. The parent process should still wait4(child) to avoid zombie, but signal SIGCHLD can be safely ignored.