cri-o: CreateContainerError running lstat on namespace path

Description I have a MongoDB StatefulSet running fine for awhile, then for unknown reasons the pod restarts. When it attempts to start back up, this error starts occurring. This occurs only rarely as most times it starts up fine. After restarting the crio.service, the error goes away and the container creates successfully.

Steps to reproduce the issue: Not sure

Describe the results you received: Pod is never re-created and shows an error like CreateContainerError in kubectl get pods. The restarts are not increasing, it’s just that error permanently.

$ sudo journalctl -fu crio.service
Nov 22 21:50:06 fsprdce1c04 crio[77667]: time="2018-11-22 21:50:06.734886735Z" level=error msg="Container creation error: container_linux.go:330: creating new parent process caused "container_linux.go:1759: running lstat on namespace path \"/proc/134447/ns/ipc\" caused \"lstat /proc/134447/ns/ipc: no such file or directory\""
Nov 22 21:50:06 fsprdce1c04 crio[77667]: "
Nov 22 21:50:22 fsprdce1c04 crio[77667]: time="2018-11-22 21:50:22.738732429Z" level=error msg="Container creation error: container_linux.go:330: creating new parent process caused "container_linux.go:1759: running lstat on namespace path \"/proc/134447/ns/ipc\" caused \"lstat /proc/134447/ns/ipc: no such file or directory\""
Nov 22 21:50:22 fsprdce1c04 crio[77667]: "
Nov 22 21:50:33 fsprdce1c04 crio[77667]: time="2018-11-22 21:50:33.735476632Z" level=error msg="Container creation error: container_linux.go:330: creating new parent process caused "container_linux.go:1759: running lstat on namespace path \"/proc/134447/ns/ipc\" caused \"lstat /proc/134447/ns/ipc: no such file or directory\""
Nov 22 21:50:33 fsprdce1c04 crio[77667]: "
Nov 22 21:50:46 fsprdce1c04 crio[77667]: time="2018-11-22 21:50:46.734345766Z" level=error msg="Container creation error: container_linux.go:330: creating new parent process caused "container_linux.go:1759: running lstat on namespace path \"/proc/134447/ns/ipc\" caused \"lstat /proc/134447/ns/ipc: no such file or directory\""
Nov 22 21:50:46 fsprdce1c04 crio[77667]: "
Nov 22 21:51:00 fsprdce1c04 crio[77667]: time="2018-11-22 21:51:00.733602480Z" level=error msg="Container creation error: container_linux.go:330: creating new parent process caused "container_linux.go:1759: running lstat on namespace path \"/proc/134447/ns/ipc\" caused \"lstat /proc/134447/ns/ipc: no such file or directory\""
Nov 22 21:51:00 fsprdce1c04 crio[77667]: "
Nov 22 21:51:16 fsprdce1c04 crio[77667]: time="2018-11-22 21:51:16.735359304Z" level=error msg="Container creation error: container_linux.go:330: creating new parent process caused "container_linux.go:1759: running lstat on namespace path \"/proc/134447/ns/ipc\" caused \"lstat /proc/134447/ns/ipc: no such file or directory\""
Nov 22 21:51:16 fsprdce1c04 crio[77667]: "

$ sudo journalctl -fu kubelet.service
Nov 22 21:52:26 fsprdce1c04 kubelet[77800]: I1122 21:52:26.572204   77800 kuberuntime_manager.go:513] Container {Name:mongodb Image:mongo:3.2 Command:[mongod] Args:[--config=/data/configdb/mongod.conf --dbpath=/data/db --replSet=rs0 --port=27017 --bind_ip=0.0.0.0 --auth --keyFile=/data/configdb/key.txt] WorkingDir: Ports:[{Name:mongodb HostPort:0 ContainerPort:27017 Protocol:TCP HostIP:}] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:2684354560 scale:0} d:{Dec:<nil>} s: Format:BinarySI} cpu:{i:{value:1500 scale:-3} d:{Dec:<nil>} s:1500m Format:DecimalSI}] Requests:map[cpu:{i:{value:1500 scale:-3} d:{Dec:<nil>} s:1500m Format:DecimalSI} memory:{i:{value:2684354560 scale:0} d:{Dec:<nil>} s: Format:BinarySI}]} VolumeMounts:[{Name:datadir ReadOnly:false MountPath:/data/db SubPath: MountPropagation:<nil>} {Name:configdir ReadOnly:false MountPath:/data/configdb SubPath: MountPropagation:<nil>} {Name:workdir ReadOnly:false MountPath:/work-dir SubPath: MountPropagation:<nil>} {Name:default-token-dxzpm ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:&ExecAction{Command:[mongo --eval db.adminCommand('ping')],},HTTPGet:nil,TCPSocket:nil,},InitialDelaySeconds:30,TimeoutSeconds:4,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:&Probe{Handler:Handler{Exec:&ExecAction{Command:[mongo --eval db.adminCommand('ping')],},HTTPGet:nil,TCPSocket:nil,},InitialDelaySeconds:5,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Nov 22 21:52:26 fsprdce1c04 kubelet[77800]: E1122 21:52:26.572332   77800 dns.go:121] Search Line limits were exceeded, some search paths have been omitted, the applied search line is: production.svc.cluster.local svc.cluster.local cluster.local corp.kwiktrip.com kwiktrip.com dmz.kwiktrip.com
Nov 22 21:52:26 fsprdce1c04 kubelet[77800]: I1122 21:52:26.572416   77800 kuberuntime_manager.go:757] checking backoff for container "mongodb" in pod "edge-mongodb-2_production(d8627797-ee89-11e8-b96d-0050568593cc)"
Nov 22 21:52:26 fsprdce1c04 kubelet[77800]: W1122 21:52:26.728328   77800 container.go:406] Failed to get RecentStats("/libcontainer_235510_systemd_test_default.slice") while determining the next housekeeping: unable to find data for container /libcontainer_235510_systemd_test_default.slice
Nov 22 21:52:26 fsprdce1c04 kubelet[77800]: E1122 21:52:26.852610   77800 remote_runtime.go:187] CreateContainer in sandbox "43299b80a8de388a98e904f15893686e0ae4f1803bf61f4a706015c742a7248e" from runtime service failed: rpc error: code = Unknown desc = container create failed: container_linux.go:330: creating new parent process caused "container_linux.go:1759: running lstat on namespace path \"/proc/134447/ns/ipc\" caused \"lstat /proc/134447/ns/ipc: no such file or directory\""
Nov 22 21:52:26 fsprdce1c04 kubelet[77800]: E1122 21:52:26.852688   77800 kuberuntime_manager.go:733] container start failed: CreateContainerError: container create failed: container_linux.go:330: creating new parent process caused "container_linux.go:1759: running lstat on namespace path \"/proc/134447/ns/ipc\" caused \"lstat /proc/134447/ns/ipc: no such file or directory\""
Nov 22 21:52:26 fsprdce1c04 kubelet[77800]: E1122 21:52:26.852719   77800 pod_workers.go:186] Error syncing pod d8627797-ee89-11e8-b96d-0050568593cc ("edge-mongodb-2_production(d8627797-ee89-11e8-b96d-0050568593cc)"), skipping: failed to "StartContainer" for "mongodb" with CreateContainerError: "container create failed: container_linux.go:330: creating new parent process caused \"container_linux.go:1759: running lstat on namespace path \\\"/proc/134447/ns/ipc\\\" caused \\\"lstat /proc/134447/ns/ipc: no such file or directory\\\"\"\n"

Describe the results you expected: Container to be created

Additional information you deem important (e.g. issue happens only occasionally):

Output of crio --version: v1.11.10

Additional environment details (AWS, VirtualBox, physical, etc.): Ubuntu 18.04 Kubernetes v1.11.4 VMware VM