moby: docker-runc did not terminate sucessfully: unknown

Problem

I recently upgraded to Docker 17.12.0. Previously all my containers were working on Docker 17.09.1 and now they’re no longer operating due to this issue:

docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/225edd3d808116d3cc5992849e60bf5369ace67c291a066ebae4ca5784bcce7a/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown.

Looking into the journalctl yields the following output after running hello-world multiple times:

Dec 31 01:39:42 sjcccsdvl01 dockerd[3944]: time="2017-12-31T01:39:42.518754201-07:00" level=error msg="6a1f3c9dba12d1ddd7ea01c470d2572bbd70d96fb567dfb94e9813870eb7c775 cleanup: failed to delete container from c
Dec 31 01:39:42 sjcccsdvl01 dockerd[3944]: time="2017-12-31T01:39:42.518799413-07:00" level=error msg="Handler for POST /v1.35/containers/6a1f/start returned error: OCI runtime create failed: unable to retrieve
Dec 31 01:42:29 sjcccsdvl01 dockerd[3944]: time="2017-12-31T01:42:29.412054472-07:00" level=warning msg="failed to retrieve docker-runc version: exit status 127"
Dec 31 01:53:04 sjcccsdvl01 dockerd[3944]: time="2017-12-31T01:53:04.724169179-07:00" level=warning msg="failed to retrieve docker-runc version: exit status 127"
Dec 31 01:55:51 sjcccsdvl01 dockerd[3944]: time="2017-12-31T01:55:51.236281894-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Dec 31 01:55:51 sjcccsdvl01 dockerd[3944]: time="2017-12-31T01:55:51-07:00" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/225edd3d808116d3cc5992849e60bf5369ace67c291a066eba
Dec 31 01:55:51 sjcccsdvl01 dockerd[3944]: time="2017-12-31T01:55:51-07:00" level=info msg="shim reaped" id=225edd3d808116d3cc5992849e60bf5369ace67c291a066ebae4ca5784bcce7a module="containerd/tasks"
Dec 31 01:55:51 sjcccsdvl01 dockerd[3944]: time="2017-12-31T01:55:51.257438767-07:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 31 01:55:51 sjcccsdvl01 dockerd[3944]: time="2017-12-31T01:55:51.257440317-07:00" level=error msg="stream copy error: reading from a closed fifo"
Dec 31 01:55:51 sjcccsdvl01 dockerd[3944]: time="2017-12-31T01:55:51.259543869-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
Dec 31 01:55:51 sjcccsdvl01 dockerd[3944]: time="2017-12-31T01:55:51.291427442-07:00" level=error msg="225edd3d808116d3cc5992849e60bf5369ace67c291a066ebae4ca5784bcce7a cleanup: failed to delete container from c
Dec 31 01:55:51 sjcccsdvl01 dockerd[3944]: time="2017-12-31T01:55:51.297187928-07:00" level=error msg="Handler for POST /v1.35/containers/225edd3d808116d3cc5992849e60bf5369ace67c291a066ebae4ca5784bcce7a/start r
lines 676-734/734 (END)

Steps

1. yum update (went from 17.09.1 to 17.12.0)
2. docker pull hello-world
3. docker run --rm hello-world

Information

Kernel info:

root•lib/systemd/system» uname -rs                                                                                                                                                                      [1:55:51]
Linux 3.10.0-327.el7.x86_64

OS info:

root•lib/systemd/system» cat /etc/os-release                                                                                                                                                            [1:56:18]
NAME="Red Hat Enterprise Linux Server"
VERSION="7.2 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.2"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.2 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.2:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.2"

Runc data:

root•lib/systemd/system» find / -name "*runc"                                                                                                                                                           [1:51:44]
/usr/bin/docker-runc
/data/lib/docker/overlay/3de52c470950e27c36062435f195d1224c088807c556fc0bacc595b13696481a/root/usr/bin/docker-runc
/data/lib/docker/overlay/9317a01304b29ee7cbd3196067acd3405f3e299d549a571bbd55e8bf71527c1c/root/usr/bin/docker-runc
/run/runc

Runc info:

root•lib/systemd/system» docker-runc                                                                                                                                                                    
[1:52:35]
docker-runc: symbol lookup error: docker-runc: undefined symbol: seccomp_version

Docker CE info:

root•lib/systemd/system» rpm -qi docker-ce                                                                                                                                                              [1:52:51]
Name        : docker-ce
Version     : 17.12.0.ce
Release     : 1.el7.centos
Architecture: x86_64
Install Date: Sun 31 Dec 2017 01:27:43 AM MST
Group       : Tools/Docker
Size        : 128453687
License     : ASL 2.0
Signature   : RSA/SHA512, Wed 27 Dec 2017 01:29:11 PM MST, Key ID c52feb6b621e9f35
Source RPM  : docker-ce-17.12.0.ce-1.el7.centos.src.rpm
Build Date  : Wed 27 Dec 2017 01:14:05 PM MST
Build Host  : ed25a6230885
Relocations : (not relocatable)
Packager    : Docker <support@docker.com>
Vendor      : Docker
URL         : https://www.docker.com
Summary     : The open-source application container engine
Description :
Docker is an open source project to build, ship and run any application as a
lightweight container.

Docker containers are both hardware-agnostic and platform-agnostic. This means
they can run anywhere, from your laptop to the largest EC2 compute instance and
everything in between - and they don't require you to use a particular
language, framework or packaging system. That makes them great building blocks
for deploying and scaling web apps, databases, and backend services without
depending on a particular stack or provider.

Docker system info:

root•lib/systemd/system» docker info                                                                                                                                                                    [1:53:02]
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 40
Server Version: 17.12.0-ce
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: N/A (expected: b2567b37d7b75eb4cf325b77297b140ea686ce8f)
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.26GiB
Name: sjcccsdvl01
ID: F74D:P3G7:6LVM:QGAN:4T5B:MLT3:7QIX:VOFE:OCQM:GIIH:FZJL:TBFE
Docker Root Dir: /data/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Note

Seems like the docker-runc binary exists. But it is not working as expected. I wonder if it got corrupted during the upgrade. Or that key system files for docker-runc do not exist. Would appreciate help for this matter.