moby: Unable to remove container

see also #2714 .

I’m using CentOS 7 and kernel is 3.10.0

$ uname -r
3.10.0-123.6.3.el7.x86_64

docker information is

$ docker version
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070

the problem mention in #2174 did occur again. here’s the message:

Aug 8 17:09:49 kili0 docker: 2014/08/08 17:09:49 GET /v1.13/containers/json Aug 8 17:09:49 kili0 docker: [1fa20b28] +job containers() Aug 8 17:09:49 kili0 docker: [1fa20b28] -job containers() = OK (0) Aug 8 17:09:51 kili0 docker: 2014/08/08 17:09:51 GET /v1.13/containers/json?all=1 Aug 8 17:09:51 kili0 docker: [1fa20b28] +job containers() Aug 8 17:09:51 kili0 docker: [1fa20b28] -job containers() = OK (0) Aug 8 17:10:01 kili0 systemd: Starting Session 11 of user root. Aug 8 17:10:01 kili0 systemd: Started Session 11 of user root. Aug 8 17:10:10 kili0 docker: 2014/08/08 17:10:10 POST /v1.13/containers/create Aug 8 17:10:10 kili0 docker: [1fa20b28] +job create() Aug 8 17:10:10 kili0 kernel: device-mapper: thin: Creation of new snapshot 0 of device 23 failed. Aug 8 17:10:10 kili0 kernel: device-mapper: thin: Creation of new snapshot 1 of device 23 failed. Aug 8 17:10:10 kili0 kernel: device-mapper: thin: Creation of new snapshot 2 of device 23 failed. Aug 8 17:10:10 kili0 kernel: device-mapper: thin: Creation of new snapshot 3 of device 23 failed. Aug 8 17:10:10 kili0 kernel: device-mapper: thin: Creation of new snapshot 4 of device 23 failed. Aug 8 17:10:10 kili0 kernel: device-mapper: thin: Creation of new snapshot 5 of device 23 failed. Aug 8 17:10:10 kili0 kernel: device-mapper: thin: Creation of new snapshot 6 of device 23 failed. Aug 8 17:10:10 kili0 kernel: device-mapper: thin: Creation of new snapshot 7 of device 23 failed. Aug 8 17:10:10 kili0 kernel: device-mapper: thin: Creation of new snapshot 8 of device 23 failed. Aug 8 17:10:10 kili0 kernel: device-mapper: thin: Creation of new snapshot 9 of device 23 failed. Aug 8 17:10:10 kili0 kernel: device-mapper: thin: Creation of new snapshot 10 of device 23 failed. Aug 8 17:10:10 kili0 kernel: device-mapper: thin: Creation of new snapshot 11 of device 23 failed. Aug 8 17:10:10 kili0 kernel: device-mapper: thin: Creation of new snapshot 12 of device 23 failed. Aug 8 17:10:10 kili0 kernel: device-mapper: thin: Creation of new snapshot 13 of device 23 failed. Aug 8 17:10:10 kili0 kernel: bio: create slab <bio-2> at 2 Aug 8 17:10:10 kili0 kernel: EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: discard Aug 8 17:10:10 kili0 systemd-udevd: conflicting device node ‘/dev/mapper/docker-8:2-134226242-0d47d59a250ea2d8bbf6d8d368b5498e8964e3108f8c05b0900a1b00a4360cdb-init’ found, link to ‘/dev/dm-1’ will not be created Aug 8 17:10:10 kili0 systemd-udevd: conflicting device node ‘/dev/mapper/docker-8:2-134226242-0d47d59a250ea2d8bbf6d8d368b5498e8964e3108f8c05b0900a1b00a4360cdb-init’ found, link to ‘/dev/dm-1’ will not be created Aug 8 17:10:10 kili0 docker: [1fa20b28] -job create() = OK (0) Aug 8 17:10:10 kili0 docker: 2014/08/08 17:10:10 POST /v1.13/containers/0d47d59a250ea2d8bbf6d8d368b5498e8964e3108f8c05b0900a1b00a4360cdb/start Aug 8 17:10:10 kili0 docker: [1fa20b28] +job start(0d47d59a250ea2d8bbf6d8d368b5498e8964e3108f8c05b0900a1b00a4360cdb) Aug 8 17:10:10 kili0 kernel: bio: create slab <bio-2> at 2 Aug 8 17:10:10 kili0 docker: [1fa20b28] +job allocate_interface(0d47d59a250ea2d8bbf6d8d368b5498e8964e3108f8c05b0900a1b00a4360cdb) Aug 8 17:10:10 kili0 docker: [1fa20b28] -job allocate_interface(0d47d59a250ea2d8bbf6d8d368b5498e8964e3108f8c05b0900a1b00a4360cdb) = OK (0) Aug 8 17:10:10 kili0 kernel: EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: discard Aug 8 17:10:10 kili0 systemd-udevd: conflicting device node ‘/dev/mapper/docker-8:2-134226242-0d47d59a250ea2d8bbf6d8d368b5498e8964e3108f8c05b0900a1b00a4360cdb’ found, link to ‘/dev/dm-1’ will not be created Aug 8 17:10:10 kili0 systemd: Starting docker container 0d47d59a250ea2d8bbf6d8d368b5498e8964e3108f8c05b0900a1b00a4360cdb. Aug 8 17:10:10 kili0 systemd: Started docker container 0d47d59a250ea2d8bbf6d8d368b5498e8964e3108f8c05b0900a1b00a4360cdb. Aug 8 17:10:10 kili0 kernel: netlink: 1 bytes leftover after parsing attributes. Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth2fad): carrier is OFF Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth2fad): new Veth device (driver: ‘unknown’ ifindex: 15) Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth2fad): exported as /org/freedesktop/NetworkManager/Devices/14 Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth2fad): device state change: unmanaged -> unavailable (reason ‘managed’) [10 20 2] Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth2fad): link connected Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth2fad): preparing device Aug 8 17:10:10 kili0 systemd-sysctl: Overwriting earlier assignment of kernel/sysrq in file ‘/etc/sysctl.d/99-sysctl.conf’. Aug 8 17:10:10 kili0 systemd-sysctl: Overwriting earlier assignment of kernel/shmmax in file ‘/etc/sysctl.d/99-sysctl.conf’. Aug 8 17:10:10 kili0 systemd-sysctl: Overwriting earlier assignment of kernel/shmall in file ‘/etc/sysctl.d/99-sysctl.conf’. Aug 8 17:10:10 kili0 systemd-sysctl: Overwriting earlier assignment of kernel/sysrq in file ‘/etc/sysctl.d/99-sysctl.conf’. Aug 8 17:10:10 kili0 systemd-sysctl: Overwriting earlier assignment of kernel/shmmax in file ‘/etc/sysctl.d/99-sysctl.conf’. Aug 8 17:10:10 kili0 systemd-sysctl: Overwriting earlier assignment of kernel/shmall in file ‘/etc/sysctl.d/99-sysctl.conf’. Aug 8 17:10:10 kili0 kernel: device veth0670 entered promiscuous mode Aug 8 17:10:10 kili0 NetworkManager[1226]: ifcfg-rh: read connection ‘Wired connection 1’ Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth2fad): created default wired connection ‘Wired connection 1’ Aug 8 17:10:10 kili0 kernel: IPv6: ADDRCONF(NETDEV_UP): veth0670: link is not ready Aug 8 17:10:10 kili0 avahi-daemon[1219]: Withdrawing workstation service for veth2fad. Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth0670): link connected Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth0670): carrier is ON Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth0670): new Veth device (driver: ‘unknown’ ifindex: 16) Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth0670): exported as /org/freedesktop/NetworkManager/Devices/15 Aug 8 17:10:10 kili0 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth0670: link becomes ready Aug 8 17:10:10 kili0 kernel: docker0: port 1(veth0670) entered forwarding state Aug 8 17:10:10 kili0 kernel: docker0: port 1(veth0670) entered forwarding state Aug 8 17:10:10 kili0 NetworkManager[1226]: ifcfg-rh: read connection ‘veth0670’ Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth0670): device state change: unmanaged -> unavailable (reason ‘connection-assumed’) [10 20 41] Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (docker0): link connected Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth0670): device state change: unavailable -> disconnected (reason ‘connection-assumed’) [20 30 41] Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> Activation (veth0670) starting connection ‘veth0670’ Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> Activation (veth0670) Stage 1 of 5 (Device Prepare) scheduled… Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth2fad): device state change: unavailable -> unmanaged (reason ‘removed’) [20 10 36] Aug 8 17:10:10 kili0 NetworkManager[1226]: init_link: assertion rtnl_link_get_name (rtnllink)' failed Aug 8 17:10:10 kili0 NetworkManager[1226]: file platform/nm-linux-platform.c: line 1026 (to_string_link): should not be reached Aug 8 17:10:10 kili0 NetworkManager[1226]: <error> [1407489010.579443] [platform/nm-linux-platform.c:1841] link_change(): Netlink error changing link (invalid link 0x7ff04798e090): No such device Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> Activation (veth0670) Stage 1 of 5 (Device Prepare) started... Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth0670): device state change: disconnected -> prepare (reason 'none') [30 40 0] Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> Activation (veth0670) Stage 2 of 5 (Device Configure) scheduled... Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> Activation (veth0670) Stage 1 of 5 (Device Prepare) complete. Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> Activation (veth0670) Stage 2 of 5 (Device Configure) starting... Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth0670): device state change: prepare -> config (reason 'none') [40 50 0] Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> Activation (veth0670) Stage 2 of 5 (Device Configure) successful. Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> Activation (veth0670) Stage 3 of 5 (IP Configure Start) scheduled. Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> Activation (veth0670) Stage 2 of 5 (Device Configure) complete. Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> Activation (veth0670) Stage 3 of 5 (IP Configure Start) started... Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth0670): device state change: config -> ip-config (reason 'none') [50 70 0] Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (docker0): attached bridge port veth0670 Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> Activation (veth0670) connection 'veth0670' enslaved, continuing activation Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> Activation (veth0670) Stage 3 of 5 (IP Configure Start) complete. Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth0670): device state change: ip-config -> secondaries (reason 'none') [70 90 0] Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth0670): device state change: secondaries -> activated (reason 'none') [90 100 0] Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> Activation (veth0670) successful, device activated. Aug 8 17:10:10 kili0 dbus[1244]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' Aug 8 17:10:10 kili0 dbus-daemon: dbus[1244]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' Aug 8 17:10:10 kili0 systemd: Starting Network Manager Script Dispatcher Service... Aug 8 17:10:10 kili0 dbus-daemon: dbus[1244]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' Aug 8 17:10:10 kili0 dbus[1244]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' Aug 8 17:10:10 kili0 systemd: Started Network Manager Script Dispatcher Service. Aug 8 17:10:10 kili0 docker: [1fa20b28] -job start(0d47d59a250ea2d8bbf6d8d368b5498e8964e3108f8c05b0900a1b00a4360cdb) = OK (0) Aug 8 17:10:10 kili0 docker: [1fa20b28] +job release_interface(0d47d59a250ea2d8bbf6d8d368b5498e8964e3108f8c05b0900a1b00a4360cdb) Aug 8 17:10:10 kili0 docker: [1fa20b28] -job release_interface(0d47d59a250ea2d8bbf6d8d368b5498e8964e3108f8c05b0900a1b00a4360cdb) = OK (0) Aug 8 17:10:10 kili0 avahi-daemon[1219]: Withdrawing workstation service for veth0670. Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth0670): device state change: activated -> unmanaged (reason 'removed') [100 10 36] Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (veth0670): deactivating device (reason 'removed') [36] Aug 8 17:10:10 kili0 NetworkManager[1226]: init_link: assertionrtnl_link_get_name (rtnllink)’ failed Aug 8 17:10:10 kili0 NetworkManager[1226]: file platform/nm-linux-platform.c: line 1026 (to_string_link): should not be reached Aug 8 17:10:10 kili0 NetworkManager[1226]: <error> [1407489010.641837] [platform/nm-linux-platform.c:1841] link_change(): Netlink error changing link (invalid link 0x7ff047989c50): No such device Aug 8 17:10:10 kili0 NetworkManager[1226]: init_link: assertion `rtnl_link_get_name (rtnllink)’ failed Aug 8 17:10:10 kili0 NetworkManager[1226]: file platform/nm-linux-platform.c: line 1026 (to_string_link): should not be reached Aug 8 17:10:10 kili0 NetworkManager[1226]: <error> [1407489010.642472] [platform/nm-linux-platform.c:1841] link_change(): Netlink error changing link (invalid link 0x7ff047989c50): No such device Aug 8 17:10:10 kili0 NetworkManager[1226]: <info> (docker0): detached bridge port veth0670 (success 0) Aug 8 17:10:10 kili0 NetworkManager[1226]: (devices/nm-device.c:985):nm_device_release_one_slave: runtime check failed: (success) Aug 8 17:10:10 kili0 kernel: docker0: port 1(veth0670) entered disabled state Aug 8 17:10:10 kili0 kernel: device veth0670 left promiscuous mode Aug 8 17:10:10 kili0 kernel: docker0: port 1(veth0670) entered disabled state Aug 8 17:10:11 kili0 NetworkManager[1226]: <info> (docker0): link disconnected (deferring action for 4 seconds) Aug 8 17:10:15 kili0 NetworkManager[1226]: <info> (docker0): link disconnected (calling deferred action)