moby: Error "Network sandbox join failed: could not get network sandbox (oper true): failed get network namespace "": no such file or directory"

Output of docker version:

$ sudo docker version
Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 22:00:36 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 22:00:36 2016
 OS/Arch:      linux/amd64

Output of docker info:

$ sudo docker info
Containers: 6
 Running: 0
 Paused: 0
 Stopped: 6
Images: 4
Server Version: 1.12.0
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay null bridge host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 4.3.0-040300-generic
Operating System: Ubuntu 14.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 14.94 GiB
Name: ip-10-3-0-92
ID: NQRY:TQDU:MZ7P:242T:S24G:6PNJ:I3HH:OTVY:IAHK:O5GY:2OVY:P7KP
Docker Root Dir: /home/ubuntu/hgdata/deployments/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 19
 Goroutines: 51
 System Time: 2016-07-29T09:49:26.373617108Z
 EventsListeners: 1
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Cluster Store: consul://localhost:8500
Cluster Advertise: 10.3.0.92:2375
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

8 machine Consul cluster in AWS runnning Docker 1.11.2.

Steps to reproduce the issue:

All machines have running containers with restart policies set (unless-stopped or always)
Concurrent upgrade to 1.12 for all machines (we use Ansible to run the following command sudo apt-get -o Dpkg::Options::=“–force-confold” -o Dpkg::Options::=“–force-confdef” install --reinstall -y docker-engine ``` on all machines )

Describe the results you received:

Got the following error network sandbox join failed: could not get network sandbox (oper true): failed get network namespace \"\": no such file or directory for all containers from just a single host

Describe the results you expected:

No errors during container restarts

Additional information you deem important (e.g. issue happens only occasionally):

It happened only on a single node.

$ sudo docker ps -a
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS                           PORTS               NAMES
220e9b95844c        hgdata1/modsecurity:65f576adca8f   "./trap.sh"              20 hours ago        Exited (0) 20 hours ago                              modsecurity
65e7a632bbd8        hgdata1/haproxy:44c12be8862e       "/docker-entrypoint.s"   44 hours ago        Exited (128) About an hour ago                       haproxy-ops-o1-b
c98ab340c9e0        hgdata1/api:44c12be8862e           "./trap.sh"              44 hours ago        Exited (128) About an hour ago                       api-ops-o1-b-blue
02c3138ebbbd        hgdata1/api:44c12be8862e           "./trap.sh"              45 hours ago        Exited (0) 44 hours ago                              api-ops-o1-b-green_previous
d962e63be086        hgdata1/api:44c12be8862e           "./trap.sh"              45 hours ago        Exited (0) 45 hours ago                              api-ops-o1-b-blue_previous
65ff1b44494a        hgdata1/httpd:44c12be8862e         "./trap.sh"              45 hours ago        Exited (128) About an hour ago                       httpd-b

$ sudo docker inspect haproxy-ops-o1-b
[
    {
        "Id": "65e7a632bbd8813ad9b18b18dfd75579e1afb57809e13a7fcb3fb4804bfcfd6f",
        "Created": "2016-07-27T13:08:51.696950169Z",
        "Path": "/docker-entrypoint.sh",
        "Args": [
            "haproxy",
            "-f",
            "/usr/local/etc/haproxy/haproxy_global.cfg",
            "-f",
            "/usr/local/etc/haproxy/api.cfg",
            "-f",
            "/usr/local/etc/haproxy/ldap.cfg",
            "-f",
            "/usr/local/etc/haproxy/ui.cfg",
            "-f",
            "/usr/local/etc/haproxy/db.cfg"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 128,
            "Error": "network sandbox join failed: could not get network sandbox (oper true): failed get network namespace \"\": no such file or directory",
            "StartedAt": "2016-07-29T08:44:28.800313984Z",
            "FinishedAt": "2016-07-29T08:57:28.24702013Z"
        },
        "Image": "sha256:8dfa093839496da3025f1ab0e4492f0cb43823a79879e1a4f30075a1449775d9",
        "ResolvConfPath": "/home/ubuntu/hgdata/deployments/docker/containers/65e7a632bbd8813ad9b18b18dfd75579e1afb57809e13a7fcb3fb4804bfcfd6f/resolv.conf",
        "HostnamePath": "/home/ubuntu/hgdata/deployments/docker/containers/65e7a632bbd8813ad9b18b18dfd75579e1afb57809e13a7fcb3fb4804bfcfd6f/hostname",
        "HostsPath": "/home/ubuntu/hgdata/deployments/docker/containers/65e7a632bbd8813ad9b18b18dfd75579e1afb57809e13a7fcb3fb4804bfcfd6f/hosts",
        "LogPath": "/home/ubuntu/hgdata/deployments/docker/containers/65e7a632bbd8813ad9b18b18dfd75579e1afb57809e13a7fcb3fb4804bfcfd6f/65e7a632bbd8813ad9b18b18dfd75579e1afb57809e13a7fcb3fb4804bfcfd6f-json.log",
        "Name": "/haproxy-ops-o1-b",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/home/ubuntu/hgdata/deployments/ops/o1/haproxy:/usr/local/etc/haproxy/"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "backbone2",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/home/ubuntu/hgdata/deployments/docker/overlay/40a16bae1f1360008ecb289cc9d9994d6c101fdb3abf74b3ec49e4d874cd98c0/root",
                "MergedDir": "/home/ubuntu/hgdata/deployments/docker/overlay/4e29d7ecebfd17faf2f6a7d8a830220fc2ed58acd03ba50f8688ae06031d5352/merged",
                "UpperDir": "/home/ubuntu/hgdata/deployments/docker/overlay/4e29d7ecebfd17faf2f6a7d8a830220fc2ed58acd03ba50f8688ae06031d5352/upper",
                "WorkDir": "/home/ubuntu/hgdata/deployments/docker/overlay/4e29d7ecebfd17faf2f6a7d8a830220fc2ed58acd03ba50f8688ae06031d5352/work"
            }
        },
        "Mounts": [
            {
                "Source": "/home/ubuntu/hgdata/deployments/ops/o1/haproxy",
                "Destination": "/usr/local/etc/haproxy",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "65e7a632bbd8",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": true,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "HAPROXY_MAJOR=1.6",
                "HAPROXY_VERSION=1.6.5",
                "HAPROXY_MD5=5290f278c04e682e42ab71fed26fc082"
            ],
            "Cmd": [
                "haproxy",
                "-f",
                "/usr/local/etc/haproxy/haproxy_global.cfg",
                "-f",
                "/usr/local/etc/haproxy/api.cfg",
                "-f",
                "/usr/local/etc/haproxy/ldap.cfg",
                "-f",
                "/usr/local/etc/haproxy/ui.cfg",
                "-f",
                "/usr/local/etc/haproxy/db.cfg"
            ],
            "Image": "hgdata1/haproxy:44c12be8862e",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "counter": "b",
                "service": "haproxy-ops-o1"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "bc9b69b5db3bd8f4e11c3c64c781ae108e5a55e84c68d3309422c4dc4bbbb34e",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": null,
            "SandboxKey": "/var/run/docker/netns/bc9b69b5db3b",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "backbone2": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "haproxy-ops-o1",
                        "65e7a632bbd8"
                    ],
                    "NetworkID": "61c181cca3cf90c428b3360c503398c587395bf16ec3f314ecb734240250f203",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": ""
                }
            }
        }
    }
]

$ sudo docker start haproxy-ops-o1-b
Error response from daemon: network sandbox join failed: could not get network sandbox (oper true): failed get network namespace "": no such file or directory
Error: failed to start containers: haproxy-ops-o1-b

Daemon log:

time="2016-07-29T08:57:36.410353378Z" level=debug msg="Using default logging driver json-file" 
time="2016-07-29T08:57:36.410425943Z" level=debug msg="Golang's threads limit set to 109980" 
time="2016-07-29T08:57:36.575057676Z" level=debug msg="[graphdriver] trying provided driver \"overlay\"" 
time="2016-07-29T08:57:36.582073752Z" level=debug msg="Using graph driver overlay" 
time="2016-07-29T08:57:36.598917317Z" level=debug msg="Max Concurrent Downloads: 3" 
time="2016-07-29T08:57:36.599150282Z" level=debug msg="Max Concurrent Uploads: 5" 
time="2016-07-29T08:57:36.615731333Z" level=info msg="Graph migration to content-addressability took 0.00 seconds" 
time="2016-07-29T08:57:36.615982284Z" level=debug msg="Initializing discovery service" name=consul uri="localhost:8500" 
time="2016-07-29T08:57:36.616014306Z" level=info msg="Initializing discovery without TLS" 
time="2016-07-29T08:57:36.616397192Z" level=warning msg="Your kernel does not support swap memory limit." 
time="2016-07-29T08:57:36.617982419Z" level=debug msg="Loaded container 02c3138ebbbd586daa043eca74a53029b281734237cf217dc97393268989c245" 
time="2016-07-29T08:57:36.618516963Z" level=debug msg="Loaded container 220e9b95844c132f5f672fc59402e5383d3b1f47535b2864b09b052583d8d23e" 
time="2016-07-29T08:57:36.619125163Z" level=debug msg="Loaded container 65e7a632bbd8813ad9b18b18dfd75579e1afb57809e13a7fcb3fb4804bfcfd6f" 
time="2016-07-29T08:57:36.619813526Z" level=debug msg="Loaded container 65ff1b44494ae51257a684169336db7b0085c51bc74cfb0a6e5fbe2e2474042c" 
time="2016-07-29T08:57:36.620683163Z" level=debug msg="Loaded container c98ab340c9e07355c89f2237f64ab03c9e2faae03720db2f2b3a556152e17034" 
time="2016-07-29T08:57:36.621685691Z" level=debug msg="Loaded container d962e63be0861a68ea7b2c7db109825428eb8fe8b4ccb2151349c8d634b9a83c" 
time="2016-07-29T08:57:36.621983909Z" level=debug msg="Option DefaultDriver: bridge" 
time="2016-07-29T08:57:36.622105987Z" level=debug msg="Option DefaultNetwork: bridge" 
time="2016-07-29T08:57:36.622202965Z" level=debug msg="Option OptionKVProvider: consul" 
time="2016-07-29T08:57:36.622312818Z" level=debug msg="Option OptionKVProviderURL: localhost:8500" 
time="2016-07-29T08:57:36.636492609Z" level=info msg="Firewalld running: false" 
time="2016-07-29T08:57:36.638227012Z" level=debug msg="/sbin/iptables, [--wait --version]" 
time="2016-07-29T08:57:36.643047095Z" level=debug msg="/sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]" 
time="2016-07-29T08:57:36.646917498Z" level=debug msg="/sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]" 
time="2016-07-29T08:57:36.650951552Z" level=debug msg="/sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]" 
time="2016-07-29T08:57:36.652516386Z" level=debug msg="/sbin/iptables, [--wait -t nat -D PREROUTING]" 
time="2016-07-29T08:57:36.658954104Z" level=debug msg="/sbin/iptables, [--wait -t nat -D OUTPUT]" 
time="2016-07-29T08:57:36.662946484Z" level=debug msg="/sbin/iptables, [--wait -t nat -F DOCKER]" 
time="2016-07-29T08:57:36.666921197Z" level=debug msg="/sbin/iptables, [--wait -t nat -X DOCKER]" 
time="2016-07-29T08:57:36.670918488Z" level=debug msg="/sbin/iptables, [--wait -t filter -F DOCKER]" 
time="2016-07-29T08:57:36.674921400Z" level=debug msg="/sbin/iptables, [--wait -t filter -X DOCKER]" 
time="2016-07-29T08:57:36.678938066Z" level=debug msg="/sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION]" 
time="2016-07-29T08:57:36.682921996Z" level=debug msg="/sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION]" 
time="2016-07-29T08:57:36.685393359Z" level=debug msg="/sbin/iptables, [--wait -t nat -n -L DOCKER]" 
time="2016-07-29T08:57:36.690918247Z" level=debug msg="/sbin/iptables, [--wait -t nat -N DOCKER]" 
time="2016-07-29T08:57:36.694912483Z" level=debug msg="/sbin/iptables, [--wait -t filter -n -L DOCKER]" 
time="2016-07-29T08:57:36.698923382Z" level=debug msg="/sbin/iptables, [--wait -t filter -n -L DOCKER-ISOLATION]" 
time="2016-07-29T08:57:36.702952501Z" level=debug msg="/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION -j RETURN]" 
time="2016-07-29T08:57:36.710969379Z" level=debug msg="/sbin/iptables, [--wait -I DOCKER-ISOLATION -j RETURN]" 
time="2016-07-29T08:57:36.725502370Z" level=debug msg="/sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.18.0.0/16 ! -o docker_gwbridge -j MASQUERADE]" 
time="2016-07-29T08:57:36.728255150Z" level=debug msg="/sbin/iptables, [--wait -t nat -C POSTROUTING -m addrtype --src-type LOCAL -o docker_gwbridge -j MASQUERADE]" 
time="2016-07-29T08:57:36.734972259Z" level=debug msg="/sbin/iptables, [--wait -D FORWARD -i docker_gwbridge -o docker_gwbridge -j ACCEPT]" 
time="2016-07-29T08:57:36.738958189Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -i docker_gwbridge -o docker_gwbridge -j DROP]" 
time="2016-07-29T08:57:36.742937228Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -i docker_gwbridge ! -o docker_gwbridge -j ACCEPT]" 
time="2016-07-29T08:57:36.746938064Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o docker_gwbridge -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT]" 
time="2016-07-29T08:57:36.750964920Z" level=debug msg="/sbin/iptables, [--wait -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]" 
time="2016-07-29T08:57:36.752778672Z" level=debug msg="/sbin/iptables, [--wait -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]" 
time="2016-07-29T08:57:36.754479724Z" level=debug msg="/sbin/iptables, [--wait -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]" 
time="2016-07-29T08:57:36.756029444Z" level=debug msg="/sbin/iptables, [--wait -t nat -A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]" 
time="2016-07-29T08:57:36.757627864Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o docker_gwbridge -j DOCKER]" 
time="2016-07-29T08:57:36.759032660Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o docker_gwbridge -j DOCKER]" 
time="2016-07-29T08:57:36.760451130Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-ISOLATION]" 
time="2016-07-29T08:57:36.761837268Z" level=debug msg="/sbin/iptables, [--wait -D FORWARD -j DOCKER-ISOLATION]" 
time="2016-07-29T08:57:36.763308962Z" level=debug msg="/sbin/iptables, [--wait -I FORWARD -j DOCKER-ISOLATION]" 
time="2016-07-29T08:57:36.764978658Z" level=debug msg="Network (3430034) restored" 
time="2016-07-29T08:57:36.765392252Z" level=debug msg="/sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE]" 
time="2016-07-29T08:57:36.766939063Z" level=debug msg="/sbin/iptables, [--wait -t nat -C POSTROUTING -m addrtype --src-type LOCAL -o docker0 -j MASQUERADE]" 
time="2016-07-29T08:57:36.768556631Z" level=debug msg="/sbin/iptables, [--wait -D FORWARD -i docker0 -o docker0 -j DROP]" 
time="2016-07-29T08:57:36.770024198Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT]" 
time="2016-07-29T08:57:36.771496595Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT]" 
time="2016-07-29T08:57:36.772825742Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT]" 
time="2016-07-29T08:57:36.774341646Z" level=debug msg="/sbin/iptables, [--wait -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]" 
time="2016-07-29T08:57:36.775939945Z" level=debug msg="/sbin/iptables, [--wait -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]" 
time="2016-07-29T08:57:36.777579301Z" level=debug msg="/sbin/iptables, [--wait -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]" 
time="2016-07-29T08:57:36.779122874Z" level=debug msg="/sbin/iptables, [--wait -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]" 
time="2016-07-29T08:57:36.780659511Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER]" 
time="2016-07-29T08:57:36.782060396Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER]" 
time="2016-07-29T08:57:36.783532237Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-ISOLATION]" 
time="2016-07-29T08:57:36.784960011Z" level=debug msg="/sbin/iptables, [--wait -D FORWARD -j DOCKER-ISOLATION]" 
time="2016-07-29T08:57:36.786639322Z" level=debug msg="/sbin/iptables, [--wait -I FORWARD -j DOCKER-ISOLATION]" 
time="2016-07-29T08:57:36.788174851Z" level=debug msg="/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION -i docker0 -o docker_gwbridge -j DROP]" 
time="2016-07-29T08:57:36.789858465Z" level=debug msg="/sbin/iptables, [--wait -I DOCKER-ISOLATION -i docker0 -o docker_gwbridge -j DROP]" 
time="2016-07-29T08:57:36.791309541Z" level=debug msg="/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION -i docker_gwbridge -o docker0 -j DROP]" 
time="2016-07-29T08:57:36.792668345Z" level=debug msg="/sbin/iptables, [--wait -I DOCKER-ISOLATION -i docker_gwbridge -o docker0 -j DROP]" 
time="2016-07-29T08:57:36.794173671Z" level=debug msg="Network (a67d42e) restored" 
time="2016-07-29T08:57:36.910315146Z" level=debug msg="Watch triggered with 8 nodes" discovery=consul 
time="2016-07-29T08:57:36.911244077Z" level=info msg="2016/07/29 08:57:36 [INFO] serf: EventMemberJoin: ip-10-3-0-92 10.3.0.92\n" 
time="2016-07-29T08:57:36.912673342Z" level=debug msg="Allocating IPv4 pools for network docker_gwbridge (3430034048bbfaf02b5033d74e97ef652df97997b7b2c16f1394461b2cd9c150)" 
time="2016-07-29T08:57:36.912742600Z" level=debug msg="RequestPool(LocalDefault, 172.18.0.0/16, , map[], false)" 
time="2016-07-29T08:57:36.912817756Z" level=debug msg="RequestAddress(LocalDefault/172.18.0.0/16, 172.18.0.1, map[RequestAddressType:com.docker.network.gateway])" 
time="2016-07-29T08:57:36.913046459Z" level=debug msg="2016/07/29 08:57:36 [DEBUG] memberlist: Failed to join 10.3.0.127: dial tcp 10.3.0.127:7946: getsockopt: connection refused\n" 
time="2016-07-29T08:57:36.913103437Z" level=error msg="joining serf neighbor 10.3.0.127 failed: Failed to join the cluster at neigh IP 10.3.0.127: 1 error(s) occurred:\n\n* Failed to join 10.3.0.127: dial tcp 10.3.0.127:7946: getsockopt: connection refused" 
time="2016-07-29T08:57:36.913561284Z" level=debug msg="2016/07/29 08:57:36 [DEBUG] memberlist: Failed to join 10.3.0.217: dial tcp 10.3.0.217:7946: getsockopt: connection refused\n" 
time="2016-07-29T08:57:36.913644304Z" level=error msg="joining serf neighbor 10.3.0.217 failed: Failed to join the cluster at neigh IP 10.3.0.217: 1 error(s) occurred:\n\n* Failed to join 10.3.0.217: dial tcp 10.3.0.217:7946: getsockopt: connection refused" 
time="2016-07-29T08:57:36.914133128Z" level=debug msg="2016/07/29 08:57:36 [DEBUG] memberlist: Failed to join 10.3.0.226: dial tcp 10.3.0.226:7946: getsockopt: connection refused\n" 
time="2016-07-29T08:57:36.914200121Z" level=error msg="joining serf neighbor 10.3.0.226 failed: Failed to join the cluster at neigh IP 10.3.0.226: 1 error(s) occurred:\n\n* Failed to join 10.3.0.226: dial tcp 10.3.0.226:7946: getsockopt: connection refused" 
time="2016-07-29T08:57:36.914857855Z" level=debug msg="2016/07/29 08:57:36 [DEBUG] memberlist: Failed to join 10.3.0.227: dial tcp 10.3.0.227:7946: getsockopt: connection refused\n" 
time="2016-07-29T08:57:36.914945163Z" level=error msg="joining serf neighbor 10.3.0.227 failed: Failed to join the cluster at neigh IP 10.3.0.227: 1 error(s) occurred:\n\n* Failed to join 10.3.0.227: dial tcp 10.3.0.227:7946: getsockopt: connection refused" 
time="2016-07-29T08:57:36.915607010Z" level=debug msg="2016/07/29 08:57:36 [DEBUG] memberlist: Failed to join 10.3.0.235: dial tcp 10.3.0.235:7946: getsockopt: connection refused\n" 
time="2016-07-29T08:57:36.915652188Z" level=error msg="joining serf neighbor 10.3.0.235 failed: Failed to join the cluster at neigh IP 10.3.0.235: 1 error(s) occurred:\n\n* Failed to join 10.3.0.235: dial tcp 10.3.0.235:7946: getsockopt: connection refused" 
time="2016-07-29T08:57:36.916108148Z" level=debug msg="2016/07/29 08:57:36 [DEBUG] memberlist: Initiating push/pull sync with: 10.3.0.32:7946\n" 
time="2016-07-29T08:57:36.917181600Z" level=info msg="2016/07/29 08:57:36 [INFO] serf: EventMemberJoin: ip-10-3-0-32 10.3.0.32\n" 
time="2016-07-29T08:57:36.922811728Z" level=debug msg="Allocating IPv4 pools for network bridge (a67d42ee838952d681d8ede5cd9ebed5d809866c376fe840a55c6779a2b8ec9c)" 
time="2016-07-29T08:57:36.922862840Z" level=debug msg="RequestPool(LocalDefault, 172.17.42.1/16, , map[], false)" 
time="2016-07-29T08:57:36.922961802Z" level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, 172.17.42.1, map[RequestAddressType:com.docker.network.gateway])" 
time="2016-07-29T08:57:36.989982425Z" level=debug msg="/sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE]" 
time="2016-07-29T08:57:36.992319457Z" level=debug msg="/sbin/iptables, [--wait -t nat -D POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE]" 
time="2016-07-29T08:57:36.994097159Z" level=debug msg="/sbin/iptables, [--wait -t nat -C POSTROUTING -m addrtype --src-type LOCAL -o docker0 -j MASQUERADE]" 
time="2016-07-29T08:57:36.995846566Z" level=debug msg="/sbin/iptables, [--wait -t nat -D POSTROUTING -m addrtype --src-type LOCAL -o docker0 -j MASQUERADE]" 
time="2016-07-29T08:57:36.997624561Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT]" 
time="2016-07-29T08:57:36.999143919Z" level=debug msg="/sbin/iptables, [--wait -D FORWARD -i docker0 -o docker0 -j ACCEPT]" 
time="2016-07-29T08:57:37.000744761Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT]" 
time="2016-07-29T08:57:37.002269216Z" level=debug msg="/sbin/iptables, [--wait -D FORWARD -i docker0 ! -o docker0 -j ACCEPT]" 
time="2016-07-29T08:57:37.003906492Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT]" 
time="2016-07-29T08:57:37.006340380Z" level=debug msg="/sbin/iptables, [--wait -D FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT]" 
time="2016-07-29T08:57:37.008118305Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER]" 
time="2016-07-29T08:57:37.009768137Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER]" 
time="2016-07-29T08:57:37.011396881Z" level=debug msg="/sbin/iptables, [--wait -D FORWARD -o docker0 -j DOCKER]" 
time="2016-07-29T08:57:37.013091354Z" level=debug msg="/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION -i docker0 -o docker_gwbridge -j DROP]" 
time="2016-07-29T08:57:37.014700528Z" level=debug msg="/sbin/iptables, [--wait -D DOCKER-ISOLATION -i docker0 -o docker_gwbridge -j DROP]" 
time="2016-07-29T08:57:37.026987917Z" level=debug msg="/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION -i docker_gwbridge -o docker0 -j DROP]" 
time="2016-07-29T08:57:37.035027625Z" level=debug msg="/sbin/iptables, [--wait -D DOCKER-ISOLATION -i docker_gwbridge -o docker0 -j DROP]" 
time="2016-07-29T08:57:37.047514046Z" level=debug msg="releasing IPv4 pools from network bridge (a67d42ee838952d681d8ede5cd9ebed5d809866c376fe840a55c6779a2b8ec9c)" 
time="2016-07-29T08:57:37.047735557Z" level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.42.1)" 
time="2016-07-29T08:57:37.047870412Z" level=debug msg="ReleasePool(LocalDefault/172.17.0.0/16)" 
time="2016-07-29T08:57:37.064822845Z" level=debug msg="Allocating IPv4 pools for network bridge (b8be3ccbc25a3e6b536ba4a09d16212bd616cbb5fce2f79b4d2fb925ec8b1c8c)" 
time="2016-07-29T08:57:37.065034400Z" level=debug msg="RequestPool(LocalDefault, 172.17.42.1/16, , map[], false)" 
time="2016-07-29T08:57:37.065222185Z" level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, 172.17.42.1, map[RequestAddressType:com.docker.network.gateway])" 
time="2016-07-29T08:57:37.065676816Z" level=debug msg="/sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE]" 
time="2016-07-29T08:57:37.067846625Z" level=debug msg="/sbin/iptables, [--wait -t nat -I POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE]" 
time="2016-07-29T08:57:37.069406523Z" level=debug msg="/sbin/iptables, [--wait -t nat -C POSTROUTING -m addrtype --src-type LOCAL -o docker0 -j MASQUERADE]" 
time="2016-07-29T08:57:37.071046040Z" level=debug msg="/sbin/iptables, [--wait -t nat -I POSTROUTING -m addrtype --src-type LOCAL -o docker0 -j MASQUERADE]" 
time="2016-07-29T08:57:37.072690892Z" level=debug msg="/sbin/iptables, [--wait -D FORWARD -i docker0 -o docker0 -j DROP]" 
time="2016-07-29T08:57:37.074266451Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT]" 
time="2016-07-29T08:57:37.086907858Z" level=debug msg="/sbin/iptables, [--wait -I FORWARD -i docker0 -o docker0 -j ACCEPT]" 
time="2016-07-29T08:57:37.094938035Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT]" 
time="2016-07-29T08:57:37.097563215Z" level=debug msg="/sbin/iptables, [--wait -I FORWARD -i docker0 ! -o docker0 -j ACCEPT]" 
time="2016-07-29T08:57:37.099139530Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT]" 
time="2016-07-29T08:57:37.100784038Z" level=debug msg="/sbin/iptables, [--wait -I FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT]" 
time="2016-07-29T08:57:37.102417256Z" level=debug msg="/sbin/iptables, [--wait -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]" 
time="2016-07-29T08:57:37.104077999Z" level=debug msg="/sbin/iptables, [--wait -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]" 
time="2016-07-29T08:57:37.105806238Z" level=debug msg="/sbin/iptables, [--wait -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]" 
time="2016-07-29T08:57:37.107424313Z" level=debug msg="/sbin/iptables, [--wait -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]" 
time="2016-07-29T08:57:37.108986292Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER]" 
time="2016-07-29T08:57:37.110494135Z" level=debug msg="/sbin/iptables, [--wait -I FORWARD -o docker0 -j DOCKER]" 
time="2016-07-29T08:57:37.112072970Z" level=debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-ISOLATION]" 
time="2016-07-29T08:57:37.113563442Z" level=debug msg="/sbin/iptables, [--wait -D FORWARD -j DOCKER-ISOLATION]" 
time="2016-07-29T08:57:37.115234272Z" level=debug msg="/sbin/iptables, [--wait -I FORWARD -j DOCKER-ISOLATION]" 
time="2016-07-29T08:57:37.116867000Z" level=debug msg="/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION -i docker0 -o docker_gwbridge -j DROP]" 
time="2016-07-29T08:57:37.118341490Z" level=debug msg="/sbin/iptables, [--wait -I DOCKER-ISOLATION -i docker0 -o docker_gwbridge -j DROP]" 
time="2016-07-29T08:57:37.120445515Z" level=debug msg="/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION -i docker_gwbridge -o docker0 -j DROP]" 
time="2016-07-29T08:57:37.123246878Z" level=debug msg="/sbin/iptables, [--wait -I DOCKER-ISOLATION -i docker_gwbridge -o docker0 -j DROP]" 
time="2016-07-29T08:57:37.132411557Z" level=debug msg="Watch triggered with 8 nodes" discovery=consul 
time="2016-07-29T08:57:37.147044574Z" level=debug msg="2016/07/29 08:57:37 [DEBUG] serf: messageJoinType: ip-10-3-0-92\n" 
time="2016-07-29T08:57:37.159530405Z" level=debug msg="Starting container c98ab340c9e07355c89f2237f64ab03c9e2faae03720db2f2b3a556152e17034" 
time="2016-07-29T08:57:37.159686605Z" level=debug msg="Starting container 65e7a632bbd8813ad9b18b18dfd75579e1afb57809e13a7fcb3fb4804bfcfd6f" 
time="2016-07-29T08:57:37.159920100Z" level=debug msg="Starting container 65ff1b44494ae51257a684169336db7b0085c51bc74cfb0a6e5fbe2e2474042c" 
time="2016-07-29T08:57:37.171579661Z" level=debug msg="container mounted via layerStore: /home/ubuntu/hgdata/deployments/docker/overlay/4e29d7ecebfd17faf2f6a7d8a830220fc2ed58acd03ba50f8688ae06031d5352/merged" 
time="2016-07-29T08:57:37.172686652Z" level=debug msg="container mounted via layerStore: /home/ubuntu/hgdata/deployments/docker/overlay/940b01df699784d9ff0a66d9b04560a78d7ca5c82fa4a0bf13b7b0931aa2d90c/merged" 
time="2016-07-29T08:57:37.179409896Z" level=debug msg="container mounted via layerStore: /home/ubuntu/hgdata/deployments/docker/overlay/52650b1bf550396875643f5772399750c6e12eddbc8b4ef008351333548aa846/merged" 
time="2016-07-29T08:57:37.215739479Z" level=debug msg="Assigning addresses for endpoint httpd-b's interface on network backbone2" 
time="2016-07-29T08:57:37.215845287Z" level=debug msg="RequestAddress(GlobalDefault/10.0.0.0/24, <nil>, map[])" 
time="2016-07-29T08:57:37.217445392Z" level=debug msg="Assigning addresses for endpoint haproxy-ops-o1-b's interface on network backbone2" 
time="2016-07-29T08:57:37.217537454Z" level=debug msg="RequestAddress(GlobalDefault/10.0.0.0/24, <nil>, map[])" 
time="2016-07-29T08:57:37.225728621Z" level=debug msg="Assigning addresses for endpoint api-ops-o1-b-blue's interface on network backbone2" 
time="2016-07-29T08:57:37.225776649Z" level=debug msg="RequestAddress(GlobalDefault/10.0.0.0/24, <nil>, map[])" 
time="2016-07-29T08:57:37.234783387Z" level=debug msg="Assigning addresses for endpoint httpd-b's interface on network backbone2" 
time="2016-07-29T08:57:37.240746864Z" level=debug msg="Assigning addresses for endpoint api-ops-o1-b-blue's interface on network backbone2" 
time="2016-07-29T08:57:37.257392189Z" level=debug msg="Assigning addresses for endpoint haproxy-ops-o1-b's interface on network backbone2" 
time="2016-07-29T08:57:37.347124347Z" level=debug msg="2016/07/29 08:57:37 [DEBUG] serf: messageJoinType: ip-10-3-0-92\n" 
time="2016-07-29T08:57:37.370529481Z" level=debug msg="Releasing addresses for endpoint httpd-b's interface on network backbone2" 
time="2016-07-29T08:57:37.370691252Z" level=debug msg="ReleaseAddress(GlobalDefault/10.0.0.0/24, 10.0.0.2)" 
time="2016-07-29T08:57:37.401833645Z" level=debug msg="Releasing addresses for endpoint haproxy-ops-o1-b's interface on network backbone2" 
time="2016-07-29T08:57:37.401903119Z" level=debug msg="ReleaseAddress(GlobalDefault/10.0.0.0/24, 10.0.0.4)" 
time="2016-07-29T08:57:37.408772774Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /home/ubuntu/hgdata/deployments/docker/containers/65ff1b44494ae51257a684169336db7b0085c51bc74cfb0a6e5fbe2e2474042c/shm: invalid argument" 
time="2016-07-29T08:57:37.416758906Z" level=debug msg="Releasing addresses for endpoint api-ops-o1-b-blue's interface on network backbone2" 
time="2016-07-29T08:57:37.416850279Z" level=debug msg="ReleaseAddress(GlobalDefault/10.0.0.0/24, 10.0.0.3)" 
time="2016-07-29T08:57:37.423211346Z" level=error msg="Failed to start container 65ff1b44494ae51257a684169336db7b0085c51bc74cfb0a6e5fbe2e2474042c: network sandbox join failed: could not get network sandbox (oper true): failed get network namespace \"\": no such file or directory" 
time="2016-07-29T08:57:37.425044859Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /home/ubuntu/hgdata/deployments/docker/containers/65e7a632bbd8813ad9b18b18dfd75579e1afb57809e13a7fcb3fb4804bfcfd6f/shm: invalid argument" 
time="2016-07-29T08:57:37.431759608Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /home/ubuntu/hgdata/deployments/docker/containers/c98ab340c9e07355c89f2237f64ab03c9e2faae03720db2f2b3a556152e17034/shm: invalid argument" 
time="2016-07-29T08:57:37.434924754Z" level=error msg="Failed to start container 65e7a632bbd8813ad9b18b18dfd75579e1afb57809e13a7fcb3fb4804bfcfd6f: network sandbox join failed: could not get network sandbox (oper true): failed get network namespace \"\": no such file or directory" 
time="2016-07-29T08:57:37.442971692Z" level=error msg="Failed to start container c98ab340c9e07355c89f2237f64ab03c9e2faae03720db2f2b3a556152e17034: network sandbox join failed: could not get network sandbox (oper true): failed get network namespace \"\": no such file or directory" 
time="2016-07-29T08:57:37.443081837Z" level=info msg="Daemon has completed initialization" 
time="2016-07-29T08:57:37.443102973Z" level=info msg="Docker daemon" commit=8eab29e graphdriver=overlay version=1.12.0

About this issue

Original URL
State: closed
Created 8 years ago
Reactions: 4
Comments: 48 (14 by maintainers)

Most upvoted comments

How long to wait for the next patch release? Containers can not be started. This is a critical issue.

vingrad on Sep 6, 2016

Releasing a fix for this issue is also getting really urgent for us. We have this on a newly setup staging swarm with overlay networking. When the above error occurs the whole swarm is getting into an uncontrollable state - the only workaround so far is to restart all engines in the swarm. Absolutely unusable for production.

schmunk42 on Sep 20, 2016

Any updates here? Seeing something along the same lines, i.e.

upgrade to 1.12
consul
network sandbox join failed: could not get network sandbox

KramKroc on Aug 3, 2016

The question is “has a fix been released”, and the answer “yes, it will be available in the next docker release” means “no, it hasn’t been released”.

You shouldn’t close this issue until the fix is released (and tested). But anyway, let’s continue hiding issues, it’s no different from having docs from a not-released version on the website for 3 months.

randunel on Sep 15, 2016

Seems like if you get an error like this on a node, the workaround that I found for it is to recreate the overlay network… which you cant really do in a prod env… it’s an upgrade showstopper for us

sebi-hgdata on Aug 11, 2016

Can anyone provide some workaround until this fix is released?

This happens for us all the time in production. Absolutely frustrating. Currently the only option I know is physically remove the VM from the cloud and create a new one.

groyee on Sep 27, 2016

@randunel

You shouldn’t close this issue until the fix is released (and tested).

Common workflow in github is that an issue gets closed when the PR containing the fix for it has been merged. This step is even automated when the PR description contains the word “fixes” along with the issue number. Then user looks at the fix PR and derives which release contains the fix.

In this case, the PR which brought the fix in docker/docker did not have the reference to this issue. This is why I suggested to close it manually.

Regarding the testing, it is not always possible to recreate the exact scenario the user was in. In this specific case, though, we knew from the logs that the root cause was the missing namespace path and a scenario with missing namespace path was recreated and the fix tested against it.

But anyway, let’s continue hiding issues,

I understand you may feel this way because certain error messages are recurring in different issues opened at different times and across different docker versions, sometimes months apart from each other. Most of the times, at least from what I witnessed so far, they come from very different scenario, different exercised code path which happen to lead to similar (not always same) error messages.

This is so common that when developers see two issues with same error messages, they reject a priori the idea (I’d call it temptation) that the two issues have the same root cause, to focus instead on what was happening when the issue was hit.

Cheers

aboch on Sep 15, 2016