moby: Overlay network X not found | Swarm 1.12.2

Description

Overlay network randomly disappear. I had the network sql_net for 5 days now and I start to get this error. "network sql_net not found"

I deleted it and tried with a different name … Same story. I had the same issue with previous Swarm setup as well.

Steps to reproduce the issue:

docker network create --driver overlay percona_net

docker service create \
    --name $CTN_NAME \
    --constraint node.labels.type==db \
    --mount type=bind,src=/var/lib/mysql,dst=/var/lib/mysql \
    --network percona_net \
    --publish 3306 \
    --restart-condition any \
    -e MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD \
    -e MYSQL_DATABASE=$MYSQL_DATABASE \
    -e MYSQL_USER=$MYSQL_USER \
    -e MYSQL_PASSWORD=$MYSQL_PASSWORD \
    $ENV_DOCKER_IMAGE

Describe the results you received:**

root@swm-aa:~/deploy-setup# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
e87ba909e780        bridge              bridge              local
c8f22dde7607        docker_gwbridge     bridge              local
ecr8r7o8y7n4        front101            overlay             swarm
235f5fda0270        host                host                local
5yq6mrg8sqv5        ingress             overlay             swarm
5g7vkq8oey5m        logentries_net      overlay             swarm
1ba2ae3618c8        none                null                local
5l4a1lze8r8i        percona_net         overlay             swarm
590obyq253of        router-management   overlay             swarm
dhtu1p5vg7dg        sema_net            overlay             swarm

docker service ps percona

ID                         NAME           IMAGE           NODE    DESIRED STATE  CURRENT STATE            ERROR
1ghc5lowz5oerpuo4d40m3epn  percona.1      percona:5.7.14  swm-db  Ready          Rejected 1 seconds ago   "network percona_net not found"
b87zzyet9h8p7su8jjm3slrfh   \_ percona.1  percona:5.7.14  swm-db  Shutdown       Rejected 6 seconds ago   "network percona_net not found"
c068bbeuxyghza57fljygmcm2   \_ percona.1  percona:5.7.14  swm-db  Shutdown       Rejected 11 seconds ago  "network percona_net not found"
6h9idsch7kgbcstvgqq96m0uo   \_ percona.1  percona:5.7.14  swm-db  Shutdown       Rejected 16 seconds ago  "network percona_net not found"
9ouq0e9lfs904yn4ngb5ctrtl   \_ percona.1  percona:5.7.14  swm-db  Shutdown       Rejected 21 seconds ago  "network percona_net not found"

Describe the results you expected:**

common sense

Additional information you deem important (e.g. issue happens only occasionally)😗*

It’s random

Output of docker version😗*

root@swm-aa:~/deploy-setup# docker version
Client:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:29:41 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:29:41 2016
 OS/Arch:      linux/amd64

Output of docker info😗*

root@swm-aa:~/deploy-setup# docker info
Containers: 12
 Running: 4
 Paused: 0
 Stopped: 8
Images: 10
Server Version: 1.12.2
Storage Driver: devicemapper
 Pool Name: docker-253:1-1047149-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.266 GB
 Data Space Total: 107.4 GB
 Data Space Available: 14.97 GB
 Metadata Space Used: 3.854 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.144 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.110 (2015-10-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay null host bridge
Swarm: active
 NodeID: 3btni1tzvoyjwufyqbiea45hq
 Is Manager: true
 ClusterID: ey0d2zl6oydps6c4qntzb06v1
 Managers: 1
 Nodes: 5
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 123.123.123.123
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-42-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 488.5 MiB
Name: swm-aa
ID: WPYK:WT4B:FCSO:WPYU:6N67:MCB7:XTSQ:6DOS:3MN4:I4WU:OW6N:RPZM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

Output of Docker Bench for Security v1.1.0😗*

[INFO] 1 - Host Configuration
[WARN] 1.1  - Create a separate partition for containers
[PASS] 1.2  - Use an updated Linux Kernel
[WARN] 1.4  - Remove all non-essential services from the host - Network
[WARN]      * Host listening on: 6 ports
[PASS] 1.5  - Keep Docker up to date
[INFO]       * Using 1.12.2 which is current as of 2016-10-06
[INFO]       * Check with your operating system vendor for support and security maintenance for docker
[INFO] 1.6  - Only allow trusted users to control Docker daemon
[INFO]      * docker:x:999
[WARN] 1.7  - Failed to inspect: auditctl command not found.
[WARN] 1.8  - Failed to inspect: auditctl command not found.
[WARN] 1.9  - Failed to inspect: auditctl command not found.
[INFO] 1.10 - Audit Docker files and directories - docker.service
[INFO]      * File not found
[INFO] 1.11 - Audit Docker files and directories - docker.socket
[INFO]      * File not found
[WARN] 1.12 - Failed to inspect: auditctl command not found.
[INFO] 1.13 - Audit Docker files and directories - /etc/docker/daemon.json
[INFO]      * File not found
[INFO] 1.14 - Audit Docker files and directories - /usr/bin/docker-containerd
[INFO]      * File not found
[INFO] 1.15 - Audit Docker files and directories - /usr/bin/docker-runc
[INFO]      * File not found


[INFO] 2 - Docker Daemon Configuration
[WARN] 2.1  - Restrict network traffic between containers
[WARN] 2.2  - Set the logging level
[PASS] 2.3  - Allow Docker to make changes to iptables
[PASS] 2.4  - Do not use insecure registries
[PASS] 2.5  - Do not use the aufs storage driver
[INFO] 2.6  - Configure TLS authentication for Docker daemon
[INFO]      * Docker daemon not listening on TCP
[INFO] 2.7 - Set default ulimit as appropriate
[INFO]      * Default ulimit doesn't appear to be set
[WARN] 2.8  - Enable user namespace support
[PASS] 2.9  - Confirm default cgroup usage
[PASS] 2.10 - Do not change base device size until needed
[WARN] 2.11 - Use authorization plugin
[WARN] 2.12 - Configure centralized and remote logging
[WARN] 2.13 - Disable operations on legacy registry (v1)


[INFO] 3 - Docker Daemon Configuration Files
[INFO] 3.1  - Verify that docker.service file ownership is set to root:root
[INFO]      * File not found
[INFO] 3.2  - Verify that docker.service file permissions are set to 644
[INFO]      * File not found
[INFO] 3.3  - Verify that docker.socket file ownership is set to root:root
[INFO]      * File not found
[INFO] 3.4  - Verify that docker.socket file permissions are set to 644
[INFO]      * File not found
[PASS] 3.5  - Verify that /etc/docker directory ownership is set to root:root
[PASS] 3.6  - Verify that /etc/docker directory permissions are set to 755
[INFO] 3.7  - Verify that registry certificate file ownership is set to root:root
[INFO]      * Directory not found
[INFO] 3.8  - Verify that registry certificate file permissions are set to 444
[INFO]      * Directory not found
[INFO] 3.9  - Verify that TLS CA certificate file ownership is set to root:root
[INFO]      * No TLS CA certificate found
[INFO] 3.10 - Verify that TLS CA certificate file permissions are set to 444
[INFO]      * No TLS CA certificate found
[INFO] 3.11 - Verify that Docker server certificate file ownership is set to root:root
[INFO]      * No TLS Server certificate found
[INFO] 3.12 - Verify that Docker server certificate file permissions are set to 444
[INFO]      * No TLS Server certificate found
[INFO] 3.13 - Verify that Docker server key file ownership is set to root:root
[INFO]      * No TLS Key found
[INFO] 3.14 - Verify that Docker server key file permissions are set to 400
[INFO]      * No TLS Key found
[PASS] 3.15 - Verify that Docker socket file ownership is set to root:docker
[PASS] 3.16 - Verify that Docker socket file permissions are set to 660
[INFO] 3.17 - Verify that daemon.json file ownership is set to root:root
[INFO]      * File not found
[INFO] 3.18 - Verify that daemon.json file permissions are set to 644
[INFO]      * File not found
[PASS] 3.19 - Verify that /etc/default/docker file ownership is set to root:root
[PASS] 3.20 - Verify that /etc/default/docker file permissions are set to 644


[INFO] 4 - Container Images and Build Files
[WARN] 4.1  - Create a user for the container
[WARN]      * Running as root: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN]      * Running as root: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN]      * Running as root: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN]      * Running as root: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[WARN] 4.5  - Enable Content trust for Docker


[INFO] 5  - Container Runtime
[WARN] 5.1  - Verify AppArmor Profile, if applicable
[WARN]      * No AppArmorProfile Found: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN]      * No AppArmorProfile Found: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN]      * No AppArmorProfile Found: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN]      * No AppArmorProfile Found: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[WARN] 5.2  - Verify SELinux security options, if applicable
[WARN]      * No SecurityOptions Found: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN]      * No SecurityOptions Found: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN]      * No SecurityOptions Found: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN]      * No SecurityOptions Found: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[PASS] 5.3  - Restrict Linux Kernel Capabilities within containers
[PASS] 5.4  - Do not use privileged containers
[PASS] 5.5  - Do not mount sensitive host system directories on containers
[PASS] 5.6  - Do not run ssh within containers
[PASS] 5.7  - Do not map privileged ports within containers
[PASS] 5.9 - Do not share the host's network namespace
[WARN] 5.10 - Limit memory usage for container
[WARN]      * Container running without memory restrictions: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN]      * Container running without memory restrictions: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN]      * Container running without memory restrictions: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN]      * Container running without memory restrictions: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[WARN] 5.11 - Set container CPU priority appropriately
[WARN]      * Container running without CPU restrictions: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN]      * Container running without CPU restrictions: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN]      * Container running without CPU restrictions: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN]      * Container running without CPU restrictions: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[WARN] 5.12 - Mount container's root filesystem as read only
[WARN]      * Container running with root FS mounted R/W: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN]      * Container running with root FS mounted R/W: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN]      * Container running with root FS mounted R/W: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN]      * Container running with root FS mounted R/W: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[PASS] 5.13 - Bind incoming container traffic to a specific host interface
[WARN] 5.14 - Set the 'on-failure' container restart policy to 5
[WARN]      * MaximumRetryCount is not set to 5: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN]      * MaximumRetryCount is not set to 5: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN]      * MaximumRetryCount is not set to 5: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN]      * MaximumRetryCount is not set to 5: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[PASS] 5.15 - Do not share the host's process namespace
[PASS] 5.16 - Do not share the host's IPC namespace
[PASS] 5.17 - Do not directly expose host devices to containers
[INFO] 5.18 - Override default ulimit at runtime only if needed
[INFO]      * Container no default ulimit override: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[INFO]      * Container no default ulimit override: router.0.3auya7kd42o0vhv9s4ms07lcc
[INFO]      * Container no default ulimit override: router-backend.1.3aslakd616evw0aw80o222fm7
[INFO]      * Container no default ulimit override: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[PASS] 5.19 - Do not set mount propagation mode to shared
[PASS] 5.20 - Do not share the host's UTS namespace
[PASS] 5.21 - Do not disable default seccomp profile
[PASS] 5.24 - Confirm cgroup usage
[WARN] 5.25 - Restrict container from acquiring additional privileges
[WARN]      * Privileges not restricted: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN]      * Privileges not restricted: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN]      * Privileges not restricted: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN]      * Privileges not restricted: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr


[INFO] 6  - Docker Security Operations
[INFO] 6.4 - Avoid image sprawl
[INFO]      * There are currently: 11 images
[INFO] 6.5 - Avoid container sprawl
[INFO]      * There are currently a total of 13 containers, with 5 of them currently running

Cluster setup

5 nodes (1 manager, 4 nodes)

Those labels are applied:

    docker node update --label-add region=do_nyc3 swm-aa
    docker node update --label-add type=elected swm-aa
    docker node update --label-add ram=512m swm-aa

    docker node update --label-add region=do_nyc3 swm-db
    docker node update --label-add type=db swm-db
    docker node update --label-add ram=1g swm-db

    docker node update --label-add region=do_nyc3 swm-01
    docker node update --label-add type=apps swm-01
    docker node update --label-add ram=512m swm-01

    docker node update --label-add region=do_nyc3 swm-02
    docker node update --label-add type=apps swm-02
    docker node update --label-add ram=512m swm-02

    docker node update --label-add region=do_nyc3 swm-03
    docker node update --label-add type=apps swm-03
    docker node update --label-add ram=512m swm-03

Datacenter

Digital Ocean Ubuntu 16.04