moby: Overlay network X not found | Swarm 1.12.2
Description
Overlay network randomly disappear. I had the network sql_net
for 5 days now and I start to get this error. "network sql_net not found"
I deleted it and tried with a different name β¦ Same story. I had the same issue with previous Swarm setup as well.
Steps to reproduce the issue:
docker network create --driver overlay percona_net
docker service create \
--name $CTN_NAME \
--constraint node.labels.type==db \
--mount type=bind,src=/var/lib/mysql,dst=/var/lib/mysql \
--network percona_net \
--publish 3306 \
--restart-condition any \
-e MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD \
-e MYSQL_DATABASE=$MYSQL_DATABASE \
-e MYSQL_USER=$MYSQL_USER \
-e MYSQL_PASSWORD=$MYSQL_PASSWORD \
$ENV_DOCKER_IMAGE
Describe the results you received:**
root@swm-aa:~/deploy-setup# docker network ls
NETWORK ID NAME DRIVER SCOPE
e87ba909e780 bridge bridge local
c8f22dde7607 docker_gwbridge bridge local
ecr8r7o8y7n4 front101 overlay swarm
235f5fda0270 host host local
5yq6mrg8sqv5 ingress overlay swarm
5g7vkq8oey5m logentries_net overlay swarm
1ba2ae3618c8 none null local
5l4a1lze8r8i percona_net overlay swarm
590obyq253of router-management overlay swarm
dhtu1p5vg7dg sema_net overlay swarm
docker service ps percona
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
1ghc5lowz5oerpuo4d40m3epn percona.1 percona:5.7.14 swm-db Ready Rejected 1 seconds ago "network percona_net not found"
b87zzyet9h8p7su8jjm3slrfh \_ percona.1 percona:5.7.14 swm-db Shutdown Rejected 6 seconds ago "network percona_net not found"
c068bbeuxyghza57fljygmcm2 \_ percona.1 percona:5.7.14 swm-db Shutdown Rejected 11 seconds ago "network percona_net not found"
6h9idsch7kgbcstvgqq96m0uo \_ percona.1 percona:5.7.14 swm-db Shutdown Rejected 16 seconds ago "network percona_net not found"
9ouq0e9lfs904yn4ngb5ctrtl \_ percona.1 percona:5.7.14 swm-db Shutdown Rejected 21 seconds ago "network percona_net not found"
Describe the results you expected:**
common sense
Additional information you deem important (e.g. issue happens only occasionally)π*
Itβs random
Output of docker version
π*
root@swm-aa:~/deploy-setup# docker version
Client:
Version: 1.12.2
API version: 1.24
Go version: go1.6.3
Git commit: bb80604
Built: Tue Oct 11 18:29:41 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.2
API version: 1.24
Go version: go1.6.3
Git commit: bb80604
Built: Tue Oct 11 18:29:41 2016
OS/Arch: linux/amd64
Output of docker info
π*
root@swm-aa:~/deploy-setup# docker info
Containers: 12
Running: 4
Paused: 0
Stopped: 8
Images: 10
Server Version: 1.12.2
Storage Driver: devicemapper
Pool Name: docker-253:1-1047149-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 2.266 GB
Data Space Total: 107.4 GB
Data Space Available: 14.97 GB
Metadata Space Used: 3.854 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.144 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.110 (2015-10-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: overlay null host bridge
Swarm: active
NodeID: 3btni1tzvoyjwufyqbiea45hq
Is Manager: true
ClusterID: ey0d2zl6oydps6c4qntzb06v1
Managers: 1
Nodes: 5
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: 123.123.123.123
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-42-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 488.5 MiB
Name: swm-aa
ID: WPYK:WT4B:FCSO:WPYU:6N67:MCB7:XTSQ:6DOS:3MN4:I4WU:OW6N:RPZM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
Output of Docker Bench for Security v1.1.0
π*
[INFO] 1 - Host Configuration
[WARN] 1.1 - Create a separate partition for containers
[PASS] 1.2 - Use an updated Linux Kernel
[WARN] 1.4 - Remove all non-essential services from the host - Network
[WARN] * Host listening on: 6 ports
[PASS] 1.5 - Keep Docker up to date
[INFO] * Using 1.12.2 which is current as of 2016-10-06
[INFO] * Check with your operating system vendor for support and security maintenance for docker
[INFO] 1.6 - Only allow trusted users to control Docker daemon
[INFO] * docker:x:999
[WARN] 1.7 - Failed to inspect: auditctl command not found.
[WARN] 1.8 - Failed to inspect: auditctl command not found.
[WARN] 1.9 - Failed to inspect: auditctl command not found.
[INFO] 1.10 - Audit Docker files and directories - docker.service
[INFO] * File not found
[INFO] 1.11 - Audit Docker files and directories - docker.socket
[INFO] * File not found
[WARN] 1.12 - Failed to inspect: auditctl command not found.
[INFO] 1.13 - Audit Docker files and directories - /etc/docker/daemon.json
[INFO] * File not found
[INFO] 1.14 - Audit Docker files and directories - /usr/bin/docker-containerd
[INFO] * File not found
[INFO] 1.15 - Audit Docker files and directories - /usr/bin/docker-runc
[INFO] * File not found
[INFO] 2 - Docker Daemon Configuration
[WARN] 2.1 - Restrict network traffic between containers
[WARN] 2.2 - Set the logging level
[PASS] 2.3 - Allow Docker to make changes to iptables
[PASS] 2.4 - Do not use insecure registries
[PASS] 2.5 - Do not use the aufs storage driver
[INFO] 2.6 - Configure TLS authentication for Docker daemon
[INFO] * Docker daemon not listening on TCP
[INFO] 2.7 - Set default ulimit as appropriate
[INFO] * Default ulimit doesn't appear to be set
[WARN] 2.8 - Enable user namespace support
[PASS] 2.9 - Confirm default cgroup usage
[PASS] 2.10 - Do not change base device size until needed
[WARN] 2.11 - Use authorization plugin
[WARN] 2.12 - Configure centralized and remote logging
[WARN] 2.13 - Disable operations on legacy registry (v1)
[INFO] 3 - Docker Daemon Configuration Files
[INFO] 3.1 - Verify that docker.service file ownership is set to root:root
[INFO] * File not found
[INFO] 3.2 - Verify that docker.service file permissions are set to 644
[INFO] * File not found
[INFO] 3.3 - Verify that docker.socket file ownership is set to root:root
[INFO] * File not found
[INFO] 3.4 - Verify that docker.socket file permissions are set to 644
[INFO] * File not found
[PASS] 3.5 - Verify that /etc/docker directory ownership is set to root:root
[PASS] 3.6 - Verify that /etc/docker directory permissions are set to 755
[INFO] 3.7 - Verify that registry certificate file ownership is set to root:root
[INFO] * Directory not found
[INFO] 3.8 - Verify that registry certificate file permissions are set to 444
[INFO] * Directory not found
[INFO] 3.9 - Verify that TLS CA certificate file ownership is set to root:root
[INFO] * No TLS CA certificate found
[INFO] 3.10 - Verify that TLS CA certificate file permissions are set to 444
[INFO] * No TLS CA certificate found
[INFO] 3.11 - Verify that Docker server certificate file ownership is set to root:root
[INFO] * No TLS Server certificate found
[INFO] 3.12 - Verify that Docker server certificate file permissions are set to 444
[INFO] * No TLS Server certificate found
[INFO] 3.13 - Verify that Docker server key file ownership is set to root:root
[INFO] * No TLS Key found
[INFO] 3.14 - Verify that Docker server key file permissions are set to 400
[INFO] * No TLS Key found
[PASS] 3.15 - Verify that Docker socket file ownership is set to root:docker
[PASS] 3.16 - Verify that Docker socket file permissions are set to 660
[INFO] 3.17 - Verify that daemon.json file ownership is set to root:root
[INFO] * File not found
[INFO] 3.18 - Verify that daemon.json file permissions are set to 644
[INFO] * File not found
[PASS] 3.19 - Verify that /etc/default/docker file ownership is set to root:root
[PASS] 3.20 - Verify that /etc/default/docker file permissions are set to 644
[INFO] 4 - Container Images and Build Files
[WARN] 4.1 - Create a user for the container
[WARN] * Running as root: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN] * Running as root: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN] * Running as root: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN] * Running as root: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[WARN] 4.5 - Enable Content trust for Docker
[INFO] 5 - Container Runtime
[WARN] 5.1 - Verify AppArmor Profile, if applicable
[WARN] * No AppArmorProfile Found: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN] * No AppArmorProfile Found: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN] * No AppArmorProfile Found: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN] * No AppArmorProfile Found: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[WARN] 5.2 - Verify SELinux security options, if applicable
[WARN] * No SecurityOptions Found: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN] * No SecurityOptions Found: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN] * No SecurityOptions Found: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN] * No SecurityOptions Found: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[PASS] 5.3 - Restrict Linux Kernel Capabilities within containers
[PASS] 5.4 - Do not use privileged containers
[PASS] 5.5 - Do not mount sensitive host system directories on containers
[PASS] 5.6 - Do not run ssh within containers
[PASS] 5.7 - Do not map privileged ports within containers
[PASS] 5.9 - Do not share the host's network namespace
[WARN] 5.10 - Limit memory usage for container
[WARN] * Container running without memory restrictions: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN] * Container running without memory restrictions: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN] * Container running without memory restrictions: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN] * Container running without memory restrictions: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[WARN] 5.11 - Set container CPU priority appropriately
[WARN] * Container running without CPU restrictions: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN] * Container running without CPU restrictions: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN] * Container running without CPU restrictions: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN] * Container running without CPU restrictions: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[WARN] 5.12 - Mount container's root filesystem as read only
[WARN] * Container running with root FS mounted R/W: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN] * Container running with root FS mounted R/W: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN] * Container running with root FS mounted R/W: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN] * Container running with root FS mounted R/W: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[PASS] 5.13 - Bind incoming container traffic to a specific host interface
[WARN] 5.14 - Set the 'on-failure' container restart policy to 5
[WARN] * MaximumRetryCount is not set to 5: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN] * MaximumRetryCount is not set to 5: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN] * MaximumRetryCount is not set to 5: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN] * MaximumRetryCount is not set to 5: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[PASS] 5.15 - Do not share the host's process namespace
[PASS] 5.16 - Do not share the host's IPC namespace
[PASS] 5.17 - Do not directly expose host devices to containers
[INFO] 5.18 - Override default ulimit at runtime only if needed
[INFO] * Container no default ulimit override: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[INFO] * Container no default ulimit override: router.0.3auya7kd42o0vhv9s4ms07lcc
[INFO] * Container no default ulimit override: router-backend.1.3aslakd616evw0aw80o222fm7
[INFO] * Container no default ulimit override: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[PASS] 5.19 - Do not set mount propagation mode to shared
[PASS] 5.20 - Do not share the host's UTS namespace
[PASS] 5.21 - Do not disable default seccomp profile
[PASS] 5.24 - Confirm cgroup usage
[WARN] 5.25 - Restrict container from acquiring additional privileges
[WARN] * Privileges not restricted: logentries.0.ev84fq8gswjb7zy8s1mama5ah
[WARN] * Privileges not restricted: router.0.3auya7kd42o0vhv9s4ms07lcc
[WARN] * Privileges not restricted: router-backend.1.3aslakd616evw0aw80o222fm7
[WARN] * Privileges not restricted: router-storage.1.4l2csgqh4opyn6hoq2nnb6sbr
[INFO] 6 - Docker Security Operations
[INFO] 6.4 - Avoid image sprawl
[INFO] * There are currently: 11 images
[INFO] 6.5 - Avoid container sprawl
[INFO] * There are currently a total of 13 containers, with 5 of them currently running
Cluster setup
5 nodes (1 manager, 4 nodes)
Those labels are applied:
docker node update --label-add region=do_nyc3 swm-aa
docker node update --label-add type=elected swm-aa
docker node update --label-add ram=512m swm-aa
docker node update --label-add region=do_nyc3 swm-db
docker node update --label-add type=db swm-db
docker node update --label-add ram=1g swm-db
docker node update --label-add region=do_nyc3 swm-01
docker node update --label-add type=apps swm-01
docker node update --label-add ram=512m swm-01
docker node update --label-add region=do_nyc3 swm-02
docker node update --label-add type=apps swm-02
docker node update --label-add ram=512m swm-02
docker node update --label-add region=do_nyc3 swm-03
docker node update --label-add type=apps swm-03
docker node update --label-add ram=512m swm-03
Datacenter
Digital Ocean Ubuntu 16.04
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 31 (9 by maintainers)
I just started a 7 nodes cluster (3m 4w) over 3 regions. Toronto, NY, Amsterdam. Letβs see β¦