moby: Swarm service, with overlay network, fails to remove all containers
Output of docker version
:
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:02:53 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:02:53 2016
OS/Arch: linux/amd64
Output of docker info
:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 4
Server Version: 1.12.1
Storage Driver: overlay2
Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null bridge host overlay
Swarm: active
NodeID: 0phf7j5medrlnam24hcki16db
Is Manager: true
ClusterID: 0ol98k8pblzgb1xpqeuuqn27f
Managers: 1
Nodes: 3
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: 10.20.21.1
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 4.6.0-0.bpo.1-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.911 GiB
Name: mc-docker1
ID: RG6N:UCJN:AT5K:AYYN:ZPH6:6IAN:ARJZ:TWAD:ZFTZ:BSYC:A4AN:TTP3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
Insecure Registries:
127.0.0.0/8
Steps to reproduce the issue:
docker network create -d overlay sleeper_nw
docker service create --name sleeper --replicas 20 --network sleeper_nw urlund/sleeper
docker service rm sleeper
Describe the results you received:
Output from docker ps -a
:
Node 1:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9b79d94bcebb urlund/sleeper:latest "/run.sh" 41 seconds ago Exited (137) 7 seconds ago sleeper.20.5shcg82g0p2sn8hkqidx6yuim
239a00862bc4 urlund/sleeper:latest "/run.sh" 41 seconds ago Exited (137) 7 seconds ago sleeper.2.7vm1a1u41fwpf3mjrviphsfyo
274ccd13294e urlund/sleeper:latest "/run.sh" 41 seconds ago Exited (137) 7 seconds ago sleeper.18.1qzllrci7j6rzpjdvk01fxip7
7837971e8e4d urlund/sleeper:latest "/run.sh" 41 seconds ago Exited (137) 7 seconds ago sleeper.6.dzrkydszgnvd4mzyy37fyxarc
Node 2:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
854ef7a322b7 urlund/sleeper:latest "/run.sh" 44 seconds ago Exited (137) 10 seconds ago sleeper.19.609h64iauf7oa1i95o0qi7qqk
fcfcf0208ebf urlund/sleeper:latest "/run.sh" 44 seconds ago Exited (137) 10 seconds ago sleeper.15.dpxzl9lysf6koofzhld6o8hnq
Node 3:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
32518f0c9ce2 urlund/sleeper:latest "/run.sh" 45 seconds ago Exited (137) 12 seconds ago sleeper.9.1k6snzf3xnjjfayoq3fhfvdtu
bf15e1d18a8a urlund/sleeper:latest "/run.sh" 45 seconds ago Exited (137) 12 seconds ago sleeper.5.bqf8dn47yqcosd45uk0skzvyj
fc406aef2f9c urlund/sleeper:latest "/run.sh" 45 seconds ago Exited (137) 12 seconds ago sleeper.12.68hlgm025wtur6xa57v8gfi8i
d5a9359d120e urlund/sleeper:latest "/run.sh" 45 seconds ago Exited (137) 12 seconds ago sleeper.7.0ql6whgya5753ckk12nahqx5u
46035a959fbe urlund/sleeper:latest "/run.sh" 45 seconds ago Exited (137) 12 seconds ago sleeper.10.e5z07uzu46z9c051iusx6qpqf
Describe the results you expected: I would expect all containers to be removed.
Additional information you deem important (e.g. issue happens only occasionally): You should not be able to reproduce this without creating/configuring an overlay network.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 26 (20 by maintainers)
Seems not find appropriate place for test.
So directly submit a PR here. Please let me know if there is any problem. @xiaods @urlund @mrjana @mostolog @cpuguy83
All debs have been uploaded to here, please try it based on your contrib. Thanks!
@urlund Yes, this is a bug caused by multiple go routines racing to delete the network and only one of them wins and everybody else fails to delete the network and because of that fails to remove the container. Will fix it. But other than some stale containers it will not cause any other issues in terms of functionality.
Great, I think it is exact. I will verify and submit a PR then.