moby: Error response from daemon: configuration network 8-2 is in use

Description

It seems when an IPAM driver fails and the container is removed the network that used the IPAM driver becomes stuck and can no longer be removed. rm -rf /var/lib/docker is necessary (unless some way to edit the .db file?).

Steps to reproduce the issue:

1. docker network create -d macvlan --subnet 8.2.250.0/24 --gateway 8.2.250.3 -o parent=public --ipam-driver arp-ipam3 --config-only 8-2
2. docker network create -d macvlan --scope swarm --config-from 8-2 --attachable public-8-2
3. docker service create … --network public-8-2 … my-service
4. ipam driver crashes during RequestAddress.
5. service create is in pending state for a long time.
6. CTRL-C
7. docker service rm my-service. Succeeds.
8. docker network rm public-8-2. Succeeds but errors in logs that network doesn’t exist?
9. docker network rm 8-2. This produces the titled error. Only way to remove network now is to rm -rf /var/lib/docker/

Describe the results you received: Cannot remove network.

Describe the results you expected: Network should be removed.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:18 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:56 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: vz74i89jega7k0ccyekpplgwq
 Is Manager: true
 ClusterID: qrpx9z4664ts79a6eamvrrxnw
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 10.0.21.39
 Manager Addresses:
  10.0.21.39:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.11.0-14-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 40
Total Memory: 141.6GiB
Name: container1
ID: IMGD:TF6D:E4FH:AT2H:SZ5C:YNIU:IB57:ATV7:Y6KU:JBIG:Z5PF:ZBMF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

Logs:

level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.RequestAddress: Post http://8.2.250.40:8181/IpamDriver.RequestAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 2s"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.RequestAddress: Post http://8.2.250.40:8181/IpamDriver.RequestAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 4s"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.RequestAddress: Post http://8.2.250.40:8181/IpamDriver.RequestAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 8s"
level=info msg="Node join event for container2-ba0d08b08afe/10.0.21.40"
level=error msg="fatal task error" error="starting container failed: Post http://8.2.250.40:8181/IpamDriver.RequestAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused" module="node/agent/taskmanager" node.id=vz74i89jega7k0ccyekpplgwq service.id=scogq490jtph6f5ahigrmvxl7 task.id=f97hf1oyokft44dbusddw3os1
level=warning msg="underweighting node vz74i89jega7k0ccyekpplgwq for service scogq490jtph6f5ahigrmvxl7 because it experienced 5 failures or rejections within 5m0s" module=node node.id=vz74i89jega7k0ccyekpplgwq
level=warning msg="underweighting node awvo1u8zp17i30jb0tze4cebj for service scogq490jtph6f5ahigrmvxl7 because it experienced 5 failures or rejections within 5m0s" module=node node.id=vz74i89jega7k0ccyekpplgwq
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleaseAddress: Post http://8.2.250.40:8181/IpamDriver.ReleaseAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 1s"
level=info msg="Node join event for container2-ba0d08b08afe/10.0.21.40"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleaseAddress: Post http://8.2.250.40:8181/IpamDriver.ReleaseAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 2s"
level=error msg="fatal task error" error="starting container failed: No such network: public-8-2" module="node/agent/taskmanager" node.id=vz74i89jega7k0ccyekpplgwq service.id=scogq490jtph6f5ahigrmvxl7 task.id=y3w4ai1tyirxtqumrp98bzadv
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleaseAddress: Post http://8.2.250.40:8181/IpamDriver.ReleaseAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 4s"
level=error msg="network public-8-2 remove failed: No such network: public-8-2" module="node/agent" node.id=vz74i89jega7k0ccyekpplgwq
level=error msg="remove task failed" error="No such network: public-8-2" module="node/agent" node.id=vz74i89jega7k0ccyekpplgwq task.id=njtqop3cb81v080k7f4lg5x88
level=error msg="network public-8-2 remove failed: No such network: public-8-2" module="node/agent" node.id=vz74i89jega7k0ccyekpplgwq
level=error msg="remove task failed" error="No such network: public-8-2" module="node/agent" node.id=vz74i89jega7k0ccyekpplgwq task.id=jizmlx0lgbmdo5bjg5ae2dkfs
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleaseAddress: Post http://8.2.250.40:8181/IpamDriver.ReleaseAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 8s"
level=warning msg="Failed to release gateway ip address 8.2.250.3 on delete of network public-8-2 (8jcmhqcq34jaeoa8qy2iguiw2): Post http://8.2.250.40:8181/IpamDriver.ReleaseAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleasePool: Post http://8.2.250.40:8181/IpamDriver.ReleasePool: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 1s"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleasePool: Post http://8.2.250.40:8181/IpamDriver.ReleasePool: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 2s"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleasePool: Post http://8.2.250.40:8181/IpamDriver.ReleasePool: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 4s"
level=info msg="Node join event for container2-ba0d08b08afe/10.0.21.40"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleasePool: Post http://8.2.250.40:8181/IpamDriver.ReleasePool: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 8s"
level=warning msg="Failed to release address pool 8.2.250.0/24 on delete of network public-8-2 (8jcmhqcq34jaeoa8qy2iguiw2): Post http://8.2.250.40:8181/IpamDriver.ReleasePool: dial tcp 8.2.250.40:8181: getsockopt: connection refused"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.RequestPool: Post http://8.2.250.40:8181/IpamDriver.RequestPool: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 1s"