moby: docker network rm `networkname` has error has active endpoints

We have the same issue on a non proxmox environment

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 24
 Server Version: 20.10.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.19.0-14-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 3
 Total Memory: 9.528GiB
 Name: <censore>
 ID: ES76:Q3LQ:A2MI:AHZ3:K7RS:WELI:7OYF:7SWB:CNLB:6ZDF:ONZ5:NCGR
 Docker Root Dir: /mnt/data/docker/data
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

while no container is attached

docker inspect testing_default
[
    {
        "Name": "testing_default",
        "Id": "ba43b3a8d952d30bceb36d9f76e110958065cb605c033e4282696e5df7e2ca4f",
        "Created": "2021-03-23T20:09:26.959545689+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.249.249.128/25"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.name": "dw1"
        },
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "testing",
            "com.docker.compose.version": "1.28.4"
        }
    }
]

and not even a single container exists

docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Seems to be happening with the new docker engine, 18.xx seems to work just fine. Fairly blocking since it reoccurs frequently.

There is no way to fix this issue besides restarting the docker-engine itself. We neither can do either of this

• docker network prune --force
• docker network rm testing_default
• readd some random container to the network, remove the container and try to remove the network then

@sonersivri you should probably rename the issue to include ‘while no containers exist’ because there are a couple of other issues with ‘still active endpoints’ which actually have attached containes - which we do not have here and that should be an important difference

docker --version Docker version 20.10.7, build f0df350

on Ubuntu 18.04.5

docker network rm netA
Error response from daemon: error while removing network: network netA id 881a8b7416c1d0f432699c4bbd0ae5da68cd53b8df08229ef9d2a57568cb1b9d has active endpoints

docker container ls --all
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

sudo systemctl restart docker 

docker network rm netA

Restating the daemon works

Just happened to me as well. I solved it by restarting docker service, I can do that right now since this is a new server setup but wont be able to in the future once we have services running.

@jest: Thanks for the detailed look into it. I also had the hunch, that is was tracking the number of containers in the network via some variable which it fails to decrement to zero and then it refuses to remove the container because it only checks that value at that point.

I also think there are concurrent database accesses which mess up that value resulting in an inconsistency.

You are also correct that this issue exists for quite some time already. You can find people all over the internet running into it and the only “solution” is to restart dockerd…

Maybe with your detailed information someone is willing to dig into it and find the offending database interactions.

Some other workaround that I have found:

docker compose down --remove-orphans --timeout 30 # this might fail with the removal error
docker system prune --force # this seems to get rid of the broken network
docker compose down --remove-orphans --timeout 30 # this does not fail anymore

We are doing this for 6 weeks now and haven’t hit the bug again. We might be lucky, or this is another workaround without shutting down dockerd.

We’re seeing this same issue as well with docker 20.10.5. We have a CI/CD that is spawning hundreds of dockers and with docker 19.03 we never had issues with “docker-compose down” reporting active endpoints. We recently upgraded to docker 20.10.5 and we’re hitting this problem tens of times per day on that CI/CD pipeline.

Let me know if there’s any log I can collect to help debugging this issue.