moby: Unable to remove network "has active endpoints"

@keithbentrup This is a stale endpoint case. Do you happen to have the error log when that container that was originally removed (which left the endpoint in this state). BTW, if the container is removed, but the endpoint is still seen, then one can force disconnect the endpoint using docker network disconnect -f {network} {endpoint-name} . You can get the endpoint-name from the docker network inspect {network} command.

This issue seems to be very intermittent and does not happen very often.

Encountered this issue. I’m on linux (mint), and restarting docker fixed the issue for me:

sudo service docker restart
docker network rm my-network

Awesome. Thanks.

docker service restart fixed the problem for me.

sudo service docker restart

docker network rm <network name>

@keithbentrup thats correct. as I suggested earlier. the docker network disconnect -f {network} {endpoint-name} … pls use the endpoint-name. We can enhance this to support endpoint-id as well. But I wanted to confirm that by using the force option, were you able to make progress.

Or sudo reboot -f. Works 100%.

I may have the same problem with docker 1.13.0.

Since no-one in this thread have given an example of just what I did, I’ll post it.

For completes, this is the error that starts it. It may be because I have codekitchen/dinghy-http-proxy:2.5.0 which listens on port 80.

$ docker-compose -f deploy/docker-compose/docker-compose.yml
Creating network "dockercompose_default" with the default driver
Creating dockercompose_front-end_1
# and so on..

ERROR: for edge-router  Cannot start service edge-router: driver failed programming external connectivity on endpoint dockercompose_edge-router_1 (3ed8fb6cf4bc221dce615a9a3c5b8e4f0f8332e00e6c6d9b9f9bf0b09da57b36): Bind for 0.0.0.0:80 failed: port is already allocated
ERROR: Encountered errors while bringing up the project.

And trying to bring it all down:

$ docker-compose -f deploy/docker-compose/docker-compose.yml down
Stopping dockercompose_front-end_1
# and so on..
ERROR: network dockercompose_default has active endpoints

And how I killed the network:

$ docker network inspect dockercompose_default
[
    {
        "Name": "dockercompose_default", # <--- Param 1
        "Id": "dd1326487a637df8a4a7a11856864a0059fca45cb63e8363bfe5196082d42d6e",
        "Created": "2017-02-08T00:22:41.341653339Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "ea7a142c113700145e894c950b18fd4dec8a53e04a45045f1fb71c47eae1a13b": {
                "Name": "dinghy_http_proxy", # <--- Param 2
                "EndpointID": "38f362af8b22e575cc987f68399a97f3ed10abf2c4cc365460dba768f2df8daa",
                "MacAddress": "02:42:ac:12:00:0d",
                "IPv4Address": "172.18.0.13/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

$ docker network disconnect -f dockercompose_default dinghy_http_proxy
$ docker network rm dockercompose_default
dockercompose_default

why is this issue closed? It has never been fixed and still happens very regularly, still after multiple docker-compose down -v

~ > docker network rm up_default
Error response from daemon: error while removing network: 
network up_default id fe9eb06460d5a7cab71af2f21b4a95efa9c3251dd4d57dfbff7d9aaee49ebdff has active endpoints

~ > docker network inspect  up_default
[
    {
        "Name": "up_default",
        "Id": "fe9eb06460d5a7cab71af2f21b4a95efa9c3251dd4d57dfbff7d9aaee49ebdff",
        "Created": "2021-02-24T09:08:31.748003007Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "33.33.33.0/24"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "up",
            "com.docker.compose.version": "1.27.2"
        }
    }
]

what exactly is the mentioned endpoint considered here?

@danwdart Another reason for this is when there are dangling containers. in order to remove them, use the command docker-compose down --remove-orphans that should do the trick.

@mavenugo

docker network disconnect -f  [Network-Name] [Endpoint-Name] 

This worked for me .

I just experienced the bug that makes impossible to remove a network that has no containers attached to. Restarting the docker daemon solved the issue. Thank you @davidroeca!

@keithbentrup thats correct. as I suggested earlier. the docker network disconnect -f {network} {endpoint-name} … pls use the endpoint-name. We can enhance this to support endpoint-id as well. But I wanted to confirm that by using the force option, were you able to make progress.

What if the “endpoint” no longer exists? What if there are NO containers whatsoever (docker container ls -a shows NOTHING), but I STILL get this error?

When no command works then do sudo service docker restart your problem will be solved

I’ve just reproduced this issue with docker v1.11.2 while attempting docker-compose down. An earlier attempt to run docker-compose down did close the app_front network.

$ docker-compose down
Removing network app_front
WARNING: Network app_front not found.
Removing network app_back
ERROR: network app_back has active endpoints

$ docker network inspect app_back                                                            
[                                                                                                    
    {                                                                                                
        "Name": "app_back",                                                                  
        "Id": "4a8d557eda7ce06d222fc0a9053069f44e75d25147300796686522a872261245",                    
        "Scope": "local",                                                                            
        "Driver": "bridge",                                                                          
        "EnableIPv6": false,                                                                         
        "IPAM": {                                                                                    
            "Driver": "default",                                                                     
            "Options": null,                                                                         
            "Config": [                                                                              
                {                                                                                    
                    "Subnet": "172.22.0.0/16",                                                       
                    "Gateway": "172.22.0.1/16"                                                       
                }                                                                                    
            ]                                                                                        
        },                                                                                           
        "Internal": false,                                                                           
        "Containers": {                                                                              
            "702e9916e86b7f77af363014134f160a8dcd189399719e062069c10f735cb927": {                    
                "Name": "app_db_1",                                                          
                "EndpointID": "1decedbca5bc704be84f19e287926361d196d20fe2a9bbf092ab15b37b856b3a",    
                "MacAddress": "02:42:ac:16:00:02",                                                   
                "IPv4Address": "172.22.0.2/16",                                                      
                "IPv6Address": ""                                                                    
            }                                                                                        
        },                                                                                           
        "Options": {},                                                                               
        "Labels": {}                                                                                 
    }                                                                                                
]                                                                                                    

docker info

Containers: 17                                                                                   
 Running: 1                                                                                      
 Paused: 0                                                                                       
 Stopped: 16                                                                                     
Images: 140                                                                                      
Server Version: 1.11.2                                                                           
Storage Driver: aufs                                                                             
 Root Dir: /mnt/sda1/var/lib/docker/aufs                                                         
 Backing Filesystem: extfs                                                                       
 Dirs: 245                                                                                       
 Dirperm1 Supported: true                                                                        
Logging Driver: json-file                                                                        
Cgroup Driver: cgroupfs                                                                          
Plugins:                                                                                         
 Volume: local                                                                                   
 Network: bridge null host                                                                       
Kernel Version: 4.4.12-boot2docker                                                               
Operating System: Boot2Docker 1.11.2 (TCL 7.1); HEAD : a6645c3 - Wed Jun  1 22:59:51 UTC 2016    
OSType: linux                                                                                    
Architecture: x86_64                                                                             
CPUs: 1                                                                                          
Total Memory: 1.955 GiB                                                                          
Name: default                                                                                    
ID: LKRP:E2TX:KNVZ:UD4M:FIGG:ZROO:CIA5:WBKH:RNUB:KXTQ:E6DC:545P                                  
Docker Root Dir: /mnt/sda1/var/lib/docker                                                        
Debug mode (client): false                                                                       
Debug mode (server): true                                                                        
 File Descriptors: 18                                                                            
 Goroutines: 38                                                                                  
 System Time: 2016-06-15T22:44:13.34779866Z                                                      
 EventsListeners: 0                                                                              
Username: tohagan                                                                                
Registry: https://index.docker.io/v1/                                                            
Labels:                                                                                          
 provider=virtualbox                                                                             

See this ref for faster resolution Docker removal examples

@mavenugo If by docker network disconnect -f {network} {endpoint-name} you mean docker network disconnect [OPTIONS] NETWORK CONTAINER per docker network disconnect --help, I tried that, but it complained (not surprisingly) with No such container.

If you meant the EndpointID instead of the container name/id, I did not try that (but will next time) because that’s not what the --help suggested.

For a closed issue this ticket is rather active. Just had a situation where docker-compose -f file down finished with the error in question but there was no active (or any) endpoint. Restarting docker service fixed allowed removing the network.

Was brought here because of the same issue. I appended --remove-orphans and it was fixed

I’m seeing the same problem when attempting to remove a stack:

> sudo docker stack rm my-stack
Removing network my-stack_default
Failed to remove network g0450dknntdsfj1o055mk4efm: Error response from daemon: network my-stack_default has active endpointsFailed to remove some resources

I had first created the stack like so:

sudo docker stack deploy -c docker-compose.yml --with-registry-auth my-stack

Am using this version:

Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:14:09 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:14:09 2017
 OS/Arch:      linux/amd64
 Experimental: false

Luckily, sudo service docker restart fixes it, but still not ideal behavior.

@mavenugo sorry for the delay. I’m not using docker multi-host networking afaik. Its a single node raspberry pi and I haven’t done anything other than install docker via hypriot.

Here’s the output you requested (network is the network I can’t delete)

$ docker network ls
NETWORK ID          NAME                DRIVER
d22a34456cb9        bridge              bridge              
ef922c6e861e        network             bridge              
c2859ad8bda4        none                null                
150ed62cfc44        host                host 

The kv file is attached, I had to name it .txt to get around github filters, but its the binary file.

local-kv.db.txt

I created the network via direct API calls (dockerode)

This has worked (create and delete) numerous times, I think in this instance, I docker rm -f <container-id> but I’m not positive, I might have power-cycled the machine…

Hope that helps. –brendan

Restarting docker didn’t do the trick for me. Added --remove-orphans option to docker-compose down and the network was successfully removed.

Hello from 2019, @mavenugo I would like to pass on my sincere, sincere thanks for having the solution on this one back in 2016.

Instead, if I include project name with the down then the containers are torn down first and then the custom network: docker-compose --file docker-compose.tests.yml --project-name myprojecttests down

Thanks for sharing your fix, @Wallace-Kelly. Adding a --project-name to my docker-compose up and docker-compose down commands fixed the “network has active endpoints” error for me. 👍

still seeing these “orphaned networks” can you reopen @rmb938 @thaJeztah

Error response from daemon: network abcd_default id 3f2f1a6cb1cee2d82f2e2e59d10a099834a12b18eb7e3e48d6482d758bd93617 has active endpoints

docker version
Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:23:31 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:19:04 2017
 OS/Arch:      linux/amd64

only way to prune them seems to be restarting the engine

@mavenugo Yes, I’m using the overlay driver via docker-compose with a swarm host managing 2 nodes.

When I docker network inspect the network on each individual node, 1 node had 1 container listed that no longer existed and so could not be removed by docker rm -fv using the container name or id.

Same issue happening in our ci pipeline. This is happening too often now a days and failing out pipelines. docker-compose down with --remove-orphans also same issue. Restarting docker is the only solution now and retriggering the jobs .

I have this error running airflow.

Running

sudo aa-remove-unknown

fixed it.

I needed to just run docker-compose rm - docker-compose down is what I normally do and ps -a showed no containers so this really tripped me up until I ran the rm cmd. Thought I’d share.

i had a similar issue today. what i did was i ran “docker container ls -a” and saw few containers still running were utilizing the network which i had launched via docker stack. so manually when i killed those containers i was able to delete the network

Encountered it in 17.07.0-ce, disconnect approach did not work, then restarted docker and run rm again with success.

I also just reproduced this in 1.10.3 and landed here via google looking for a work around. I can’t force disconnect the active endpoints b/c none of the containers listed via docker network inspect still exist.

I eventually had to recreate my consul container and restart the docker daemon.

Another thank you @mavenugo here from 2020

This is still reproducable on Docker version 18.06.1-ce, build e68fc7a Seems that even when the containers of a compose file are removed, their endpoints are sometimes not removed, which can sometimes happen on power loss, so composes fail to either completely start or completely remove.

Good luck today

docker-compose down
Removing network gen365cms_default
ERROR: network gen365cms_default id b6c51b1a83ee2b938ee1c7f7148347dc9ef80a8d8ed93334873f1f84b3f27c04 has active endpoints

docker version
Client:
 Version:	17.12.0-ce-rc4
 API version:	1.35
 Go version:	go1.9.2
 Git commit:	6a2c058
 Built:	Wed Dec 20 15:53:52 2017
 OS/Arch:	darwin/amd64

Server:
 Engine:
  Version:	17.12.0-ce-rc4
  API version:	1.35 (minimum version 1.12)
  Go version:	go1.9.2
  Git commit:	6a2c058
  Built:	Wed Dec 20 15:59:49 2017
  OS/Arch:	linux/amd64
  Experimental:	true

I’ve run into this with a 17.06-ce swarm cluster too; running out of options besides rebooting.

@nicolaiskogheim has a valid solution. However, my team has a docker-compose file with ~20 containers. So I found another solution.

I’d like to throw in that you can also restart the docker daemon (e.g. systemctl restart docker on centos) and then the links between network and containers will be gone. You can then docker system prune -f with success.

docker-compose down --remove-orphans Removing network test_default ERROR: error while removing network: network test_default id 4169314d8c79786bd7ed0a9d8df9a153ee26827f9b00f582c4fd981019bc2137 has active endpoints

seeing a similar issue, albeit very very intermittently in CI (for https://github.com/envoyproxy/envoy) for an example that scales backend services

for ref, the related compose file is here https://github.com/envoyproxy/envoy/blob/main/examples/locality-load-balancing/docker-compose.yaml

and the script that is testing it in CI is here https://github.com/envoyproxy/envoy/blob/main/examples/locality-load-balancing/verify.sh

im going to add the --remove-orphans flag, altho as this happens so infrequently it wont be easy to see if it helps

@lkaupp from the output of your error, I suspect you have the docker “snap” installed, which are packaged and maintained by Canonical / Ubuntu. I know there’s various issues with those; are you seeing the same problem when running the official docker packages (https://docs.docker.com/engine/install/ubuntu/)?

yes you are correct, I just found the solution myself and uninstalled the snap version. Added a description for others that run into the problem. Thank you for the fast response @thaJeztah 😃

Solution: The different server version made it suspicious. Found a working solution for me: https://askubuntu.com/a/1315837/1199816 .In my case, our admin installed docker during the ubuntu setup, which made a snap installation. But, as an experienced long-term ubuntu / debian user, I installed docker using the command line as usual. Apt list will only show the packages installed via apt and do not show the snap packages! Now, two separate versions of the docker server run side-by-side and the docker client connects either to one of the competing background servers - with the side-effect that only one has the running containers and the other told me there are no running containers with docker ps. I just uninstalled the snap version (we are not using snap by default) and everything is working fine and is up-to-date. I just leave my post here to help others.

Old post with error behavior: Still an issue. I can specify it further. The containers seem to run “hidden” and are accessible from the internet. But, docker ps just show one or two containers (out of 7) in my application. Now, if I want to docker-compose down the file, the network cannot be removed because of the containers running “hidden”. What I have recognized earlier is, if you uninstall docker completely and reinstall it from the ground, the containers are attached backed and you can simply down them and remove the network. Nevertheless, this cannot be the intended behavior. As a side-effect, you cannot “up” the compose file because it throws an error that the ports are in use or: ERROR: for db Cannot create container for service db: failed to mount local volume: mount /digLab/GG/db:/var/snap/docker/common/var-lib-docker/volumes/dbdata/_data, flags: 0x1000: no such file or directory

I am on Ubuntu 20.04 LTS, with:

Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.40
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:54:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       bd33bbf
  Built:            Fri Feb  5 15:58:24 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

This is still an issue after more than four years. Is there some simpler way of disconnecting every container from every network they are connected to than a 10±line shell script? FWIW this seems to work:

#!/usr/bin/env bash

set -o errexit -o nounset -o pipefail

trap 'rm --recursive "$workspace"' EXIT
workspace="$(mktemp --directory)"
error_log="${workspace}/error.log"

for container_id in $(docker ps --all --quiet)
do
    readarray -t network_names < <(docker inspect "$container_id" | jq --raw-output '.[] | .NetworkSettings.Networks | if . == null then empty else keys | .[] end')
    for network_name in "${network_names[@]}"
    do
        echo "Disconnecting container ${container_id} from network ${network_name}."
        exit_code=0
        docker network disconnect "$network_name" "$container_id" 2> "$error_log" || exit_code="$?"
        if [[ "$exit_code" -ne 0 ]]
        then
            if grep --fixed-strings --quiet --regexp 'not connected to network' --regexp 'not connected to the network' "$error_log"
            then
                echo 'Ignoring "not connected" error…'
            else
                cat "$error_log" >&2
                exit "$exit_code"
            fi
        fi
    done
done

In summary:

1. Set up a trap to remove the workspace at exit.
2. Create the workspace.
3. For each container:
   1. For each network the container is associated with:
      1. Try to disconnect.
      2. If the disconnect fails because it is already not connected to the network, ignore the error (unfortunately it seems to be random whether “the” is part of that error message). Otherwise fail.

I end up with the same problem, as the network was not able to remove with all acctions , noting helped. my version is Docker version 18.09.6, build 481bc77

To fix, i have restart docker services. by “sudo service docker restart” after that i able to remove with “docker network rm {network}”

sudo service docker restart gets rid of it for me on ubuntu, allowing me to deploy & start my containers again.

Also works if one of the containers refuses to get killed (happens more than I’d hope). Annoying as it makes me bring all services down because of one mischievous container.

@keithbentrup @brendandburns thanks for raising the issue. Couple of questions

1. Are you using any multi-host network driver (such as Overlay driver). Can you please share the docker network ls output.
2. If you dont’ use multi-host driver, then can you please share the /var/lib/docker/network/files/local-kv.db file (via some file-sharing website) and which network are you trying to remove ? And how was the network originally created ?

FYI. for a multi-host network driver, docker maintains the endpoints for a network across the cluster in the KV-Store. Hence, if any host in that cluster still has an endpoint alive in that network, we will see this error and this is an expected condition.