moby: [1.13.0-rc3 -- Compose V3] 'network not found' on re-creating previously created stack

Is still happening in docker version 17.09.0-ce

@biels You just need to explicitly wait for the services and networks to be removed before deploying again. Here’s an example of how I’m doing i (from #30942).

until [ -z "$(docker service ls --filter label=com.docker.stack.namespace=$COMPOSE_PROJECT_NAME -q)" ] || [ "$limit" -lt 0 ]; do
  sleep 1;
done

until [ -z "$(docker network ls --filter label=com.docker.stack.namespace=$COMPOSE_PROJECT_NAME -q)" ] || [ "$limit" -lt 0 ]; do
  sleep 1;
done

The $COMPOSE_PROJECT_NAME is whatever you named the stack. The two loops run until there are no services and networks belonging to the named stack. It’s generally safe to re-deploy once that’s true.

This issue is over two years old, curious why it gets no attention or replies from code owners?

I have the same issue with docker 18.06.0-ce

Still in Docker version 18.09.1

I have my vm restarted and the problem solved.

It seems like this is probably because swarm mode is eventually consistent. When you run the network rm it returns before the network is fully removed. So if you attempt to deploy again, you might find the network that is being removed.

Do you think that’s the case?

Could be, but why would a rm request update “desired” state and return immediately, whereas a create request would actually refuse to go through if the network already exists (i.e., check actual state)?

Shouldn’t the creation of a network also modify only “desired” state and let swarm mode converge the actual state in the background?

Still in Docker version 18.09.0, build 4d60db4

Reproduced on RC5

docker stack rm project
Removing service project_service_a
Removing service project_service_b
Removing service project_service_c
Removing network project_default

docker stack deploy --compose-file stack.yml project
Creating service project_service_a
Error response from daemon: network project_default not found

As trying again work that’s clear to me the creation check is done before network actually deleted. Anything I could try to help?

Is still happening in docker version 18.03.1-ce

@dnephin - there are cases in which we would like to rm + redeploy for example on DEV and QA environments (as oppose to production in which we redeploy on top of existing deployment). The result is interrupted work processed for developers and automated testing.

I tried this, it worked for me

• Inspect the network which we are unable to delete

docker network inspect <id> or <name>

• Disconnect the network

docker network disconnect -f <networkID> <endpointName> or <endpointId>

• Delete unused networks

docker network prune

I am using Docker for Mac, this consistently happens for a year in my dev env.

This happens consistently on Linux with docker-ce 19.03.2

Just saw this on Windows 2019 with Docker 18.09.6. I’m going to try to create the networks before deploying the stack with ‘docker network create’ and reference them as external in the stack file.

Yes i have taken a look in the logs of docker for windows and the docker client send the rest call /DELETE network and get the response but the deletion is asynchronous. I think the network is removed when all services are removed

Thanks @kinghuang. Just implemented this logic in a gradle buildscript. I share it in case it can help someone because I think it is a sensible thing to want to do (be able to redeploy your dev environments in one command):

task undeployDev(type: Exec, group: 'application'){
    commandLine 'docker', 'stack', 'rm', 'sm'
}

private void waitForRemoval(String dockerEntity) {
    while (true) {
        ByteArrayOutputStream stdout = new ByteArrayOutputStream()
        exec {
            commandLine 'docker', dockerEntity, 'ls', '--filter', 'label=com.docker.stack.namespace=sm', '-q'
            standardOutput = stdout;
        }
        def length = stdout.toString().length()
        if (length == 0) break;
    }
}

task deployDev(type: Exec, group: 'application'){
    dependsOn ':sm-manager:buildDocker', ':sm-worker:buildDocker'
    mustRunAfter undeployDev
    doFirst{
        println 'Waiting for stack to finish undeploying...'
        waitForRemoval('service')
        waitForRemoval('network')
        println 'Stack undeploy finished'
    }
    commandLine 'docker', 'stack', 'deploy', '-c', 'docker-compose.yml', 'sm'
}

task redeployDev(group: 'application'){
    dependsOn undeployDev, deployDev
}

In this case it works with a stack named sm declared in a docker-compose.yml file at the root of the project but you can adapt it to your needs.

Why are you using stack rm for deployments? It should be just another stack deploy to send updates.

@eyal-lupu In my automated deployments, I explicitly wait for services and networks to be removed before redeploying. There’s a basic example of this in #30942 (the two until blocks). I have them between the docker stack rm … and docker stack deploy … commands.