moby: swarm mode duplicate ip addresses
Description We recently had several issues with our cloud provider which lead us to several docker swarms failing with strange errors. One thing we observied that from time to time IP adresses in the swarm overlay network are assigned duplicate. We made several attempts to reproduce this issue and finally found a way to reproduce the duplicate ip adress issue.
Steps to reproduce the issue:
- Create a docker swarm with several nodes (five in our example)
- Start some Services in the swarm with at least two replicas
- Start some containers on swarm nodes, that are attached to the swarm overlay network
- kill the docker process (kill -9 <main_pid_from systemctl status docker>) and/or kill some service tasks running on the same node (kill -9 <main_pid_from docker inspect>)
- do a “docker network inspect -v <overlay_network>” on the node
- start and stop some services in the swarm
Describe the results you received: I can see that ip adresses are assigned to more than one service task.
Describe the results you expected: I would expect that every service task is assigned a unique ip adress.
Additional information you deem important (e.g. issue happens only occasionally):
What we tried to resolve this issue:
- let affected node leave the swarm (docker swarm leave, docker node demote, docker node rm)
- stop the docker daemon on the affected node
- rm -rf /var/lib/docker/*
- start the docker daemon
- rejoin the swarm
there are still duplicate ip adresses (ocker network inspect -v <overlay_network>)
next try:
- let node leave the swarm again
- stop the host
- delete the host
- create a new host (new host name, new ip address)
- let new host join swarm as manager
again, there are duplicate ip adresses which are assigned to the host that was completely removed from the swarm (see attached file overlay.txt)
overlay.txt
Output of docker version:
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:06:25 2017
OS/Arch: linux/amd64
Server: Version: 17.05.0-ce API version: 1.29 (minimum version 1.12) Go version: go1.7.5 Git commit: 89658be Built: Thu May 4 22:06:25 2017 OS/Arch: linux/amd64 Experimental: false
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:06:25 2017
OS/Arch: linux/amd64
Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:06:25 2017
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
see attached file docker_info.txt
docker_info.txt
Additional environment details (AWS, VirtualBox, physical, etc.):
Environment: OS: CentOS 7 running in OpenStack in a public cloud.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 2
- Comments: 36 (14 by maintainers)
@selfisch can you try 17.10, the IP allocation is now serialized, so till you don’t get close to the exhaustion of the IP space, you should not see this behavior anymore.
Just seen this issue again using 17.12.0. Debian 9, Docker 17.12.0, 5 Node Swarm. Leader was node 1, I stopped the docker service on node 1 and node 3 got elected leader with this log from node 3. 172.20.0.0/16 is a Swarm overlay network. I can also confirm that by explicitly setting EndpointSpec to ‘vip’ this problem doesn’t occur.
@fcrisciani yeah I was gonna say that maybe the issue is something more superficial. anyways, that is good to know and i believe the issue is resolved for us.
Ok, short feedback.
With existing duplicated ip in swarm at version 17.09, performing update to 17.10, the issue will not be resolved automatically. But after updating all nodes to 17.10 and reinitializing the whole swarm, everything seems to be fine.
I restarted several nodes, several times this morning and there are no doub ips coming up. hurray 😃
Ok, found a fairly straight forward way to reproduce this on 17.06.2: reboot one of the nodes.
What I do is dump the state of the services before and after the reboot, like so:
After the reboot we often get duplicate IP addresses for services. When diff the result of the above command you get the output below. Without any docker commands being executed, simply restarting one of the nodes causes the service IPs to be changed and duplicated. This makes no sense, but it’s what happens. And the IPs that get allocated overlap other existing services.
Note that with debugging enabled I can see no messages for ReleaseAddress or RequestAddress for these IPs. Which basically makes it look like the IP address are being made from broken state? But the entire address allocation code seems to be skipped?
I found the tool swarm-raftlog to dump the logs, and so I found that the index 88195 is when the node leadership changes:
And the next entry updates all the running services, several to bogus IPs (service ekfjd93y2qgg15v1tare811ab had this IP already and it wasn’t changed in this update)
So it seems that right after the node leadership changes the services are updated with bogus Service IPs in one single large update. The IP 10.0.2.2/24 is now duplicated.
BTW, it looks like the network allocations are not logged, so the allocation status bitmap must be either transmitted some other way, or the new master must regenerate it from the data it has?
For completeness the original raft records that created the service: