moby: Cannot join a node after previously removing it

Description I initially joined a Windows node to a swarm mode master running on Linux. Things worked ok, but I wanted to run docker service to create a service from the Windows node. I left the swarm, then tried to join again using the master key.

I tried a few different things to fix it, none of them worked:

• docker node rm ... on Linux master. This changed the error from the rpc error to permission denied (see below)
• docker swarm join-token --recycle No change
• docker init ... to reinitialize the cluster. No change in behavior

Steps to reproduce the issue:

1. docker swarm init on Linux node
2. docker join ... on Windows node with worker key
3. docker leave ... on Windows node
4. docker node rm ... to remove Windows node from Linux master
5. docker join ... on Windows node with master key

Describe the results you received:

PS C:\Users\patrick\master\MusicStore-master>  docker swarm join --token ... 10.0.0.10:2377
Error response from daemon: This node is already part of a swarm. Use "docker swarm leave" to leave this swarm and join
another one.
PS C:\Users\patrick\master\MusicStore-master> docker swarm leave
Node left the swarm.

I then removed the node from the master, and tried adding it:

PS C:\Users\patrick\master\MusicStore-master>     docker swarm join     --token ... 10.0.0.10:2377
Error response from daemon: can't initialize raft node: rpc error: code = 7 desc = Permission denied: unauthorized peer
role: rpc error: code = 7 desc = Permission denied: node npb0ng2w9iz27lhnmtgsvr1o6 was removed from swarm

I tried to fix this by cycling the keys. No change

Manager:

patrick@plang-v13-l:~$ docker swarm join-token --rotate manager
Successfully rotated manager join token.

To add a manager to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-49jm4viud3wq4w7vyxxv2zob6l78f4iidrp7ux2ba70twn5lo7-ekus743xun2wpxujei6hqp34f \
    10.0.0.10:2377

patrick@plang-v13-l:~$ docker swarm join-token --rotate worker
Successfully rotated worker join token.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-49jm4viud3wq4w7vyxxv2zob6l78f4iidrp7ux2ba70twn5lo7-6z29tt8wv2xexc7lg9ox6qad1 \
    10.0.0.10:2377

Worker:

PS C:\Users\patrick\master\MusicStore-master>     docker swarm join     --token SWMTKN-1-49jm4viud3wq4w7vyxxv2zob6l78f4i
idrp7ux2ba70twn5lo7-ekus743xun2wpxujei6hqp34f     10.0.0.10:2377
Error response from daemon: can't initialize raft node: rpc error: code = 7 desc = Permission denied: unauthorized peer
role: rpc error: code = 7 desc = Permission denied: node npb0ng2w9iz27lhnmtgsvr1o6 was removed from swarm

I tried to fix this by reinitializing the swarm. No luck

Manager:

patrick@plang-v13-l:~$ docker swarm init --force-new-cluster
Swarm initialized: current node (iz6q2tviob6eilm18hdm2i5yj) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token ... \
    10.0.0.10:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

patrick@plang-v13-l:~$ docker swarm join-token manager
To add a manager to this swarm, run the following command:

    docker swarm join \
    --token ... \
    10.0.0.10:2377

Worker node

PS C:\Users\patrick\master\MusicStore-master>     docker swarm join     --token ...     10.0.0.10:2377
Error response from daemon: can't initialize raft node: rpc error: code = 7 desc = Permission denied: unauthorized peer
role: rpc error: code = 7 desc = Permission denied: node npb0ng2w9iz27lhnmtgsvr1o6 was removed from swarm

Describe the results you expected: I would have expected my first attempt to join the cluster as a manager to succeed

Output of docker version:

patrick@plang-v13-l:~$ docker version
Client:
 Version:      1.13.0-rc3
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   4d92237
 Built:        Mon Dec  5 19:05:57 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.0-rc3
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.3
 Git commit:   4d92237
 Built:        Mon Dec  5 19:05:57 2016
 OS/Arch:      linux/amd64
 Experimental: false

Client:
 Version:      1.13.0-rc3
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   4d92237
 Built:        Tue Dec  6 01:15:44 2016
 OS/Arch:      windows/amd64

Server:
 Version:      1.13.0-rc3
 API version:  1.25 (minimum version 1.24)
 Go version:   go1.7.3
 Git commit:   4d92237
 Built:        Tue Dec  6 01:15:44 2016
 OS/Arch:      windows/amd64
 Experimental: true

Output of docker info:

patrick@plang-v13-l:~$ docker info
Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 2
Server Version: 1.13.0-rc3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 6
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: iz6q2tviob6eilm18hdm2i5yj
 Is Manager: true
 ClusterID: em9ruw7r35umshccoydxesrgu
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.0.0.10
 Manager Addresses:
  10.0.0.10:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 51371867a01c467f08af739783b8beafc154c4d7
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-47-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.359 GiB
Name: plang-v13-l
ID: EKON:4RK4:6MF3:YB5P:DP4K:XDEV:GDD3:OGUK:TUXN:34U6:2SIZ:JP5E
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

PS C:\Users\patrick\master\MusicStore-master> docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 15
Server Version: 1.13.0-rc3
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.447.amd64fre.rs1_release_inmarket.161102-0100)
Operating System: Windows Server 2016 Datacenter
OSType: windows
Architecture: x86_64
CPUs: 1
Total Memory: 3.5 GiB
Name: plang-v13-win
ID: GBCF:6HCI:JXSN:WXDG:K366:SYWB:AEB5:T66E:5WPK:THYW:7ZOP:VZGU
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

• Both are VMs running on Azure