etcd: unable to add member after removing it
@xiang90 I have a 3 machine dedicated etcd cluster. I try removing a member, then I re-add it. But the etcd service can not run. I am running etcd 2.0.9
# etcdctl member list
3aaed538a690319f: name=test-1-2 peerURLs=http://10.0.1.2:2380 clientURLs=http://0.0.0.0:2379,http://0.0.0.0:4001
76bd3f6d68582602: name=test-1-1 peerURLs=http://10.0.1.1:2380 clientURLs=http://0.0.0.0:2379,http://0.0.0.0:4001
b98b8b1bf1b01436: name=test-1-3 peerURLs=http://10.0.1.3:2380 clientURLs=http://0.0.0.0:2379,http://0.0.0.0:4001
# etcdctl member remove 76bd3f6d68582602
Removed member 76bd3f6d68582602 from cluster
# etcdctl member add test-1-1 http://10.0.1.1:2380
Added member named test-1-1 with ID a7ce7ae74a67d4f9 to cluster
ETCD_NAME="test-1-1"
ETCD_INITIAL_CLUSTER="test-1-2=http://10.0.1.2:2380,test-1-1=http://10.0.1.1:2380,test-1-3=http://10.0.1.3:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
remove etcd data-dir
# rm -rf /data/etcd/test-1-1.etcd/
# systemctl start etcd
Also it can not run when I delete etcd data-dir. the node log this:
Apr 14 15:47:42 test-1-1 etcd[10806]: 2015/04/14 15:47:42 etcdserver: the data-dir used by this member must be removed.
Apr 14 15:47:42 test-1-1 etcd[10806]: 2015/04/14 15:47:42 etcdserver: the member has been permanently removed from the cluster
Apr 14 15:47:42 test-1-1 etcd[10806]: 2015/04/14 15:47:42 raft: 76bd3f6d68582602 [logterm: 1, index: 3] sent vote request to 3aaed538a690319f at term 2
Apr 14 15:47:42 test-1-1 etcd[10806]: 2015/04/14 15:47:42 raft: 76bd3f6d68582602 [logterm: 1, index: 3] sent vote request to b98b8b1bf1b01436 at term 2
Apr 14 15:47:42 test-1-1 etcd[10806]: 2015/04/14 15:47:42 raft: 76bd3f6d68582602 received vote from 76bd3f6d68582602 at term 2
Apr 14 15:47:42 test-1-1 etcd[10806]: 2015/04/14 15:47:42 raft: 76bd3f6d68582602 became candidate at term 2
Apr 14 15:47:42 test-1-1 etcd[10806]: 2015/04/14 15:47:42 raft: 76bd3f6d68582602 is starting a new election at term 1
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcdserver: added member b98b8b1bf1b01436 [http://10.0.1.3:2380] to cluster a67a3b7f0fcea14e
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcdserver: added local member 76bd3f6d68582602 [http://10.0.1.1:2380] to cluster a67a3b7f0fcea14e
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcdserver: added member 3aaed538a690319f [http://10.0.1.2:2380] to cluster a67a3b7f0fcea14e
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 raft: 76bd3f6d68582602 became follower at term 1
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 raft: newRaft 76bd3f6d68582602 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 raft: 76bd3f6d68582602 became follower at term 0
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcdserver: start member 76bd3f6d68582602 in cluster a67a3b7f0fcea14e
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcdserver: initial cluster = test-1-1=http://10.0.1.1:2380,test-1-2=http://10.0.1.2:2380,test-1-3=http://10.0.1.3:2380
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcdserver: initial advertise peer URLs = http://10.0.1.1:2380
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcdserver: advertise client URLs = http://0.0.0.0:2379,http://0.0.0.0:4001
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcdserver: snapshot count = 10000
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcdserver: election = 1000ms
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcdserver: heartbeat = 100ms
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcdserver: member dir = /data/etcd/test-1-1.etcd/member
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcdserver: data dir = /data/etcd/test-1-1.etcd
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcdserver: name = test-1-1
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcdserver: datadir is valid for the 2.0.1 format
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcd: listening for client requests on http://0.0.0.0:4001
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcd: listening for client requests on http://0.0.0.0:2379
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcd: listening for peers on http://0.0.0.0:7001
Apr 14 15:47:41 test-1-1 etcd[10806]: 2015/04/14 15:47:41 etcd: listening for peers on http://0.0.0.0:2380
Apr 14 15:47:41 test-1-1 systemd[1]: Started Etcd Server.
Apr 14 15:47:41 test-1-1 systemd[1]: Starting Etcd Server...
thanks.
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Comments: 17 (7 by maintainers)
The new join member MUST set
ETCD_INITIAL_CLUSTER_STATE="existing"
I think you may just use
--initial-cluster-state new
or leave it unset.