etcd: etcdctl member add sometimes fails with 3.3.15
We recently updated our etcd from 3.3.13 to 3.3.15 and since then etcdctl member add
sometimes fails with following error:
Member ca83b0fdcfd3f509 added to cluster 7f02dc1fca0457e
{"level":"warn","ts":"2019-09-27T16:56:16.191+0200","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-.../etcdmaster1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Though it seems the member add actually worked fine as a etcdctl member list shows the newly added member as unstarted
# endpoints are three existing and running etcd nodes
etcdctl member add "newmember" --peer-urls=https://newmember:2380
# errors out with above message
etcdctl member list
...
ca83b0fdcfd3f509, unstarted, , https://newmember:2380,
Starting the etcd process then also works fine.
This behaviour seems to be new in 3.3.15, it did not happen in 3.3.13.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 24 (17 by maintainers)
Let me send out a quick fix on client side. Long term, I agree we should make
MemberList
consistent, so we can drop the logic on client side.