etcd: etcdctl member add sometimes fails with 3.3.15

We recently updated our etcd from 3.3.13 to 3.3.15 and since then etcdctl member add sometimes fails with following error:

Member ca83b0fdcfd3f509 added to cluster  7f02dc1fca0457e
{"level":"warn","ts":"2019-09-27T16:56:16.191+0200","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-.../etcdmaster1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}

Though it seems the member add actually worked fine as a etcdctl member list shows the newly added member as unstarted

# endpoints are three existing and running etcd nodes
etcdctl member add "newmember" --peer-urls=https://newmember:2380
# errors out with above message
etcdctl member list
...
ca83b0fdcfd3f509, unstarted, , https://newmember:2380,

Starting the etcd process then also works fine.

This behaviour seems to be new in 3.3.15, it did not happen in 3.3.13.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 24 (17 by maintainers)

Most upvoted comments

Let me send out a quick fix on client side. Long term, I agree we should make MemberList consistent, so we can drop the logic on client side.