etcd: Continuous election due to timed out waiting for read index response

env info:

[root@csv-xzcs01 tools]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[root@csv-xzcs01 tools]# etcdctl --version
etcdctl version: 3.0.17
API version: 2
[root@csv-xzcs01 tools]# etcdctl member list
9756c65a035a74e3: name=master3 peerURLs=http://20.26.28.85:7001 clientURLs=http://20.26.28.85:4001 isLeader=true
be56b052e10c15f5: name=master2 peerURLs=http://20.26.28.84:7001 clientURLs=http://20.26.28.84:4001 isLeader=false
f262b8a2d1159226: name=master1 peerURLs=http://20.26.28.83:7001 clientURLs=http://20.26.28.83:4001 isLeader=false

log info (with debug):

2017-05-23 02:31:35.689952 I | raft: f262b8a2d1159226 is starting a new election at term 94638
2017-05-23 02:31:35.690019 I | raft: f262b8a2d1159226 became candidate at term 94639
2017-05-23 02:31:35.690038 I | raft: f262b8a2d1159226 received MsgVoteResp from f262b8a2d1159226 at term 94639
2017-05-23 02:31:35.690053 I | raft: f262b8a2d1159226 [logterm: 94634, index: 633473] sent MsgVote request to be56b052e10c15f5 at term 94639
2017-05-23 02:31:35.690089 I | raft: f262b8a2d1159226 [logterm: 94634, index: 633473] sent MsgVote request to 9756c65a035a74e3 at term 94639
2017-05-23 02:31:36.597685 W | etcdserver: timed out waiting for read index response
2017-05-23 02:31:37.417364 D | etcdserver/api/v2http: [GET] /v2/keys/calico/?quorum=false&recursive=false&sorted=false remote:20.26.28.84:41285
2017-05-23 02:31:37.689966 I | raft: f262b8a2d1159226 is starting a new election at term 94639
2017-05-23 02:31:37.690048 I | raft: f262b8a2d1159226 became candidate at term 94640
2017-05-23 02:31:37.690069 I | raft: f262b8a2d1159226 received MsgVoteResp from f262b8a2d1159226 at term 94640
2017-05-23 02:31:37.690084 I | raft: f262b8a2d1159226 [logterm: 94634, index: 633473] sent MsgVote request to be56b052e10c15f5 at term 94640
2017-05-23 02:31:37.690102 I | raft: f262b8a2d1159226 [logterm: 94634, index: 633473] sent MsgVote request to 9756c65a035a74e3 at term 94640
2017-05-23 02:31:38.641612 D | etcdserver/api/v2http: [GET] /version remote:20.26.28.83:54486
2017-05-23 02:31:39.143354 I | raft: f262b8a2d1159226 [term: 94640] ignored a MsgVote message with lower term from be56b052e10c15f5 [term: 94638]
2017-05-23 02:31:40.388408 I | raft: f262b8a2d1159226 [term: 94640] ignored a MsgVoteResp message with lower term from be56b052e10c15f5 [term: 94639]
2017-05-23 02:31:40.388477 I | raft: f262b8a2d1159226 received MsgVoteResp rejection from be56b052e10c15f5 at term 94640
2017-05-23 02:31:40.388494 I | raft: f262b8a2d1159226 [quorum:2] has received 1 MsgVoteResp votes and 1 vote rejections
2017-05-23 02:31:40.393781 I | raft: f262b8a2d1159226 [term: 94640] ignored a MsgVote message with lower term from 9756c65a035a74e3 [term: 94635]
2017-05-23 02:31:40.397896 I | raft: f262b8a2d1159226 [term: 94640] ignored a MsgVoteResp message with lower term from 9756c65a035a74e3 [term: 94639]
2017-05-23 02:31:40.397930 I | raft: f262b8a2d1159226 received MsgVoteResp rejection from 9756c65a035a74e3 at term 94640
2017-05-23 02:31:40.397943 I | raft: f262b8a2d1159226 [quorum:2] has received 1 MsgVoteResp votes and 2 vote rejections
2017-05-23 02:31:40.397962 I | raft: f262b8a2d1159226 became follower at term 94640
2017-05-23 02:31:43.289988 I | raft: f262b8a2d1159226 is starting a new election at term 94640
2017-05-23 02:31:43.290067 I | raft: f262b8a2d1159226 became candidate at term 94641
2017-05-23 02:31:43.290088 I | raft: f262b8a2d1159226 received MsgVoteResp from f262b8a2d1159226 at term 94641
2017-05-23 02:31:43.290104 I | raft: f262b8a2d1159226 [logterm: 94634, index: 633473] sent MsgVote request to 9756c65a035a74e3 at term 94641
2017-05-23 02:31:43.290121 I | raft: f262b8a2d1159226 [logterm: 94634, index: 633473] sent MsgVote request to be56b052e10c15f5 at term 94641
2017-05-23 02:31:45.110998 D | etcdserver/api/v2http: [GET] /v2/keys/calico/?quorum=false&recursive=false&sorted=false remote:20.26.28.83:47978
2017-05-23 02:31:45.257413 D | etcdserver/api/v2http: [GET] /v2/keys/calico/?quorum=false&recursive=false&sorted=false remote:20.26.28.85:41603
2017-05-23 02:31:45.597899 W | etcdserver: timed out waiting for read index response
2017-05-23 02:31:46.290097 I | raft: f262b8a2d1159226 is starting a new election at term 94641
2017-05-23 02:31:46.290172 I | raft: f262b8a2d1159226 became candidate at term 94642
2017-05-23 02:31:46.290190 I | raft: f262b8a2d1159226 received MsgVoteResp from f262b8a2d1159226 at term 94642
2017-05-23 02:31:46.290206 I | raft: f262b8a2d1159226 [logterm: 94634, index: 633473] sent MsgVote request to 9756c65a035a74e3 at term 94642
2017-05-23 02:31:46.290222 I | raft: f262b8a2d1159226 [logterm: 94634, index: 633473] sent MsgVote request to be56b052e10c15f5 at term 94642
2017-05-23 02:31:47.481998 D | etcdserver/api/v2http: [GET] /v2/keys/calico/?quorum=false&recursive=false&sorted=false remote:20.26.28.84:41285
2017-05-23 02:31:48.641675 D | etcdserver/api/v2http: [GET] /version remote:20.26.28.83:54494
2017-05-23 02:31:49.490042 I | raft: f262b8a2d1159226 is starting a new election at term 94642
2017-05-23 02:31:49.490161 I | raft: f262b8a2d1159226 became candidate at term 94643
2017-05-23 02:31:49.490189 I | raft: f262b8a2d1159226 received MsgVoteResp from f262b8a2d1159226 at term 94643
2017-05-23 02:31:49.490217 I | raft: f262b8a2d1159226 [logterm: 94634, index: 633473] sent MsgVote request to 9756c65a035a74e3 at term 94643
2017-05-23 02:31:49.490240 I | raft: f262b8a2d1159226 [logterm: 94634, index: 633473] sent MsgVote request to be56b052e10c15f5 at term 94643

when the warn etcdserver: timed out waiting for read index response will happened? curl works well:

[root@csv-xzcs01 tools]# etcdctl member list
9756c65a035a74e3: name=master3 peerURLs=http://20.26.28.85:7001 clientURLs=http://20.26.28.85:4001 isLeader=true
be56b052e10c15f5: name=master2 peerURLs=http://20.26.28.84:7001 clientURLs=http://20.26.28.84:4001 isLeader=false
f262b8a2d1159226: name=master1 peerURLs=http://20.26.28.83:7001 clientURLs=http://20.26.28.83:4001 isLeader=false
[root@csv-xzcs01 tools]# curl http://20.26.28.85:7001/version
{"etcdserver":"3.1.5","etcdcluster":"3.1.0"}[root@csv-xzcs01 tools]# 
[root@csv-xzcs01 tools]# curl http://20.26.28.85:4001/version
{"etcdserver":"3.1.5","etcdcluster":"3.1.0"}[root@csv-xzcs01 tools]# 

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 16 (7 by maintainers)

Most upvoted comments

ok. we found the root cause. and closing this now.