etcd: Commit is out of range

We’ve observed a problem with clustered etcd, that after recreating one of the instances, it is entering crashloop and panicing with:

2017-11-29 05:47:07.905742 I | etcdserver: advertise client URLs = http://127.0.0.1:2379 
2017-11-29 05:47:08.076040 I | etcdserver: restarting member 9af8ed310dba8214 in cluster 488ce3f49fab5e77 at commit index 20707 
2017-11-29 05:47:08.077815 C | raft: 9af8ed310dba8214 state.commit 20707 is out of range [10001, 11535] 
panic: 9af8ed310dba8214 state.commit 20707 is out of range [10001, 11535] 

What exactly is happening is that we are recreating a VM where the etcd is running (without calling remove/add member) preserving all the addresses, etcd data etc. So after recreation, it the etcd is being started as the old member (similarly as it was restarted, though with a big bigger downtime).

We are running 3.0.17 version.

I found this https://github.com/coreos/etcd/issues/5664, which seems to be a similar (same?) problem. Has this been fixed? If so, in what release?

@xiang90 @gyuho @heyitsanthony @jpbetz

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 21 (19 by maintainers)

Most upvoted comments

Yes. As glad as I am to see the problem go away. It makes me uncomfortable not knowing why. We’ll get to the bottom of it.

On Mon, Dec 11, 2017 at 11:50 AM Wojciech Tyczynski < notifications@github.com> wrote:

Definitely - Joe already said that he will be looking into that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/coreos/etcd/issues/8935#issuecomment-350837442, or mute the thread https://github.com/notifications/unsubscribe-auth/AAf9Ru-vB6Qu67H0sSyUCIgs30UirK1jks5s_Yd6gaJpZM4Qu1ol .

Definitely - Joe already said that he will be looking into that.