etcd: inconsistent data in etcd 3.5.4

What happened?

I encounter a serious data inconsistent problem when using etcd 3.5.4。I have a three node etcd cluster, the problem will be happened occasionally when three node reboot at the same time. In the CHANGELOG-3.5, I see the data inconsistent problem was fixed in pull/13908, but I still encounter the data inconsistent problem。

What did you expect to happen?

There is no problem when three node reboot at the same time.

How can we reproduce it (as minimally and precisely as possible)?

The problem maybe happen when the three node of etcd cluster reboot at the same time.

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
# paste output here

$ etcdctl version
# paste output here

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 16 (7 by maintainers)

Most upvoted comments

One more question, can you easily reproduce this issue when rebooting all 3 members at the same time?

I tried a couple of times, but couldn’t reproduce this issue. But the traffic in my environment may not be that big as yours.

@ahrtr @serathius thank you very much, we have found the root cause for the problem. As your suggestion, it is readly due to our filesystem(we use xfs filesystem). Our filesystem was damaged when node reboot. The problem was not happen when we repari the xfs filesystem. So, it’s not etcd problem.

Thanks for a lot again, wo solve the problem troubled me many days due to your help. @ahrtr @serathius

It’s not etcd problem ,can i close the issue?

chaohan1 on Jun 22, 2022