etcd: Inconsistent Revisions Across Members ( v3.3.3 )

Etcd Version: 3.3.3 OS: Linux 14.04.1 Ubuntu

Health

http://10.213.214.4:2379 is healthy: successfully committed proposal: took = 743.509µs
http://10.213.214.2:2379 is healthy: successfully committed proposal: took = 1.00709ms
http://10.213.214.3:2379 is healthy: successfully committed proposal: took = 1.011886ms

Table output

+--------------------------+------------------+---------+---------+-----------+-----------+------------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+--------------------------+------------------+---------+---------+-----------+-----------+------------+
| http://10.213.214.2:2379 | 372c0ce563084c78 |   3.3.3 |   95 MB |     false |        18 |   12165951 |
| http://10.213.214.3:2379 | 8546c864475e6358 |   3.3.3 |  136 MB |     false |        18 |   12165952 |
| http://10.213.214.4:2379 | e030b17253d751e9 |   3.3.3 |  175 MB |      true |        18 |   12165953 |
+--------------------------+------------------+---------+---------+-----------+-----------+------------+

JSON Ouput

[ 
   {
      "Endpoint":"http://10.213.214.2:2379",
      "Status": {
         "header {
                 "cluster_id":16002117407141323128,"
                 "member_id":3975566750409837688,
                 "revision":1377021,
                 "raft_term":18
           },
          "version":"3.3.3",
          "dbSize":95182848,
          "leader":16154606967968977385,
          "raftIndex":12165645,
          "raftTerm":18
      }
 },
{
  "Endpoint":"http://10.213.214.3:2379",
  "Status":{
     "header":{
     "cluster_id":16002117407141323128,
     "member_id":9603583588433814360,
     "revision":1801804,
     "raft_term":18
  },
  "version":"3.3.3",
  "dbSize":136417280,
  "leader":16154606967968977385, 
  "raftIndex":12165646,
  "raftTerm":18
  }
},
{
  "Endpoint":"http://10.213.214.4:2379",
  "Status":{
     "header"{
        "cluster_id":16002117407141323128,
        "member_id":16154606967968977385,
        "revision":891479,
        "raft_term":18
     },
     "version":"3.3.3",
     "dbSize":174526464,
     "leader":16154606967968977385,
     "raftIndex":12165647,
     "raftTerm":18
   }
}
]

As you can see the revisions across members are vastly different.

Each of these nodes are configured with --auto-compaction-retention 1.

Any thoughts on what’s going on here?