etcd: Auth failed with error "auth: revision in header is old"

etcd version

3.4.16

Background

The original issue is 13503. There is a long back-and-forth conversation, but not much progress, so I raise a new issue to get the latest error message included.

There are 3 members in the etcd cluster, and the original version was 3.3.11. In the beginning, the users saw the data inconsistency issue, which could be caused by 11651. The issue was fixed in 3.3.21, 3.4.8 and 3.5.0. So the user upgraded to 3.4.16, and then see the error “auth: revision in header is old” (see detailed log below).

Please refer to the following two files to get the detailed info on how the user configures & deploys the etcd cluster, configmap.txt statefulset.txt

Help wanted

If you run into the same issue, please provide the etcd version & detailed steps to reproduce this issue. Thanks.

It also would be helpful if anyone can reproduce this issue in 3.5 and main branch.

Error message

**2022-03-24 02:15:14.037286 I | rafthttp: added peer af60d29c10528d99
raft2022/03/24 02:15:14 INFO: e83baa9da4c00535 switched to configuration voters=(4962605495301377754 12637332122132188569 16734156434475844917)
2022-03-24 02:15:14.037486 I | etcdserver/membership: added member e83baa9da4c00535 [https://eric-data-distributed-coordinator-ed-2.eric-data-distributed-coordinator-ed-peer.eiffelesc.svc.cluster.local:2380] to cluster 1d35520433a8d4ea
2022-03-24 02:15:14.038398 W | etcdserver: failed to apply request "header:<ID:1070308296661102795 username:\"root\" auth_revision:11 > lease_revoke:<id:0d997fb99eb3f316>" with response "size:28" took (112.397µs) to execute, err is lease not found
2022-03-24 02:15:14.038441 W | etcdserver: failed to apply request "header:<ID:1070308296661102796 username:\"root\" auth_revision:11 > lease_revoke:<id:0d997fb99eb3f314>" with response "size:28" took (18.443µs) to execute, err is lease not found
2022-03-24 02:15:14.038796 W | etcdserver: failed to apply request "header:<ID:10203326865992843127 username:\"root\" auth_revision:11 > auth_role_revoke_permission:<role:\"kms_role\" key:\"/kms/\" range_end:\"\\000\" > " with response "" took (7.084µs) to execute, err is auth: permission is not granted to the role
2022-03-24 02:15:14.038829 W | etcdserver: failed to apply request "header:<ID:10203326865992843128 username:\"root\" auth_revision:12 > auth_role_revoke_permission:<role:\"kms_role\" key:\"/shelter/\" range_end:\"\\000\" > " with response "" took (3.462µs) to execute, err is auth: permission is not granted to the role
2022-03-24 02:15:14.038923 N | auth: role kms_role's permission of key /kms/ is updated as READWRITE
2022-03-24 02:15:14.039001 N | auth: role kms_role's permission of key /shelter/ is updated as READWRITE
2022-03-24 02:15:14.039060 W | auth: user eric-sec-key-management is already granted role kms_role**
2022-03-24 02:15:14.041660 I | embed: ClientTLS: cert = /run/sec/certs/server/srvcert.pem, key = /run/sec/certs/server/srvprivkey.pem, trusted-ca = /data/combinedca/cacertbundle.pem, client-cert-auth = true, crl-file = 
2022-03-24 02:15:14.041741 I | embed: listening for peers on [::]:2380
2022-03-24 02:15:14.089214 I | rafthttp: peer 44deb767abd70eda became active
2022-03-24 02:15:14.089282 I | rafthttp: established a TCP streaming connection with peer 44deb767abd70eda (stream Message reader)
raft2022/03/24 02:15:14 INFO: raft.node: e83baa9da4c00535 elected leader 44deb767abd70eda at term 4
2022-03-24 02:15:14.167093 I | rafthttp: established a TCP streaming connection with peer 44deb767abd70eda (stream MsgApp v2 reader)
2022-03-24 02:15:14.167519 I | rafthttp: peer af60d29c10528d99 became active
2022-03-24 02:15:14.167548 I | rafthttp: established a TCP streaming connection with peer af60d29c10528d99 (stream MsgApp v2 reader)
2022-03-24 02:15:14.169870 I | rafthttp: established a TCP streaming connection with peer af60d29c10528d99 (stream Message reader)
2022-03-24 02:15:14.175822 I | etcdserver: e83baa9da4c00535 initialized peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s)
2022-03-24 02:15:14.216796 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/counters/requests/2022/03, err is auth: revision in header is old
2022-03-24 02:15:14.216948 W | etcdserver: failed to apply request "header:<ID:1070308296661102798 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/sys/counters/requests/2022/03\" value_size:47 >" with response "" took (40.243µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.216959 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/sys/counters/requests/2022/03, err is auth: revision in header is old
2022-03-24 02:15:14.216975 W | etcdserver: failed to apply request "header:<ID:1070308296661102799 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/kms/sys/counters/requests/2022/03\" value_size:47 >" with response "" took (5.927µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217077 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/core/lock/d997fb99eb3ff8a, err is auth: revision in header is old
2022-03-24 02:15:14.217138 W | etcdserver: failed to apply request "header:<ID:10203326865992843149 username:\"eric-sec-key-management\" auth_revision:15 > txn:<compare:<target:CREATE key:\"/shelter/core/lock/d997fb99eb3ff8a\" create_revision:0 > success:<request_put:<key:\"/shelter/core/lock/d997fb99eb3ff8a\" lease:979954829138067338 >> failure:<>>" with response "" took (21.485µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217211 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/core/lock/d997fb99eb3ff91, err is auth: revision in header is old
2022-03-24 02:15:14.217239 W | etcdserver: failed to apply request "header:<ID:10203326865992843156 username:\"eric-sec-key-management\" auth_revision:15 > txn:<compare:<target:CREATE key:\"/kms/core/lock/d997fb99eb3ff91\" create_revision:0 > success:<request_put:<key:\"/kms/core/lock/d997fb99eb3ff91\" lease:979954829138067345 >> failure:<>>" with response "" took (12.327µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217249 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/core/leader/dd6333b5-361d-feca-e0e1-705f79d02baf, err is auth: revision in header is old
2022-03-24 02:15:14.217274 W | etcdserver: failed to apply request "header:<ID:1070308296661102800 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/shelter/core/leader/dd6333b5-361d-feca-e0e1-705f79d02baf\" > " with response "" took (7.386µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217280 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/core/leader/358aa84d-03ff-32fa-35b1-e597af763118, err is auth: revision in header is old
2022-03-24 02:15:14.217293 W | etcdserver: failed to apply request "header:<ID:1070308296661102801 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/kms/core/leader/358aa84d-03ff-32fa-35b1-e597af763118\" > " with response "" took (4.155µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217299 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/core/lock/eda7fb99d081ad9, err is auth: revision in header is old
2022-03-24 02:15:14.217312 W | etcdserver: failed to apply request "header:<ID:1070308296661102802 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/shelter/core/lock/eda7fb99d081ad9\" > " with response "" took (4.091µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217317 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/core/lock/eda7fb99d081adb, err is auth: revision in header is old
2022-03-24 02:15:14.217330 W | etcdserver: failed to apply request "header:<ID:1070308296661102803 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/kms/core/lock/eda7fb99d081adb\" > " with response "" took (4.027µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217339 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/core/lock/d997fb99eb3ff8a, err is auth: revision in header is old
2022-03-24 02:15:14.217356 W | etcdserver: failed to apply request "header:<ID:10203326865992843159 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/core/lock/d997fb99eb3ff8a\" value_size:36 lease:979954829138067338 >" with response "" took (5.035µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217362 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/core/lock/d997fb99eb3ff91, err is auth: revision in header is old
2022-03-24 02:15:14.217375 W | etcdserver: failed to apply request "header:<ID:10203326865992843161 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/kms/core/lock/d997fb99eb3ff91\" value_size:36 lease:979954829138067345 >" with response "" took (4.078µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217385 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/core/leader/b5f8487c-ce11-a394-7122-4ddbccc3399c, err is auth: revision in header is old
2022-03-24 02:15:14.217398 W | etcdserver: failed to apply request "header:<ID:10203326865992843178 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/core/leader/b5f8487c-ce11-a394-7122-4ddbccc3399c\" value_size:1608 >" with response "" took (4.385µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217403 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/core/leader/da7d6abf-5261-d22f-4213-b11115faa115, err is auth: revision in header is old
2022-03-24 02:15:14.217417 W | etcdserver: failed to apply request "header:<ID:10203326865992843179 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/kms/core/leader/da7d6abf-5261-d22f-4213-b11115faa115\" value_size:1609 >" with response "" took (4.821µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217423 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/core/cluster/feature-flags, err is auth: revision in header is old
2022-03-24 02:15:14.217439 W | etcdserver: failed to apply request "header:<ID:10203326865992843181 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/core/cluster/feature-flags\" value_size:69 >" with response "" took (4.149µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217447 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/core/cluster/feature-flags, err is auth: revision in header is old
2022-03-24 02:15:14.217459 W | etcdserver: failed to apply request "header:<ID:10203326865992843183 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/kms/core/cluster/feature-flags\" value_size:69 >" with response "" took (4.032µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217466 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/logical/94a702ce-a10f-2931-94fd-e62d565eee60/casesensitivity, err is auth: revision in header is old
2022-03-24 02:15:14.217482 W | etcdserver: failed to apply request "header:<ID:10203326865992843226 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/logical/94a702ce-a10f-2931-94fd-e62d565eee60/casesensitivity\" value_size:69 >" with response "" took (4.1µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217488 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/logical/e4977d9c-9daf-6397-fef1-8c985643721f/casesensitivity, err is auth: revision in header is old
2022-03-24 02:15:14.217500 W | etcdserver: failed to apply request "header:<ID:10203326865992843244 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/kms/logical/e4977d9c-9daf-6397-fef1-8c985643721f/casesensitivity\" value_size:69 >" with response "" took (4.273µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217509 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/logical/94a702ce-a10f-2931-94fd-e62d565eee60/packer/buckets/4, err is auth: revision in header is old
2022-03-24 02:15:14.217522 W | etcdserver: failed to apply request "header:<ID:10203326865992843251 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/logical/94a702ce-a10f-2931-94fd-e62d565eee60/packer/buckets/4\" value_size:480 >" with response "" took (4.214µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217530 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/token/accessor/0ad5472062556c01aece0f5a735b0e243c83504e, err is auth: revision in header is old
2022-03-24 02:15:14.217543 W | etcdserver: failed to apply request "header:<ID:10203326865992843257 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/sys/token/accessor/0ad5472062556c01aece0f5a735b0e243c83504e\" value_size:138 >" with response "" took (4.351µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217550 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/token/id/hb6c0289ac4c999cf88e1c4d9cea317e5892d17fec06a5dbe599aefe7dc280d6e, err is auth: revision in header is old
2022-03-24 02:15:14.217568 W | etcdserver: failed to apply request "header:<ID:10203326865992843260 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/sys/token/id/hb6c0289ac4c999cf88e1c4d9cea317e5892d17fec06a5dbe599aefe7dc280d6e\" value_size:828 >" with response "" took (6.893µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217574 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/expire/id/auth/kubernetes/login/hb6c0289ac4c999cf88e1c4d9cea317e5892d17fec06a5dbe599aefe7dc280d6e, err is auth: revision in header is old
2022-03-24 02:15:14.217588 W | etcdserver: failed to apply request "header:<ID:10203326865992843261 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/sys/expire/id/auth/kubernetes/login/hb6c0289ac4c999cf88e1c4d9cea317e5892d17fec06a5dbe599aefe7dc280d6e\" value_size:1644 >" with response "" took (4.058µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217598 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/token/id/hd8f8461e4534812c6d951a00aca52f0865944f8d8f8feed028d5640ae34a92d6, err is auth: revision in header is old
2022-03-24 02:15:14.217624 W | etcdserver: failed to apply request "header:<ID:10203326865992843280 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/sys/token/id/hd8f8461e4534812c6d951a00aca52f0865944f8d8f8feed028d5640ae34a92d6\" value_size:871 >" with response "" took (21.234µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217630 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/expire/id/auth/kubernetes/login/hd8f8461e4534812c6d951a00aca52f0865944f8d8f8feed028d5640ae34a92d6, err is auth: revision in header is old
2022-03-24 02:15:14.217645 W | etcdserver: failed to apply request "header:<ID:10203326865992843284 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/shelter/sys/expire/id/auth/kubernetes/login/hd8f8461e4534812c6d951a00aca52f0865944f8d8f8feed028d5640ae34a92d6\" > " with response "" took (4.364µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217651 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/token/accessor/a64aa78d1e78507b67e15a55fe466d7de086ed9f, err is auth: revision in header is old
2022-03-24 02:15:14.217665 W | etcdserver: failed to apply request "header:<ID:10203326865992843285 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/shelter/sys/token/accessor/a64aa78d1e78507b67e15a55fe466d7de086ed9f\" > " with response "" took (4.522µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217670 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/token/id/hd8f8461e4534812c6d951a00aca52f0865944f8d8f8feed028d5640ae34a92d6, err is auth: revision in header is old
2022-03-24 02:15:14.217684 W | etcdserver: failed to apply request "header:<ID:10203326865992843286 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/shelter/sys/token/id/hd8f8461e4534812c6d951a00aca52f0865944f8d8f8feed028d5640ae34a92d6\" > " with response "" took (4.133µs) to execute, err is auth: revision in header is old