rancher: helm upgrade/delete intermittently fails with errors like "unable to create impersonator account: error setting up impersonation for user u-xxxxx..Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io \"cattle-impersonation-u-8j72f\": the object has been modified; please apply your changes to the latest version and try again."

Thanks for all the hints, @ctrox and @andrew-landsverk-win, I was able to reproduce it and I will have a patch incoming soon.

@anupama2501 here’s what I did to reproduce:

1. Deploy Rancher 2.6.0 via helm (no need to upgrade from 2.5.9 to 2.6.0)
2. Import a downstream cluster, I used a kind 1.21.1 cluster but it shouldn’t matter what version or distribution
3. Create a local user and add them as a cluster owner
4. As the new user, download a kubeconfig for the downstream cluster (doesn’t matter if you use the rancher CLI or the UI to get the kubeconfig)
5. Using that kubeconfig, run any kubectl command against it in parallel, I used xargs for this:

$ for i in `seq 1 10` ; do echo kubectl get pod -A ; done | xargs -I {} -P 10 bash -c "{}"
Error from server (InternalError): an error on the server ("unable to create impersonator account: error setting up impersonation for user u-wrhhk: Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io \"cattle-impersonation-u-wrhhk\": the object has been modified; please apply your changes to the latest version and try again") has prevented the request from succeeding (get pods)
Error from server (InternalError): an error on the server ("unable to create impersonator account: error setting up impersonation for user u-wrhhk: Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io \"cattle-impersonation-u-wrhhk\": the object has been modified; please apply your changes to the latest version and try again") has prevented the request from succeeding (get pods)
NAMESPACE             NAME                                         READY   STATUS    RESTARTS   AGE
cattle-fleet-system   fleet-agent-b94869475-hqjlw                  1/1     Running   0          41m
cattle-system         cattle-cluster-agent-6cbdc4f846-qfms5        1/1     Running   0          41m
kube-system           coredns-558bd4d5db-mn8fv                     1/1     Running   0          63m
kube-system           coredns-558bd4d5db-vlbkr                     1/1     Running   0          63m
kube-system           etcd-kind-control-plane                      1/1     Running   0          64m
kube-system           kindnet-6mdsf                                1/1     Running   0          63m
kube-system           kube-apiserver-kind-control-plane            1/1     Running   0          64m
kube-system           kube-controller-manager-kind-control-plane   1/1     Running   0          64m
kube-system           kube-proxy-td748                             1/1     Running   0          63m
kube-system           kube-scheduler-kind-control-plane            1/1     Running   0          64m
local-path-storage    local-path-provisioner-547f784dff-bsnqw      1/1     Running   0          63m
NAMESPACE             NAME                                         READY   STATUS    RESTARTS   AGE
cattle-fleet-system   fleet-agent-b94869475-hqjlw                  1/1     Running   0          41m
cattle-system         cattle-cluster-agent-6cbdc4f846-qfms5        1/1     Running   0          41m
kube-system           coredns-558bd4d5db-mn8fv                     1/1     Running   0          63m
<snipped>

After the first initial errors it will go away, you can remove and re-add the user to the cluster to see the errors again.

I haven’t reproduced @ctrox’s exact issue with the reordering of groups but I imagine it should be doable by setting up an auth provider where the user has multiple groups coming from the auth provider, or doing what @ctrox did and switching from one auth provider to another.

1. The app may or may not exist when you run helm that way, if it doesn’t exist, it gets created, if it does, it gets upgraded
2. This is all I get when it failed:

W0907 12:43:53.267131      51 helpers.go:557] --dry-run is deprecated and can be replaced with --dry-run=client.
error: unable to recognize "/tmp/kubectl_replace_486731274/resource.stdin": an error on the server ("unable to create impersonator account: error setting up impersonation for user u-8j72f: Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io \"cattle-impersonation-u-8j72f\": the object has been modified; please apply your changes to the latest version and try again") has prevented the request from succeeding

Verified on the v2.6-head 223ec88

Upgrade scenario Followed the same steps as above reproduction steps to create rancher server on v2.6.0 Upgraded the rancher server to v2.6-head version As the auth user, tried running the command for i in seq 1 10 ; do echo kubectl get pod -A --kubeconfig rkeimport.yaml; done | xargs -I {} -P 10 bash -c "{}" Verified no errors were seen

New user on an upgraded setup Created a new user Added the user as a cluster member and ran the kubectl command Changed the permission from cluster member to cluster owner tried running the command and no errors were seen

On a fresh install of v2.6-head Created a new user as an admin, Added the user to the cluster as cluster member, and then later changed to cluster owner Ran the kubectl command, no errors were seen

Able to reproduce the issue on v2.6.0

1. Bring up an HA rancher server of v2.6.0 version
2. Create an rke1 cluster with 3 nodes all roles in each [rke version: v1.3.0]
3. Once the cluster is up, from the rancher server, use import an existing cluster Generic type option to create a cluster.
4. Run the command curl --insecure -sfL https://<redacted> | kubectl apply -f - against the cluster created in step2
5. imported cluster registers with the rancher server.
6. Create a local user from users & auth - user1
7. Add the new user to the imported cluster as a Cluster owner
8. Log in as the user1 and download the kube config file
9. Run the kubectl command with the kubeconfig file downloaded for i in seq 1 10 ; do echo kubectl get pod -A --kubeconfig rkeimport.yaml; done | xargs -I {} -P 10 bash -c "{}"
10. Error is seen in the output:

for i in `seq 1 10` ; do echo kubectl get pod -A --kubeconfig rkeimport.yaml; done | xargs -I {} -P 10 bash -c "{}"
Error from server (InternalError): an error on the server ("unable to create impersonator account: error setting up impersonation for user u-d57bn: Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io \"cattle-impersonation-u-d57bn\": the object has been modified; please apply your changes to the latest version and try again") has prevented the request from succeeding (get pods)
Error from server (InternalError): an error on the server ("unable to create impersonator account: error setting up impersonation for user u-d57bn: Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io \"cattle-impersonation-u-d57bn\": the object has been modified; please apply your changes to the latest version and try again") has prevented the request from succeeding (get pods)
NAMESPACE             NAME                                       READY   STATUS      RESTARTS   AGE
cattle-fleet-system   fleet-agent-b94869475-zkwzr                1/1     Running     0          5m21s

Not sure if that’s exactly the same issue, but it may be:

I just upgraded a Rancher cluster from 2.5.9 to 2.6.0. It now fails to upgrade the agent in the downstream cluster.

No keycloak is in use, but we use FreeIPA authentication, if that matters.

Just to add some more information, I added a logrus.Info(cmp.Diff(role.Rules, rules)) in this block to see what is constantly changing in that ClusterRole. This is the diff I got over and over, one update call would empty out the ResourceNames while the next one would add them back again.

  {
    Verbs:           {"impersonate"},
    APIGroups:       {"authentication.k8s.io"},
    Resources:       {"userextras/principalid"},
-     ResourceNames:   []string{""},
+     ResourceNames:   []string{"local://u-wf2jf"},
      NonResourceURLs: nil,
    },
    {
      Verbs:           {"impersonate"},
      APIGroups:       {"authentication.k8s.io"},
      Resources:       {"userextras/username"},
-     ResourceNames:   []string{""},
+     ResourceNames:   []string{"username"},
      NonResourceURLs: nil,
    },
  }

This is a local user, so nothing related to Keycloak.

So out of curiosity I scaled Rancher to a single replica (from 3) and now I see way less of these diffs and so far I did not see the “unable to create impersonator account” error again.

Interesting, so no keycloak auth provider. In the meantime I have created a new local user and while I still see the error unable to create impersonator account appear from time to time it is way less frequent than with the keycloak user. I can see that the clusterRole is still frequently updated by rancher so I guess there is some other bug in that update logic that causes this.

@andrew-landsverk-win , this should help: https://rancher.com/docs/rancher/v2.6/en/troubleshooting/logging/