rancher: Rancher pod fails with error: `panic: indexer conflict: map[byPod:{}]`

Rancher Server Setup

• Rancher version: v2.6-44b8030a00b29f9f5354c645f3a90ede2eea53e0-head (supposed to be (v2.6.3-rc2))
• Installation option (Docker install/Helm Chart): HelmChart (rancherd)
  • If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE2
• Proxy/Cert Details:

Information about the Cluster

• Kubernetes version: v1.21.6+rke2r1
• rancher/fleet:v0.3.8-rc2
• Cluster Type (Local/Downstream): local
  • If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): A local cluster provisioned with rancherd

Describe the bug

After deploying the rancher, the rancher pod continues failing with panic:

2021/11/23 10:48:54 [INFO] Watching metadata for upgrade.cattle.io/v1, Kind=Plan
2021/11/23 10:48:54 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet
2021/11/23 10:48:54 [INFO] Watching metadata for networking.k8s.io/v1, Kind=IngressClass
2021/11/23 10:48:54 [INFO] Watching metadata for /v1, Kind=ServiceAccount
2021/11/23 10:48:54 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode
2021/11/23 10:48:54 [INFO] Watching metadata for /v1, Kind=Pod
2021/11/23 10:48:54 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding
2021/11/23 10:48:58 [INFO] rkecluster fleet-local/local: provisioning bootstrap node(s) custom-72af5bcab767: waiting for agent to check in and apply initial plan
panic: indexer conflict: map[byPod:{}]

goroutine 508 [running]:
k8s.io/apimachinery/pkg/util/runtime.Must(...)
        /go/pkg/mod/k8s.io/apimachinery@v0.22.3/pkg/util/runtime/runtime.go:171
github.com/rancher/rancher/pkg/generated/controllers/catalog.cattle.io/v1.(*operationCache).AddIndexer(0xc0056a5e60, 0x44ebf21, 0x5, 0x472ce40)
        /go/src/github.com/rancher/rancher/pkg/generated/controllers/catalog.cattle.io/v1/operation.go:247 +0x127
github.com/rancher/rancher/pkg/controllers/dashboard/helm.RegisterOperations(0x4cc6bc8, 0xc0045ec040, 0x4d30238, 0xc0010fda20, 0x4d16bc8, 0xc008765a40, 0x4d145b8, 0xc008765b20)
        /go/src/github.com/rancher/rancher/pkg/controllers/dashboard/helm/operations.go:42 +0x1b8
github.com/rancher/rancher/pkg/controllers/dashboard/helm.Register(0x4cc6bc8, 0xc0045ec040, 0xc0017c2820)
        /go/src/github.com/rancher/rancher/pkg/controllers/dashboard/helm/register.go:22 +0x3c5
github.com/rancher/rancher/pkg/controllers/dashboard.Register(0x4cc6bc8, 0xc0045ec040, 0xc0017c2820, 0xc0056a5b00, 0x3cc4b00, 0x6f60f20)
        /go/src/github.com/rancher/rancher/pkg/controllers/dashboard/controller.go:23 +0x5a
github.com/rancher/rancher/pkg/rancher.(*Rancher).Start.func1.1(0x4cc6bc8, 0xc0045ec040, 0x3cc4b00, 0x6f60f20)
        /go/src/github.com/rancher/rancher/pkg/rancher/rancher.go:245 +0x4e
github.com/rancher/rancher/pkg/wrangler.(*Context).StartWithTransaction(0xc0017c2820, 0x4cc66f8, 0xc0023be440, 0xc0043a5f58, 0x0, 0x0)
        /go/src/github.com/rancher/rancher/pkg/wrangler/context.go:152 +0xdc
github.com/rancher/rancher/pkg/rancher.(*Rancher).Start.func1(0x4cc66f8, 0xc0023be440, 0x4561112, 0x1e)
        /go/src/github.com/rancher/rancher/pkg/rancher/rancher.go:245 +0x107
github.com/rancher/wrangler/pkg/leader.(*Manager).OnLeader.func1(0xc000b08000, 0xc000d54300)
        /go/pkg/mod/github.com/rancher/wrangler@v0.8.9/pkg/leader/manager.go:51 +0xcc
created by github.com/rancher/wrangler/pkg/leader.(*Manager).OnLeader
        /go/pkg/mod/github.com/rancher/wrangler@v0.8.9/pkg/leader/manager.go:48 +0x49

To Reproduce

Use rancherd to deploy a RKE2 cluster:

mkdir -p /etc/rancher/rke2/config.yaml.d/
cat > /etc/rancher/rke2/config.yaml.d/90-harvester-server.yaml <<EOF
disable: rke2-ingress-nginx
EOF

mkdir -p /etc/rancher/rancherd
cat > /etc/rancher/rancherd/config.yaml << EOF
role: cluster-init
token: somethingrandom
kubernetesVersion: stable:rke2
rancherVersion: v2.6-44b8030a00b29f9f5354c645f3a90ede2eea53e0-head
rancherValues:
  features: multi-cluster-management=false
EOF

curl -fL https://raw.githubusercontent.com/rancher/rancherd/master/install.sh | sh -

Wait for the rancher pod to be deployed, the rancher pod will fail with panic.

Or use this vagrant file to start a VM: https://github.com/bk201/try-rancherd/blob/c00956d219a54786929c014d2344176b42037fc1/Vagrantfile

Result

Expected Result

The rancher pod should run without error.

Screenshots

Additional context

There is a similar error in https://github.com/rancher/rancher/issues/34971