rancher: private registry for workload cluster - let RancherUI freeze

I have also reproduced this in v2.3.4 and v2.3.5, Rancher appears to be creating a badly formatted kubernetes.io/dockerconfigjson secret which the UI then fails to parse.

After dockerhub download limits being in place adding private registry to an existing cluster is very highly needed feature. We run into this problem as well.

Rancher v2.4.8, Kubernetes 1.18.12, Docker 19.03.13, Debian 10, 14 nodes, 3 masters.

Reproduced in v.2.4.5: Cluster gets stuck in updating state after updating the private registry credentials or switching to a new registry.

1. Create custom cluster with private registry enabled using authentication (registry.acme.com/docker-local)
2. Provision at least 5 nodes to reliably reproduce (possible race condition)
3. After the cluster provisioned, edit the cluster settings and update the registry information (registry.acme.com/docker). note: It could also be a different registry hostname.

Result:

• Cluster stuck forever in updating state.
• On multiple nodes the agent deployment is stuck in upgrading with ImagePullBackoff because it fails to do an authenticated pull from the new registry.
• Inspecting the Docker config (/var/lib/kubelet/config.json) on these nodes shows that the new registry configuration has not been applied (it still contains credentials for the old registry).
• The root cause seems to be that the agents sometimes are redeployed before recreating the kubelet (which would update the credentials).

As requested in the ticket, just updating below with my reproduction steps for the frozen UI in v2.3.4, although I see Max has subsequently reproduced:

1. Provision a single node custom cluster via Rancher v2.3.4
2. Edit the custom cluster to add a private registry URL: registry.example.com, username: username, password: password
3. Observe the System project -> secrets -> registry credentials view freezes
4. Observe following contents of the secret:

kubectl -n cattle-system get secret cattle-private-registry -o jsonpath="{.data.\.dockerconfigjson}" | base64 -d
{"username":"username","password":"password"}

By comparison, create the secret manually, observe the System project registry credentials view does not freeze and contents of the secret is as follows:

kubectl create secret docker-registry -n cattle-system cattle-private-registry --docker-server registry.example.com --docker-username=username --docker-password=password
kubectl -n cattle-system get secret cattle-private-registry -o jsonpath="{.data.\.dockerconfigjson}" | base64 -d
{"auths":{"registry.example.com":{"username":"username","password":"password","auth":"dXNlcm5hbWU6cGFzc3dvcmQ="}}}

Thanks @crux!