rook: Ceph: ObjectStoreUser failures on update to 1.6

Is this a bug report or feature request?

• Bug Report

Upgrading from Rook v1.5.x (tested v1.5.0 and v1.5.9) to release-1.6 branch yields the below errors updating the CephObjectStoreUser.

...
2021-04-09 22:35:31.538358 D | ceph-spec: "ceph-object-store-user-controller": CephCluster resource "my-cluster" found in namespace "rook-ceph"
2021-04-09 22:35:31.538507 D | ceph-spec: "ceph-object-store-user-controller": ceph status is "HEALTH_WARN", operator is ready to run ceph command, reconciling
2021-04-09 22:35:31.547612 D | op-mon: found existing monitor secrets for cluster rook-ceph
2021-04-09 22:35:31.552045 I | op-mon: parsing mon endpoints: a=10.96.186.103:6789
2021-04-09 22:35:31.552681 D | op-mon: loaded: maxMonID=0, mons=map[a:0xc000e91260], assignment=&{Schedule:map[a:0xc000a1e280]}
2021-04-09 22:35:31.553507 I | ceph-object-store-user-controller: CephObjectStore "my-store" found
2021-04-09 22:35:31.553589 D | ceph-object-store-user-controller: CephObjectStore exists
2021-04-09 22:35:31.554851 D | ceph-object-store-user-controller: CephObjectStore "my-user" is running with 1 pods
2021-04-09 22:35:31.554892 I | ceph-object-store-user-controller: CephObjectStore "my-store" found
2021-04-09 22:35:31.554902 I | ceph-object-store-user-controller: creating ceph object user "my-user" in namespace "rook-ceph"
2021-04-09 22:35:31.554907 D | ceph-object-controller: creating s3 user "my-user"
2021-04-09 22:35:31.554928 D | exec: Running command: radosgw-admin user create --uid my-user --display-name my display name --rgw-realm=my-store --rgw-zonegroup=my-store --rgw-zone=my-store --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring
2021-04-09 22:35:31.796259 D | ceph-object-store-user-controller: object store user "rook-ceph/my-user" status updated to "ReconcileFailed"
2021-04-09 22:35:31.796286 E | ceph-object-store-user-controller: failed to reconcile failed to create/update object store user "my-user": failed to create ceph object user "my-user". error code 1: failed to create s3 user: exit status 5
...

This issues is not fixed if the operator is restarted. It does not seem to be an issue with the cluster not responding while an upgrade is in progress.

This does not seem to be an issue with CRDs. I updated the CephObjectStoreUser CRD from the v1.5 version to the v1.6 version while v1.5.0 was still installed with no issues. I also updated to v1.5.9 with the CRDs updated with no issue.

It’s possible this has something to do with version mismatch in the radosgw-admin utility… Ceph version in the operator: v16.2.0 Ceph version in the cluster: v15.2.5