rook: CephObjectStore - failed to commit RGW configuration period changes%!(EXTRA []string=[])

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior: When deploying a CephCluster and a CephObjectStore on a cluster in VSphere using RHEL 8.6 (other RHEL versions seem to have the same issue), we’ve observed the CephObjectStore failing due to this error:

 Warning  ReconcileFailed     7m55s (x19 over 52m)  rook-ceph-object-controller  failed to reconcile CephObjectStore "kommander/dkp-object-store". failed to create object store deployments: failed to configure multisite for object store: failed create ceph multisite for object-store ["dkp-object-store"]: failed to commit config changes after creating multisite config for CephObjectStore "kommander/dkp-object-store": failed to commit RGW configuration period changes%!(EXTRA []string=[]): signal: interrupt

This issue happens intermittently and results in the RGW deployment/service never being created. While debugging a successful cluster vs a failed cluster we found this:

  • On healthy cluster, the zonegroup was set as the default:
bash-4.4$ radosgw-admin zonegroup list
{
   "default_info": "4982b07b-23eb-4a89-9ba9-7c186cb7c520",
   "zonegroups": [
       "name-of-object-store"
   ]
}
  • On a failed cluster, the zonegroup was not set as the default:
bash-4.4$ radosgw-admin zonegroup list
{
   "default_info": "",
   "zonegroups": [
       "name-of-object-store"
   ]
}

With that info, we ran these commands to set the zonegroup as the default and that ultimately made the CephObjectStore healthy shortly after (we restarted the operator to be sure):

radosgw-admin zonegroup default --rgw-zonegroup=<object store name>
radosgw-admin period update --commit

Expected behavior:

After creating a CephCluster that reports as healthy, creating a CephObjectStore should be successful.

How to reproduce it (minimal and precise):

  1. Create a CAPV cluster running on VSphere using RHEL 8.6 as the operating system
  2. Install the rook-ceph and rook-ceph-cluster Helm charts to create a cluster and an object store
  3. Observe any errors on the CephObjectStore object and in the operator logs (unfortunately this may take a few attempts to reproduce)

File(s) to submit: Let me know if this is needed for debugging!

Logs to submit:

  • Operator’s logs (small subset of logs, I can grab more if needed):
2023-03-30 18:42:08.351829 D | ceph-spec: CephCluster "kommander" status: "Ready". "Cluster created successfully"
2023-03-30 18:42:08.893192 D | ceph-spec: found 1 ceph clusters in namespace "kommander"
2023-03-30 18:42:08.893215 D | ceph-cluster-controller: update event on CephCluster CR
E0330 18:42:11.207612       1 controller.go:204] error syncing 'kommander/dkp-loki': error provisioning bucket: failed to set admin ops api client: failed to retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "dkp-object-store": failed to create s3 user. 2023-03-30T18:42:11.200+0000 7fcf954df500  0 ERROR: could not find zone (dkp-object-store)
2023-03-30T18:42:11.200+0000 7fcf954df500  0 ERROR: failed to start notify service ((2) No such file or directory
2023-03-30T18:42:11.200+0000 7fcf954df500  0 ERROR: failed to init services (ret=(2) No such file or directory)
couldn't init storage provider. : exit status 5, requeuing
I0330 18:42:11.207654       1 controller.go:217]  "msg"="reconciling claim" "key"="kommander/dkp-velero"
I0330 18:42:11.207663       1 helpers.go:107]  "msg"="getting claim for key" "key"="kommander/dkp-velero"
2023-03-30 18:42:11.389720 D | exec: Running command: radosgw-admin zone create --master --endpoints=http://rook-ceph-rgw-dkp-object-store.kommander.svc:80 --rgw-realm=dkp-object-store --rgw-zonegroup=dkp-object-store --rgw-zone=dkp-object-store --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring
I0330 18:42:11.627780       1 helpers.go:213]  "msg"="getting ObjectBucketClaim's StorageClass" "key"="kommander/dkp-velero"
I0330 18:42:11.994319       1 helpers.go:218]  "msg"="got StorageClass" "key"="kommander/dkp-velero" "name"="dkp-object-store"
I0330 18:42:11.994363       1 controller.go:270]  "msg"="syncing obc creation" "key"="kommander/dkp-velero"
I0330 18:42:11.994402       1 controller.go:552]  "msg"="updating OBC metadata" "key"="kommander/dkp-velero"
I0330 18:42:11.994438       1 resourcehandlers.go:277]  "msg"="updating" "key"="kommander/dkp-velero" "obc"="kommander/dkp-velero"
I0330 18:42:12.732377       1 controller.go:341]  "msg"="provisioning" "bucket"="dkp-velero" "key"="kommander/dkp-velero"
2023-03-30 18:42:12.732409 D | op-bucket-prov: Provision event for OB options: &{ReclaimPolicy:0xc0015542d0 BucketName:dkp-velero UserID:obc-kommander-dkp-velero ObjectBucketClaim:0xc001abc6e0 Parameters:map[objectStoreName:dkp-object-store objectStoreNamespace:kommander]}
2023-03-30 18:42:12.732417 I | op-bucket-prov: initializing and setting CreateOrGrant services
2023-03-30 18:42:12.732422 I | op-bucket-prov: getting storage class "dkp-object-store"
2023-03-30 18:42:14.280496 D | ceph-bucket-notification: reconciling ObjectBucketClaim kommander/dkp-loki labels for bucket notifications
2023-03-30 18:42:14.280596 I | ceph-bucket-notification: ObjectBucketClaim "kommander/dkp-loki" resource did not create the bucket yet. will retry
2023-03-30 18:42:14.280659 D | ceph-bucket-notification: reconciling ObjectBucketClaim kommander/dkp-velero labels for bucket notifications
2023-03-30 18:42:14.280693 I | ceph-bucket-notification: ObjectBucketClaim "kommander/dkp-velero" resource did not create the bucket yet. will retry
2023-03-30 18:42:15.306528 D | ceph-object-controller: creating s3 user object "rgw-admin-ops-user" for object store "dkp-object-store"
2023-03-30 18:42:15.306547 D | ceph-object-controller: creating s3 user "rgw-admin-ops-user"
2023-03-30 18:42:15.306562 D | exec: Running command: radosgw-admin user create --uid rgw-admin-ops-user --display-name RGW Admin Ops User --caps buckets=*;users=*;usage=read;metadata=read;zone=read --rgw-realm=dkp-object-store --rgw-zonegroup=dkp-object-store --rgw-zone=dkp-object-store --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring
2023-03-30 18:42:21.184509 D | ceph-spec: update event from a CR: "dkp-loki"
2023-03-30 18:42:21.184537 D | ceph-spec: update event on ObjectBucketClaim CR
2023-03-30 18:42:21.184548 D | ceph-spec: no change in CR "dkp-loki"
2023-03-30 18:42:21.186297 D | ceph-spec: update event from a CR: "dkp-velero"
2023-03-30 18:42:21.186310 D | ceph-spec: update event on ObjectBucketClaim CR
2023-03-30 18:42:21.186317 D | ceph-spec: no change in CR "dkp-velero"
2023-03-30 18:42:24.280828 D | ceph-bucket-notification: reconciling ObjectBucketClaim kommander/dkp-loki labels for bucket notifications
2023-03-30 18:42:24.280898 D | ceph-bucket-notification: ObjectBucketClaim "kommander/dkp-loki" was deleted
2023-03-30 18:42:24.280915 D | ceph-bucket-notification: reconciling ObjectBucketClaim kommander/dkp-velero labels for bucket notifications
2023-03-30 18:42:24.280925 D | ceph-bucket-notification: ObjectBucketClaim "kommander/dkp-velero" was deleted
2023-03-30 18:42:24.370915 D | ceph-object-controller: retrying 'radosgw-admin' command with OMAP backend to work around FIFO file I/O issue. 2023-03-30T18:42:24.364+0000 7f9fdcba1500  0 ERROR: could not find zone (dkp-object-store)
2023-03-30T18:42:24.364+0000 7f9fdcba1500  0 ERROR: failed to start notify service ((2) No such file or directory
2023-03-30T18:42:24.364+0000 7f9fdcba1500  0 ERROR: failed to init services (ret=(2) No such file or directory)
couldn't init storage provider.
2023-03-30 18:42:24.370945 D | exec: Running command: radosgw-admin user create --uid rgw-admin-ops-user --display-name RGW Admin Ops User --caps buckets=*;users=*;usage=read;metadata=read;zone=read --rgw-realm=dkp-object-store --rgw-zonegroup=dkp-object-store --rgw-zone=dkp-object-store --rgw-data-log-backing=omap --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring
2023-03-30 18:42:24.555622 D | ceph-object-controller: created zone "dkp-object-store"
2023-03-30 18:42:24.555680 D | exec: Running command: radosgw-admin period get --rgw-realm=dkp-object-store --rgw-zonegroup=dkp-object-store --rgw-zone=dkp-object-store --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring
2023-03-30 18:42:25.652362 D | exec: Running command: radosgw-admin user create --uid rgw-admin-ops-user --display-name RGW Admin Ops User --caps buckets=*;users=*;usage=read;metadata=read;zone=read --rgw-realm=dkp-object-store --rgw-zonegroup=dkp-object-store --rgw-zone=dkp-object-store --rgw-default-data-log-backing=omap --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring
2023-03-30 18:42:27.548473 D | op-mon: checking health of mons
2023-03-30 18:42:27.548498 D | op-mon: Acquiring lock for mon orchestration
2023-03-30 18:42:27.548501 D | op-mon: Acquired lock for mon orchestration
2023-03-30 18:42:27.985006 D | op-mon: Checking health for mons in cluster "kommander"
2023-03-30 18:42:27.985035 D | exec: Running command: ceph quorum_status --connect-timeout=15 --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring --format json
2023-03-30 18:42:32.004993 D | op-mon: Mon quorum status: {Quorum:[0 1 2] MonMap:{Mons:[{Name:a Rank:0 Address:10.106.246.205:6789/0 PublicAddr:10.106.246.205:6789/0 PublicAddrs:{Addrvec:[{Type:v2 Addr:10.106.246.205:3300 Nonce:0} {Type:v1 Addr:10.106.246.205:6789 Nonce:0}]}} {Name:b Rank:1 Address:10.102.156.80:6789/0 PublicAddr:10.102.156.80:6789/0 PublicAddrs:{Addrvec:[{Type:v2 Addr:10.102.156.80:3300 Nonce:0} {Type:v1 Addr:10.102.156.80:6789 Nonce:0}]}} {Name:c Rank:2 Address:10.103.79.235:6789/0 PublicAddr:10.103.79.235:6789/0 PublicAddrs:{Addrvec:[{Type:v2 Addr:10.103.79.235:3300 Nonce:0} {Type:v1 Addr:10.103.79.235:6789 Nonce:0}]}}]}}
2023-03-30 18:42:32.005027 D | op-mon: targeting the mon count 3
2023-03-30 18:42:32.005038 D | op-mon: mon "a" found in quorum
2023-03-30 18:42:32.005042 D | op-mon: mon "b" found in quorum
2023-03-30 18:42:32.005045 D | op-mon: mon "c" found in quorum
2023-03-30 18:42:32.005048 D | op-mon: mon cluster is healthy, removing any existing canary deployment
2023-03-30 18:42:32.400559 D | op-mon: Released lock for mon orchestration
2023-03-30 18:42:32.400579 D | op-mon: ceph mon status in namespace "kommander" check interval "45s"
2023-03-30 18:42:37.623618 D | exec: Running command: radosgw-admin period update --rgw-realm=dkp-object-store --rgw-zonegroup=dkp-object-store --rgw-zone=dkp-object-store --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring
2023-03-30 18:42:40.653679 I | exec: exec timeout waiting for process radosgw-admin to return. Sending interrupt signal to the process
E0330 18:42:40.656977       1 controller.go:204] error syncing 'kommander/dkp-velero': error provisioning bucket: failed to set admin ops api client: failed to retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "dkp-object-store": failed to create s3 user. 2023-03-30T18:42:33.350+0000 7fa77aa61500  0 period (0de8e13b-1cd3-4743-8601-6c9b538847be does not have zone 15b21aae-3dfd-4d7a-9a22-2c332c60d901 configured. : signal: interrupt, requeuing
2023-03-30 18:42:52.624144 I | exec: exec timeout waiting for process radosgw-admin to return. Sending interrupt signal to the process
2023-03-30 18:42:52.987189 D | ceph-object-controller: object store "kommander/dkp-object-store" status updated to "Failure"
2023-03-30 18:42:52.987609 D | clusterdisruption-controller: reconciling "kommander/"
2023-03-30 18:42:52.987774 D | ceph-spec: update event from a CR: "dkp-object-store"
2023-03-30 18:42:52.987787 D | ceph-spec: update event on CephObjectStore CR
2023-03-30 18:42:52.987861 D | clusterdisruption-controller: could not match failure domain. defaulting to "host"
2023-03-30 18:42:53.162451 D | op-k8sutil: kubernetes version fetched 1.25.4
2023-03-30 18:42:53.337174 D | op-k8sutil: kubernetes version fetched 1.25.4
2023-03-30 18:42:53.337459 D | exec: Running command: ceph status --connect-timeout=15 --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring --format json
2023-03-30 18:42:53.968349 D | ceph-object-controller: object store "kommander/dkp-object-store" status updated to "Failure"
2023-03-30 18:42:53.968399 E | ceph-object-controller: failed to reconcile CephObjectStore "kommander/dkp-object-store". failed to create object store deployments: failed to configure multisite for object store: failed create ceph multisite for object-store ["dkp-object-store"]: failed to commit config changes after creating multisite config for CephObjectStore "kommander/dkp-object-store": failed to stage the current RGW configuration period%!!(MISSING)(EXTRA []string=[]): signal: interrupt

Cluster Status to submit:

bash-4.4$ ceph status
 cluster:
   id:     690da43d-4fb0-4ea7-9630-700fdc75ba77
   health: HEALTH_OK
 services:
   mon: 3 daemons, quorum a,b,c (age 67m)
   mgr: a(active, since 65m), standbys: b
   osd: 4 osds: 4 up (since 64m), 4 in (since 65m)
 data:
   pools:   9 pools, 89 pgs
   objects: 23 objects, 580 KiB
   usage:   173 MiB used, 160 GiB / 160 GiB avail
   pgs:     89 active+clean
 io:
   client:   3.2 KiB/s rd, 170 B/s wr, 3 op/s rd, 0 op/s wr

Environment:

  • OS (e.g. from /etc/os-release): RHEL 8.6 / 8.4 / 7.9
  • Kernel (e.g. uname -a):
  • Cloud provider or hardware configuration: VSphere
  • Rook version (use rook version inside of a Rook Pod): v1.10.11
  • Storage backend version (e.g. for ceph do ceph -v): ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
  • Kubernetes version (use kubectl version): v1.25.4
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Cluster API / CAPV
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 3
  • Comments: 19 (7 by maintainers)

Most upvoted comments

In the manifest you posted here, the zone is commented out, which means that in the operator code s.Zone.Name == "" already.

You may need to use radosgw-admin to delete the realm manually, since there seems to be some corruption.

Encountering the same issue with Rook v1.10.0: whenever I delete and redeploy an RGW with the same name, the reconciliation of the CephObjectStore consistently fails. I’ve ensured that zones, zone groups, and pools are all removed before redeploy. Confirmation that that PGs are not reaching the limit, Ceph status is healthy, and all OSDs are up and running. The setup involves EC2 and a single RGW on a 3-node cluster with 5 OSDs (replica 3 RGW faces the same problem).

It appears that when I delete the RGW, the realm doesn’t get removed, even though I haven’t set the multisite config in the manifest.

Any insights or guidance would be greatly appreciated.

2023-12-08 07:49:51.321215 I | ceph-object-controller: ceph object store gateway service running at 10.43.92.71
2023-12-08 07:49:51.321239 I | ceph-object-controller: reconciling object store pools
2023-12-08 07:49:54.026041 I | cephclient: setting pool property "compression_mode" to "none" on pool "store-test-2.rgw.control"
2023-12-08 07:49:57.194233 I | cephclient: application "rook-ceph-rgw" is already set on pool "store-test-2.rgw.control"
2023-12-08 07:49:57.194252 I | cephclient: reconciling replicated pool store-test-2.rgw.control succeeded
2023-12-08 07:49:59.577966 I | cephclient: creating a new crush rule for changed deviceClass on crush rule "store-test-2.rgw.control_host"
2023-12-08 07:49:59.577984 I | cephclient: updating pool "store-test-2.rgw.control" failure domain from "host" to "host" with new crush rule "store-test-2.rgw.control_host"
2023-12-08 07:49:59.577994 I | cephclient: crush rule "store-test-2.rgw.control_host" will no longer be used by pool "store-test-2.rgw.control"
2023-12-08 07:50:02.763830 I | cephclient: Successfully updated pool "store-test-2.rgw.control" failure domain to "host"
2023-12-08 07:50:02.763855 I | cephclient: setting pool property "pg_num_min" to "8" on pool "store-test-2.rgw.control"
2023-12-08 07:50:09.980791 I | cephclient: setting pool property "compression_mode" to "none" on pool "store-test-2.rgw.meta"
2023-12-08 07:50:15.202815 I | cephclient: application "rook-ceph-rgw" is already set on pool "store-test-2.rgw.meta"
2023-12-08 07:50:15.202905 I | cephclient: reconciling replicated pool store-test-2.rgw.meta succeeded
2023-12-08 07:50:17.691915 I | cephclient: creating a new crush rule for changed deviceClass on crush rule "store-test-2.rgw.meta_host"
2023-12-08 07:50:17.691943 I | cephclient: updating pool "store-test-2.rgw.meta" failure domain from "host" to "host" with new crush rule "store-test-2.rgw.meta_host"
2023-12-08 07:50:17.691949 I | cephclient: crush rule "store-test-2.rgw.meta_host" will no longer be used by pool "store-test-2.rgw.meta"
2023-12-08 07:50:22.819562 I | cephclient: Successfully updated pool "store-test-2.rgw.meta" failure domain to "host"
2023-12-08 07:50:22.819593 I | cephclient: setting pool property "pg_num_min" to "8" on pool "store-test-2.rgw.meta"
2023-12-08 07:50:27.298854 I | cephclient: setting pool property "compression_mode" to "none" on pool "store-test-2.rgw.log"
2023-12-08 07:50:31.817121 I | cephclient: application "rook-ceph-rgw" is already set on pool "store-test-2.rgw.log"
2023-12-08 07:50:31.817187 I | cephclient: reconciling replicated pool store-test-2.rgw.log succeeded
2023-12-08 07:50:34.794242 I | cephclient: creating a new crush rule for changed deviceClass on crush rule "store-test-2.rgw.log_host"
2023-12-08 07:50:34.794266 I | cephclient: updating pool "store-test-2.rgw.log" failure domain from "host" to "host" with new crush rule "store-test-2.rgw.log_host"
2023-12-08 07:50:34.794272 I | cephclient: crush rule "store-test-2.rgw.log_host" will no longer be used by pool "store-test-2.rgw.log"
2023-12-08 07:50:37.722045 I | cephclient: Successfully updated pool "store-test-2.rgw.log" failure domain to "host"
2023-12-08 07:50:37.722075 I | cephclient: setting pool property "pg_num_min" to "8" on pool "store-test-2.rgw.log"
2023-12-08 07:50:42.486525 I | cephclient: setting pool property "compression_mode" to "none" on pool "store-test-2.rgw.buckets.index"
2023-12-08 07:50:45.422757 I | cephclient: application "rook-ceph-rgw" is already set on pool "store-test-2.rgw.buckets.index"
2023-12-08 07:50:45.422818 I | cephclient: reconciling replicated pool store-test-2.rgw.buckets.index succeeded
2023-12-08 07:50:48.275870 I | cephclient: creating a new crush rule for changed deviceClass on crush rule "store-test-2.rgw.buckets.index_host"
2023-12-08 07:50:48.275890 I | cephclient: updating pool "store-test-2.rgw.buckets.index" failure domain from "host" to "host" with new crush rule "store-test-2.rgw.buckets.index_host"
2023-12-08 07:50:48.275895 I | cephclient: crush rule "store-test-2.rgw.buckets.index_host" will no longer be used by pool "store-test-2.rgw.buckets.index"
2023-12-08 07:50:50.929612 I | cephclient: Successfully updated pool "store-test-2.rgw.buckets.index" failure domain to "host"
2023-12-08 07:50:50.929633 I | cephclient: setting pool property "pg_num_min" to "8" on pool "store-test-2.rgw.buckets.index"
2023-12-08 07:50:55.378817 I | cephclient: setting pool property "compression_mode" to "none" on pool "store-test-2.rgw.buckets.non-ec"
2023-12-08 07:50:58.321689 I | cephclient: application "rook-ceph-rgw" is already set on pool "store-test-2.rgw.buckets.non-ec"
2023-12-08 07:50:58.321710 I | cephclient: reconciling replicated pool store-test-2.rgw.buckets.non-ec succeeded
2023-12-08 07:51:00.711659 I | cephclient: creating a new crush rule for changed deviceClass on crush rule "store-test-2.rgw.buckets.non-ec_host"
2023-12-08 07:51:00.711677 I | cephclient: updating pool "store-test-2.rgw.buckets.non-ec" failure domain from "host" to "host" with new crush rule "store-test-2.rgw.buckets.non-ec_host"
2023-12-08 07:51:00.711681 I | cephclient: crush rule "store-test-2.rgw.buckets.non-ec_host" will no longer be used by pool "store-test-2.rgw.buckets.non-ec"
2023-12-08 07:51:03.851324 I | cephclient: Successfully updated pool "store-test-2.rgw.buckets.non-ec" failure domain to "host"
2023-12-08 07:51:03.851347 I | cephclient: setting pool property "pg_num_min" to "8" on pool "store-test-2.rgw.buckets.non-ec"
2023-12-08 07:51:08.421667 I | cephclient: setting pool property "compression_mode" to "none" on pool "store-test-2.rgw.otp"
2023-12-08 07:51:11.722882 I | cephclient: application "rook-ceph-rgw" is already set on pool "store-test-2.rgw.otp"
2023-12-08 07:51:11.722911 I | cephclient: reconciling replicated pool store-test-2.rgw.otp succeeded
2023-12-08 07:51:15.097644 I | cephclient: creating a new crush rule for changed deviceClass on crush rule "store-test-2.rgw.otp_host"
2023-12-08 07:51:15.097663 I | cephclient: updating pool "store-test-2.rgw.otp" failure domain from "host" to "host" with new crush rule "store-test-2.rgw.otp_host"
2023-12-08 07:51:15.097667 I | cephclient: crush rule "store-test-2.rgw.otp_host" will no longer be used by pool "store-test-2.rgw.otp"
2023-12-08 07:51:27.259172 I | cephclient: Successfully updated pool "store-test-2.rgw.otp" failure domain to "host"
2023-12-08 07:51:27.259190 I | cephclient: setting pool property "pg_num_min" to "8" on pool "store-test-2.rgw.otp"
2023-12-08 07:51:31.778140 I | cephclient: setting pool property "compression_mode" to "none" on pool ".rgw.root"
2023-12-08 07:51:35.086706 I | cephclient: application "rook-ceph-rgw" is already set on pool ".rgw.root"
2023-12-08 07:51:35.086723 I | cephclient: reconciling replicated pool .rgw.root succeeded
2023-12-08 07:51:37.705122 I | cephclient: creating a new crush rule for changed deviceClass on crush rule ".rgw.root_host"
2023-12-08 07:51:37.705139 I | cephclient: updating pool ".rgw.root" failure domain from "host" to "host" with new crush rule ".rgw.root_host"
2023-12-08 07:51:37.705143 I | cephclient: crush rule ".rgw.root_host" will no longer be used by pool ".rgw.root"
2023-12-08 07:51:41.098131 I | cephclient: Successfully updated pool ".rgw.root" failure domain to "host"
2023-12-08 07:51:41.098154 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2023-12-08 07:51:46.916964 I | cephclient: setting pool property "allow_ec_overwrites" to "true" on pool "store-test-2.rgw.buckets.data"
2023-12-08 07:51:48.463601 I | cephclient: setting pool property "compression_mode" to "none" on pool "store-test-2.rgw.buckets.data"
2023-12-08 07:51:51.817654 I | cephclient: application "rook-ceph-rgw" is already set on pool "store-test-2.rgw.buckets.data"
2023-12-08 07:51:51.817669 I | cephclient: creating EC pool store-test-2.rgw.buckets.data succeeded
2023-12-08 07:51:51.817674 I | ceph-object-controller: setting multisite settings for object store "store-test-2"
2023-12-08 07:51:52.670470 I | ceph-object-controller: committing changes to RGW configuration period for CephObjectStore "rook-ceph/store-test-2"
2023-12-08 07:51:53.099504 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/store-test-2". failed to create object store deployments: failed to configure multisite for object store: failed create ceph multisite for object-store ["store-test-2"]: failed to commit config changes after creating multisite config for CephObjectStore "rook-ceph/store-test-2": failed to commit RGW configuration period changes%!!(MISSING)(EXTRA []string=[]): exit status 22

Here’s my manifest:

# Source: rgw/templates/deployment.yaml
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: store-test-1
  namespace: rook-ceph # namespace:cluster
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 2
      requireSafeReplicaSize: true
    parameters:
      compression_mode: none
  dataPool:
    failureDomain: host
    replicated:
      size: 3
      requireSafeReplicaSize: true
    parameters:
      compression_mode: none
  preservePoolsOnDelete: false
  gateway:
    # A reference to the secret in the rook namespace where the ca bundle is stored
    # caBundleRef:
    # The port that RGW pods will listen on (http)
    port: 80
    # The number of pods in the rgw deployment
    instances: 3
    # The affinity rules to apply to the rgw deployment.
    placement:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: ceph_daemon_id
              operator: In
              values:
              - store-test-1
          topologyKey: kubernetes.io/hostname
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - rook-ceph-rgw
              topologyKey: kubernetes.io/hostname
    # A key/value list of annotations
    #  topologySpreadConstraints:
    #  tolerations:
    #  - key: rgw-node
    #    operator: Exists
    #  podAffinity:
    #  podAntiAffinity:
    # A key/value list of annotations
    annotations:
    #  key: value
    # A key/value list of labels
    labels:
    #  key: value
    resources:
    # The requests and limits set here, allow the object store gateway Pod(s) to use half of one CPU core and 1 gigabyte of memory
    #  limits:
    #    cpu: "500m"
    #    memory: "1024Mi"
    #  requests:
    #    cpu: "500m"
    #    memory: "1024Mi"
    priorityClassName: system-cluster-critical
  #zone:
    #name: zone-a
  # service endpoint healthcheck
  healthCheck:
    bucket:
      disabled: false
      interval: 60s
    # Configure the pod probes for the rgw daemon
    startupProbe:
      disabled: false
    livenessProbe:
      disabled: false
    readinessProbe:
      disabled: false