rook: CephObjectStore - failed to commit RGW configuration period changes%!(EXTRA []string=[])
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior:
When deploying a CephCluster and a CephObjectStore on a cluster in VSphere using RHEL 8.6 (other RHEL versions seem to have the same issue), we’ve observed the CephObjectStore failing due to this error:
Warning ReconcileFailed 7m55s (x19 over 52m) rook-ceph-object-controller failed to reconcile CephObjectStore "kommander/dkp-object-store". failed to create object store deployments: failed to configure multisite for object store: failed create ceph multisite for object-store ["dkp-object-store"]: failed to commit config changes after creating multisite config for CephObjectStore "kommander/dkp-object-store": failed to commit RGW configuration period changes%!(EXTRA []string=[]): signal: interrupt
This issue happens intermittently and results in the RGW deployment/service never being created. While debugging a successful cluster vs a failed cluster we found this:
- On healthy cluster, the zonegroup was set as the default:
bash-4.4$ radosgw-admin zonegroup list
{
"default_info": "4982b07b-23eb-4a89-9ba9-7c186cb7c520",
"zonegroups": [
"name-of-object-store"
]
}
- On a failed cluster, the zonegroup was not set as the default:
bash-4.4$ radosgw-admin zonegroup list
{
"default_info": "",
"zonegroups": [
"name-of-object-store"
]
}
With that info, we ran these commands to set the zonegroup as the default and that ultimately made the CephObjectStore healthy shortly after (we restarted the operator to be sure):
radosgw-admin zonegroup default --rgw-zonegroup=<object store name>
radosgw-admin period update --commit
Expected behavior:
After creating a CephCluster that reports as healthy, creating a CephObjectStore should be successful.
How to reproduce it (minimal and precise):
- Create a CAPV cluster running on VSphere using RHEL 8.6 as the operating system
- Install the
rook-cephandrook-ceph-clusterHelm charts to create a cluster and an object store - Observe any errors on the
CephObjectStoreobject and in the operator logs (unfortunately this may take a few attempts to reproduce)
File(s) to submit: Let me know if this is needed for debugging!
Logs to submit:
- Operator’s logs (small subset of logs, I can grab more if needed):
2023-03-30 18:42:08.351829 D | ceph-spec: CephCluster "kommander" status: "Ready". "Cluster created successfully"
2023-03-30 18:42:08.893192 D | ceph-spec: found 1 ceph clusters in namespace "kommander"
2023-03-30 18:42:08.893215 D | ceph-cluster-controller: update event on CephCluster CR
E0330 18:42:11.207612 1 controller.go:204] error syncing 'kommander/dkp-loki': error provisioning bucket: failed to set admin ops api client: failed to retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "dkp-object-store": failed to create s3 user. 2023-03-30T18:42:11.200+0000 7fcf954df500 0 ERROR: could not find zone (dkp-object-store)
2023-03-30T18:42:11.200+0000 7fcf954df500 0 ERROR: failed to start notify service ((2) No such file or directory
2023-03-30T18:42:11.200+0000 7fcf954df500 0 ERROR: failed to init services (ret=(2) No such file or directory)
couldn't init storage provider. : exit status 5, requeuing
I0330 18:42:11.207654 1 controller.go:217] "msg"="reconciling claim" "key"="kommander/dkp-velero"
I0330 18:42:11.207663 1 helpers.go:107] "msg"="getting claim for key" "key"="kommander/dkp-velero"
2023-03-30 18:42:11.389720 D | exec: Running command: radosgw-admin zone create --master --endpoints=http://rook-ceph-rgw-dkp-object-store.kommander.svc:80 --rgw-realm=dkp-object-store --rgw-zonegroup=dkp-object-store --rgw-zone=dkp-object-store --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring
I0330 18:42:11.627780 1 helpers.go:213] "msg"="getting ObjectBucketClaim's StorageClass" "key"="kommander/dkp-velero"
I0330 18:42:11.994319 1 helpers.go:218] "msg"="got StorageClass" "key"="kommander/dkp-velero" "name"="dkp-object-store"
I0330 18:42:11.994363 1 controller.go:270] "msg"="syncing obc creation" "key"="kommander/dkp-velero"
I0330 18:42:11.994402 1 controller.go:552] "msg"="updating OBC metadata" "key"="kommander/dkp-velero"
I0330 18:42:11.994438 1 resourcehandlers.go:277] "msg"="updating" "key"="kommander/dkp-velero" "obc"="kommander/dkp-velero"
I0330 18:42:12.732377 1 controller.go:341] "msg"="provisioning" "bucket"="dkp-velero" "key"="kommander/dkp-velero"
2023-03-30 18:42:12.732409 D | op-bucket-prov: Provision event for OB options: &{ReclaimPolicy:0xc0015542d0 BucketName:dkp-velero UserID:obc-kommander-dkp-velero ObjectBucketClaim:0xc001abc6e0 Parameters:map[objectStoreName:dkp-object-store objectStoreNamespace:kommander]}
2023-03-30 18:42:12.732417 I | op-bucket-prov: initializing and setting CreateOrGrant services
2023-03-30 18:42:12.732422 I | op-bucket-prov: getting storage class "dkp-object-store"
2023-03-30 18:42:14.280496 D | ceph-bucket-notification: reconciling ObjectBucketClaim kommander/dkp-loki labels for bucket notifications
2023-03-30 18:42:14.280596 I | ceph-bucket-notification: ObjectBucketClaim "kommander/dkp-loki" resource did not create the bucket yet. will retry
2023-03-30 18:42:14.280659 D | ceph-bucket-notification: reconciling ObjectBucketClaim kommander/dkp-velero labels for bucket notifications
2023-03-30 18:42:14.280693 I | ceph-bucket-notification: ObjectBucketClaim "kommander/dkp-velero" resource did not create the bucket yet. will retry
2023-03-30 18:42:15.306528 D | ceph-object-controller: creating s3 user object "rgw-admin-ops-user" for object store "dkp-object-store"
2023-03-30 18:42:15.306547 D | ceph-object-controller: creating s3 user "rgw-admin-ops-user"
2023-03-30 18:42:15.306562 D | exec: Running command: radosgw-admin user create --uid rgw-admin-ops-user --display-name RGW Admin Ops User --caps buckets=*;users=*;usage=read;metadata=read;zone=read --rgw-realm=dkp-object-store --rgw-zonegroup=dkp-object-store --rgw-zone=dkp-object-store --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring
2023-03-30 18:42:21.184509 D | ceph-spec: update event from a CR: "dkp-loki"
2023-03-30 18:42:21.184537 D | ceph-spec: update event on ObjectBucketClaim CR
2023-03-30 18:42:21.184548 D | ceph-spec: no change in CR "dkp-loki"
2023-03-30 18:42:21.186297 D | ceph-spec: update event from a CR: "dkp-velero"
2023-03-30 18:42:21.186310 D | ceph-spec: update event on ObjectBucketClaim CR
2023-03-30 18:42:21.186317 D | ceph-spec: no change in CR "dkp-velero"
2023-03-30 18:42:24.280828 D | ceph-bucket-notification: reconciling ObjectBucketClaim kommander/dkp-loki labels for bucket notifications
2023-03-30 18:42:24.280898 D | ceph-bucket-notification: ObjectBucketClaim "kommander/dkp-loki" was deleted
2023-03-30 18:42:24.280915 D | ceph-bucket-notification: reconciling ObjectBucketClaim kommander/dkp-velero labels for bucket notifications
2023-03-30 18:42:24.280925 D | ceph-bucket-notification: ObjectBucketClaim "kommander/dkp-velero" was deleted
2023-03-30 18:42:24.370915 D | ceph-object-controller: retrying 'radosgw-admin' command with OMAP backend to work around FIFO file I/O issue. 2023-03-30T18:42:24.364+0000 7f9fdcba1500 0 ERROR: could not find zone (dkp-object-store)
2023-03-30T18:42:24.364+0000 7f9fdcba1500 0 ERROR: failed to start notify service ((2) No such file or directory
2023-03-30T18:42:24.364+0000 7f9fdcba1500 0 ERROR: failed to init services (ret=(2) No such file or directory)
couldn't init storage provider.
2023-03-30 18:42:24.370945 D | exec: Running command: radosgw-admin user create --uid rgw-admin-ops-user --display-name RGW Admin Ops User --caps buckets=*;users=*;usage=read;metadata=read;zone=read --rgw-realm=dkp-object-store --rgw-zonegroup=dkp-object-store --rgw-zone=dkp-object-store --rgw-data-log-backing=omap --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring
2023-03-30 18:42:24.555622 D | ceph-object-controller: created zone "dkp-object-store"
2023-03-30 18:42:24.555680 D | exec: Running command: radosgw-admin period get --rgw-realm=dkp-object-store --rgw-zonegroup=dkp-object-store --rgw-zone=dkp-object-store --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring
2023-03-30 18:42:25.652362 D | exec: Running command: radosgw-admin user create --uid rgw-admin-ops-user --display-name RGW Admin Ops User --caps buckets=*;users=*;usage=read;metadata=read;zone=read --rgw-realm=dkp-object-store --rgw-zonegroup=dkp-object-store --rgw-zone=dkp-object-store --rgw-default-data-log-backing=omap --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring
2023-03-30 18:42:27.548473 D | op-mon: checking health of mons
2023-03-30 18:42:27.548498 D | op-mon: Acquiring lock for mon orchestration
2023-03-30 18:42:27.548501 D | op-mon: Acquired lock for mon orchestration
2023-03-30 18:42:27.985006 D | op-mon: Checking health for mons in cluster "kommander"
2023-03-30 18:42:27.985035 D | exec: Running command: ceph quorum_status --connect-timeout=15 --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring --format json
2023-03-30 18:42:32.004993 D | op-mon: Mon quorum status: {Quorum:[0 1 2] MonMap:{Mons:[{Name:a Rank:0 Address:10.106.246.205:6789/0 PublicAddr:10.106.246.205:6789/0 PublicAddrs:{Addrvec:[{Type:v2 Addr:10.106.246.205:3300 Nonce:0} {Type:v1 Addr:10.106.246.205:6789 Nonce:0}]}} {Name:b Rank:1 Address:10.102.156.80:6789/0 PublicAddr:10.102.156.80:6789/0 PublicAddrs:{Addrvec:[{Type:v2 Addr:10.102.156.80:3300 Nonce:0} {Type:v1 Addr:10.102.156.80:6789 Nonce:0}]}} {Name:c Rank:2 Address:10.103.79.235:6789/0 PublicAddr:10.103.79.235:6789/0 PublicAddrs:{Addrvec:[{Type:v2 Addr:10.103.79.235:3300 Nonce:0} {Type:v1 Addr:10.103.79.235:6789 Nonce:0}]}}]}}
2023-03-30 18:42:32.005027 D | op-mon: targeting the mon count 3
2023-03-30 18:42:32.005038 D | op-mon: mon "a" found in quorum
2023-03-30 18:42:32.005042 D | op-mon: mon "b" found in quorum
2023-03-30 18:42:32.005045 D | op-mon: mon "c" found in quorum
2023-03-30 18:42:32.005048 D | op-mon: mon cluster is healthy, removing any existing canary deployment
2023-03-30 18:42:32.400559 D | op-mon: Released lock for mon orchestration
2023-03-30 18:42:32.400579 D | op-mon: ceph mon status in namespace "kommander" check interval "45s"
2023-03-30 18:42:37.623618 D | exec: Running command: radosgw-admin period update --rgw-realm=dkp-object-store --rgw-zonegroup=dkp-object-store --rgw-zone=dkp-object-store --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring
2023-03-30 18:42:40.653679 I | exec: exec timeout waiting for process radosgw-admin to return. Sending interrupt signal to the process
E0330 18:42:40.656977 1 controller.go:204] error syncing 'kommander/dkp-velero': error provisioning bucket: failed to set admin ops api client: failed to retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "dkp-object-store": failed to create s3 user. 2023-03-30T18:42:33.350+0000 7fa77aa61500 0 period (0de8e13b-1cd3-4743-8601-6c9b538847be does not have zone 15b21aae-3dfd-4d7a-9a22-2c332c60d901 configured. : signal: interrupt, requeuing
2023-03-30 18:42:52.624144 I | exec: exec timeout waiting for process radosgw-admin to return. Sending interrupt signal to the process
2023-03-30 18:42:52.987189 D | ceph-object-controller: object store "kommander/dkp-object-store" status updated to "Failure"
2023-03-30 18:42:52.987609 D | clusterdisruption-controller: reconciling "kommander/"
2023-03-30 18:42:52.987774 D | ceph-spec: update event from a CR: "dkp-object-store"
2023-03-30 18:42:52.987787 D | ceph-spec: update event on CephObjectStore CR
2023-03-30 18:42:52.987861 D | clusterdisruption-controller: could not match failure domain. defaulting to "host"
2023-03-30 18:42:53.162451 D | op-k8sutil: kubernetes version fetched 1.25.4
2023-03-30 18:42:53.337174 D | op-k8sutil: kubernetes version fetched 1.25.4
2023-03-30 18:42:53.337459 D | exec: Running command: ceph status --connect-timeout=15 --cluster=kommander --conf=/var/lib/rook/kommander/kommander.config --name=client.admin --keyring=/var/lib/rook/kommander/client.admin.keyring --format json
2023-03-30 18:42:53.968349 D | ceph-object-controller: object store "kommander/dkp-object-store" status updated to "Failure"
2023-03-30 18:42:53.968399 E | ceph-object-controller: failed to reconcile CephObjectStore "kommander/dkp-object-store". failed to create object store deployments: failed to configure multisite for object store: failed create ceph multisite for object-store ["dkp-object-store"]: failed to commit config changes after creating multisite config for CephObjectStore "kommander/dkp-object-store": failed to stage the current RGW configuration period%!!(MISSING)(EXTRA []string=[]): signal: interrupt
Cluster Status to submit:
bash-4.4$ ceph status
cluster:
id: 690da43d-4fb0-4ea7-9630-700fdc75ba77
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 67m)
mgr: a(active, since 65m), standbys: b
osd: 4 osds: 4 up (since 64m), 4 in (since 65m)
data:
pools: 9 pools, 89 pgs
objects: 23 objects, 580 KiB
usage: 173 MiB used, 160 GiB / 160 GiB avail
pgs: 89 active+clean
io:
client: 3.2 KiB/s rd, 170 B/s wr, 3 op/s rd, 0 op/s wr
Environment:
- OS (e.g. from /etc/os-release):
RHEL 8.6 / 8.4 / 7.9 - Kernel (e.g.
uname -a): - Cloud provider or hardware configuration:
VSphere - Rook version (use
rook versioninside of a Rook Pod):v1.10.11 - Storage backend version (e.g. for ceph do
ceph -v):ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) - Kubernetes version (use
kubectl version):v1.25.4 - Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
Cluster API / CAPV - Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox):
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 3
- Comments: 19 (7 by maintainers)
In the manifest you posted here, the
zoneis commented out, which means that in the operator codes.Zone.Name == ""already.You may need to use
radosgw-adminto delete the realm manually, since there seems to be some corruption.Encountering the same issue with Rook v1.10.0: whenever I delete and redeploy an RGW with the same name, the reconciliation of the CephObjectStore consistently fails. I’ve ensured that zones, zone groups, and pools are all removed before redeploy. Confirmation that that PGs are not reaching the limit, Ceph status is healthy, and all OSDs are up and running. The setup involves EC2 and a single RGW on a 3-node cluster with 5 OSDs (replica 3 RGW faces the same problem).
It appears that when I delete the RGW, the realm doesn’t get removed, even though I haven’t set the multisite config in the manifest.
Any insights or guidance would be greatly appreciated.
Here’s my manifest: