rook: Object Store unable to provision via admin ops endpoint -- NoSuchBucket errors
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior:
After deploying a new CephObjectStore, the object store reconciliation process encounters two distinct issues:
- Initially, a timeout occurs when attempting to reach the RGW service:
failed to get COSI user "cosi": Get "http://rook-ceph-rgw-my-store.rook-ceph.svc:80/admin/user?format=json&uid=cosi": dial tcp 10.104.148.223:80: i/o timeout
- Once the RGW service is reachable, the reconciliation gets stuck in a loop with repeated NoSuchBucket errors:
failed to get COSI user "cosi": NoSuchBucket tx0000036d36ca989b8e107-00674ff69b-358c2fd-my-store 358c2fd-my-store-my-store
The RGW pod itself is running and responding to basic requests, but the COSI user creation consistently fails, preventing the object store from completing its reconciliation.
Expected behavior:
The CephObjectStore reconciliation should:
- Wait for the RGW service to be fully ready before attempting COSI user creation
- Successfully create the COSI user without encountering NoSuchBucket errors
- Complete the reconciliation process without entering a loop
Potentially Related Issues
#13904 [Creating COSI user causes the object store reconcile to fail several times before finally succeeding] Had the same problem, but after a couple of retries it was resolved automatically
#13614 [ceph_assert error on rgw start in rook-ceph-rgw-ceph-objectstore pod] Saw the same Problem in the logs, but was caused from another issue.
Background: The issue occurred during a planned upgrade of a Rook-Ceph cluster. The upgrade was executed in multiple steps (unfortunately it was not a step-by-step upgrade, i jumped to the latest version and removed the static images keys in another step. Helm values values are at the end of the issue):
-
Initial upgrade from v1.11.8 to v1.15.6:
- Upgraded rook-ceph operator to v1.15.6
- Upgraded rook-ceph-cluster to v1.15.6
-
Configuration cleanup:
- Removed static image keys from helm configuration files
-
Final verification upgrade at v1.15.6:
- Re-applied rook-ceph operator v1.15.6
- Re-applied rook-ceph-cluster v1.15.6
The cluster was healthy throughout and after the upgrade process. The issue manifested only when attempting to deploy a new CephObjectStore after the upgrade was complete. The cluster runs on 3 nodes with NVMe drives (4TB each) and serves existing RBD and CephFS workloads without issues.
How to reproduce it (minimal and precise):
- Start with a healthy Ceph cluster running Rook v1.11.8
- Upgrade Rook-Ceph operator and cluster to v1.15.6
- Deploy a new CephObjectStore with default configuration:
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
name: my-store
namespace: rook-ceph
spec:
metadataPool:
failureDomain: host
replicated:
size: 3
dataPool:
failureDomain: host
erasureCoded:
dataChunks: 2
codingChunks: 1
gateway:
port: 80
instances: 1
File(s) to submit:
Logs to submit:
Container Logs:
rook-operator_install.log rook-opreator_startup.log rook-rgw-pod.log
YAML configurations:
rook_cluster_values_after_upgrade.yaml.txt rook_cluster_values_before_upgrade.yaml.txt rook_values_after_upgrade.yaml.txt rook_values_before_upgrade.yaml.txt
Cluster Status to submit:
Before the objectstore creation:
bash-5.1$ ceph status
cluster:
id: 5f71074a-67b8-4372-9d7b-d35fa4be7035
health: HEALTH_OK
services:
mon: 3 daemons, quorum j,m,o (age 24m)
mgr: b(active, since 24m), standbys: a
mds: 1/1 daemons up, 1 hot standby
osd: 3 osds: 3 up (since 22m), 3 in (since 6M)
data:
volumes: 1/1 healthy
pools: 8 pools, 185 pgs
objects: 444.95k objects, 1.5 TiB
usage: 4.4 TiB used, 6.1 TiB / 10 TiB avail
pgs: 185 active+clean
io:
client: 1.3 MiB/s rd, 2.3 MiB/s wr, 333 op/s rd, 58 op/s wr
bash-5.1$ ceph osd pool ls
.mgr
ceph-blockpool
ceph-filesystem-metadata
ceph-filesystem-data0
default.rgw.log
default.rgw.control
default.rgw.meta
.rgw.root
After creation:
bash-5.1$ ceph health
HEALTH_WARN too many PGs per OSD (265 > max 250)
bash-5.1$ ceph status
cluster:
id: 5f71074a-67b8-4372-9d7b-d35fa4be7035
health: HEALTH_WARN
too many PGs per OSD (265 > max 250)
services:
mon: 3 daemons, quorum j,m,o (age 45m)
mgr: b(active, since 45m), standbys: a
mds: 1/1 daemons up, 1 hot standby
osd: 3 osds: 3 up (since 43m), 3 in (since 6M)
rgw: 1 daemon active (1 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 15 pools, 265 pgs
objects: 445.22k objects, 1.5 TiB
usage: 4.4 TiB used, 6.0 TiB / 10 TiB avail
pgs: 265 active+clean
io:
client: 1.5 MiB/s rd, 2.4 MiB/s wr, 391 op/s rd, 69 op/s wr
bash-5.1$ ceph osd pool ls
.mgr
ceph-blockpool
ceph-filesystem-metadata
ceph-filesystem-data0
default.rgw.log
default.rgw.control
default.rgw.meta
.rgw.root
my-store.rgw.control
my-store.rgw.meta
my-store.rgw.log
my-store.rgw.buckets.index
my-store.rgw.buckets.non-ec
my-store.rgw.otp
my-store.rgw.buckets.data
Environment:
- Kubernetes version: v1.28.2
- Rook version: v1.15.6
- Ceph version: v18.2.4 (reef)
- OS: Debian with Linux 5.10.0-26-amd64
- Cluster type: On-premises Kubernetes
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 26 (12 by maintainers)
OMG. Thats it. Thank you so much for your support! For anyone who might have similar symptoms, here is what i finally did:
checked the user via
deleted the entry and verified it via
Then i restarted my rook-operator and the rgw pod. Afterwards the error-message were gone and the pending bucket was finally created. The specific bucket user is also being created and i can see the bucket user also listed as a user in ceph
And it also fixed my bug that i wasn’t able to see any objectstore on my dashboard if another port than 8080 was defined.