rook: Object Store unable to provision via admin ops endpoint -- NoSuchBucket errors

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

After deploying a new CephObjectStore, the object store reconciliation process encounters two distinct issues:

  1. Initially, a timeout occurs when attempting to reach the RGW service:
failed to get COSI user "cosi": Get "http://rook-ceph-rgw-my-store.rook-ceph.svc:80/admin/user?format=json&uid=cosi": dial tcp 10.104.148.223:80: i/o timeout
  1. Once the RGW service is reachable, the reconciliation gets stuck in a loop with repeated NoSuchBucket errors:
failed to get COSI user "cosi": NoSuchBucket tx0000036d36ca989b8e107-00674ff69b-358c2fd-my-store 358c2fd-my-store-my-store

The RGW pod itself is running and responding to basic requests, but the COSI user creation consistently fails, preventing the object store from completing its reconciliation.

Expected behavior:

The CephObjectStore reconciliation should:

  • Wait for the RGW service to be fully ready before attempting COSI user creation
  • Successfully create the COSI user without encountering NoSuchBucket errors
  • Complete the reconciliation process without entering a loop

Potentially Related Issues

#13904 [Creating COSI user causes the object store reconcile to fail several times before finally succeeding] Had the same problem, but after a couple of retries it was resolved automatically

#13614 [ceph_assert error on rgw start in rook-ceph-rgw-ceph-objectstore pod] Saw the same Problem in the logs, but was caused from another issue.

Background: The issue occurred during a planned upgrade of a Rook-Ceph cluster. The upgrade was executed in multiple steps (unfortunately it was not a step-by-step upgrade, i jumped to the latest version and removed the static images keys in another step. Helm values values are at the end of the issue):

  1. Initial upgrade from v1.11.8 to v1.15.6:

    • Upgraded rook-ceph operator to v1.15.6
    • Upgraded rook-ceph-cluster to v1.15.6
  2. Configuration cleanup:

    • Removed static image keys from helm configuration files
  3. Final verification upgrade at v1.15.6:

    • Re-applied rook-ceph operator v1.15.6
    • Re-applied rook-ceph-cluster v1.15.6

The cluster was healthy throughout and after the upgrade process. The issue manifested only when attempting to deploy a new CephObjectStore after the upgrade was complete. The cluster runs on 3 nodes with NVMe drives (4TB each) and serves existing RBD and CephFS workloads without issues.

How to reproduce it (minimal and precise):

  • Start with a healthy Ceph cluster running Rook v1.11.8
  • Upgrade Rook-Ceph operator and cluster to v1.15.6
  • Deploy a new CephObjectStore with default configuration:
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: my-store
  namespace: rook-ceph
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
  dataPool:
    failureDomain: host
    erasureCoded:
      dataChunks: 2
      codingChunks: 1
  gateway:
    port: 80
    instances: 1

File(s) to submit:

Logs to submit:

Container Logs:

rook-operator_install.log rook-opreator_startup.log rook-rgw-pod.log

YAML configurations:

rook_cluster_values_after_upgrade.yaml.txt rook_cluster_values_before_upgrade.yaml.txt rook_values_after_upgrade.yaml.txt rook_values_before_upgrade.yaml.txt

Cluster Status to submit:

Before the objectstore creation:

bash-5.1$ ceph status
  cluster:
    id:     5f71074a-67b8-4372-9d7b-d35fa4be7035
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum j,m,o (age 24m)
    mgr: b(active, since 24m), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 22m), 3 in (since 6M)

  data:
    volumes: 1/1 healthy
    pools:   8 pools, 185 pgs
    objects: 444.95k objects, 1.5 TiB
    usage:   4.4 TiB used, 6.1 TiB / 10 TiB avail
    pgs:     185 active+clean

  io:
    client:   1.3 MiB/s rd, 2.3 MiB/s wr, 333 op/s rd, 58 op/s wr
bash-5.1$ ceph osd pool ls
.mgr
ceph-blockpool
ceph-filesystem-metadata
ceph-filesystem-data0
default.rgw.log
default.rgw.control
default.rgw.meta
.rgw.root

After creation:

bash-5.1$ ceph health
HEALTH_WARN too many PGs per OSD (265 > max 250)
bash-5.1$ ceph status
  cluster:
    id:     5f71074a-67b8-4372-9d7b-d35fa4be7035
    health: HEALTH_WARN
            too many PGs per OSD (265 > max 250)
 
  services:
    mon: 3 daemons, quorum j,m,o (age 45m)
    mgr: b(active, since 45m), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 43m), 3 in (since 6M)
    rgw: 1 daemon active (1 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   15 pools, 265 pgs
    objects: 445.22k objects, 1.5 TiB
    usage:   4.4 TiB used, 6.0 TiB / 10 TiB avail
    pgs:     265 active+clean
 
  io:
    client:   1.5 MiB/s rd, 2.4 MiB/s wr, 391 op/s rd, 69 op/s wr
bash-5.1$ ceph osd pool ls
.mgr
ceph-blockpool
ceph-filesystem-metadata
ceph-filesystem-data0
default.rgw.log
default.rgw.control
default.rgw.meta
.rgw.root
my-store.rgw.control
my-store.rgw.meta
my-store.rgw.log
my-store.rgw.buckets.index
my-store.rgw.buckets.non-ec
my-store.rgw.otp
my-store.rgw.buckets.data

Environment:

  • Kubernetes version: v1.28.2
  • Rook version: v1.15.6
  • Ceph version: v18.2.4 (reef)
  • OS: Debian with Linux 5.10.0-26-amd64
  • Cluster type: On-premises Kubernetes

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 26 (12 by maintainers)

Most upvoted comments

OMG. Thats it. Thank you so much for your support! For anyone who might have similar symptoms, here is what i finally did:

checked the user via

bash-5.1$ ceph config dump | grep dns
global                                 advanced  rgw_dns_name                             rook-ceph-rgw-ceph-objectstore            * 

deleted the entry and verified it via

bash-5.1$ ceph config rm global rgw_dns_name
bash-5.1$ ceph config dump | grep dns

Then i restarted my rook-operator and the rgw pod. Afterwards the error-message were gone and the pending bucket was finally created. The specific bucket user is also being created and i can see the bucket user also listed as a user in ceph

bash-5.1$ radosgw-admin user list
[
    "dashboard-admin",
    "test-user",
    "myuser",
    "obc-rook-ceph-ceph-bucket-fece5de2-6871-48a0-a430-a107e581cd76",
    "rgw-admin-ops-user"
]

And it also fixed my bug that i wasn’t able to see any objectstore on my dashboard if another port than 8080 was defined.