ceph-csi: Rados Namespace: "failed to create rbd image: rbd: ret=-1, Operation not permitted"

Hi,

i’m trying to use the new rados namespace feature for RBD images which was introduced in ceph-csi 3.1 release. I already configured a new ceph-csi configmap with the radosNamespace like in the documentation. I also created a new ceph pool with a new user.

As far as i understood from the ceph documentation i cannot create a namespace within a pool - it’s just kind of a prefix and gets created automatically when i try to add data to the namespace.

If i try to create rbd-volumes from k8s / ceph-csi i get the following error:

I1118 13:47:40.259809       1 controller.go:562] CreateVolumeRequest {Name:pvc-08368567-417c-4aa6-9216-348f2f8cd918 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:"ext4" mount_flags:"discard" > access_mode: ] Parameters:map[clusterID:01578d80-6c97-46ba-9327-cb2b13980916 csi.storage.k8s.io/controller-expand-secret-name:csi-rbd-secret-ssd csi.storage.k8s.io/controller-expand-secret-namespace:ceph-csi csi.storage.k8s.io/fstype:ext4 csi.storage.k8s.io/node-stage-secret-name:csi-rbd-secret-ssd csi.storage.k8s.io/node-stage-secret-namespace:ceph-csi csi.storage.k8s.io/provisioner-secret-name:csi-rbd-secret-ssd csi.storage.k8s.io/provisioner-secret-namespace:ceph-csi imageFeatures:layering imageFormat:2 mounter:rbd-nbd pool:k8s.ssd] Secrets:map[] VolumeContentSource: AccessibilityRequirements: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1118 13:47:40.260158       1 event.go:281] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pvc", UID:"08368567-417c-4aa6-9216-348f2f8cd918", APIVersion:"v1", ResourceVersion:"422256", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/rbd-pvc"
I1118 13:47:40.262301       1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume
I1118 13:47:40.262321       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-08368567-417c-4aa6-9216-348f2f8cd918","parameters":{"clusterID":"01578d80-6c97-46ba-9327-cb2b13980916","imageFeatures":"layering","imageFormat":"2","mounter":"rbd-nbd","pool":"k8s.ssd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I1118 13:47:40.287853       1 connection.go:185] GRPC response: {}
I1118 13:47:40.288338       1 connection.go:186] GRPC error: rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted
I1118 13:47:40.288390       1 controller.go:634] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Finished: rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted
I1118 13:47:40.288423       1 controller.go:1051] Final error received, removing PVC 08368567-417c-4aa6-9216-348f2f8cd918 from claims in progress
W1118 13:47:40.288432       1 controller.go:916] Retrying syncing claim "08368567-417c-4aa6-9216-348f2f8cd918", failure 0
E1118 13:47:40.288450       1 controller.go:939] error syncing claim "08368567-417c-4aa6-9216-348f2f8cd918": failed to provision volume with StorageClass "csi-rbd-ssd": rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted
I1118 13:47:40.288480       1 controller.go:1284] provision "default/rbd-pvc" class "csi-rbd-ssd": started
I1118 13:47:40.288671       1 event.go:281] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pvc", UID:"08368567-417c-4aa6-9216-348f2f8cd918", APIVersion:"v1", ResourceVersion:"422256", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-ssd": rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted

I thought this tells me that my ceph user capabilities are not correct - but i didn’t find a proper documentation. I tried a few different settings but none did work:

caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=k8s.ssd namespace=services"

caps mon = "profile rbd"
caps osd = "profile rbd pool=k8s.ssd namespace=services"

Am i right that the error relies on wrong ceph capabilities? If yes, can anyone tell me what capabilities are needed for rbd images with rados namespaces?

Thanks