Describe the bug

I deploy ceph-csi in k8s and use cephfs to provide pvc. PVC created fail when I use a normal ceph user but succeed if I use admin ceph user.

Environment details

• Image/version of Ceph CSI driver : v3.2.0
• OS version: Ubuntu 20.04.1
• Kernel version : 5.4.0-58
• Mounter used for mounting PVC (for cephfs its fuse or kernel. for rbd its krbd or rbd-nbd) : kernel
• Kubernetes cluster version : v1.20.0
• Containerd version: 1.4.3
• Ceph cluster version : 14.2.8

Steps to reproduce

Steps to reproduce the behavior:

1. create ceph user ceph auth caps client.k8sfs mon 'allow r' mgr 'allow rw' mds 'allow rw' osd 'allow rw tag cephfs *=*'
2. dowload yaml from https://github.com/ceph/ceph-csi/tree/release-v3.2/deploy/cephfs/kubernetes
3. modify ceph information in csi-config-map.yaml
4. add kms-config.yaml and create from it

---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    {}
metadata:
  name: ceph-csi-encryption-kms-config

1. add secret.yaml and create from it

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-secret
  namespace: ceph-csi
stringData:
  # Required for statically provisioned volumes
  #userID: <plaintext ID>
  #userKey: <Ceph auth key corresponding to ID above>

  # Required for dynamically provisioned volumes
  adminID: k8sfs
  adminKey: AQDuM+xfXz0zNRAAnxeJaWdmR2J5I/QxMR9gLQ==

1. add storage class

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
  # String representing a Ceph cluster to provision storage from.
  # Should be unique across all Ceph clusters in use for provisioning,
  # cannot be greater than 36 bytes in length, and should remain immutable for
  # the lifetime of the StorageClass in use.
  # Ensure to create an entry in the config map named ceph-csi-config, based on
  # csi-config-map-sample.yaml, to accompany the string chosen to
  # represent the Ceph cluster in clusterID below
  clusterID: d9693b9b-8988-44bb-8bf9-ccb2c2733eec

  # CephFS filesystem name into which the volume shall be created
  fsName: cephfs

  # (optional) Ceph pool into which volume data shall be stored
  # pool: cephfs_data

  # (optional) Comma separated string of Ceph-fuse mount options.
  # For eg:
  # fuseMountOptions: debug

  # (optional) Comma separated string of Cephfs kernel mount options.
  # Check man mount.ceph for mount options. For eg:
  # kernelMountOptions: readdir_max_bytes=1048576,norbytes

  # The secrets have to contain user and/or Ceph admin credentials.
  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi
  csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi
  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi

  # (optional) The driver can use either ceph-fuse (fuse) or
  # ceph kernelclient (kernel).
  # If omitted, default volume mounter will be used - this is
  # determined by probing for ceph-fuse and mount.ceph
  mounter: kernel
reclaimPolicy: Retain
allowVolumeExpansion: true
mountOptions:
  - debug

1. create pvc

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-cephfs-sc

Actual results

# kubectl get sc
NAME            PROVISIONER           RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-cephfs-sc   cephfs.csi.ceph.com   Retain          Immediate           true                   7h1m

# kubectl get pvc
NAME             STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
csi-cephfs-pvc   Pending                                      csi-cephfs-sc   57m

# kubectl get pv
No resources found

Expected behavior

PVC should be created successfully and bound to a PV.

Logs

If the issue is in PVC creation, deletion, cloning please attach complete logs of below containers.

• csi-provisioner and csi-rbdplugin/csi-cephfsplugin container logs from the provisioner pod.

I1230 09:46:02.448025       1 controller.go:1317] provision "default/csi-cephfs-pvc" class "csi-cephfs-sc": started                                                                                                                                               
 I1230 09:46:02.448273       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"csi-cephfs-pvc", UID:"4bd0ecf9-d613-4f8f-998d-d0b204e8352d", APIVersion:"v1", ResourceVersion:"1814335", FieldPath:""}): type: 'Nor 
 mal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/csi-cephfs-pvc"                                                                                                                                                        
 I1230 09:46:02.448202       1 controller.go:573] CreateVolumeRequest {Name:pvc-4bd0ecf9-d613-4f8f-998d-d0b204e8352d CapacityRange:required_bytes:5368709120  VolumeCapabilities:[mount:<mount_flags:"debug" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > ] Param 
 eters:map[clusterID:d9693b9b-8988-44bb-8bf9-ccb2c2733eec csi.storage.k8s.io/controller-expand-secret-name:csi-cephfs-secret csi.storage.k8s.io/controller-expand-secret-namespace:ceph-csi csi.storage.k8s.io/node-stage-secret-name:csi-cephfs-secret csi.storag 
 e.k8s.io/node-stage-secret-namespace:ceph-csi csi.storage.k8s.io/provisioner-secret-name:csi-cephfs-secret csi.storage.k8s.io/provisioner-secret-namespace:ceph-csi fsName:cephfs mounter:kernel] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequiremen 
 ts:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}                                                                                                                                                                                             
 I1230 09:46:02.467093       1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume                                                                                                                                                                       
 I1230 09:46:02.467124       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-4bd0ecf9-d613-4f8f-998d-d0b204e8352d","parameters":{"clusterID":"d9693b9b-8988-44bb-8bf9-ccb2c2733eec","fsName":"cephfs","mounter":"ke 
 rnel"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["debug"]}},"access_mode":{"mode":5}}]}                                                                                                                            
 I1230 09:46:02.473398       1 connection.go:185] GRPC response: {}                                                                                                                                                                                                
 I1230 09:46:02.473461       1 connection.go:186] GRPC error: rpc error: code = Internal desc = rados: ret=-1, Operation not permitted                                                                                                                             
 I1230 09:46:02.473515       1 controller.go:645] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Finished: rpc error: code = Internal desc = rados: ret=-1, Operation not permitted                      
 I1230 09:46:02.473588       1 controller.go:1084] Final error received, removing PVC 4bd0ecf9-d613-4f8f-998d-d0b204e8352d from claims in progress                                                                                                                 
 W1230 09:46:02.473608       1 controller.go:943] Retrying syncing claim "4bd0ecf9-d613-4f8f-998d-d0b204e8352d", failure 16                                                                                                                                        
 E1230 09:46:02.473644       1 controller.go:966] error syncing claim "4bd0ecf9-d613-4f8f-998d-d0b204e8352d": failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Internal desc = rados: ret=-1, Operation not permitted               
 I1230 09:46:02.473699       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"csi-cephfs-pvc", UID:"4bd0ecf9-d613-4f8f-998d-d0b204e8352d", APIVersion:"v1", ResourceVersion:"1814335", FieldPath:""}): type: 'War 
 ning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Internal desc = rados: ret=-1, Operation not permitted

Additional context

the ceph user ‘k8sfs’ caps:

client.k8sfs
	key: AQDuM+xfXz0zNRAAnxeJaWdmR2J5I/QxMR9gLQ==
	caps: [mds] allow rw
	caps: [mgr] allow rw
	caps: [mon] allow r
	caps: [osd] allow rw tag cephfs *=*

this user has ability to create subvolume and subvolumegroup as well.

# ceph --id k8sfs fs subvolume create cephfs test 
# ceph --id k8sfs fs subvolume ls cephfs
[
    {
        "name": "test"
    }
]

# ceph --id k8sfs fs subvolumegroup ls cephfs
[
    {
        "name": "_nogroup"
    }, 
    {
        "name": "csi"
    }
]

# ceph --id k8sfs fs subvolumegroup create cephfs testgroup
# ceph --id k8sfs fs subvolumegroup ls cephfs
[
    {
        "name": "_nogroup"
    }, 
    {
        "name": "csi"
    }, 
    {
        "name": "testgroup"
    }
]

# ceph --id k8sfs fs subvolume create cephfs testsubvolume csi
# ceph --id k8sfs fs subvolume ls cephfs csi
[
    {
        "name": "testsubvolume"
    }, 
    {
        "name": "csi-vol-eac5a168-4a70-11eb-b23a-8e1756c5ca33"
    }
]

the ‘csi’ subvolumegroup is created when I use admin keyring in ceph-csi.