rook: cephfs storageclass does not work

Solution

• you need to create the filesystem (Rook operator is not supposed to create that )

kubectl apply -f https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/filesystem.yaml

• Create storage class
• Create pvc,pv,po or whatever

Thanks @Madhu-1

ceph fs subvolumegroup create myfs csi ceph fs subvolume create myfs subvol00 --group_name csi ceph fs subvolume getpath myfs subvol00 --group_name csi ceph fs subvolume rm myfs subvol00 --group_name csi

Please help me understand the issue, and why the above resolved the issue???

As i understood, it, we created a CephFilesystem with the following: https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/filesystem.yaml

Afterwards, we create a storageclass “rook-cephfs” which makes use the filesystem called “myfs” created above. https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/csi/cephfs/storageclass.yaml

Lastly, we create the PVC and test deployment https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/csi/cephfs/kube-registry.yaml

Why is it that we had to execute the above quoted commands to create the subvolumegroup called csi ?? Is this something we need to do every time we need to create a new filesystem? If so, I cannot find it documented here: https://rook.io/docs/rook/v1.4/ceph-filesystem.html If it is required to manually create the subvolumegroup called csi, should it not be automatically done by rook?

@ajarr The command(ceph fs subvolumegroup create mycephfs csi) gets stuck.

Having the same problem with rook 1.2.6, ceph 14.2.7, kubernetes 1.16.8 and calico cni (tried both ipip and vxlan backends) 3.13.1. Using 3 Nodes and the example production manifests (with changed names and references).

Cephfs-csi-provisioner is hanging on process: /usr/bin/python2.7 /usr/bin/ceph fs subvolumegroup create cephfs csi -m 10.99.154.70:6789,10.102.229.119:6789,10.98.121.172:6789 -c /etc/ceph/ceph.conf -n client.csi-cephfs-provisioner --keyfile=/tmp/csi/keys/keyfile-739574613

failing after a few minutes with: E0321 12:19:41.405149 1 controller.go:910] error syncing claim "840cbb9b-fa9a-4026-b9a3-efccc48513af": failed to provision volume with StorageClass "rook-cephfs": rpc error: code = DeadlineExceeded desc = context deadline exceeded

and then looping forever with: W0321 12:19:41.926998 1 controller.go:887] Retrying syncing claim "840cbb9b-fa9a-4026-b9a3-efccc48513af", failure 2 E0321 12:19:41.927087 1 controller.go:910] error syncing claim "840cbb9b-fa9a-4026-b9a3-efccc48513af": failed to provision volume with StorageClass "rook-cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-840cbb9b-fa9a-4026-b9a3-efccc48513af already exists

while “ceph fs subvolumegroup create cephfs csi” is still hanging in the background.

Ceph OSD/Mon Pod IPs and ports are all reachable from the nodes and pods. When using hostNetwork: true in the CephCluster the CephFS PVC creation actually works, for whatever reason…

Edit: PVC creation also works with hostNetwork: false and kube-proxy in ipvs mode

Hi guys, This seems to be a problem with the CNI Provider that I use, previously I used CNI weave net and currently I use CNI calico, a result the cephfs run perfectly.

#cheers

Strange but true, tnx a lot!

I have had these same issues, both with hostNetwork on and hostNetwork off. Rook @ 1.1.7, Ceph at 14.2.5. Cannot complete instructions since ceph fs subvolumegroup create cephfs csi hangs.

Hi guys, This seems to be a problem with the CNI Provider that I use, previously I used CNI weave net and currently I use CNI calico, a result the cephfs run perfectly.

#cheers

@satwikk : UnboundLocalError: local variable 'ret' referenced before assignment This happens because you pressed Ctrl+c. Additionally, ret was not initialized in do_command as it is supposed to be ret = 0, "", ""

The issue we need to check here is why was it hanging. Looking at the logs provided here and ceph status output, I see that the volume myfs was created by default but the below operation is not able to identify it (ENOENT) and you are trying to use myfs later, which should be the cause of this problem.

Could you please explain what the below operation is and are you passing the parameters correctly?

I1010 07:11:52.254715       1 utils.go:119] ID: 6768 GRPC call: /csi.v1.Controller/CreateVolume
I1010 07:11:52.254745       1 utils.go:120] ID: 6768 GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-3be36efb-eb2d-11e9-bead-ac1f6b84bde2","parameters":{"clusterID":"rook-ceph","fsName":"myfs","pool":"myfs-data0"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":5}}]}
I1010 07:11:52.258214       1 util.go:48] ID: 6768 cephfs: EXEC ceph [-m 10.152.183.224:6789 --id admin --keyfile=***stripped*** -c /etc/ceph/ceph.conf fs get myfs --format=json]
E1010 07:11:52.802795       1 controllerserver.go:84] ID: 6768 validation and extraction of volume options failed: an error occurred while running (21749) ceph [-m 10.152.183.224:6789 --id admin --keyfile=***stripped*** -c /etc/ceph/ceph.conf fs get myfs --format=json]: exit status 2: Error ENOENT: filesystem 'myfs' not found

@veezhang from the rook ceph-tool box pod can you try issuing the following command, and check the output of,

$ ceph fs subvolumegroup create mycephfs csi

that was the last ceph fs volumes command (https://docs.ceph.com/docs/master/cephfs/fs-volumes/#fs-subvolume-groups) executed as per logs in https://github.com/rook/rook/issues/4006#issuecomment-540428989

Can you share the corresponding ceph-mgr log for that command?

If the above commands succeeds, try issuing the following commands too,

$ ceph fs subvolume create mycephfs subvol00 --group_name csi
$ ceph fs subvolume getpath mycephfs subvol00 --group_name csi
$ ceph fs subvolume rm mycephfs subvol00 --group_name csi

The only critical issue I’m aware that some of us hit in Ceph v14.2.4, and that will be fixed in Ceph v14.2.5 is https://tracker.ceph.com/issues/41933 But you don’t seem to hit this issue either as fs subvolume getpath is not executed by your csi-cephfs plugin pod?