ceph-csi: Failed to create cephfs PVC

Describe the bug

Failed to create cephfs PVC

Environment details

  • Image/version of Ceph CSI driver: canary
  • helm chart version
  • Kubernetes cluster version: 1.15.0
  • Logs
E0722 08:26:57.932549       1 volume.go:100] failed to create subvolume group csi, for the vol csi-vol-77ecd37e-ac5a-11e9-8a38-22008e3300de(an error occurred while running (597) ceph [fs subvolumegroup create myfs csi --mode 777 --pool_layout myfs-data0 -m 10.106.133.206:6789 -c /etc/ceph/ceph.conf -n client.admin --key=***stripped***]: exit status 22: 2019-07-22 08:26:57.543 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.543 7f97e93ba700 -1 AuthRegistry(0x7f97e4063b98) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2019-07-22 08:26:57.580 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.580 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.580 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.614 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.614 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.614 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.614 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 909, in _handle_command
    return self.handle_command(inbuf, cmd)
  File "/usr/share/ceph/mgr/volumes/module.py", line 182, in handle_command
    return handler(inbuf, cmd)
  File "/usr/share/ceph/mgr/volumes/module.py", line 208, in _cmd_fs_subvolumegroup_create
    return self.vc.create_subvolume_group(vol_name, group_name, mode=mode, pool=pool_layout)
  File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 305, in create_subvolume_group
    sv.create_group(spec, pool=pool, mode=self.octal_str_to_decimal_int(mode))
  File "/usr/share/ceph/mgr/volumes/fs/subvolume.py", line 178, in create_group
    self._mkdir_p(path, mode)
  File "/usr/share/ceph/mgr/volumes/fs/subvolume.py", line 58, in _mkdir_p
    self.fs.mkdir(subpath, mode)
  File "cephfs.pyx", line 801, in cephfs.LibCephFS.mkdir (/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1-636-g64f035c/rpm/el7/BUILD/ceph-14.2.1-636-g64f035c/build/src/pybind/cephfs/pyrex/cephfs.c:8613)
Error: (13, "error in mkdir 'volumes': error code 13")

Steps to reproduce

deploy cephfs plugin and create PVC

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 31 (15 by maintainers)

Commits related to this issue

Most upvoted comments

This hit us too when testing with our existing bare-metal ceph-14.2.2 cluster.

I can confirm that editing /lib/systemd/system/ceph-mgr@.service to remove the setuser/setgroup options to ceph-mgr and then restarting the process fixes it, but that hack will be overwritten any time we update ceph.

I have this far not been able to fix it by adjusting the ceph auth capabilities of the client.mgr certificates to give them all permissions like the admin, and giving the ‘ceph’ user full ACL permissions to the cephfs root directory did not help either.

To fix this we need to provide the user ‘ceph’, RW permission to the root of the cephfs filesystem, either chmod or acl or ceph auth?, @ajarr will have more insight into fixing this.