ceph-csi: Failed to create cephfs PVC
Describe the bug
Failed to create cephfs PVC
Environment details
- Image/version of Ceph CSI driver: canary
- helm chart version
- Kubernetes cluster version: 1.15.0
- Logs
E0722 08:26:57.932549 1 volume.go:100] failed to create subvolume group csi, for the vol csi-vol-77ecd37e-ac5a-11e9-8a38-22008e3300de(an error occurred while running (597) ceph [fs subvolumegroup create myfs csi --mode 777 --pool_layout myfs-data0 -m 10.106.133.206:6789 -c /etc/ceph/ceph.conf -n client.admin --key=***stripped***]: exit status 22: 2019-07-22 08:26:57.543 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.543 7f97e93ba700 -1 AuthRegistry(0x7f97e4063b98) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2019-07-22 08:26:57.580 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.580 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.580 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.614 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.614 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.614 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.614 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
Error EINVAL: Traceback (most recent call last):
File "/usr/share/ceph/mgr/mgr_module.py", line 909, in _handle_command
return self.handle_command(inbuf, cmd)
File "/usr/share/ceph/mgr/volumes/module.py", line 182, in handle_command
return handler(inbuf, cmd)
File "/usr/share/ceph/mgr/volumes/module.py", line 208, in _cmd_fs_subvolumegroup_create
return self.vc.create_subvolume_group(vol_name, group_name, mode=mode, pool=pool_layout)
File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 305, in create_subvolume_group
sv.create_group(spec, pool=pool, mode=self.octal_str_to_decimal_int(mode))
File "/usr/share/ceph/mgr/volumes/fs/subvolume.py", line 178, in create_group
self._mkdir_p(path, mode)
File "/usr/share/ceph/mgr/volumes/fs/subvolume.py", line 58, in _mkdir_p
self.fs.mkdir(subpath, mode)
File "cephfs.pyx", line 801, in cephfs.LibCephFS.mkdir (/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1-636-g64f035c/rpm/el7/BUILD/ceph-14.2.1-636-g64f035c/build/src/pybind/cephfs/pyrex/cephfs.c:8613)
Error: (13, "error in mkdir 'volumes': error code 13")
Steps to reproduce
deploy cephfs plugin and create PVC
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 31 (15 by maintainers)
Commits related to this issue
- ceph: mgr fix startup Because the manager is running under the 'ceph' user the cephfs subvolume plugin can not create directories anymore. It fails with a permission denied because of the usage of th... — committed to leseb/rook by leseb 5 years ago
- docs: add special config steps to make CephFS CSI v1.1.0 ... work with Ceph v14.2.2 cluster that is not deployed by rook. Fixes: #486 Signed-off-by: Ramana Raja <rraja@redhat.com> — committed to ajarr/ceph-csi by ajarr 5 years ago
- docs: add special config steps to make CephFS CSI v1.1.0 ... work with Ceph v14.2.2 cluster that is not deployed by rook. Fixes: #486 Signed-off-by: Ramana Raja <rraja@redhat.com> — committed to ajarr/ceph-csi by ajarr 5 years ago
- docs: add special config steps to make CephFS CSI v1.1.0 ... work with Ceph v14.2.2 cluster that is not deployed by rook. Fixes: #486 Signed-off-by: Ramana Raja <rraja@redhat.com> — committed to ajarr/ceph-csi by ajarr 5 years ago
- docs: add special config steps to make CephFS CSI v1.1.0 ... work with Ceph v14.2.2 cluster that is not deployed by rook. Fixes: #486 Signed-off-by: Ramana Raja <rraja@redhat.com> — committed to ajarr/ceph-csi by ajarr 5 years ago
- docs: add special config steps to make CephFS CSI v1.1.0 ... work with Ceph v14.2.2 cluster that is not deployed by rook. Fixes: #486 Signed-off-by: Ramana Raja <rraja@redhat.com> — committed to ceph/ceph-csi by ajarr 5 years ago
- docs: add special config steps to make CephFS CSI v1.1.0 ... work with Ceph v14.2.2 cluster that is not deployed by rook. Fixes: #486 Signed-off-by: Ramana Raja <rraja@redhat.com> — committed to wilmardo/ceph-csi by ajarr 5 years ago
- ceph: mgr fix startup Because the manager is running under the 'ceph' user the cephfs subvolume plugin can not create directories anymore. It fails with a permission denied because of the usage of th... — committed to leseb/rook by leseb 5 years ago
This hit us too when testing with our existing bare-metal ceph-14.2.2 cluster.
I can confirm that editing /lib/systemd/system/ceph-mgr@.service to remove the setuser/setgroup options to ceph-mgr and then restarting the process fixes it, but that hack will be overwritten any time we update ceph.
I have this far not been able to fix it by adjusting the ceph auth capabilities of the client.mgr certificates to give them all permissions like the admin, and giving the ‘ceph’ user full ACL permissions to the cephfs root directory did not help either.
@poornimag - can you look if https://github.com/rook/rook/pull/3498 and/or https://github.com/rook/rook/pull/3487 help?
To fix this we need to provide the user ‘ceph’, RW permission to the root of the cephfs filesystem, either chmod or acl or ceph auth?, @ajarr will have more insight into fixing this.