rook: cephfs mount failure.permission denied

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior: Unable to mount cephfs

Expected behavior: It is possible to mount cephfs normally

How to reproduce it (minimal and precise):

E0210 20:38:31.292481       1 utils.go:185] ID: 44990 Req-ID: 0001-0009-rook-ceph-0000000000000002-bbb57db2-4684-11ec-a965-7a1a7a095897 GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-89e0b81a-c433-4971-b446-e9341d9ab148/globalmount: permission denied
E0210 20:40:33.398387       1 nodeserver.go:117] ID: 44994 Req-ID: 0001-0009-rook-ceph-0000000000000002-bbb57db2-4684-11ec-a965-7a1a7a095897 stat failed: stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-89e0b81a-c433-4971-b446-e9341d9ab148/globalmount: permission denied
E0210 20:40:33.398454       1 utils.go:185] ID: 44994 Req-ID: 0001-0009-rook-ceph-0000000000000002-bbb57db2-4684-11ec-a965-7a1a7a095897 GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-89e0b81a-c433-4971-b446-e9341d9ab148/globalmount: permission denied
E0210 20:42:35.561473       1 nodeserver.go:117] ID: 44998 Req-ID: 0001-0009-rook-ceph-0000000000000002-bbb57db2-4684-11ec-a965-7a1a7a095897 stat failed: stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-89e0b81a-c433-4971-b446-e9341d9ab148/globalmount: permission denied
E0210 20:42:35.561539       1 utils.go:185] ID: 44998 Req-ID: 0001-0009-rook-ceph-0000000000000002-bbb57db2-4684-11ec-a965-7a1a7a095897 GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-89e0b81a-c433-4971-b446-e9341d9ab148/globalmount: permission denied
E0210 20:44:37.732460       1 nodeserver.go:117] ID: 45002 Req-ID: 0001-0009-rook-ceph-0000000000000002-bbb57db2-4684-11ec-a965-7a1a7a095897 stat failed: stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-89e0b81a-c433-4971-b446-e9341d9ab148/globalmount: permission denied
E0210 20:44:37.732528       1 utils.go:185] ID: 45002 Req-ID: 0001-0009-rook-ceph-0000000000000002-bbb57db2-4684-11ec-a965-7a1a7a095897 GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-89e0b81a-c433-4971-b446-e9341d9ab148/globalmount: permission denied
E0210 20:56:12.253793       1 nodeserver.go:117] ID: 45011 Req-ID: 0001-0009-rook-ceph-0000000000000002-bbb57db2-4684-11ec-a965-7a1a7a095897 stat failed: stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-89e0b81a-c433-4971-b446-e9341d9ab148/globalmount: permission denied
E0210 20:56:12.253885       1 utils.go:185] ID: 45011 Req-ID: 0001-0009-rook-ceph-0000000000000002-bbb57db2-4684-11ec-a965-7a1a7a095897 GRPC error: rpc error: code = Internal desc = stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-89e0b81a-c433-4971-b446-e9341d9ab148/globalmount: permission denied

Environment:

  • OS (e.g. from /etc/os-release): CentOS 7 X86
  • Kernel (e.g. uname -a): 5.4.172
  • Cloud provider or hardware configuration:
  • Rook version (use rook version inside of a Rook Pod): v1.7.7
  • Storage backend version (e.g. for ceph do ceph -v): 16.2.6
  • Kubernetes version (use kubectl version): v1.16.15
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Native Kubernetes
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): HEALTH_OK

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 15 (3 by maintainers)

Most upvoted comments

I also encountered the same problem, I found that the mount directory permissions caused by the problem, my solution is.

d75c38524eeefc831768da71832cfef ff79f94cedf30cd1dcc1e9a02fcc935 cea37e7459570149f908dd725c9e416 b037c2bf67599302d96a6cf745c86ee

I found the directory reporting the error, Remove the POD from this node

umount -lf /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1739b858-93b0-4ac4-b59e-773a84d3ea83/globalmount

Migrate the POD back to the node and the problem is resolved