rook: globalmount: permission denied

Hi Team,

we have installed rook-ceph on k8s cluster, we have installed the application as well in production, 3 worker nodes and 2 master node cluster, we are getting the permission denied to mount the path in worker1, only one pod having issue, other all pods are running fine even those are running on worker1, how i can resolve this issue ? and what is the root cause of this issue ?

below are the logs:

Events: Type Reason Age From Message


Normal Scheduled 117s default-scheduler Successfully assigned default/task-pv-pod to worker1 Warning FailedMount 53s (x8 over 117s) kubelet MountVolume.SetUp failed for volume “pvc-df9b64aa-5c90-4b17-9382-5dc677b19873” : rpc error: code = Internal desc = stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-df9b64aa-5c90-4b17-9382-5dc677b19873/globalmount: permission denied

How to reproduce it (minimal and precise): not able to reproduce it on local setup

File(s) to submit:

  • Cluster CR (custom resource), typically called cluster.yaml, if necessary

Logs to submit:

  • Operator’s logs, if necessary

  • Crashing pod(s) logs, if necessary

    To get logs, use kubectl -n <namespace> logs <pod name> When pasting logs, always surround them with backticks or use the insert code button from the Github UI. Read GitHub documentation if you need help.

Cluster Status to submit:

  • Output of kubectl commands, if necessary

    To get the health of the cluster, use kubectl rook-ceph health ceph status cluster: id: cc9da5b7-2e86-4abc-971a-701ce9b3d532 health: HEALTH_OK

    services: mon: 3 daemons, quorum c,e,f (age 8d) mgr: a(active, since 3h) mds: 1/1 daemons up, 1 hot standby osd: 3 osds: 3 up (since 8d), 3 in (since 4M)

    data: volumes: 1/1 healthy pools: 3 pools, 65 pgs objects: 3.58k objects, 439 MiB usage: 2.0 GiB used, 1.8 TiB / 1.8 TiB avail pgs: 65 active+clean

    io: client: 1.2 KiB/s rd, 2 op/s rd, 0 op/s wr

    To get the status of the cluster, use kubectl rook-ceph ceph status For more details, see the Rook kubectl Plugin

Environment:

  • OS (e.g. from /etc/os-release): ubuntu 20
  • Kernel (e.g. uname -a): Linux master1 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
  • Cloud provider or hardware configuration:
  • Rook version (use rook version inside of a Rook Pod): rook: v1.8.1
  • Storage backend version (e.g. for ceph do ceph -v): ceph version 16.2.7
  • Kubernetes version (use kubectl version): v1.21.5
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): VMware
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (5 by maintainers)

Most upvoted comments

@Rakshith-R to unmount and remount the cephfs from the node, is there any impact on data ? what is the case if once unmount and remount done if other pods not comes up, because not other pods are running the volume path is having permision denied only one pod is in crashloopbackoff

No, the mount point is already blocklisted by the ceph cluster for that pod. It doesn’t affect other pods/volumes. Just scale down and up that particular pod.

You maybe interested in using recover_session mount option too if network fluctuation in cluster is unavoidable. https://docs.ceph.com/en/reef/man/8/mount.ceph/#basic