rook: globalmount: permission denied
Hi Team,
we have installed rook-ceph on k8s cluster, we have installed the application as well in production, 3 worker nodes and 2 master node cluster, we are getting the permission denied to mount the path in worker1, only one pod having issue, other all pods are running fine even those are running on worker1, how i can resolve this issue ? and what is the root cause of this issue ?
below are the logs:
Events: Type Reason Age From Message
Normal Scheduled 117s default-scheduler Successfully assigned default/task-pv-pod to worker1 Warning FailedMount 53s (x8 over 117s) kubelet MountVolume.SetUp failed for volume “pvc-df9b64aa-5c90-4b17-9382-5dc677b19873” : rpc error: code = Internal desc = stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-df9b64aa-5c90-4b17-9382-5dc677b19873/globalmount: permission denied
How to reproduce it (minimal and precise): not able to reproduce it on local setup
File(s) to submit:
- Cluster CR (custom resource), typically called
cluster.yaml, if necessary
Logs to submit:
-
Operator’s logs, if necessary
-
Crashing pod(s) logs, if necessary
To get logs, use
kubectl -n <namespace> logs <pod name>When pasting logs, always surround them with backticks or use theinsert codebutton from the Github UI. Read GitHub documentation if you need help.
Cluster Status to submit:
-
Output of kubectl commands, if necessary
To get the health of the cluster, use
kubectl rook-ceph healthceph status cluster: id: cc9da5b7-2e86-4abc-971a-701ce9b3d532 health: HEALTH_OKservices: mon: 3 daemons, quorum c,e,f (age 8d) mgr: a(active, since 3h) mds: 1/1 daemons up, 1 hot standby osd: 3 osds: 3 up (since 8d), 3 in (since 4M)
data: volumes: 1/1 healthy pools: 3 pools, 65 pgs objects: 3.58k objects, 439 MiB usage: 2.0 GiB used, 1.8 TiB / 1.8 TiB avail pgs: 65 active+clean
io: client: 1.2 KiB/s rd, 2 op/s rd, 0 op/s wr
To get the status of the cluster, use
kubectl rook-ceph ceph statusFor more details, see the Rook kubectl Plugin
Environment:
- OS (e.g. from /etc/os-release): ubuntu 20
- Kernel (e.g.
uname -a): Linux master1 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux - Cloud provider or hardware configuration:
- Rook version (use
rook versioninside of a Rook Pod): rook: v1.8.1 - Storage backend version (e.g. for ceph do
ceph -v): ceph version 16.2.7 - Kubernetes version (use
kubectl version): v1.21.5 - Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): VMware
- Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox):
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (5 by maintainers)
No, the mount point is already blocklisted by the ceph cluster for that pod. It doesn’t affect other pods/volumes. Just scale down and up that particular pod.
You maybe interested in using recover_session mount option too if network fluctuation in cluster is unavoidable. https://docs.ceph.com/en/reef/man/8/mount.ceph/#basic