rook: Timeout expired waiting for volumes to attach/mount for pod (rook plugin not found)
k8s version: v1.9 env: VirtualBox os: Coreos
It is 1 node kubernete cluster I followed the below steps:
- Followed https://rook.io/docs/rook/v0.5/k8s-pre-reqs.html and updated the kubelet with
Environment="RKT_OPTS=--volume modprobe,kind=host,source=/usr/sbin/modprobe \
--mount volume=modprobe,target=/usr/sbin/modprobe \
--volume lib-modules,kind=host,source=/lib/modules \
--mount volume=lib-modules,target=/lib/modules \
--uuid-file-save=/var/run/kubelet-pod.uuid"
- Installed ceph utility
rbd -v
ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
All rook pods are working but MySQL pod fails with error ‘timeout expired waiting for volumes to attach/mount for pod’
➜ kubectl get pod -n rook-system
NAME READY STATUS RESTARTS AGE
rook-agent-rqw6j 1/1 Running 0 21m
rook-operator-5457d48c94-bhh2z 1/1 Running 0 22m
➜ kubectl get pod -n rook
NAME READY STATUS RESTARTS AGE
rook-api-848df956bf-fhmg2 1/1 Running 0 20m
rook-ceph-mgr0-cfccfd6b8-8brxz 1/1 Running 0 20m
rook-ceph-mon0-xdd77 1/1 Running 0 21m
rook-ceph-mon1-gntgh 1/1 Running 0 20m
rook-ceph-mon2-srmg8 1/1 Running 0 20m
rook-ceph-osd-84wmn 1/1 Running 0 20m
➜ kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-6a4c5c2a-127d-11e8-a846-080027b424ef 20Gi RWO Delete Bound default/mysql-pv-claim rook-block 15m
➜ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-6a4c5c2a-127d-11e8-a846-080027b424ef 20Gi RWO rook-block 15m
kubectl get pods
NAME READY STATUS RESTARTS AGE
wordpress-mysql-557ffc4f69-8zxsq 0/1 ContainerCreating 0 16m
Error when I describe pod : FailedMount Unable to mount volumes for pod “wordpress-mysql-557ffc4f69-8zxsq_default(6a932df1-127d-11e8-a846-080027b424ef)”: timeout expired waiting for volumes to attach/mount for pod “default”/“wordpress-mysql-557ffc4f69-8zxsq”. list of unattached/unmounted volumes=[mysql-persistent-storage]
Also added the following option to rook-operator.yaml
- name: FLEXVOLUME_DIR_PATH
value: "/var/lib/kubelet/volumeplugins"
Could you please help with this? Please let me know if you need further details. I checked the similar issues but a solution is not working.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 1
- Comments: 34 (18 by maintainers)
Hi everyone,
I’ve finally figured out why rook was not working on my setup.
Using the default deploy mode, kubespray reconfigures the location for volume plugin to
/var/lib/kubelet/volume-plugins.However, The Rook agent still deploy the flexvolume plugin in the default location:
/usr/libexec/kubernetes/kubelet-plugins/volume/exec.Thus when trying to mount the volume in the pod, the plugin is not reachable.
I try my rook manifests on a minikube or a kubeadm based kubespray deployment (which uses the default location) and it works as expected.
So it seems that rook does not support k8s cluster deployed with a custom volume plugin directory.
thx @jbw976 for pointing to the kubelet logs, it was the key to my debug.
@aolwas There is documentation available for Rook to change the flevolume plugin path, here: https://rook.io/docs/rook/master/flexvolume.html (see the
FLEXVOLUME_DIR_PATHenv var for therook-operator). After adding the env var to the operator, you need to delete therook-agentDaemonSet and restart the operator that it recreates therook-agentDaemonSet.Let us know if that fixes your issue with the flexvolume plugin path in the
rook-agent.Let’s consider backporting #2068 to 0.8 to resolve this issue
Same error here
PVC and PV creation works fine but got a timeout when mounting the PVC in a pod
We also ran into the same issue on our 1.11.2 test cluster. Same symptoms as described in this issue and restarting the kubelet on the affected node helped.
@laevilgenius I noticed you fixed https://github.com/kubernetes/kubernetes/issues/60694 locally. Can you confirm this is indeed the root cause of this issue?
Has the common issues entry on this topic been useful for anyone yet? there are a few parts of it that may give some guidance: https://rook.io/docs/rook/master/common-problems.html#pod-using-rook-storage-is-not-running
the rook-agent logs and sometimes the kubelet logs for the node that the pod is scheduled on can be especially useful here too.
here’s an example of exact same issue, same setup as @aolwas
K8S v1.9.3 calic pool with replica size: 1 made sure to delete /var/lib/rook after numerous attempts to reinstall