rook: failed to mount PVC to pod using cephfs-csi

Is this a bug report or feature request?

• Bug Report failed to attach pvc to pods

Deviation from expected behavior: Expected behavior: Mounting of PVC to pod should be successful

How to reproduce it (minimal and precise):

• deploy ceph-csi using operator
• create PVC using cephfs-csi
• mount PVC to pod

Environment:

• OS (e.g. from /etc/os-release):

• Kernel (e.g. uname -a):

• Cloud provider or hardware configuration:

• Rook version (use rook version inside of a Rook Pod):

• Kubernetes version (use kubectl version): 1.13.0

• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):

• Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

• pod descirbe output

QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason              Age                   From                     Message
  ----     ------              ----                  ----                     -------
  Normal   Scheduled           9m58s                 default-scheduler        Successfully assigned default/csicephfs-demo-pod to worker2
  Warning  FailedMount         70s (x4 over 7m55s)   kubelet, worker2         Unable to mount volumes for pod "csicephfs-demo-pod_default(6d2a6439-5f45-11e9-be2b-5254007e23ff)": timeout expired waiting for volumes to attach or mount for pod "default"/"csicephfs-demo-pod". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-p8slv]
  Warning  FailedAttachVolume  61s (x11 over 9m43s)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-5ebbb9b5-5f45-11e9-be2b-5254007e23ff" : attachment timeout for volume csi-cephfs-pvc-5ebbb9b5-5f45-11e9-be2b-5254007e23ff

logs from kubectl

[root@worker2 vagrant]# journalctl -xe|grep mypvc
Apr 15 06:16:37 worker2 kubelet[3835]: E0415 06:16:37.962016    3835 kubelet.go:1680] Unable to mount volumes for pod "csicephfs-demo-pod_default(6d2a6439-5f45-11e9-be2b-5254007e23ff)": timeout expired waiting for volumes to attach or mount for pod "default"/"csicephfs-demo-pod". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-p8slv]; skipping pod
Apr 15 06:16:37 worker2 kubelet[3835]: E0415 06:16:37.962103    3835 pod_workers.go:190] Error syncing pod 6d2a6439-5f45-11e9-be2b-5254007e23ff ("csicephfs-demo-pod_default(6d2a6439-5f45-11e9-be2b-5254007e23ff)"), skipping: timeout expired waiting for volumes to attach or mount for pod "default"/"csicephfs-demo-pod". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-p8slv]
Apr 15 06:18:53 worker2 kubelet[3835]: E0415 06:18:53.966157    3835 kubelet.go:1680] Unable to mount volumes for pod "csicephfs-demo-pod_default(6d2a6439-5f45-11e9-be2b-5254007e23ff)": timeout expired waiting for volumes to attach or mount for pod "default"/"csicephfs-demo-pod". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-p8slv]; skipping pod
Apr 15 06:18:53 worker2 kubelet[3835]: E0415 06:18:53.966196    3835 pod_workers.go:190] Error syncing pod 6d2a6439-5f45-11e9-be2b-5254007e23ff ("csicephfs-demo-pod_default(6d2a6439-5f45-11e9-be2b-5254007e23ff)"), skipping: timeout expired waiting for volumes to attach or mount for pod "default"/"csicephfs-demo-pod". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-p8slv]
Apr 15 06:21:08 worker2 kubelet[3835]: E0415 06:21:08.981803    3835 kubelet.go:1680] Unable to mount volumes for pod "csicephfs-demo-pod_default(6d2a6439-5f45-11e9-be2b-5254007e23ff)": timeout expired waiting for volumes to attach or mount for pod "default"/"csicephfs-demo-pod". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-p8slv]; skipping pod
Apr 15 06:21:08 worker2 kubelet[3835]: E0415 06:21:08.981844    3835 pod_workers.go:190] Error syncing pod 6d2a6439-5f45-11e9-be2b-5254007e23ff ("csicephfs-demo-pod_default(6d2a6439-5f45-11e9-be2b-5254007e23ff)"), skipping: timeout expired waiting for volumes to attach or mount for pod "default"/"csicephfs-demo-pod". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-p8slv]
Apr 15 06:23:24 worker2 kubelet[3835]: E0415 06:23:24.966423    3835 kubelet.go:1680] Unable to mount volumes for pod "csicephfs-demo-pod_default(6d2a6439-5f45-11e9-be2b-5254007e23ff)": timeout expired waiting for volumes to attach or mount for pod "default"/"csicephfs-demo-pod". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-p8slv]; skipping pod
Apr 15 06:23:24 worker2 kubelet[3835]: E0415 06:23:24.966443    3835 pod_workers.go:190] Error syncing pod 6d2a6439-5f45-11e9-be2b-5254007e23ff ("csicephfs-demo-pod_default(6d2a6439-5f45-11e9-be2b-5254007e23ff)"), skipping: timeout expired waiting for volumes to attach or mount for pod "default"/"csicephfs-demo-pod". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-p8slv]
Apr 15 06:25:41 worker2 kubelet[3835]: E0415 06:25:41.975890    3835 kubelet.go:1680] Unable to mount volumes for pod "csicephfs-demo-pod_default(6d2a6439-5f45-11e9-be2b-5254007e23ff)": timeout expired waiting for volumes to attach or mount for pod "default"/"csicephfs-demo-pod". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-p8slv]; skipping pod
Apr 15 06:25:41 worker2 kubelet[3835]: E0415 06:25:41.975933    3835 pod_workers.go:190] Error syncing pod 6d2a6439-5f45-11e9-be2b-5254007e23ff ("csicephfs-demo-pod_default(6d2a6439-5f45-11e9-be2b-5254007e23ff)"), skipping: timeout expired waiting for volumes to attach or mount for pod "default"/"csicephfs-demo-pod". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-p8slv]
Apr 15 06:27:58 worker2 kubelet[3835]: E0415 06:27:58.964396    3835 kubelet.go:1680] Unable to mount volumes for pod "csicephfs-demo-pod_default(6d2a6439-5f45-11e9-be2b-5254007e23ff)": timeout expired waiting for volumes to attach or mount for pod "default"/"csicephfs-demo-pod". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-p8slv]; skipping pod
Apr 15 06:27:58 worker2 kubelet[3835]: E0415 06:27:58.964484    3835 pod_workers.go:190] Error syncing pod 6d2a6439-5f45-11e9-be2b-5254007e23ff ("csicephfs-demo-pod_default(6d2a6439-5f45-11e9-be2b-5254007e23ff)"), skipping: timeout expired waiting for volumes to attach or mount for pod "default"/"csicephfs-demo-pod". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-p8slv]

@rootfs @travisn