azurefile-csi-driver: Randomly fails to mount Azure Fileshare volume

What happened: Randomly fails to mount Azure Fileshare. Pod stays in ContainerCreating state forever. Describing the Pod, found the following:

Type Reason Age From Message

---

Warning FailedMount 8m23s (x103 over 3h54m) kubelet, aks-npbalanced-14900033-vmss000001 MountVolume.WaitForAttach failed for volume “pvc-b3e955ca-e3a6-4bcf-846b-98234b22ac2b” : volume attachment is being deleted Warning FailedMount 3m3s (x79 over 3h54m) kubelet, aks-npbalanced-14900033-vmss000001 Unable to attach or mount volumes: unmounted volumes=[storage-volume], unattached volumes=[storage-volume default-token-6w4nx]: timed out waiting for the condition Warning FailedAttachVolume 2m15s (x115 over 3h54m) attachdetach-controller AttachVolume.Attach failed for volume “pvc-4a9462d2-17f4-4c32-9178-706b5b91f7bb” : volume attachment is being deleted

When I delete the pod and a new pod is scheduled, this problem goes away and the volume is mounted. If a deployment has 2 replicas, then one pod runs without any issue but the other pod is stuck with this issue most of the time. Following is a portion of log from kubectl logs -n kube-system csi-azurefile-node-d459v -c azurefile

I1014 07:56:12.787162 1 utils.go:111] GRPC call: /csi.v1.Identity/Probe I1014 07:56:12.787190 1 utils.go:112] GRPC request: I1014 07:56:12.787200 1 utils.go:118] GRPC response: ready:<value:true > I1014 07:56:38.842570 1 utils.go:111] GRPC call: /csi.v1.Node/NodeUnpublishVolume I1014 07:56:38.842621 1 utils.go:112] GRPC request: volume_id:“plt-dev-euw-rg#pltfilestoragedev#collaboration#” target_path:“/var/lib/kubelet/pods/cb0633b5-d18f-44b8-b89d-5ab9d8a9932f/volumes/kubernetes.io~csi/pvc-b3e955ca-e3a6-4bcf-846b-98234b22ac2b/mount” I1014 07:56:38.842635 1 nodeserver.go:92] NodeUnPublishVolume: called with args {VolumeId:plt-dev-euw-rg#pltfilestoragedev#collaboration# TargetPath:/var/lib/kubelet/pods/cb0633b5-d18f-44b8-b89d-5ab9d8a9932f/volumes/kubernetes.io~csi/pvc-b3e955ca-e3a6-4bcf-846b-98234b22ac2b/mount XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} I1014 07:56:38.842656 1 nodeserver.go:102] NodeUnpublishVolume: unmounting volume plt-dev-euw-rg#pltfilestoragedev#collaboration# on /var/lib/kubelet/pods/cb0633b5-d18f-44b8-b89d-5ab9d8a9932f/volumes/kubernetes.io~csi/pvc-b3e955ca-e3a6-4bcf-846b-98234b22ac2b/mount I1014 07:56:38.855715 1 mount_helper_common.go:71] “/var/lib/kubelet/pods/cb0633b5-d18f-44b8-b89d-5ab9d8a9932f/volumes/kubernetes.io~csi/pvc-b3e955ca-e3a6-4bcf-846b-98234b22ac2b/mount” is a mountpoint, unmounting I1014 07:56:38.855752 1 mount_linux.go:238] Unmounting /var/lib/kubelet/pods/cb0633b5-d18f-44b8-b89d-5ab9d8a9932f/volumes/kubernetes.io~csi/pvc-b3e955ca-e3a6-4bcf-846b-98234b22ac2b/mount I1014 07:56:38.890792 1 mount_helper_common.go:85] “/var/lib/kubelet/pods/cb0633b5-d18f-44b8-b89d-5ab9d8a9932f/volumes/kubernetes.io~csi/pvc-b3e955ca-e3a6-4bcf-846b-98234b22ac2b/mount” is unmounted, deleting the directory I1014 07:56:38.890877 1 nodeserver.go:107] NodeUnpublishVolume: unmount volume plt-dev-euw-rg#pltfilestoragedev#collaboration# on /var/lib/kubelet/pods/cb0633b5-d18f-44b8-b89d-5ab9d8a9932f/volumes/kubernetes.io~csi/pvc-b3e955ca-e3a6-4bcf-846b-98234b22ac2b/mount successfully

What you expected to happen: Azure fileshare should have been mounted successfully always. How to reproduce it:

Anything else we need to know?:

Environment:

• CSI Driver version: v0.8.0
• Kubernetes version (use kubectl version): 1.17.7