vsphere-csi-driver: FailedMount - FailedPrecondition - Volume does not appear staged

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened: While deleing and (re-)applying the kubernetes statefulset configuration. Some of the pods that use vSphere CSI based persistent volumes are failing start and are stuck at ContainerCreating state. This happens randomly to some pods. There are other pods which also use vSphere CSI based PV and are also scheduled on the same worker node which starts without any issues.

Currently, we have 2 pods in this failed state. Here is the link to the Gist containing related k8s objects describe output and vSphere CSI logs: https://gist.github.com/mohideen/3b0cfa29f03d10fb7e1170f5729cb528

What you expected to happen: The pods should start successfully with the corresponding PV mounted on it.

How to reproduce it (as minimally and precisely as possible):

1. Create a PVC with using a vSphere storage class.
2. Create a Statefulset and configure it use the above PVC
3. Verify that the pods has the volume mounted to it.
4. Delete the Statefulset
5. Redeploy the same Statefulset
6. Check the status of the pod
   • The pod usually starts correctly.
   • On some random occasion, the pod fails to start and it is stuck in ContainerCreating state with the message that MountVolume.SetUp failed for volume (Note: This has happened only thrice so far in the last month)

Anything else we need to know?:

Environment:

• csi-vsphere version: v2.0.0
• vsphere-cloud-controller-manager version: v2.0.0
• Kubernetes version: 1.18.5
• vSphere version: 6.7U3
• OS (e.g. from /etc/os-release): Red Hat Enterprise Linux Server 7.8 (Maipo)
• Kernel (e.g. uname -a): 3.10.0-1127.18.2.el7.x86_64
• Install tools: kubeadm
• Others: 6 Node (3 Controller and 3 Worker) cluster