longhorn: [BUG] MountDevice failed while expanding volume

Describe the bug

After a while my pod dies and does not start again. I get the following events

MountVolume. for volume "pvc-6911563e-37e6-4e08-a857-11267216d408" : Expander.NodeExpand found CSI plugin kubernetes.io/csi/driver.longhorn.io to not support node expansion
Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-pxgzh]: timed out waiting for the condition
Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kube-api-access-pxgzh data]: timed out waiting for the condition
MountVolume.MountDevice failed for volume "pvc-6911563e-37e6-4e08-a857-11267216d408" : rpc error: code = InvalidArgument desc = volume pvc-6911563e-37e6-4e08-a857-11267216d408 hasn't been attached yet
Successfully assigned production/document-management-api-mongodb-56fbbbd46b-rjxxc to 89.58.1.150

To Reproduce

Expected behavior

Stable Storages

Log or Support bundle

If applicable, add the Longhorn managers’ log or support bundle when the issue happens. You can generate a Support Bundle using the link at the footer of the Longhorn UI.

Environment

• Longhorn version: 1.2.3
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Kubectl
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: Server Version: version.Info{Major:“1”, Minor:“21”, GitVersion:“v1.21.7”, GitCommit:“1f86634ff08f37e54e8bfcd86bc90b61c98f84d4”, GitTreeState:“clean”, BuildDate:“2021-11-17T14:35:38Z”, GoVersion:“go1.16.10”, Compiler:“gc”, Platform:“linux/amd64”}
  • Number of management node in the cluster: 1
  • Number of worker node in the cluster: 2
• Node config
  • OS type and version:
  • CPU per node: 4
  • Memory per node: 12
  • Disk type(e.g. SSD/NVMe): SSD
  • Network bandwidth between the nodes: 500mbit vlan ( Hoster: netcup )
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): KVM
• Number of Longhorn volumes in the cluster: 16