longhorn: [BUG] RWX (ReadWriteMany) pvc stay "Detached", and never attach

Describe the bug

RWO pvc working fine, but RWX (ReadWriteMany) pvc stay “Detached”, and never attach.

To Reproduce

Create Manifest

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-amapi
  namespace: amapi
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: longhorn
---
 apiVersion: v1
 kind: Pod
 metadata:
   name: busybox-rwx
   namespace: amapi
 spec:
   nodeName: kubenodeaz0a83
   containers:
   - name: busybox-rwx
     image: xxx.localrepo/busybox:latest
     command:
       - sleep
       - "3600"
     imagePullPolicy: IfNotPresent
     volumeMounts:
     - name: data-amapi
       mountPath: /data
   volumes:
   - name: data-amapi
     persistentVolumeClaim:
       claimName: test-amapi

FYI , this the storageclass manifest

$>kubectl get sc longhorn -o yaml

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    longhorn.io/last-applied-configmap: |
      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
        name: longhorn
      provisioner: driver.longhorn.io
      allowVolumeExpansion: true
      reclaimPolicy: Delete
      volumeBindingMode: Immediate
      parameters:
        fsType: "ext4"
        numberOfReplicas: "2"
        staleReplicaTimeout: "2880"
        fromBackup: ""
  creationTimestamp: "2022-01-20T15:16:55Z"
  name: longhorn
  resourceVersion: "2277273785"
  uid: 46eb093f-ceac-4c85-9ee6-66fb3798e795
parameters:
  fromBackup: ""
  fsType: ext4
  numberOfReplicas: "2"
  staleReplicaTimeout: "2880"
provisioner: driver.longhorn.io
reclaimPolicy: Delete
volumeBindingMode: Immediate

Expected behavior

A pod is expected to be launched with its PVC.

Log or Support bundle

$> kubectl logs -f longhorn-manager-htvjq -n longhorn-system

time="2022-03-15T13:46:17Z" level=debug msg="Can't find share manager, may have been deleted" controller=longhorn-share-manager node=kubenodeaz0a83 shareManager=pvc-777149ba-7f9e-41d5-a351-ef2551c9b06f
time="2022-03-15T13:46:17Z" level=debug msg="Can't find share manager, may have been deleted" controller=longhorn-share-manager node=kubenodeaz0a83 shareManager=pvc-777149ba-7f9e-41d5-a351-ef2551c9b06f

Event log of the pod

AttachVolume.Attach failed for volume "pvc-777149ba-7f9e-41d5-a351-ef2551c9b06f" : rpc error: code = DeadlineExceeded desc = volume pvc-777149ba-7f9e-41d5-a351-ef2551c9b06f failed to attach to node kubenodeaz0a83

share-manager pod is never created, without any error message

Environment

• Longhorn version: 1.2.3
• Installation method : Kubectl
• Kubernetes distro : On Premise version 1.20.6
  • Number of management node in the cluster: 5
  • Number of worker node in the cluster: 100+
• Node config
  • OS type and version: centos 7
  • CPU per node: 8
  • Memory per node: 64
  • Disk type(e.g. SSD/NVMe): SSD
  • Network bandwidth between the nodes: 10Gbs
• Underlying Infrastructure : OpenStack
• Number of Longhorn volumes in the cluster: 10

Aditionnal context

When i look on the longhorn webui, volume il created, but “Detached”