longhorn: [BUG] PVC only works with plural annotation `volumes.kubernetes.io/storage-provisioner: driver.longhorn.io`

Describe the bug (🐛 if you encounter this issue)

When I create a PVC (for example by an restore or manually create one), the PVCs are created but get stuck in the Pending state and the PV will never be created. I noticed this bug by try to restore PVC from an Velero backup.

To Reproduce

Steps to reproduce the behavior:

1. Create a PVC, the following example that dind’t work (stuck in Pending):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.kubernetes.io/storage-provisioner: driver.longhorn.io
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: harbor
  name: harbor-registry-manual
  namespace: harbor
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: longhorn
  volumeMode: Filesystem

-> But if I change the annotation volume.kubernetes.io/storage-provisioner: driver.longhorn.io to volumes.kubernetes.io/storage-provisioner: driver.longhorn.io then it works (the PVC changes its state to Bound).

Expected behavior

After a short time, the PV will be created and the PVC changes its state to Bound.

Environment

• Longhorn version: v1.3.2
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Rancher Helm Chart
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: RKE2, v1.24.4+rke2r1
  • Number of management node in the cluster: 4
  • Number of worker node in the cluster: 3
• Node config
  • OS type and version: Ubuntu
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Proxmox
• Number of Longhorn volumes in the cluster: 18