longhorn: [BUG] RWX volume is stuck at detaching when the attached node is down

Describe the bug (🐛 if you encounter this issue)

The RWX volume is stuck at detaching when the attached node is down. The root cause is the side effect of the fix https://github.com/longhorn/longhorn/issues/5507. To prevent that nfs-ganesha sever mounts the dead volume, the engine controller tries to delete the engine instance. If the attached node is down and does not come back, the deletion won’t succeed and then retry again and again. Then, the volume will be stuck at detaching state.

To Reproduce

Steps to reproduce the behavior:

1. Go to ‘…’
2. Click on ‘…’
3. Perform ‘…’
4. See error

Expected behavior

A clear and concise description of what you expected to happen.

Log or Support bundle

If applicable, add the Longhorn managers’ log or support bundle when the issue happens. You can generate a Support Bundle using the link at the footer of the Longhorn UI.

Environment

• Longhorn version: v1.4.1
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl):
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version:
  • Number of management node in the cluster:
  • Number of worker node in the cluster:
• Node config
  • OS type and version:
  • CPU per node:
  • Memory per node:
  • Disk type(e.g. SSD/NVMe):
  • Network bandwidth between the nodes:
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
• Number of Longhorn volumes in the cluster:

Workaround

• Delete the down node or
• Delete the unknown instancemanager of the down node
  • list the instancemanager by

    kubectl -n longhorn-system get instancemanagers
    

    Then, you can see the unknown instance-manager-e-xxx which is on the down node and delete it .

Additional context

Add any other context about the problem here.

https://cloud-native.slack.com/archives/CNVPEL9U3/p1678784583120039