longhorn: [BUG] filesystem corrupted after delete instance-manager-r for a locality best-effort volume

Describe the bug (🐛 if you encounter this issue)

Filesystem corrupted after delete instance-manager-r for a locality best-effort volume.

Try to read file from the recovered volume:

cat: read error: I/O error

To Reproduce

Steps to reproduce the behavior:

1. From Longhorn UI, add tag node-1 to node-1
2. From Longhorn UI, create a volume with 1 replica, data-locality set to best-effort, and tag set to node-1
3. From Longhorn UI, create PV/PVC for the volume
4. Create a deployment with a pod uses the PVC, but set node selector to node-2 for the pod, so it will be scheduled to node-2, and causes failed to schedule to local replica, like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment
  labels:
    name: test-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      name: test-deployment
  template:
    metadata:
      labels:
        name: test-deployment
    spec:
      nodeSelector:
        kubernetes.io/hostname: node-2
      containers:
        - name: test-deployment
          image: nginx:stable-alpine
          volumeMounts:
            - name: test-pod
              mountPath: /data
      volumes:
        - name: test-pod
          persistentVolumeClaim:
            claimName: test-1

1. Wait for the pod to be in running state and write some data and sync filesystem:

/ # dd if=/dev/urandom of=/data/test-1 bs=1M count=256
256+0 records in
256+0 records out
/ # sync
/ # sync /data/test-1
/ # cksum /data/test-1 
1372275509 268435456 /data/test-1

1. Kill the instance-manager-r on node-1
2. Wait for the volume re-attaching and the pod restarting and running
3. Exec into the pod and try to read data, failed with the following error:

/ # cksum /data/test-1
cksum: /data/test-1: I/O error

Expected behavior

A clear and concise description of what you expected to happen.

Log or Support bundle

If applicable, add the Longhorn managers’ log or support bundle when the issue happens. You can generate a Support Bundle using the link at the footer of the Longhorn UI.

Environment

• Longhorn version: master-head
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl):
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version:
  • Number of management node in the cluster:
  • Number of worker node in the cluster:
• Node config
  • OS type and version:
  • CPU per node:
  • Memory per node:
  • Disk type(e.g. SSD/NVMe):
  • Network bandwidth between the nodes:
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
• Number of Longhorn volumes in the cluster:

Additional context

Add any other context about the problem here.