longhorn: [BUG] FailedAttachVolume - cannot be attached to the node A since it is already attached to the node B

I want to report a problem I have with Longhorn from the early beginning and even I have build up complete new Kuberntes clusters I get not rid of the problem.

The situation:

• running a cluster with 3 worker nodes and 1 master
• running a POD with a durable longhorn volume
• Now one of my nodes failed (e.g. a network problem)
• after 5 minutes kubernetes restarts the POD on a different node
• the new pod gets stuck in ContainerCreating state

I am running Longhorn 1.1.0 with the extra settings:

• Automatically Delete Workload Pod when The Volume Is Detached Unexpectedly = TRUE
• Volume Attachment Recovery Policy = immediate

From the official documentation I would expect that the pod and the volume will be rescheduled. But this did nto work. I have permanently the error message:

 Warning  FailedAttachVolume  117s (x16 over 32m)  attachdetach-controller  AttachVolume.Attach failed for volume "my-index" : rpc error: code = FailedPrecondition desc = The volume my-index cannot be attached to the node worker-5 since it is already attached to the node worker-6                                                                              │
 Warning  FailedMount         78s (x10 over 30m)   kubelet, worker-5  Unable to attach or mount volumes: unmounted volumes=[index], unattached volumes=[index default-token-cntp8]: timed out waiting for the condition  

Even if I try to kill the POD it will not solve the probem.

When I delete my complete deployment ($ kubectl delete ....) the affected volume is still attached.

The command “kubectl get pv” did no longer list the volume. But in the Longhorn UI I can still see the volume (as it is durable) and I can see that it is attached (!) with the following status details:

State: attached
Health: healthy
Ready for workload:Ready
Conditions:
restore
scheduled
Frontend:Block Device
Attached Node & Endpoint:
tikal-worker-6
/dev/longhorn/my-index
Size:
1 Gi
Actual Size:49.1 Mi
Data Locality:disabled
Access Mode:ReadWriteOnce
Base Image:
Engine Image:longhornio/longhorn-engine:v1.1.0
Created:2 months ago
Node Tags:
Disk Tags:
Last Backup:
Last Backup At:
Instance Manager:
instance-manager-e-fc27e69c
Last time bound with PVC:5 minutes ago
Last time used by Pod:5 minutes ago
Last Namespace:xxxxx
Last Bounded PVC Name:index
PV Name:
PV Status:
Revision Counter:False
Last Pod Name:imixs-office-workflow-c8dcc469f-5jn9r
Last Workload Name:imixs-office-workflow-c8dcc469f
Last Workload Type:ReplicaSet

When I try to recreate my deployment with kubectl apply… I am still running into the same situation.

The only solution for me is to:

• delete the POD manauell (kubectl delete …)
• go into the Longhonr UI and detach the volume manually
• redeploy the pod manually (kubectl apply …)

Now longhorn will re-attach the volume and the POD starts and everything works fine again.

But my question is: Why I am running permanently into this awful situation? What could be the reason? What would be the expected behaviour?

The only part I found in my setup which is not official documented by Longhorn is my Storage Class I use:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-durable    
provisioner: driver.longhorn.io
reclaimPolicy: Retain
allowVolumeExpansion: true
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "2880" # 48 hours in minutes
  fromBackup: ""

Is there something wrong with this storage class?