longhorn: [BUG] Persistent volume is not ready for workloads

• Log: https://github.com/longhorn/longhorn/issues/6776#issuecomment-1734829109
• Problematic volume: pvc-506e824d-414f-43ce-af59-5821b2b9accf
• Initial Statue
  • volume pvc-506e824d-414f-43ce-af59-5821b2b9accf
    • status.state: detached
  • sharemanager share-manager-pvc-506e824d-414f-43ce-af59-5821b2b9accf
    • status.state = stopped
    • status.ownerID = ip-x-x-x-57

When the volume is requested to attach to node 142. In the meantime, the volume is triggered to detach from 57. The newly created replicas are mistakenly deleted by the detachment operation.

The script repeatedly attaches and detaches a pod using the share manager.

#!/bin/bash

ATTACH_WAIT_SECONDS=120
DETACH_WAIT_SECONDS=300

for ((i=0;i<200;i++))
do
        node=$((i%3+1))
        echo "Iteration #$i at `date` node ${node}"

				# Change sharemanager owner ID and trigger volumeattachment to request a new node
        status="status: {ownerID:  rancher60-worker${node}}"
        kubectl -n longhorn-system patch sharemanager pvc-1418f411-ccad-4df9-a66a-050cb80d4645  --type=merge --subresource status --patch "$status"

        kubectl apply -f pod1.yaml

        echo "Attaching"
        c=0
        while [ $c -lt $ATTACH_WAIT_SECONDS ]
        do
                phase=`kubectl get pod volume-test-1 -o=jsonpath="{['status.phase']}"  2>/dev/null`
                if [ x"$phase" == x"Running" ]; then
                        break
                fi

                sleep 1

                c=$((c+1))
                if [ x"$c" = x"$ATTACH_WAIT_SECONDS" ]; then
                        echo "Found error"
                        exit 1
                fi
        done

        echo "Detaching"
        kubectl delete -f pod1.yaml

        while [ $c -lt $DETACH_WAIT_SECONDS ]
        do
                phase=`kubectl -n longhorn-system get volumes pvc-1418f411-ccad-4df9-a66a-050cb80d4645 -o=jsonpath="{['status.state']}" 2>/dev/null`
                if [ x"$phase" == x"detached" ]; then
                        break
                fi

                sleep 1

                if [ x"$c" = x"DETACH_WAIT_SECONDS" ]; then
                        echo "Failed to detach"
                        exit 1
                fi
        done
done

• sc_pvc.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-test
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "2880"
  fromBackup: ""
  fsType: "ext4"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-volv-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: longhorn-test
  resources:
    requests:
      storage: 2Gi

• pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: volume-test-1
  namespace: default
spec:
  restartPolicy: Always
  containers:
  - name: volume-test-1
    image: ubuntu:20.04
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: longhorn-volv
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: longhorn-volv
    persistentVolumeClaim:
      claimName: longhorn-volv-pvc

Currently, I cannot reproduce the faulty volume issue (all replicas are mistakenly deleted). But, from the test, we can sometimes observe one of the attaching volume’s replica instances is mistakenly deleted, so deleting all replicas mistakenly is possible.

• What we can improve
  • Improve the log in share manager controller and volume_attachment_controller for help debugging process
  • Make sure there is no on-going replica deletion operation before attaching a volume
  • Starting a share manger in current design is lengthy. Is it possible to simply the flow?
    • share manager is picked by the status.ownerID
    • volume is requested to attach to status.ownerID
    • share manager controller creates a share manager pod
    • If the pod is created on another node, share-manager ownership is transferred to another node
    • volume is requested to detach
    • volume is requested to attach

@james-munson Can you help take a look on this issue and see if I missed something? If possible, see if we can have a fix on the race issue.

cc @shuo-wu @innobead

As we discussed last time, this part is problematic, which may lead to unexpected and unnecessary detachment after a temporary node being unavailable (kubelet/network down). In fact, keeping volume.Spec.NodeID the same as ShareManager.Status.OwnerID is unnecessary.

The share-manager-controller workflow can be like the following:

1. Start the share manager pod scheduled by Kubernetes.
2. Set the volume attachment ticket to the pod node
3. Unset the volume attachment ticket when the volume or the share manager pod is error or stopping/stopped

Does the CI/CD process involve restarting nodes or pods? The share manager recovery backend logs that it is removing NFS client entries on multiple occasions.

Each step in pipeline is a separated pod with shared longhorn volume. It means that if you have 10 steps then you think about them like 10 pods with mounted shared longhorn volume.

@ajoskowski Could we have:

• the yaml output of kubectl get volumes.longhorn.io,volumeattachments.longhorn.io,engines.longhorn.io,replicas.longhorn.io -n longhorn-system -oyaml
• logs from longhorn-csi-plugin-xxx pods in longhorn-system namespace
• Is the workload pod ci-state-pr-2463-env-doaks-prod6-uaenorth-v2wnw-override-refs-in-tf-modules-407269516 stuck right now?

• Regarding kubectl get volumes.longhorn.io,volumeattachments.longhorn.io,engines.longhorn.io,replicas.longhorn.io -n longhorn-system -oyaml - no problem, but I am able to do it for current cluster shape (problematic volume was already removed) - volumes_attachments_engines_replicas.log
• logs from longhorn-csi-plugin-xxx pods in longhorn-system namespace - I do not see any logs which have information about pvc-506e824d-414f-43ce-af59-5821b2b9accf
• Pod ci-state-pr-2463-env-doaks-prod6-uaenorth-v2wnw-override-refs-in-tf-modules-407269516 was not able to start due to problem which I’ve described in the bug description and it was removed

In an effort to reproduce, do you think the following steps similar to your CI pipeline @ajoskowski ?

1. Create a RWX PVC
2. Create a deployment using the PVC
3. Repeatedly quickly scale up and down the deplooyment from 0 to 5 and back to 0
4. Verify if any pod stuck to come up

Yeah, it’s similar. On our side we create new pod definitions (new steps) instead of scaling of deployment but logic is the same - creating and removing pods with mounting/unmounting longhorn volume.

The question is it valid for Longhorn? Should it support such cases? We use longhorn ~2 years in this way and we didn’t observer such problems in the past.

Got it. Longhorn introduces a new Attachment/Detachment mechanism since v1.5.0. Not sure if it is related and still under investigation. Ref: https://github.com/longhorn/longhorn/issues/3715

cc @PhanLe1010

Thanks guys, we will verify this fix in 1.5.4

@roger-ryao, that looks good. I would be up for building a 1.5.1-based private build (this fix is also being backported to 1.5.4) if @ajoskowski would be up for pre-testing it before 1.5.4 releases.

I think this is a good candidate for backport to 1.4 and 1.5. @innobead do you agree?

I also tried a repro with @Phan’s idea of using a deployment with an RWX volume, and scaling it up and down quickly. Specifically, used the rwx example from https://github.com/longhorn/longhorn/examples/rwx/rwx-nginx-deployment.yaml, although I modified the container slightly to do this

      containers:
        - image: ubuntu:xenial
          imagePullPolicy: IfNotPresent
          command: [ "/bin/sh", "-c" ]
          args:
            - sleep 10; touch /data/index.html; while true; do  echo "`hostname` `date`" >> /data/index.html; sleep 1; done;

to include the hostname in the periodic writes to the shared volume.

Even with scaling up to 3 and down to 0 at 10-second intervals (far faster than the attach and detach can be accomplished) no containers crashed, and no replicas were broken. Kubernetes and Longhorn are untroubled by the fact that creating and terminating resources overlap. In fact, I revised the time after scale up to 60 seconds, and the time after scale down to 0, so new pods were created immediately, and that just had the effect of attaching and writing from the new pods while the old ones were still detaching. So for some interval, there were 6 pods writing to the volume without trouble. I conclude from that test that this is likely not a representative repro of the situation in this issue.

Thanks @ajoskowski ! The provided yaml https://github.com/longhorn/longhorn/files/12734412/volumes_attachments_engines_replicas.log doesn’t have anything abnormal as it is taken when the problem doesn’t exist.

We will try to reproduce the issue in lab.

Longhorn manager logs from all (3) instances: all-longhorn-manager.log

@ajoskowski Sorry, I got confused when reading the log file. Does the all-longhorn-manager.log include all longhorn-manager pods’ logs or only one longhorn-manager pod?

All logs from all instances of longhorn-managers - it means set of logs from 3 instances. Do you need to have it in separated files?

Yeah, some messages are mixed together. Can you help provide separate files? Thank you.