longhorn: [BUG] GET error for volume attachment on node reboot

@diamonwiggins @hedefalk @simonreddy2001 @Orhideous @rajivml I tried to reproduce the issue using Longhorn v1.3.2 and a StatefulSet with 2 replicas on a 2-node cluster. Reboot the two nodes repeatedly, but still cannot reproduce the issue.

Could you please provide the reproducing steps? If you run into the issue again, could you provide a support bundle as well? Thanks.

Hi I have same issue

MountVolume.WaitForAttach failed for volume “pvc-xx” : volume vol-xx has GET error for volume attachment csi-xx: volumeattachments.storage.k8s.io “csi-xx” is forbidden: User “system:node:ip-xx.compute.internal” cannot get resource “volumeattachments” in API group “storage.k8s.io” at the cluster scope: no relationship found between node ‘ip-xx.compute.internal’ and this object

But we scaledown the statefulset replicas to 0 and scale it back such that volume attachment flow gets triggered again

Next action

Even though we are not able to reproduce the upstream issue, from code analysis, I do think that the race condition in the upstream issue COULD be the root cause of this ticket. The upstream issue is fixed in:

• Kubernetes v1.26 line: >= v1.26.10
• Kubernetes v1.27 line: >= v1.27.7
• Kubernetes v1.28 line:>= v1.28.3
• Kubernetes v1.29 line:>= v1.29.0

Therefore, I think the next step for this ticket would be:

1. Ask the user to try with these fixed Kubernetes versions to see if the issue still persists. (cc @diamonwiggins Could you try to upgrade Kubenetes to the fixed versions?)
2. Close this ticket
3. If the user still hits the issue after upgrading Kubenetes to the fixed versions, we can reopen the ticket

WDYT @derekbit @innobead @ejweber ?

Workaround:

Additionally, from code analysis, I think the workaround may be to scale down the workload, wait for the workload to be fully terminated, then scale back the workload again. Kube-controller-manager should be able to recreate the VolumeAttachment for the new pod

HI @PhanLe1010

We are seeing it on both single node and multi-node environments

I will share an environment for your offline analysis via DM over slack

Longhorn version: 1.3.1 Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: RKE2 Number of management node in the cluster: 3 nodes which acts as both master + worker Number of worker node in the cluster: 3 nodes which acts as both master + worker Node config: 32 Core 128GB RAM OS type and version: RHEL CPU per node: 32 Memory per node: 128GB RAM Disk type(e.g. SSD/NVMe): SSD Network bandwidth between the nodes: Azure Provided Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Azure Disks Number of Longhorn volumes in the cluster: Around 20

we see this issue so often with longhorn and today also we had a repro where on node restart in a multi-node environment alertmanager statefulset related pods were not able to mount PVCs even after 30-40 minutes and we see this issue with both deployments and statefulsets

This is happening with longhorn 1.3.1 also and this repro is with longhorn 1.3.1 itself

When ever this happens we scaledown the workload replicas to 0 and scale it back such that volume attachment flow gets triggered again but this is not an acceptable solution while running production workloads

       {
            "apiVersion": "v1",
            "count": 60,
            "eventTime": null,
            "firstTimestamp": "2022-10-12T06:05:07Z",
            "involvedObject": {
                "apiVersion": "v1",
                "kind": "Pod",
                "name": "alertmanager-rancher-monitoring-alertmanager-1",
                "namespace": "cattle-monitoring-system",
                "resourceVersion": "83537",
                "uid": "2088fca6-b6cb-458f-8297-44fa477b0e81"
            },
            "kind": "Event",
            "lastTimestamp": "2022-10-12T07:50:59Z",
            "message": "MountVolume.WaitForAttach failed for volume \"pvc-84933541-a66d-4ca2-a710-6db17e6643ba\" : volume pvc-84933541-a66d-4ca2-a710-6db17e6643ba has GET error for volume attachment csi-0c400de43ff27c65fa12afab1248675317dbb2b8fc07ae6582df5ce218fa6ff7: volumeattachments.storage.k8s.io \"csi-0c400de43ff27c65fa12afab1248675317dbb2b8fc07ae6582df5ce218fa6ff7\" is forbidden: User \"system:node:server1\" cannot get resource \"volumeattachments\" in API group \"storage.k8s.io\" at the cluster scope: no relationship found between node 'server1' and this object",
            "metadata": {
                "creationTimestamp": "2022-10-12T06:05:07Z",
                "name": "alertmanager-rancher-monitoring-alertmanager-1.171d3d2e89354c2e",
                "namespace": "cattle-monitoring-system",
                "resourceVersion": "167371",
                "uid": "1874d80b-43e9-4242-9df3-bc39b68c0cc1"
            },
            "reason": "FailedMount",
            "reportingComponent": "",
            "reportingInstance": "",
            "source": {
                "component": "kubelet",
                "host": "server1"
            },
            "type": "Warning"
        },

@PhanLe1010 The node went down for only minutes. Maybe 5 minutes or so. However it was a full 24 hours before the user manually created the VolumeAttachment objects.

Hi @diamonwiggins, ref:issues-2629 Did it ‘FailedMount’ happen for a long time before you created the volumeattachment object?