longhorn: [BUG] Listing Backups in "Backup" tab fails

Describe the bug

When I click on “Backup” in the Longhorn dashboard. A toast shows up with the following error message:

error listing backup volume names: timeout executing: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.3.0/longhorn [backup ls --volume-only nfs://<IP-OF-MY-NAS>:/Longhorn], output , stderr, , error <nil>

Backups are successfully stored on the NAS in that nfs volume ‘Longhorn’ and folder ‘backupstore’. But do not show up in the Longhorn dashboard.

To Reproduce

Steps to reproduce the behavior:

Go to ‘Settings’
Scroll down to ‘Backup Target’
Insert the backup target ‘nfs://<IP-OF-MY-NAS>:/Longhorn’
Click on ‘Save’
Go to ‘Backup’ tab
See error

Expected behavior

All created backups should be listed

Log or Support bundle

longhorn-support-bundle_2583f89e-dc70-45d0-abf7-3af7093064b3_2022-07-03T17-03-17Z.zip

Environment

Longhorn version: 1.3.0
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm (flux)
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: K3s v1.24.2+k3s1
- Number of management node in the cluster: 3
- Number of worker node in the cluster: 5
Node config
- OS type and version: Ubuntu 22.04 LTS
- CPU per node: 4
- Memory per node: 4 GB
- Disk type(e.g. SSD/NVMe): Samsung 256 GB 400 MB/s USB 3.1 Flash Drive
- Network bandwidth between the nodes:
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal (Raspberry Pi 4B)
Number of Longhorn volumes in the cluster: 1 with 3 replica

Additional context

Add any other context about the problem here.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 16 (9 by maintainers)

Most upvoted comments

@derekbit it did (at least with Ubuntu 21.10)

I stumbled on a similar issue on Ubuntu 22.04 LTS with the nfs-provisioner. I had to add additional mount options:

    nfs:
      mountOptions:
      - nfsvers=4.0

Only with that the nfs-provisioner was able to work on my cluster.

untcha on Jul 4, 2022

Probably because “nfsvers” was changed to “mountvers” in Ubuntu 22.04 nfs-common.

derekbit on Jul 4, 2022

You can copy the backup folder on the QNAP NAS to another machine. Then, export the folder via NFS protocol and set the new path to the Longhorn backup target.

derekbit on Jul 5, 2022

/var/lib/longhorn-backupstore-mounts/192_168_178_110/Longhorn already exists. You can check if it’s already mounted. If it’s empty, you can delete it and try again.

derekbit on Jul 5, 2022

Got it. I can reproduce this using my own QNAP NAS. Mounting the NAS folder by the following command on Ubuntu 22.04 will hit the issue. mount -t nfs -o nfsvers=4.2 10.20.90.100:/Longhorn /root/test nfsvers=4.1 or 4.2 hangs the mount operation forever, and server does not return the error code.

Although nfsvers=4.0 can work correctly, but the nfsvers fallback mechanism in Longhorn relies on error code from the server side. No error code returned leads to fallback failure.

I think you can open a ticket with QNAP to raise their awareness of the problem and check if it can provide any solution.

Update: Just saw QNAP has released a new firmware fixing the NFS v4.1 and 4.2 issue. You can take a look and check if the new firmware can fix this issue.

derekbit on Jul 5, 2022