longhorn: [QUESTION] Restored volume does not appear in k8s

Describe the bug A clear and concise description of what the bug is. I have two clusters which use the same s3 backup bucket. I would like to restore a volume in the second cluster which was backed up in the first cluster. The volume appears health in the longhorn gui but does not appear in the k8s api.

To Reproduce Steps to reproduce the behavior:

Create two clusters
Connect both clusters to the same s3 bucket
Backup volume in the first cluster
Restore volume in the second cluster (create pv/pvc in longhorn gui)

Expected behavior The restored volume (which is reported as healthy) should appear in the pv/pvc list.

Log If applicable, add the Longhorn managers’ log when the issue happens.

[hass-home-assistant-influxdb-e-4b74ae5f] time="2021-02-24T09:11:18Z" level=warning msg="Socket server connection closed"
[hass-home-assistant-influxdb-e-4b74ae5f] time="2021-02-24T09:11:18Z" level=info msg="device hass-home-assistant-influxdb: SCSI device /dev/longhorn/hass-home-assistant-influxdb shutdown"
time="2021-02-24T09:11:18Z" level=info msg="Closing: 10.42.2.246:10015"
[hass-home-assistant-influxdb-e-4b74ae5f] time="2021-02-24T09:11:18Z" level=info msg="Closing: 10.42.0.85:10030"
[hass-home-assistant-influxdb-e-4b74ae5f] time="2021-02-24T09:11:18Z" level=info msg="Closing: 10.42.1.221:10015"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=info msg="Process Manager: process hass-home-assistant-influxdb-e-4b74ae5f stopped"
[pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006-e-64c73f07] time="2021-02-24T09:11:18Z" level=info msg="Shutdown SCSI device for target iqn.2019-10.io.longhorn:pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=debug msg="Process Manager: wait for process hass-home-assistant-mariadb-e-ed8e9b96 to shutdown before unregistering process"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=info msg="wait for process hass-home-assistant-mariadb-e-ed8e9b96 to shutdown"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=info msg="Process Manager: successfully unregistered process hass-home-assistant-influxdb-e-4b74ae5f"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=debug msg="Process Manager: wait for process hass-home-assistant-mariadb-e-ed8e9b96 to shutdown before unregistering process"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=info msg="wait for process hass-home-assistant-config-e-e8cbeb7b to shutdown"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=debug msg="Process Manager: wait for process hass-home-assistant-config-e-e8cbeb7b to shutdown before unregistering process"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=info msg="wait for process pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006-e-64c73f07 to shutdown"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=debug msg="Process Manager: wait for process pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006-e-64c73f07 to shutdown before unregistering process"
[pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006-e-64c73f07] time="2021-02-24T09:11:18Z" level=info msg="Shutdown SCSI target iqn.2019-10.io.longhorn:pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006"
lh_client_close_conn: Closing connection
[pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006-e-64c73f07] time="2021-02-24T09:11:18Z" level=warning msg="Socket server connection closed"
lh_client_close_conn: Connection close complete
[pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006-e-64c73f07] time="2021-02-24T09:11:18Z" level=info msg="device pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006: SCSI device /dev/longhorn/pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006 shutdown"
time="2021-02-24T09:11:18Z" level=info msg="Closing: 10.42.1.221:10060"
[pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006-e-64c73f07] time="2021-02-24T09:11:18Z" level=info msg="Closing: 10.42.2.246:10030"
[pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006-e-64c73f07] time="2021-02-24T09:11:18Z" level=info msg="Closing: 10.42.0.85:10060"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:18Z" level=info msg="Shutdown SCSI device for target iqn.2019-10.io.longhorn:hass-home-assistant-mariadb"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=info msg="Process Manager: process pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006-e-64c73f07 stopped"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=debug msg="Process Manager: wait for process hass-home-assistant-mariadb-e-ed8e9b96 to shutdown before unregistering process"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=debug msg="Process Manager: prepare to delete process hass-home-assistant-influxdb-e-4b74ae5f"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=debug msg="Process Manager: wait for process hass-home-assistant-mariadb-e-ed8e9b96 to shutdown before unregistering process"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=info msg="wait for process hass-home-assistant-mariadb-e-ed8e9b96 to shutdown"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=info msg="wait for process hass-home-assistant-config-e-e8cbeb7b to shutdown"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=debug msg="Process Manager: wait for process hass-home-assistant-config-e-e8cbeb7b to shutdown before unregistering process"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=info msg="Process Manager: successfully unregistered process pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006-e-64c73f07"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:18Z" level=info msg="Shutdown SCSI target iqn.2019-10.io.longhorn:hass-home-assistant-mariadb"
lh_client_close_conn: Closing connection
lh_client_close_conn: Connection close complete
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:18Z" level=warning msg="Socket server connection closed"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:18Z" level=info msg="device hass-home-assistant-mariadb: SCSI device /dev/longhorn/hass-home-assistant-mariadb shutdown"
time="2021-02-24T09:11:18Z" level=info msg="Closing: 10.42.1.221:10030"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:18Z" level=info msg="Closing: 10.42.0.85:10045"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:18Z" level=info msg="Closing: 10.42.2.246:10060"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=info msg="Process Manager: process hass-home-assistant-mariadb-e-ed8e9b96 stopped"
[hass-home-assistant-config-e-e8cbeb7b] time="2021-02-24T09:11:18Z" level=info msg="Shutdown SCSI device for target iqn.2019-10.io.longhorn:hass-home-assistant-config"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=info msg="Process Manager: successfully unregistered process hass-home-assistant-mariadb-e-ed8e9b96"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=debug msg="Process Manager: prepare to delete process pvc-44b2d6e9-e5ba-439f-859c-d2e587d8e006-e-64c73f07"
[hass-home-assistant-config-e-e8cbeb7b] time="2021-02-24T09:11:18Z" level=info msg="Shutdown SCSI target iqn.2019-10.io.longhorn:hass-home-assistant-config"
lh_client_close_conn: Closing connection
[hass-home-assistant-config-e-e8cbeb7b] time="2021-02-24T09:11:18Z" level=warning msg="Socket server connection closed"
lh_client_close_conn: Connection close complete
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=debug msg="Process Manager: wait for process hass-home-assistant-config-e-e8cbeb7b to shutdown before unregistering process"
[hass-home-assistant-config-e-e8cbeb7b] time="2021-02-24T09:11:18Z" level=info msg="device hass-home-assistant-config: SCSI device /dev/longhorn/hass-home-assistant-config shutdown"
time="2021-02-24T09:11:18Z" level=info msg="Closing: 10.42.1.221:10045"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=info msg="wait for process hass-home-assistant-config-e-e8cbeb7b to shutdown"
[hass-home-assistant-config-e-e8cbeb7b] time="2021-02-24T09:11:18Z" level=info msg="Closing: 10.42.0.85:10015"
[hass-home-assistant-config-e-e8cbeb7b] time="2021-02-24T09:11:18Z" level=info msg="Closing: 10.42.2.246:10045"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=info msg="Process Manager: process hass-home-assistant-config-e-e8cbeb7b stopped"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=info msg="Process Manager: successfully unregistered process hass-home-assistant-config-e-e8cbeb7b"
[longhorn-instance-manager] time="2021-02-24T09:11:18Z" level=debug msg="Process Manager: prepare to delete process hass-home-assistant-config-e-e8cbeb7b"
[longhorn-instance-manager] time="2021-02-24T09:11:58Z" level=info msg="Process Manager: prepare to create process hass-home-assistant-mariadb-e-ed8e9b96"
[longhorn-instance-manager] time="2021-02-24T09:11:58Z" level=info msg="Process Manager: created process hass-home-assistant-mariadb-e-ed8e9b96"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:58Z" level=info msg="Starting with replicas [\"tcp://10.42.1.221:10060\" \"tcp://10.42.0.85:10060\" \"tcp://10.42.2.246:10060\"]"
time="2021-02-24T09:11:58Z" level=info msg="Connecting to remote: 10.42.1.221:10060"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:58Z" level=info msg="Opening: 10.42.1.221:10060"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:58Z" level=info msg="Connecting to remote: 10.42.0.85:10060"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:58Z" level=info msg="Opening: 10.42.0.85:10060"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:58Z" level=info msg="Connecting to remote: 10.42.2.246:10060"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:58Z" level=info msg="Opening: 10.42.2.246:10060"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:59Z" level=info msg="Adding backend: tcp://10.42.1.221:10060"
time="2021-02-24T09:11:59Z" level=info msg="Adding backend: tcp://10.42.0.85:10060"
time="2021-02-24T09:11:59Z" level=info msg="Adding backend: tcp://10.42.2.246:10060"
time="2021-02-24T09:11:59Z" level=info msg="Start monitoring tcp://10.42.0.85:10060"
time="2021-02-24T09:11:59Z" level=info msg="Start monitoring tcp://10.42.2.246:10060"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:59Z" level=info msg="Start monitoring tcp://10.42.1.221:10060"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:59Z" level=info msg="Get backend tcp://10.42.1.221:10060 revision counter 11767"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:59Z" level=info msg="Get backend tcp://10.42.0.85:10060 revision counter 11767"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:59Z" level=info msg="Get backend tcp://10.42.2.246:10060 revision counter 11767"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:11:59Z" level=info msg="device hass-home-assistant-mariadb: SCSI device /dev/longhorn/hass-home-assistant-mariadb shutdown"
[hass-home-assistant-mariadb-e-ed8e9b96] go-iscsi-helper: tgtd is already running
[longhorn-instance-manager] time="2021-02-24T09:12:00Z" level=info msg="wait for gRPC service of process hass-home-assistant-mariadb-e-ed8e9b96 to start at localhost:10000"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:12:00Z" level=info msg="go-iscsi-helper: found available target id 1"
tgtd: device_mgmt(246) sz:96 params:path=/var/run/longhorn-hass-home-assistant-mariadb.sock,bstype=longhorn,bsopts=size=10737418240
tgtd: bs_thread_open(409) 16
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:12:00Z" level=info msg="New data socket connnection established"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:12:00Z" level=info msg="default: automatically rescan all LUNs of all iscis sessions"
[longhorn-instance-manager] time="2021-02-24T09:12:01Z" level=info msg="wait for gRPC service of process hass-home-assistant-mariadb-e-ed8e9b96 to start at localhost:10000"
[hass-home-assistant-mariadb-e-ed8e9b96] time="2021-02-24T09:12:01Z" level=info msg="Creating device /dev/longhorn/hass-home-assistant-mariadb 8:16"
time="2021-02-24T09:12:01Z" level=info msg="device hass-home-assistant-mariadb: SCSI device sdb created"
time="2021-02-24T09:12:01Z" level=info msg="Listening on gRPC Controller server: 0.0.0.0:10000"
[longhorn-instance-manager] time="2021-02-24T09:12:02Z" level=info msg="wait for gRPC service of process hass-home-assistant-mariadb-e-ed8e9b96 to start at localhost:10000"
[longhorn-instance-manager] time="2021-02-24T09:12:02Z" level=info msg="Process hass-home-assistant-mariadb-e-ed8e9b96 has started at localhost:10000"
[longhorn-instance-manager] time="2021-02-24T09:27:01Z" level=info msg="Process Manager: prepare to create process hass-home-assistant-config-restore-e-36e8830d"
[longhorn-instance-manager] time="2021-02-24T09:27:01Z" level=info msg="Process Manager: created process hass-home-assistant-config-restore-e-36e8830d"
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:01Z" level=info msg="Starting with replicas [\"tcp://10.42.1.221:10075\" \"tcp://10.42.2.246:10075\" \"tcp://10.42.0.85:10075\"]"
time="2021-02-24T09:27:01Z" level=info msg="Connecting to remote: 10.42.1.221:10075"
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:01Z" level=info msg="Opening: 10.42.1.221:10075"
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:01Z" level=info msg="Connecting to remote: 10.42.2.246:10075"
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:01Z" level=info msg="Opening: 10.42.2.246:10075"
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:01Z" level=info msg="Connecting to remote: 10.42.0.85:10075"
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:01Z" level=info msg="Opening: 10.42.0.85:10075"
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:01Z" level=info msg="Adding backend: tcp://10.42.1.221:10075"
time="2021-02-24T09:27:01Z" level=info msg="Adding backend: tcp://10.42.2.246:10075"
time="2021-02-24T09:27:01Z" level=info msg="Adding backend: tcp://10.42.0.85:10075"
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:01Z" level=info msg="Start monitoring tcp://10.42.1.221:10075"
time="2021-02-24T09:27:01Z" level=info msg="Start monitoring tcp://10.42.2.246:10075"
time="2021-02-24T09:27:01Z" level=info msg="Start monitoring tcp://10.42.0.85:10075"
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:01Z" level=info msg="Get backend tcp://10.42.1.221:10075 revision counter 0"
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:01Z" level=info msg="Get backend tcp://10.42.2.246:10075 revision counter 0"
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:01Z" level=info msg="Get backend tcp://10.42.0.85:10075 revision counter 0"
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:01Z" level=info msg="device hass-home-assistant-config-restore: SCSI device /dev/longhorn/hass-home-assistant-config-restore shutdown"
[hass-home-assistant-config-restore-e-36e8830d] go-iscsi-helper: tgtd is already running
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:02Z" level=info msg="go-iscsi-helper: found available target id 2"
tgtd: device_mgmt(246) sz:102 params:path=/var/run/longhorn-hass-home-assistant-config-restore.sock,bstype=longhorn,bsopts=size=2147483648
tgtd: bs_thread_open(409) 16
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:02Z" level=info msg="New data socket connnection established"
[longhorn-instance-manager] time="2021-02-24T09:27:02Z" level=info msg="wait for gRPC service of process hass-home-assistant-config-restore-e-36e8830d to start at localhost:10001"
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:02Z" level=info msg="default: automatically rescan all LUNs of all iscis sessions"
[longhorn-instance-manager] time="2021-02-24T09:27:03Z" level=info msg="wait for gRPC service of process hass-home-assistant-config-restore-e-36e8830d to start at localhost:10001"
[hass-home-assistant-config-restore-e-36e8830d] time="2021-02-24T09:27:03Z" level=info msg="Creating device /dev/longhorn/hass-home-assistant-config-restore 8:32"
time="2021-02-24T09:27:03Z" level=info msg="device hass-home-assistant-config-restore: SCSI device sdc created"
time="2021-02-24T09:27:03Z" level=info msg="Listening on gRPC Controller server: 0.0.0.0:10001"
[longhorn-instance-manager] time="2021-02-24T09:27:04Z" level=info msg="wait for gRPC service of process hass-home-assistant-config-restore-e-36e8830d to start at localhost:10001"
[longhorn-instance-manager] time="2021-02-24T09:27:04Z" level=info msg="Process hass-home-assistant-config-restore-e-36e8830d has started at localhost:10001"

You can also attach a Support Bundle here. You can generate a Support Bundle using the link at the footer of the Longhorn UI.

Environment:

Longhorn version: 1.1
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3os
- Number of management node in the cluster: 1
- Number of worker node in the cluster: 3
Node config
- OS type and version:
- CPU per node:
- Memory per node:
- Disk type(e.g. SSD/NVMe):
- Network bandwidth between the nodes:
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Virtualbox
Number of Longhorn volumes in the cluster: 7

Additional context Add any other context about the problem here.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 19 (19 by maintainers)

Most upvoted comments

In my evn, I just verified creating PV/PVC from a restored volume working fine.

Looking back at your support bundle. It is weird that there is no pvCreate or pvcCreate request in Longhorn UI pod as well as Longhorn manager pod. Is it possible that the request from UI doesn’t reach the backend?

Can you try 1 more time:

restore volume hass-home-assistant-config-restore-2

Paste the result of

kubectl get pv -o yaml
kubectl get pvc -n hass -o yaml
kubectl get volumes -n longhorn-system -o yaml

Create PV/PVC from Longhorn UI as in the picture syou sent above

Paste the result of

kubectl get pv -o yaml
kubectl get pvc -n hass -o yaml
kubectl get volumes -n longhorn-system -o yaml

Send us the support bundle.

PhanLe1010 on Mar 23, 2021

We’ll add Trafix 2.x workaround to our knowledge base doc.

jenting on Apr 20, 2021

@runningman84

The support bundle looks fine to me.

After restoring the volume, how do you create the PV/PVC using Longhorn UI? Can you take a screenshot of the parameter in Longhorn UI just before you click create PV/PVC? It might be the case that you already have the PV/PVC with the same name in the cluster.

There is also the bug that once you restore a volume it is shown as attached to some pod (which might be the pod name of my other cluster). After a restore, the volume should not be attached to anything from my opinion.

This is a feature of Longhorn. Longhorn remembers the workload that this volume was attached to the last time. It helps the users to identify which workload this newly restored volume is associated with. Notice the fields lastPVCRefAt: "2021-02-01T00:00:13Z" and lastPodRefAt: "2021-02-01T00:00:13Z". It means that the volume was associated with the workload but not currently.

PhanLe1010 on Mar 18, 2021

@PhanLe1010 I just send you the support bundle per mail…

runningman84 on Mar 13, 2021

Did you create a PV/PVC for the volume after restore?

joshimoo on Feb 24, 2021