cloudstack: Local Storage migration fails due to storage pool not found on destination KVM host

On a 4.18.1.0 RC2 adv zone KVM env using local only storage and local storage for systemvms, migration fails from kvm1 to kvm2 due to the following seen in the target kvm host

2023-09-05 21:12:06,645 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:f38dafe4) Attempting to create volume 23ef733c-c26e-405a-829e-6e0010b3cbeb (Filesystem) in pool 3a11d1ea-eff4-4f5b-9b7c-e0ff89a8c9dd with size (2.20 GB) 2361393152
2023-09-05 21:12:06,691 INFO  [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) (logid:f38dafe4) No existing libvirtd connection found. Opening a new one
2023-09-05 21:12:06,692 WARN  [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) (logid:f38dafe4) Can not find a connection for Instance i-2-453-VM. Assuming the default connection.
2023-09-05 21:12:08,499 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:f38dafe4) Trying to fetch storage pool 8aa9768c-cbcf-4e8e-8875-f94a7f9445b6 from libvirt
2023-09-05 21:12:08,500 WARN  [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:f38dafe4) Caught:
com.cloud.utils.exception.CloudRuntimeException: Could not fetch storage pool 8aa9768c-cbcf-4e8e-8875-f94a7f9445b6 from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '8aa9768c-cbcf-4e8e-8875-f94a7f9445b6'
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:278)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:264)

The above can sometimes cause the VM to crash as it fails right at the last step on the dest. kvm host end, and CloudStack may further report that domain is not running.

However, systemvm migration with storage work. I’m not sure if this is an env issue or a bug.

ISSUE TYPE

Bug Report
Improvement Request
Enhancement Request
Feature Idea
Documentation Report
Other

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

About this issue

Original URL
State: closed
Created 10 months ago
Comments: 15 (15 by maintainers)

Most upvoted comments

@weizhouapache I’m using 4.19.0.0-SNAPSHOT, that I’ve built from main. Unfortunately I’ve already rebuilt my environment with another version, but I had the same exception that you described here #7942 (comment).

Ok @JoaoJandre If you have not seen this issue in other versions, it should be caused by same root cause which is fixed by #7945

weizhouapache on Sep 6, 2023

@weizhouapache @rohityadavcloud It seems that this is not only affecting local storage, I had the same issue that @weizhouapache reported while migrating volumes of a running VM from NFS to SharedMountPoint.

JoaoJandre on Sep 6, 2023

we have found the root cause

diff --git a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtConnection.java b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/
+resource/LibvirtConnection.java
index c70a72f399c..0f8031e3aaa 100644
--- a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtConnection.java
+++ b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtConnection.java
@@ -21,6 +21,7 @@ import java.util.Map;

 import org.apache.log4j.Logger;
 import org.libvirt.Connect;
+import org.libvirt.Library;
 import org.libvirt.LibvirtException;

 import com.cloud.hypervisor.Hypervisor;
@@ -44,6 +45,7 @@ public class LibvirtConnection {
         if (conn == null) {
             s_logger.info("No existing libvirtd connection found. Opening a new one");
             conn = new Connect(hypervisorURI, false);
+            Library.initEventLoop();
             s_logger.debug("Successfully connected to libvirt at: " + hypervisorURI);
             s_connections.put(hypervisorURI, conn);
         } else {

@harikrishna-patnala is working on the fix.

weizhouapache on Sep 6, 2023