cloudstack: 4.12 add host fail ,using tls connection

ISSUE TYPE

• Other

COMPONENT NAME

INSTALL

CLOUDSTACK VERSION

4.12

CONFIGURATION

advanced networking

OS / ENVIRONMENT

CentOS7.6

SUMMARY

I am an experience user of cloudstack , we built enterprise private cloud in cloudstack two years . now we test the new 4.12 version for update . we install management ok. install agent in hypervisor ok. but we encounter an issue when add first zone. one of step is add host, 4.12 using tls comucation with agent and manager, but libvirtd connect error , tls can’t work ,the host note as “alert” in MGRUI.

we reinstall cloudstack-agent and libvirtd , same error we configure libvirtd with tcp ,not tls , can add but show unscure ,then click “prevission host key” , restart agent , error show again we checked the /var/log/message for libvirtd log , no especial info there. some times , keep “alert” state , about one or two hours later , the first host added ,and show in “UP” ,why? but it turn back “alert” state when restart management. we checked all tls certificate exists.

the key log when add a host, if no error log then host state turn “UP” :

2019-07-23 01:11:24,309 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion

full log when agent first add to zone:

2019-07-23 01:07:30,306 INFO  [utils.nio.NioClient] (main:null) (logid:) Connecting to localhost:8250
2019-07-23 01:07:30,306 WARN  [utils.nio.NioConnection] (main:null) (logid:) Unable to connect to remote: is there a server running on port 8250
2019-07-23 01:07:35,307 INFO  [cloud.agent.Agent] (main:null) (logid:) Connecting to host:localhost
2019-07-23 01:07:35,308 INFO  [utils.nio.NioClient] (main:null) (logid:) Connecting to localhost:8250
2019-07-23 01:07:35,313 WARN  [utils.nio.NioConnection] (main:null) (logid:) Unable to connect to remote: is there a server running on port 8250
2019-07-23 01:07:38,229 INFO  [cloud.agent.Agent] (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
2019-07-23 01:08:50,219 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2019-07-23 01:08:50,221 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.12.0.0
2019-07-23 01:08:50,222 INFO  [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2019-07-23 01:08:50,235 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage
2019-07-23 01:08:50,236 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm
2019-07-23 01:08:50,246 INFO  [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2019-07-23 01:08:50,254 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection
2019-07-23 01:08:50,355 INFO  [cloud.agent.Agent] (main:null) (logid:) id is 
2019-07-23 01:08:50,357 WARN  [cloud.resource.ServerResourceBase] (main:null) (logid:) Nics are not specified in properties file/db, will try to autodiscover
2019-07-23 01:08:50,361 INFO  [cloud.resource.ServerResourceBase] (main:null) (logid:) Designating private to be nic cloudbr0
2019-07-23 01:08:50,374 INFO  [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one
2019-07-23 01:08:50,529 INFO  [org.reflections.Reflections] (main:null) (logid:) Reflections took 26 ms to scan 1 urls, producing 7 keys and 10 values 
2019-07-23 01:08:50,541 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver.
2019-07-23 01:08:50,646 INFO  [cloud.agent.Agent] (main:null) (logid:) Agent [id = new : type = LibvirtComputingResource : zone = default : pod = default : workers = 5 : host = localhost : port = 8250
2019-07-23 01:08:50,650 INFO  [utils.nio.NioClient] (main:null) (logid:) Connecting to localhost:8250
2019-07-23 01:08:50,651 WARN  [utils.nio.NioConnection] (main:null) (logid:) Unable to connect to remote: is there a server running on port 8250
2019-07-23 01:08:55,652 INFO  [cloud.agent.Agent] (main:null) (logid:) Connecting to host:localhost
2019-07-23 01:08:55,655 INFO  [utils.nio.NioClient] (main:null) (logid:) Connecting to localhost:8250
2019-07-23 01:08:55,656 WARN  [utils.nio.NioConnection] (main:null) (logid:) Unable to connect to remote: is there a server running on port 8250
2019-07-23 01:09:00,539 INFO  [cloud.agent.Agent] (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
2019-07-23 01:09:00,657 INFO  [cloud.agent.Agent] (main:null) (logid:) Connecting to host:localhost
2019-07-23 01:09:00,659 INFO  [utils.nio.NioClient] (main:null) (logid:) Connecting to localhost:8250
2019-07-23 01:09:00,660 WARN  [utils.nio.NioConnection] (main:null) (logid:) Unable to connect to remote: is there a server running on port 8250
2019-07-23 01:09:07,634 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2019-07-23 01:09:07,637 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.12.0.0
2019-07-23 01:09:07,638 INFO  [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2019-07-23 01:09:07,649 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage
2019-07-23 01:09:07,651 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm
2019-07-23 01:09:07,661 INFO  [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2019-07-23 01:09:07,672 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection
2019-07-23 01:09:07,773 INFO  [cloud.agent.Agent] (main:null) (logid:) id is 
2019-07-23 01:09:07,774 WARN  [cloud.resource.ServerResourceBase] (main:null) (logid:) Nics are not specified in properties file/db, will try to autodiscover
2019-07-23 01:09:07,778 INFO  [cloud.resource.ServerResourceBase] (main:null) (logid:) Designating private to be nic cloudbr0
2019-07-23 01:09:07,795 INFO  [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one
2019-07-23 01:11:23,438 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2019-07-23 01:11:23,440 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.12.0.0
2019-07-23 01:11:23,440 INFO  [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2019-07-23 01:11:23,450 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage
2019-07-23 01:11:23,451 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm
2019-07-23 01:11:23,460 INFO  [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2019-07-23 01:11:23,470 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection
2019-07-23 01:11:23,562 INFO  [cloud.agent.Agent] (main:null) (logid:) id is 
2019-07-23 01:11:23,579 INFO  [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one
2019-07-23 01:11:23,721 INFO  [org.reflections.Reflections] (main:null) (logid:) Reflections took 26 ms to scan 1 urls, producing 7 keys and 10 values 
2019-07-23 01:11:23,732 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver.
2019-07-23 01:11:23,833 INFO  [cloud.agent.Agent] (main:null) (logid:) Agent [id = new : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : host = 192.168.100.1 : port = 8250
2019-07-23 01:11:23,836 INFO  [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.100.1:8250
2019-07-23 01:11:23,840 INFO  [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2019-07-23 01:11:24,242 INFO  [utils.nio.NioClient] (main:null) (logid:) SSL: Handshake done
2019-07-23 01:11:24,242 INFO  [utils.nio.NioClient] (main:null) (logid:) Connected to 192.168.100.1:8250
2019-07-23 01:11:24,305 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 36380b99-39b5-48e3-9202-6d69a3efabe1 (Filesystem) in libvirt
2019-07-23 01:11:24,309 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2019-07-23 01:11:24,310 WARN  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Storage pool 36380b99-39b5-48e3-9202-6d69a3efabe1 was not found running in libvirt. Need to create it.
2019-07-23 01:11:24,311 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Didn't find an existing storage pool 36380b99-39b5-48e3-9202-6d69a3efabe1 by UUID, checking for pools with duplicate paths
2019-07-23 01:11:24,314 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool 36380b99-39b5-48e3-9202-6d69a3efabe1 from libvirt
2019-07-23 01:11:35,629 INFO  [cloud.serializer.GsonHelper] (Agent-Handler-1:null) (logid:) Default Builder inited.

STEPS TO REPRODUCE

add host ->input ip -> host add

EXPECTED RESULTS

add host 

ACTUAL RESULTS

host added , but show alert, can't work,  the problem at libvirtd tls connection