cloudstack: The System VMs Agent state is not Up after upgrade to CloudStack 4.17.1.0

ISSUE TYPE

• Other

COMPONENT NAME

• System VMs

CLOUDSTACK VERSION

• CloudStack 4.17.1.0

OS / ENVIRONMENT

# rpm -qa | grep cloudstack
cloudstack-management-4.17.1.0-1.el7.x86_64
cloudstack-common-4.17.1.0-1.el7.x86_64

# rpm -qa | grep cloudstack
cloudstack-agent-4.17.1.0-shapeblue0.el8.x86_64
cloudstack-common-4.17.1.0-shapeblue0.el8.x86_64

# cat /etc/os-release 
NAME="Oracle Linux Server"
VERSION="8.3"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="8.3"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Oracle Linux Server 8.3"

SUMMARY

The System VMs Agent state is not Up after upgrade from 4.15.2.0 to CloudStack 4.17.1.0

STEPS TO REPRODUCE

we using the doc as below: upgrade the cloudstack

EXPECTED RESULTS

The System VMs Agent state is OK.

ACTUAL RESULTS

The System VMs Agent state is not OK.

---

There is no valid log error message in the log file.

2022-10-13 00:01:50,636 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) (logid:) Execution is successful.
2022-10-13 00:01:50,637 DEBUG [kvm.resource.LibvirtConnection] (UgentTask-5:null) (logid:) Looking for libvirtd connection at: qemu:///system
2022-10-13 00:01:50,648 DEBUG [cloud.agent.Agent] (UgentTask-5:null) (logid:) Sending ping: Seq 32-380:  { Cmd , MgmtId: -1, via: 32, Ver: v1, Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{"i-2-26018-VM":["26018","2"],"i-2-26020-VM":["26020","2"],"i-2-25943-VM":["25943","4"],"i-2-25969-VM":["25969","2"],"i-2-25950-VM":["25950","2"],"i-2-25937-VM":["25937","4"],"i-2-26008-VM":["26008","2"],"i-2-25931-VM":["25931","5"],"i-2-26007-VM":["26007","2"],"i-2-26005-VM":["26005","5"],"i-2-25959-VM":["25959","2"],"i-2-25948-VM":["25948","3"],"i-2-26012-VM":["26012","2"],"i-2-26017-VM":["26017","2"]},"_hostVmStateReport":{"i-2-26018-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"r-25932-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"i-2-26020-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"i-2-25943-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"i-2-25969-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"i-2-25950-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"i-2-25937-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"r-25938-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"i-2-26008-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"s-26559-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"i-2-25931-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"i-2-26007-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"i-2-26005-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"i-2-25959-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"i-2-25948-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"i-2-26012-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"v-26560-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"},"i-2-26017-VM":{"state":"PowerOn","host":"whdckvm007.cn.prod"}},"_gatewayAccessible":"true","_vnetAccessible":"true","hostType":"Routing","hostId":"32","wait":"0","bypassHostMaintenance":"false"}}] }
2022-10-13 00:01:50,913 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) (logid:) Received response: Seq 32-380:  { Ans: , MgmtId: 345052215515, via: 32, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":"32","wait":"0","bypassHostMaintenance":"false"},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2022-10-13 00:02:23,580 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) (logid:d574e7cf) Processing command: com.cloud.agent.api.GetHostStatsCommand
2022-10-13 00:02:27,142 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:621fd576) Processing command: com.cloud.agent.api.GetStorageStatsCommand
2022-10-13 00:02:27,142 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:621fd576) Trying to fetch storage pool b4fb50ca-79f2-46ba-bd1b-092f9bd14d4e from libvirt
2022-10-13 00:02:27,142 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-2:null) (logid:621fd576) Looking for libvirtd connection at: qemu:///system
2022-10-13 00:02:27,158 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:621fd576) Asking libvirt to refresh storage pool b4fb50ca-79f2-46ba-bd1b-092f9bd14d4e
2022-10-13 00:02:27,160 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:621fd576) Successfully refreshed pool b4fb50ca-79f2-46ba-bd1b-092f9bd14d4e Capacity: (49.98 GB) 53660876800 Used: (10.98 GB) 11788746752 Available: (39.00 GB) 41872130048
2022-10-13 00:02:27,244 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:621fd576) Processing command: com.cloud.agent.api.GetStorageStatsCommand
2022-10-13 00:02:27,244 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:621fd576) Trying to fetch storage pool 0edb0e58-6b90-3bee-8e7b-13007b0f7bf9 from libvirt
2022-10-13 00:02:27,244 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:621fd576) Looking for libvirtd connection at: qemu:///system

I have test all the connection:

The cloudstack manager node ip: 10.25.2.173

The agent host node ip: 10.26.246.10

The KVM vm instance IP:

• 10.26.18.78 (VLAN1)
• 10.26.15.68 (VLAN2)

The virtual routes ip:

• r-25938-VM: 10.26.18.31 (VLAN1)
• r-25932-VM: 10.26.15.31 (VLAN2)

The system vm ip:

• s-26559-VM
• v-26560-VM

The system vm can ping all the ip address:

The Second Storage VM: s-26559-VM

The console proxy VM: v-26560-VM

The virtual route can also ping all the ip:

The manager node can ping all the address:

But I don’t know why the System VMs Agent state is not Up, Can you give some suggestions on how to troubleshoot this problem? If you need more information, please ask.