cloudstack: The VPC Redundant router "Virtual routers" can not work as expected

ISSUE TYPE

• Bug Report

COMPONENT NAME

• Virtual routers

CLOUDSTACK VERSION

CloudStack 4.18.0.0

OS / ENVIRONMENT

OS: oracle linux 8

CONFIGURATION

• Advanced networking, using VXLAN (http:docs.cloudstack.apache.org/en/latest/plugins/vxlan.html?highlight=vxlan), having two hosts name is kvm001 and kvm002

• The hosts network config:

                 |---------------- cloudbr0:  10.26.128.25 ( VLAN 2128)
    kvm001 ------
                 |---------------- cloudbr1:  10.71.231.41 ( VLAN 2230)

                         
                 |---------------- cloudbr0:  10.26.128.22 ( VLAN 2128)
    kvm002 ------
                 |---------------- cloudbr1:  10.71.231.42 ( VLAN 2230)                       

• Management Network: VLAN: 2227 (10.71.227.0/24）

• The switch has enabled the following VLANs: 2128, 2230, 2227

The network test is:

# From kvm001

# ping -I cloudbr0 10.26.128.254
PING 10.26.128.254 (10.26.128.254) from 10.26.128.25 cloudbr0: 56(84) bytes of data.
64 bytes from 10.26.128.254: icmp_seq=1 ttl=255 time=1.08 ms
64 bytes from 10.26.128.254: icmp_seq=2 ttl=255 time=1.05 ms
^C
--- 10.26.128.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.046/1.062/1.079/0.036 ms

# ping -I cloudbr1 10.71.231.42
PING 10.71.231.42 (10.71.231.42) from 10.71.231.41 cloudbr1: 56(84) bytes of data.
64 bytes from 10.71.231.42: icmp_seq=1 ttl=64 time=0.191 ms
64 bytes from 10.71.231.42: icmp_seq=2 ttl=64 time=0.177 ms
64 bytes from 10.71.231.42: icmp_seq=3 ttl=64 time=0.181 ms
^C
--- 10.71.231.42 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2086ms
rtt min/avg/max/mdev = 0.177/0.183/0.191/0.005 ms

• Create a VPC name is test, the details info is

The Virtual routers is r-30912-VM and r-30911-VM and the IP is 10.71.227.33

root@r-30912-VM:/opt/cloud/bin# sh -x checkrouter.sh 
+ STATUS=UNKNOWN
+ systemctl is-active keepalived
+ [ active != active ]
+ sed -e s/[,\"]//g
+ awk {print $2;}
+ grep type
+ cat /etc/cloudstack/cmdline.json
+ ROUTER_TYPE=vpcrouter
+ [ vpcrouter = router ]
+ awk {print $9;}+ grep state

+ ip -4 addr show dev eth1
+ ROUTER_STATE=UP
+ [ UP = UP ]
+ STATUS=PRIMARY
+ echo Status: PRIMARY
Status: PRIMARY

root@r-30911-VM:/opt/cloud/bin# sh -x checkrouter.sh 
+ STATUS=UNKNOWN
+ systemctl is-active keepalived
+ [ active != active ]
+ sed -e s/[,\"]//g
+ awk {print $2;}
+ grep type
+ cat /etc/cloudstack/cmdline.json
+ ROUTER_TYPE=vpcrouter
+ [ vpcrouter = router ]
+ awk {print $9;}
+ grep state
+ ip -4 addr show dev eth1
+ ROUTER_STATE=UP
+ [ UP = UP ]
+ STATUS=PRIMARY
+ echo Status: PRIMARY
Status: PRIMARY

The Result:

• The VM can only ping the VM on same physical hosts
• The VM spawned on different physical hosts cannot ping each other directly
• The VM can ping each other when migrated all VMs to a single host
• The VPC Redundant router state are PRIMARY

I have restarted the VPC, It can not work. I have restarted the Virtual routers, It also can not work.

The cloud.log (r-30912-VM) is

cloud.log

EXPECTED RESULTS

The “Virtual routers” works fine.

Are there any commands for further debugging and How to solve this problem ?