submariner: inter cluster connection fail,calico network
cluster1:
[root@worker-01 ~]# subctl diagnose all --kubeconfig 153config
Cluster "cluster1"
✓ Checking Submariner support for the Kubernetes version
✓ Kubernetes version "v1.28.1" is supported
✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
✓ Checking DaemonSet "submariner-gateway"
✓ Checking DaemonSet "submariner-routeagent"
✓ Checking DaemonSet "submariner-metrics-proxy"
✓ Checking Deployment "submariner-lighthouse-agent"
✓ Checking Deployment "submariner-lighthouse-coredns"
⚠ Checking the status of all Submariner pods
⚠ Pod "submariner-gateway-cbrp2" has restarted 6 times
✓ Checking that gateway metrics are accessible from non-gateway nodes
✓ Checking Submariner support for the CNI network plugin
✓ The detected CNI network plugin ("calico") is supported
✓ Calico CNI detected, checking if the Submariner IPPool pre-requisites are configured
✗ Checking gateway connections
✗ Connection to cluster "member2" is not established. Connection details:
{
"status": "error",
"statusMessage": "Failed to successfully ping the remote endpoint IP \"172.22.171.0\"",
"endpoint": {
"cluster_id": "member2",
"cable_name": "submariner-cable-member2-192-168-229-156",
"healthCheckIP": "172.22.171.0",
"hostname": "member2-156",
"subnets": [
"10.70.0.0/16",
"172.22.0.0/16"
],
"private_ip": "192.168.229.156",
"public_ip": "39.189.13.59",
"nat_enabled": false,
"backend": "libreswan",
"backend_config": {
"natt-discovery-port": "4490",
"preferred-server": "false",
"udp-port": "4500"
}
},
"usingIP": "192.168.229.156",
"latencyRTT": {
"last": "0s",
"min": "0s",
"average": "0s",
"max": "0s",
"stdDev": "0s"
}
}
✓ Checking Submariner support for the kube-proxy mode
✓ The kube-proxy mode is supported
✓ Checking that firewall configuration allows intra-cluster VXLAN traffic
✓ Checking that services have been exported properly
Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.
subctl version: v0.16.2
cluster2:
[root@worker-01 ~]# subctl diagnose all --kubeconfig 156config
Cluster "cluster1"
✓ Checking Submariner support for the Kubernetes version
✓ Kubernetes version "v1.28.1" is supported
✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
✓ Checking DaemonSet "submariner-gateway"
✓ Checking DaemonSet "submariner-routeagent"
✓ Checking DaemonSet "submariner-metrics-proxy"
✓ Checking Deployment "submariner-lighthouse-agent"
✓ Checking Deployment "submariner-lighthouse-coredns"
⚠ Checking the status of all Submariner pods
⚠ Pod "submariner-gateway-q46gf" has restarted 51 times
✓ Checking that gateway metrics are accessible from non-gateway nodes
✓ Checking Submariner support for the CNI network plugin
✓ The detected CNI network plugin ("calico") is supported
✓ Calico CNI detected, checking if the Submariner IPPool pre-requisites are configured
✓ Checking gateway connections
✓ Checking Submariner support for the kube-proxy mode
✓ The kube-proxy mode is supported
✓ Checking that firewall configuration allows intra-cluster VXLAN traffic
✓ Checking that services have been exported properly
Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.
cluster1:
[root@worker-01 ~]# subctl show all --kubeconfig 153config
Cluster "cluster1"
✓ Detecting broker(s)
✓ No brokers found
✓ Showing Connections
GATEWAY CLUSTER REMOTE IP NAT CABLE DRIVER SUBNETS STATUS RTT avg.
member2-156 member2 192.168.229.156 no libreswan 10.70.0.0/16, 172.22.0.0/16 error 0s
✓ Showing Endpoints
CLUSTER ENDPOINT IP PUBLIC IP CABLE DRIVER TYPE
member1 192.168.229.153 39.189.13.59 libreswan local
member2 192.168.229.156 39.189.13.59 libreswan remote
✓ Showing Gateways
NODE HA STATUS SUMMARY
member1-153 active 0 connections out of 1 are established
✓ Showing Network details
Discovered network details via Submariner:
Network plugin: calico
Service CIDRs: [10.69.0.0/16]
Cluster CIDRs: [172.21.0.0/16]
✓ Showing versions
COMPONENT REPOSITORY CONFIGURED RUNNING ARCH
submariner-gateway quay.io/submariner 0.16.2 release-0.16-88f54d25ccfc amd64
submariner-routeagent quay.io/submariner 0.16.2 release-0.16-88f54d25ccfc amd64
submariner-metrics-proxy quay.io/submariner 0.16.2 release-0.16-3d7d32af4808 amd64
submariner-operator quay.io/submariner 0.16.2 release-0.16-d4284df96925 amd64
submariner-lighthouse-agent quay.io/submariner 0.16.2 release-0.16-847b34cf06d0 amd64
submariner-lighthouse-coredns quay.io/submariner 0.16.2 release-0.16-847b34cf06d0 amd64
cluster2:
[root@worker-01 ~]# subctl show all --kubeconfig 156config
Cluster "cluster1"
✓ Detecting broker(s)
✓ No brokers found
✓ Showing Connections
GATEWAY CLUSTER REMOTE IP NAT CABLE DRIVER SUBNETS STATUS RTT avg.
member1-153 member1 192.168.229.153 no libreswan 10.69.0.0/16, 172.21.0.0/16 connected 0s
✓ Showing Endpoints
CLUSTER ENDPOINT IP PUBLIC IP CABLE DRIVER TYPE
member2 192.168.229.156 39.189.13.59 libreswan local
member1 192.168.229.153 39.189.13.59 libreswan remote
✓ Showing Gateways
NODE HA STATUS SUMMARY
member2-156 active All connections (1) are established
✓ Showing Network details
Discovered network details via Submariner:
Network plugin: calico
Service CIDRs: [10.70.0.0/16]
Cluster CIDRs: [172.22.0.0/16]
✓ Showing versions
COMPONENT REPOSITORY CONFIGURED RUNNING ARCH
submariner-gateway quay.io/submariner 0.16.2 release-0.16-88f54d25ccfc amd64
submariner-routeagent quay.io/submariner 0.16.2 release-0.16-88f54d25ccfc amd64
submariner-metrics-proxy quay.io/submariner 0.16.2 release-0.16-3d7d32af4808 amd64
submariner-operator quay.io/submariner 0.16.2 release-0.16-d4284df96925 amd64
submariner-lighthouse-agent quay.io/submariner 0.16.2 release-0.16-847b34cf06d0 amd64
submariner-lighthouse-coredns quay.io/submariner 0.16.2 release-0.16-847b34cf06d0 amd64
submariner-cluster1-1130.zip submariner-cluster2-1130.zip
enviroment : vm create with wmware.node can connect each other.k8s 2.28.1 What happened:
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
- Diagnose information (use
subctl diagnose all): - Gather information (use
subctl gather): - Cloud provider or hardware configuration:
- Install tools:
- Others:
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (9 by maintainers)
No, the CNIs I used are some Underlay CNIs, such as Macvlan. We wanted to validate the combination of Submariner + Spiderpool to solve the multi-cluster network connectivity issue across data centers. The results show that it works perfectly, and I have written documentation on it. If you’re interested, you can see Submariner + Spiderpool.