submariner: inter cluster connection fail,calico network

cluster1:

[root@worker-01 ~]# subctl diagnose all --kubeconfig 153config 
Cluster "cluster1"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.28.1" is supported

 ✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
 ✓ Checking DaemonSet "submariner-gateway"
 ✓ Checking DaemonSet "submariner-routeagent"
 ✓ Checking DaemonSet "submariner-metrics-proxy"
 ✓ Checking Deployment "submariner-lighthouse-agent"
 ✓ Checking Deployment "submariner-lighthouse-coredns"
 ⚠ Checking the status of all Submariner pods
 ⚠ Pod "submariner-gateway-cbrp2" has restarted 6 times
 ✓ Checking that gateway metrics are accessible from non-gateway nodes 

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("calico") is supported
 ✓ Calico CNI detected, checking if the Submariner IPPool pre-requisites are configured
 ✗ Checking gateway connections
 ✗ Connection to cluster "member2" is not established. Connection details:
{
  "status": "error",
  "statusMessage": "Failed to successfully ping the remote endpoint IP \"172.22.171.0\"",
  "endpoint": {
    "cluster_id": "member2",
    "cable_name": "submariner-cable-member2-192-168-229-156",
    "healthCheckIP": "172.22.171.0",
    "hostname": "member2-156",
    "subnets": [
      "10.70.0.0/16",
      "172.22.0.0/16"
    ],
    "private_ip": "192.168.229.156",
    "public_ip": "39.189.13.59",
    "nat_enabled": false,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "udp-port": "4500"
    }
  },
  "usingIP": "192.168.229.156",
  "latencyRTT": {
    "last": "0s",
    "min": "0s",
    "average": "0s",
    "max": "0s",
    "stdDev": "0s"
  }
}
 ✓ Checking Submariner support for the kube-proxy mode 
 ✓ The kube-proxy mode is supported
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic 

 ✓ Checking that services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.16.2

cluster2:


[root@worker-01 ~]# subctl diagnose all --kubeconfig 156config 
Cluster "cluster1"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.28.1" is supported

 ✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
 ✓ Checking DaemonSet "submariner-gateway"
 ✓ Checking DaemonSet "submariner-routeagent"
 ✓ Checking DaemonSet "submariner-metrics-proxy"
 ✓ Checking Deployment "submariner-lighthouse-agent"
 ✓ Checking Deployment "submariner-lighthouse-coredns"
 ⚠ Checking the status of all Submariner pods
 ⚠ Pod "submariner-gateway-q46gf" has restarted 51 times
 ✓ Checking that gateway metrics are accessible from non-gateway nodes 

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("calico") is supported
 ✓ Calico CNI detected, checking if the Submariner IPPool pre-requisites are configured
 ✓ Checking gateway connections
 ✓ Checking Submariner support for the kube-proxy mode 
 ✓ The kube-proxy mode is supported
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic 

 ✓ Checking that services have been exported properly


Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

cluster1:

[root@worker-01 ~]# subctl show all --kubeconfig 153config 
Cluster "cluster1"
 ✓ Detecting broker(s)
 ✓ No brokers found

 ✓ Showing Connections
GATEWAY       CLUSTER   REMOTE IP         NAT   CABLE DRIVER   SUBNETS                       STATUS   RTT avg.   
member2-156   member2   192.168.229.156   no    libreswan      10.70.0.0/16, 172.22.0.0/16   error    0s         

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP       PUBLIC IP      CABLE DRIVER   TYPE     
member1   192.168.229.153   39.189.13.59   libreswan      local    
member2   192.168.229.156   39.189.13.59   libreswan      remote   

 ✓ Showing Gateways
NODE          HA STATUS   SUMMARY                                  
member1-153   active      0 connections out of 1 are established   

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  calico
        Service CIDRs:   [10.69.0.0/16]
        Cluster CIDRs:   [172.21.0.0/16]

 ✓ Showing versions 
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING                     ARCH    
submariner-gateway              quay.io/submariner   0.16.2       release-0.16-88f54d25ccfc   amd64   
submariner-routeagent           quay.io/submariner   0.16.2       release-0.16-88f54d25ccfc   amd64   
submariner-metrics-proxy        quay.io/submariner   0.16.2       release-0.16-3d7d32af4808   amd64   
submariner-operator             quay.io/submariner   0.16.2       release-0.16-d4284df96925   amd64   
submariner-lighthouse-agent     quay.io/submariner   0.16.2       release-0.16-847b34cf06d0   amd64   
submariner-lighthouse-coredns   quay.io/submariner   0.16.2       release-0.16-847b34cf06d0   amd64  

cluster2:


[root@worker-01 ~]# subctl show all --kubeconfig 156config 
Cluster "cluster1"
 ✓ Detecting broker(s)
 ✓ No brokers found

 ✓ Showing Connections
GATEWAY       CLUSTER   REMOTE IP         NAT   CABLE DRIVER   SUBNETS                       STATUS      RTT avg.   
member1-153   member1   192.168.229.153   no    libreswan      10.69.0.0/16, 172.21.0.0/16   connected   0s         

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP       PUBLIC IP      CABLE DRIVER   TYPE     
member2   192.168.229.156   39.189.13.59   libreswan      local    
member1   192.168.229.153   39.189.13.59   libreswan      remote   

 ✓ Showing Gateways
NODE          HA STATUS   SUMMARY                               
member2-156   active      All connections (1) are established   

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  calico
        Service CIDRs:   [10.70.0.0/16]
        Cluster CIDRs:   [172.22.0.0/16]

 ✓ Showing versions 
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING                     ARCH    
submariner-gateway              quay.io/submariner   0.16.2       release-0.16-88f54d25ccfc   amd64   
submariner-routeagent           quay.io/submariner   0.16.2       release-0.16-88f54d25ccfc   amd64   
submariner-metrics-proxy        quay.io/submariner   0.16.2       release-0.16-3d7d32af4808   amd64   
submariner-operator             quay.io/submariner   0.16.2       release-0.16-d4284df96925   amd64   
submariner-lighthouse-agent     quay.io/submariner   0.16.2       release-0.16-847b34cf06d0   amd64   
submariner-lighthouse-coredns   quay.io/submariner   0.16.2       release-0.16-847b34cf06d0   amd64 

submariner-cluster1-1130.zip submariner-cluster2-1130.zip

enviroment : vm create with wmware.node can connect each other.k8s 2.28.1 What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Diagnose information (use subctl diagnose all):
  • Gather information (use subctl gather):
  • Cloud provider or hardware configuration:
  • Install tools:
  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (9 by maintainers)

Most upvoted comments

No, the CNIs I used are some Underlay CNIs, such as Macvlan. We wanted to validate the combination of Submariner + Spiderpool to solve the multi-cluster network connectivity issue across data centers. The results show that it works perfectly, and I have written documentation on it. If you’re interested, you can see Submariner + Spiderpool.