submariner: Cross-Cluster Service Connectivity Fails with "Host is Unreachable" Despite Successful DNS Resolution in Submariner GlobalNet Setup

What happened: I deployed Submariner with GlobalNet across two Kubernetes clusters. DNS resolution works as expected, but connectivity to services across clusters fails with a Host is unreachable error.

More info is available in below link

https://kubernetes.slack.com/archives/C010RJV694M/p1730390376380879

What you expected to happen: curl requests from a pod in cluster2 to a service exposed via Submariner in cluster1 should succeed, indicating that cross-cluster communication is functioning.

How to reproduce it (as minimally and precisely as possible):

  • Set up two Kubernetes clusters and deploy Submariner with GlobalNet enabled.
  • cluster1 GlobalNet CIDR: 242.0.0.0/16
  • cluster2 GlobalNet CIDR: 243.0.0.0/16
  • Deploy an nginx pod in cluster1, expose it as a service, and export it using Submariner.
  • Deploy a test pod (tmp-shell) in cluster2.
  • Attempt to access the nginx service in cluster1 from tmp-shell in cluster2 using DNS (nginx-cluster1.default.svc.clusterset.local) or the resolved GlobalNet IP.

Anything else we need to know?:

Environment:

  • Diagnose information (use subctl diagnose all):
Cluster 1 info
Aswin πŸ”₯πŸ”₯πŸ”₯ $ subctl diagnose all --kubeconfig /Users/aswina/Downloads/sub1
Cluster "sub1"
 βœ“ Checking Submariner support for the Kubernetes version
 βœ“ Kubernetes version "v1.30.6" is supported

 βœ— Globalnet deployment detected - checking that globalnet CIDRs do not overlap
 βœ— Error getting the Broker's REST config: error getting auth rest config: Get "https://9.66.245.122:6443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/any": tls: failed to verify certificate: x509: "kube-apiserver" certificate is not trusted

 ⚠ Checking Submariner support for the CNI network plugin
 ⚠ Submariner could not detect the CNI network plugin and is using ("generic") plugin. It may or may not work.
 βœ“ Checking gateway connections
 βœ— Checking route agent connections
 βœ— Connection to cluster "cluster2" is not established. Connection details:
{
  "status": "error",
  "statusMessage": "Failed to successfully ping the remote endpoint IP \"243.0.255.254\"",
  "spec": {
    "cluster_id": "cluster2",
    "cable_name": "submariner-cable-cluster2-10-21-82-227",
    "healthCheckIP": "243.0.255.254",
    "hostname": "sub2-worker-1.fyre.ibm.com",
    "subnets": [
      "243.0.0.0/16"
    ],
    "private_ip": "10.21.82.227",
    "public_ip": "129.41.87.3",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "udp-port": "4500"
    }
  }
}
 βœ— Connection to cluster "cluster2" is not established. Connection details:
{
  "status": "error",
  "statusMessage": "Failed to successfully ping the remote endpoint IP \"243.0.255.254\"",
  "spec": {
    "cluster_id": "cluster2",
    "cable_name": "submariner-cable-cluster2-10-21-82-227",
    "healthCheckIP": "243.0.255.254",
    "hostname": "sub2-worker-1.fyre.ibm.com",
    "subnets": [
      "243.0.0.0/16"
    ],
    "private_ip": "10.21.82.227",
    "public_ip": "129.41.87.3",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "udp-port": "4500"
    }
  }
}
 βœ— Connection to cluster "cluster2" is not established. Connection details:
{
  "status": "error",
  "statusMessage": "Failed to successfully ping the remote endpoint IP \"243.0.255.254\"",
  "spec": {
    "cluster_id": "cluster2",
    "cable_name": "submariner-cable-cluster2-10-21-82-227",
    "healthCheckIP": "243.0.255.254",
    "hostname": "sub2-worker-1.fyre.ibm.com",
    "subnets": [
      "243.0.0.0/16"
    ],
    "private_ip": "10.21.82.227",
    "public_ip": "129.41.87.3",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "udp-port": "4500"
    }
  }
}
 βœ“ Checking Submariner support for the kube-proxy mode
 βœ“ The kube-proxy mode is supported
 βœ— Checking that firewall configuration allows intra-cluster VXLAN traffic
 βœ— The tcpdump output from the sniffer pod does not contain the expected remote endpoint IP 243.0.0.0. Please check that your firewall configuration allows UDP/4800 traffic. Actual pod output:
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vx-submariner, link-type EN10MB (Ethernet), snapshot length 262144 bytes

0 packets captured
0 packets received by filter
0 packets dropped by kernel

 βœ“ Checking that Globalnet is correctly configured and functioning

 βœ“ Checking that services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.19.0

Aswin πŸ”₯πŸ”₯πŸ”₯ $
Cluster 2 info
Aswin πŸ”₯πŸ”₯πŸ”₯ $ subctl diagnose all --kubeconfig /Users/aswina/Downloads/sub2
Cluster "sub2"
 βœ“ Checking Submariner support for the Kubernetes version
 βœ“ Kubernetes version "v1.30.6" is supported

 βœ— Globalnet deployment detected - checking that globalnet CIDRs do not overlap
 βœ— Error getting the Broker's REST config: error getting auth rest config: Get "https://9.66.245.122:6443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/any": tls: failed to verify certificate: x509: "kube-apiserver" certificate is not trusted

 ⚠ Checking Submariner support for the CNI network plugin
 ⚠ Submariner could not detect the CNI network plugin and is using ("generic") plugin. It may or may not work.
 βœ“ Checking gateway connections
 βœ— Checking route agent connections
 βœ— Connection to cluster "cluster1" is not established. Connection details:
{
  "status": "error",
  "statusMessage": "Failed to successfully ping the remote endpoint IP \"242.0.255.254\"",
  "spec": {
    "cluster_id": "cluster1",
    "cable_name": "submariner-cable-cluster1-10-21-101-75",
    "healthCheckIP": "242.0.255.254",
    "hostname": "sub1-worker-1.fyre.ibm.com",
    "subnets": [
      "242.0.0.0/16"
    ],
    "private_ip": "10.21.101.75",
    "public_ip": "129.41.87.4",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "udp-port": "4500"
    }
  }
}
 βœ— Connection to cluster "cluster1" is not established. Connection details:
{
  "status": "error",
  "statusMessage": "Failed to successfully ping the remote endpoint IP \"242.0.255.254\"",
  "spec": {
    "cluster_id": "cluster1",
    "cable_name": "submariner-cable-cluster1-10-21-101-75",
    "healthCheckIP": "242.0.255.254",
    "hostname": "sub1-worker-1.fyre.ibm.com",
    "subnets": [
      "242.0.0.0/16"
    ],
    "private_ip": "10.21.101.75",
    "public_ip": "129.41.87.4",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "udp-port": "4500"
    }
  }
}
 βœ— Connection to cluster "cluster1" is not established. Connection details:
{
  "status": "error",
  "statusMessage": "Failed to successfully ping the remote endpoint IP \"242.0.255.254\"",
  "spec": {
    "cluster_id": "cluster1",
    "cable_name": "submariner-cable-cluster1-10-21-101-75",
    "healthCheckIP": "242.0.255.254",
    "hostname": "sub1-worker-1.fyre.ibm.com",
    "subnets": [
      "242.0.0.0/16"
    ],
    "private_ip": "10.21.101.75",
    "public_ip": "129.41.87.4",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "udp-port": "4500"
    }
  }
}
 βœ“ Checking Submariner support for the kube-proxy mode
 βœ“ The kube-proxy mode is supported
 βœ— Checking that firewall configuration allows intra-cluster VXLAN traffic
 βœ— The tcpdump output from the sniffer pod does not contain the expected remote endpoint IP 242.0.0.0. Please check that your firewall configuration allows UDP/4800 traffic. Actual pod output:
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vx-submariner, link-type EN10MB (Ethernet), snapshot length 262144 bytes

0 packets captured
0 packets received by filter
0 packets dropped by kernel

 βœ“ Checking that Globalnet is correctly configured and functioning

 βœ“ Checking that services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.19.0
  • Gather information (use subctl gather):

sub1.zip

sub2.zip

  • Cloud provider or hardware configuration:

K8S is installed on ubuntu VM

OS INFO

PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@sub2-master:~# kubectl version
Client Version: v1.30.6
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.6
root@sub1-master:~# kubectl get networkpolicies --all-namespaces
No resources found
root@sub1-master:~#

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 42 (16 by maintainers)

Most upvoted comments

Since this setup is not used anymore and we cannot validate if changing the encapsulation solves it, shall we close these issue? Please feel free to reopen if occurs again.

@yboaron I found a nice blog that you have written recently, Thanks for that

https://medium.com/@yboaron/connecting-k8s-cilium-cluster-and-k8s-calico-cluster-using-submariner-d56d7c39f0cb

Instead of using Calium, I used Calico for both the clusters and I was able to test connectivity

bash-5.0# curl nginx.default.svc.clusterset.local
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
bash-5.0# exit
exit
Session ended, resume using 'kubectl attach tmp-shell -c tmp-shell -i -t' command when the pod is running
pod "tmp-shell" deleted

I used below steps

# Download the latest Kind binary
curl -Lo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64

# Make the Kind binary executable
chmod +x ./kind

# Move the binary to PATH
sudo mv ./kind /usr/local/bin/kind

# Verify the installation
kind --version


# Download the latest stable release of kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

# Make the kubectl binary executable
chmod +x kubectl

# Move the binary to  PATH
sudo mv kubectl /usr/local/bin/kubectl

# Verify the installation
kubectl version --client


# Download the latest Subctl release
curl -Ls https://get.submariner.io | bash
export PATH=$PATH:~/.local/bin
echo export PATH=\$PATH:~/.local/bin >> ~/.profile

# Verify the installation
subctl version

# Install Docker 
snap install docker

# Install Make
apt install make

# Install Kind with Calcio CNI
git clone https://github.com/submariner-io/shipyard.git
cd shipyard/


cat > deploy.two.clusters.nocni.yaml << EOF
nodes: control-plane worker
clusters:
 cluster1:
   cni: none
 cluster2:
   cni: none
EOF

make SETTINGS=deploy.two.clusters.nocni.yaml clusters

# increase  inotify resource limits.
sudo sysctl fs.inotify.max_user_watches=524288
sudo sysctl fs.inotify.max_user_instances=512


## List clusters
kind get clusters

# Check the current Context
export KUBECONFIG=$(find $(git rev-parse --show-toplevel)/output/kubeconfigs/ -type f -printf %p:)
kubectl config get-contexts

# Confirm that we have two nodes in each cluster
kubectl --context cluster1 get nodes
kubectl --context cluster2 get nodes

# Deploy Calico on cluster1

kubectl --context cluster1 create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/tigera-operator.yaml

mkdir calico_manifests
wget -O calico_manifests/custom-resources.yaml  https://raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/custom-resources.yaml

sed -i 's,cidr: 192.168.0.0/16,cidr: 10.130.0.0/16,g' calico_manifests/custom-resources.yaml

sed -i 's,VXLANCrossSubnet,VXLAN,g' calico_manifests/custom-resources.yaml

kubectl --context cluster1 apply -f calico_manifests/custom-resources.yaml

# Install Calico on Cluster 2

kubectl --context cluster2 create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/tigera-operator.yaml

wget -O calico_manifests/custom-resources.yaml https://raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/custom-resources.yaml

sed -i 's,cidr: 192.168.0.0/16,cidr: 10.131.0.0/16,g' calico_manifests/custom-resources.yaml

sed -i 's,VXLANCrossSubnet,VXLAN,g' calico_manifests/custom-resources.yaml

kubectl --context cluster2 apply -f calico_manifests/custom-resources.yaml

# Deploy Submariner

subctl deploy-broker --context cluster1

subctl join --context cluster1  broker-info.subm --clusterid cluster1 --natt=false

subctl join --context cluster2  broker-info.subm --clusterid cluster2 --natt=false

#  check Submariner inter-cluster tunnels status
subctl show connections  --context cluster2   
subctl show connections  --context cluster1

# Verify inter-cluster connectivity
kubectl --context cluster2 create deployment nginx --image=nginx

kubectl --context cluster2 expose deployment nginx --port=80

subctl export service --context cluster2 --namespace default nginx

# Run nettest pod on cluster1 to access the nginx service
kubectl --context cluster1 -n default run tmp-shell --rm -i --tty --image quay.io/submariner/nettest -- /bin/bash

I wanted to test Submariner on normal K8s , OCP etc. and try to establish Cross cluster connectivity and see if that works so I will continue my research on diff Machines and Flavors of K8s but now at least I can test it on Kind using Calico

IIRC, there were other users who also reported problems deploying Kind on Ubuntu in the past, probably due to environment configuration issues

A. I usually apply [1] script on my host before deploying Submariner on Kind, could you check that? B. Also as workaround, you can try deploying Submariner with OVNK as CNI, by running : make deploy using=ovn and see if that helps

[1] `sudo setenforce 0 sudo sysctl fs.inotify.max_user_watches=524288 sudo sysctl fs.inotify.max_user_instances=512

sudo systemctl stop firewalld sudo systemctl disable firewalld sudo dnf install -y iptables-services sudo touch /etc/sysconfig/iptables sudo touch /etc/sysconfig/ip6tables sudo systemctl start iptables sudo systemctl start ip6tables sudo systemctl enable iptables sudo systemctl enable ip6tables sudo iptables -t filter -F sudo iptables -t filter -X sudo sysctl net.bridge.bridge-nf-call-iptables=0 sudo sysctl net.bridge.bridge-nf-call-arptables=0 sudo sysctl net.bridge.bridge-nf-call-ip6tables=0 sudo systemctl restart docker`

Can you try running tcpdump -vv -penni any | grep -i icmp on nongw node and check if you get anything ?

I have created a new issue: #3210. A draft change has been pushed here: #3268.

@yboaron, I haven’t yet looked into linting, unit tests, or e2es testing; I’m just checking if the changes look something like this (Draft linked above). I also modified the loop structure from

       for k := range daemonsets.Items {
               if strings.Contains(daemonsets.Items[k].Name, "flannel") {
                       volumes = daemonsets.Items[k].Spec.Template.Spec.Volumes

to


       for _, ds := range daemonsets.Items {
               if strings.Contains(ds.Name, "flannel") {
                       flannelDaemonSet = &ds
                       volumes = ds.Spec.Template.Spec.Volumes
                       break
                }
        }

to enhance code readability and clarity. I think thid approach makes it clear that ds represents a DaemonSet obj, eliminating the need for indexing. Additionally, by storing a pointer to the found DS and breaking the loop upon finding it, I believe if we do something like this the code becomes more efficient and reduces the risk of errors associated with accessing elements via an index.