submariner: Cross-Cluster Service Connectivity Fails with "Host is Unreachable" Despite Successful DNS Resolution in Submariner GlobalNet Setup
What happened:
I deployed Submariner with GlobalNet across two Kubernetes clusters. DNS resolution works as expected, but connectivity to services across clusters fails with a Host is unreachable error.
More info is available in below link
https://kubernetes.slack.com/archives/C010RJV694M/p1730390376380879
What you expected to happen:
curl requests from a pod in cluster2 to a service exposed via Submariner in cluster1 should succeed, indicating that cross-cluster communication is functioning.
How to reproduce it (as minimally and precisely as possible):
- Set up two Kubernetes clusters and deploy Submariner with GlobalNet enabled.
- cluster1 GlobalNet CIDR: 242.0.0.0/16
- cluster2 GlobalNet CIDR: 243.0.0.0/16
- Deploy an nginx pod in cluster1, expose it as a service, and export it using Submariner.
- Deploy a test pod (tmp-shell) in cluster2.
- Attempt to access the nginx service in cluster1 from tmp-shell in cluster2 using DNS (nginx-cluster1.default.svc.clusterset.local) or the resolved GlobalNet IP.
Anything else we need to know?:
Environment:
- Diagnose information (use
subctl diagnose all):
Cluster 1 info
Aswin π₯π₯π₯ $ subctl diagnose all --kubeconfig /Users/aswina/Downloads/sub1
Cluster "sub1"
β Checking Submariner support for the Kubernetes version
β Kubernetes version "v1.30.6" is supported
β Globalnet deployment detected - checking that globalnet CIDRs do not overlap
β Error getting the Broker's REST config: error getting auth rest config: Get "https://9.66.245.122:6443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/any": tls: failed to verify certificate: x509: "kube-apiserver" certificate is not trusted
β Checking Submariner support for the CNI network plugin
β Submariner could not detect the CNI network plugin and is using ("generic") plugin. It may or may not work.
β Checking gateway connections
β Checking route agent connections
β Connection to cluster "cluster2" is not established. Connection details:
{
"status": "error",
"statusMessage": "Failed to successfully ping the remote endpoint IP \"243.0.255.254\"",
"spec": {
"cluster_id": "cluster2",
"cable_name": "submariner-cable-cluster2-10-21-82-227",
"healthCheckIP": "243.0.255.254",
"hostname": "sub2-worker-1.fyre.ibm.com",
"subnets": [
"243.0.0.0/16"
],
"private_ip": "10.21.82.227",
"public_ip": "129.41.87.3",
"nat_enabled": true,
"backend": "libreswan",
"backend_config": {
"natt-discovery-port": "4490",
"preferred-server": "false",
"udp-port": "4500"
}
}
}
β Connection to cluster "cluster2" is not established. Connection details:
{
"status": "error",
"statusMessage": "Failed to successfully ping the remote endpoint IP \"243.0.255.254\"",
"spec": {
"cluster_id": "cluster2",
"cable_name": "submariner-cable-cluster2-10-21-82-227",
"healthCheckIP": "243.0.255.254",
"hostname": "sub2-worker-1.fyre.ibm.com",
"subnets": [
"243.0.0.0/16"
],
"private_ip": "10.21.82.227",
"public_ip": "129.41.87.3",
"nat_enabled": true,
"backend": "libreswan",
"backend_config": {
"natt-discovery-port": "4490",
"preferred-server": "false",
"udp-port": "4500"
}
}
}
β Connection to cluster "cluster2" is not established. Connection details:
{
"status": "error",
"statusMessage": "Failed to successfully ping the remote endpoint IP \"243.0.255.254\"",
"spec": {
"cluster_id": "cluster2",
"cable_name": "submariner-cable-cluster2-10-21-82-227",
"healthCheckIP": "243.0.255.254",
"hostname": "sub2-worker-1.fyre.ibm.com",
"subnets": [
"243.0.0.0/16"
],
"private_ip": "10.21.82.227",
"public_ip": "129.41.87.3",
"nat_enabled": true,
"backend": "libreswan",
"backend_config": {
"natt-discovery-port": "4490",
"preferred-server": "false",
"udp-port": "4500"
}
}
}
β Checking Submariner support for the kube-proxy mode
β The kube-proxy mode is supported
β Checking that firewall configuration allows intra-cluster VXLAN traffic
β The tcpdump output from the sniffer pod does not contain the expected remote endpoint IP 243.0.0.0. Please check that your firewall configuration allows UDP/4800 traffic. Actual pod output:
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vx-submariner, link-type EN10MB (Ethernet), snapshot length 262144 bytes
0 packets captured
0 packets received by filter
0 packets dropped by kernel
β Checking that Globalnet is correctly configured and functioning
β Checking that services have been exported properly
Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.
subctl version: v0.19.0
Aswin π₯π₯π₯ $
Cluster 2 info
Aswin π₯π₯π₯ $ subctl diagnose all --kubeconfig /Users/aswina/Downloads/sub2
Cluster "sub2"
β Checking Submariner support for the Kubernetes version
β Kubernetes version "v1.30.6" is supported
β Globalnet deployment detected - checking that globalnet CIDRs do not overlap
β Error getting the Broker's REST config: error getting auth rest config: Get "https://9.66.245.122:6443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/any": tls: failed to verify certificate: x509: "kube-apiserver" certificate is not trusted
β Checking Submariner support for the CNI network plugin
β Submariner could not detect the CNI network plugin and is using ("generic") plugin. It may or may not work.
β Checking gateway connections
β Checking route agent connections
β Connection to cluster "cluster1" is not established. Connection details:
{
"status": "error",
"statusMessage": "Failed to successfully ping the remote endpoint IP \"242.0.255.254\"",
"spec": {
"cluster_id": "cluster1",
"cable_name": "submariner-cable-cluster1-10-21-101-75",
"healthCheckIP": "242.0.255.254",
"hostname": "sub1-worker-1.fyre.ibm.com",
"subnets": [
"242.0.0.0/16"
],
"private_ip": "10.21.101.75",
"public_ip": "129.41.87.4",
"nat_enabled": true,
"backend": "libreswan",
"backend_config": {
"natt-discovery-port": "4490",
"preferred-server": "false",
"udp-port": "4500"
}
}
}
β Connection to cluster "cluster1" is not established. Connection details:
{
"status": "error",
"statusMessage": "Failed to successfully ping the remote endpoint IP \"242.0.255.254\"",
"spec": {
"cluster_id": "cluster1",
"cable_name": "submariner-cable-cluster1-10-21-101-75",
"healthCheckIP": "242.0.255.254",
"hostname": "sub1-worker-1.fyre.ibm.com",
"subnets": [
"242.0.0.0/16"
],
"private_ip": "10.21.101.75",
"public_ip": "129.41.87.4",
"nat_enabled": true,
"backend": "libreswan",
"backend_config": {
"natt-discovery-port": "4490",
"preferred-server": "false",
"udp-port": "4500"
}
}
}
β Connection to cluster "cluster1" is not established. Connection details:
{
"status": "error",
"statusMessage": "Failed to successfully ping the remote endpoint IP \"242.0.255.254\"",
"spec": {
"cluster_id": "cluster1",
"cable_name": "submariner-cable-cluster1-10-21-101-75",
"healthCheckIP": "242.0.255.254",
"hostname": "sub1-worker-1.fyre.ibm.com",
"subnets": [
"242.0.0.0/16"
],
"private_ip": "10.21.101.75",
"public_ip": "129.41.87.4",
"nat_enabled": true,
"backend": "libreswan",
"backend_config": {
"natt-discovery-port": "4490",
"preferred-server": "false",
"udp-port": "4500"
}
}
}
β Checking Submariner support for the kube-proxy mode
β The kube-proxy mode is supported
β Checking that firewall configuration allows intra-cluster VXLAN traffic
β The tcpdump output from the sniffer pod does not contain the expected remote endpoint IP 242.0.0.0. Please check that your firewall configuration allows UDP/4800 traffic. Actual pod output:
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vx-submariner, link-type EN10MB (Ethernet), snapshot length 262144 bytes
0 packets captured
0 packets received by filter
0 packets dropped by kernel
β Checking that Globalnet is correctly configured and functioning
β Checking that services have been exported properly
Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.
subctl version: v0.19.0
- Gather information (use
subctl gather):
- Cloud provider or hardware configuration:
K8S is installed on ubuntu VM
OS INFO
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@sub2-master:~# kubectl version
Client Version: v1.30.6
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.6
root@sub1-master:~# kubectl get networkpolicies --all-namespaces
No resources found
root@sub1-master:~#
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 42 (16 by maintainers)
Since this setup is not used anymore and we cannot validate if changing the encapsulation solves it, shall we close these issue? Please feel free to reopen if occurs again.
@yboaron I found a nice blog that you have written recently, Thanks for that
https://medium.com/@yboaron/connecting-k8s-cilium-cluster-and-k8s-calico-cluster-using-submariner-d56d7c39f0cb
Instead of using Calium, I used Calico for both the clusters and I was able to test connectivity
I used below steps
I wanted to test Submariner on normal K8s , OCP etc. and try to establish Cross cluster connectivity and see if that works so I will continue my research on diff Machines and Flavors of K8s but now at least I can test it on Kind using Calico
IIRC, there were other users who also reported problems deploying Kind on Ubuntu in the past, probably due to environment configuration issues
A. I usually apply [1] script on my host before deploying Submariner on Kind, could you check that? B. Also as workaround, you can try deploying Submariner with OVNK as CNI, by running : make deploy using=ovn and see if that helps
[1] `sudo setenforce 0 sudo sysctl fs.inotify.max_user_watches=524288 sudo sysctl fs.inotify.max_user_instances=512
sudo systemctl stop firewalld sudo systemctl disable firewalld sudo dnf install -y iptables-services sudo touch /etc/sysconfig/iptables sudo touch /etc/sysconfig/ip6tables sudo systemctl start iptables sudo systemctl start ip6tables sudo systemctl enable iptables sudo systemctl enable ip6tables sudo iptables -t filter -F sudo iptables -t filter -X sudo sysctl net.bridge.bridge-nf-call-iptables=0 sudo sysctl net.bridge.bridge-nf-call-arptables=0 sudo sysctl net.bridge.bridge-nf-call-ip6tables=0 sudo systemctl restart docker`
Can you try running
tcpdump -vv -penni any | grep -i icmpon nongw node and check if you get anything ?I have created a new issue: #3210. A draft change has been pushed here: #3268.
@yboaron, I havenβt yet looked into linting, unit tests, or e2es testing; Iβm just checking if the changes look something like this (Draft linked above). I also modified the loop structure from
to
to enhance code readability and clarity. I think thid approach makes it clear that ds represents a DaemonSet obj, eliminating the need for indexing. Additionally, by storing a pointer to the found DS and breaking the loop upon finding it, I believe if we do something like this the code becomes more efficient and reduces the risk of errors associated with accessing elements via an index.