submariner: Exported service results in timeout from consuming onPremise cluster

Hello everyone,

I have an issue with a Submariner usecase where I want to separate the control-plane network (used by the broker to communicate with the participating clusters) and the data plane network (used by participating clusters to connect my applications through the gateways).

I have deployed 3 clusters :

  • 1 Broker cluster called e2e-mgmt
  • 2 deployment clusters called domain-2 and domain-3

Note that each deployment clusters have 3 nodes. Each node has 2 interfaces :

  • 1 for the control plane : 172.16.100.0/22 dev enp1s0
  • 1 for the data plane : 172.16.110.0/22 dev enp2s0

The nodes are actually virtual machines. Some tests showed that each VM can reach the other communicating through the control-plane or the data-plane (ex: from 172.16.100.10 to 172.16.100.11 but not from 172.16.100.10 to 172.16.110.11).

I labeled and annotated all my nodes in domain-2 and domain-3 like :

$ kubectl annotate --list node --all --context domain-3 | grep submariner.io/public-ip
gateway.submariner.io/public-ip=ipv4:172.16.110.84
gateway.submariner.io/public-ip=ipv4:172.16.110.85
gateway.submariner.io/public-ip=ipv4:172.16.110.86
$ kubectl label --list nodes --all  --context domain-3 | grep submariner
submariner.io/gateway=true
submariner.io/gateway=true
submariner.io/gateway=true

Because I want the VPN tunnels to communicate through 172.16.110.0/24 (data plane) even if the Kubernetes APIs are listening to 172.16.100.0/24 (control plane).

I created the broker and joined the clusters using :

subctl deploy-broker --globalnet --context e2e-mgmt
subctl join broker-info.subm --clustercidr "10.42.0.0/16" --globalnet --context domain-2 --clusterid domain-2
subctl join broker-info.subm --clustercidr "10.42.0.0/16" --globalnet --context domain-3 --clusterid domain-3

Indeed, the clusters joined the broker :

kubectl -n submariner-k8s-broker get clusters.submariner.io --context e2e-mgmt
NAME       AGE
domain-2   6h10m
domain-3   5h38m

The connections seem good :

$ subctl show connections
subctl show connections
Cluster "domain-2"
 ✓ Showing Connections
GATEWAY    CLUSTER    REMOTE IP       NAT   CABLE DRIVER   SUBNETS        STATUS      RTT avg.    
hermione   domain-3   172.16.100.85   no    libreswan      242.1.0.0/16   connected   998.16µs    

Cluster "domain-3"
 ✓ Showing Connections 
GATEWAY   CLUSTER    REMOTE IP       NAT   CABLE DRIVER   SUBNETS        STATUS      RTT avg.     
porthos   domain-2   172.16.100.81   no    libreswan      242.0.0.0/16   connected   859.482µs

Cluster "e2e-mgmt"
 ⚠ Submariner connectivity feature is not installed

Or maybe the remote IPs should be on 172.16.110.0/24 ?

The gateways seem good :

$ subctl show gateways 
Cluster "domain-3"
 ✓ Showing Gateways
NODE       HA STATUS   SUMMARY                               
harry      passive     There are no connections              
hermione   active      All connections (1) are established   
ronald     passive     There are no connections              

Cluster "e2e-mgmt"
 ⚠ Submariner connectivity feature is not installed

Cluster "domain-2"
 ✓ Showing Gateways 
NODE      HA STATUS   SUMMARY                               
aramis    passive     There are no connections              
athos     passive     There are no connections              
porthos   active      All connections (1) are established

I created a Nginx service in domain-2 and namespace hello-domain-2 called hello-world-svc.
Using netshoot, I can tell that the requests are working locally from domain-2

kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot --context domain-2
> curl hello-world-svc.hello-domain-2.svc.clusterset.local. -sSLI
> # HTTP/1.1 200 OK

I exported the service and domain-3 consumed properly the ServiceImport from the Broker :

$ kubectl get serviceimport -n hello-domain-2 --context domain-3
NAME              TYPE           IP    AGE
hello-world-svc   ClusterSetIP         10m

However, the same test with netshoot from domain-3 does not work :

kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot --context domain-3
> curl hello-world-svc.hello-domain-2.svc.clusterset.local. -sSLI
> # TIMEOUT

Even if dig resolves it properly :

$ dig hello-world-svc.hello-domain-2.svc.clusterset.local.
; <<>> DiG 9.18.21 <<>> hello-world-svc.hello-domain-2.svc.clusterset.local.
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57307
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: f53618936495597c (echoed)
;; QUESTION SECTION:
;hello-world-svc.hello-domain-2.svc.clusterset.local. IN        A

;; ANSWER SECTION:
hello-world-svc.hello-domain-2.svc.clusterset.local. 5 IN A 242.0.255.253

;; Query time: 8 msec
;; SERVER: 10.43.0.10#53(10.43.0.10) (UDP)
;; WHEN: Fri Mar 08 15:41:07 UTC 2024
;; MSG SIZE  rcvd: 159

What did I do wrong ?

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 27 (14 by maintainers)

Most upvoted comments

@IceManGreen , I’m going to close this topic, feel free to reopen it if it’s still relevant

Hello @yboaron , sorry for the very late answer too !

Are “10.10.0.0/16” and “10.11.0.0/16” for domain1 and “10.12.0.0/16” and “10.13.0.0/16” for domain2 the clusters CIDRS?

Yes it is !

Regarding the rest of your answer, thank you so much for guiding me through so much details, it helps a lot. Unfortunately, I did not succeed in managing the routes and make things work in the end. Since CIlium is adding its own routing logic that comes in conflicts with Submariner, it makes the solution very hard to manage because I was planning to scale things up in terms machines and clusters … So I will probably look for another solution, like Calico + Submariner if they fit together. This is what I recommend for anyone who faces the same kind of trouble with Cilium and Submariner.

Thanks again for your help ! I think we are good with this issue !

Hi @IceManGreen , Sorry for the late response,

  1. Yep, Submariner doesn’t support Cilium detection, the CNI auto detection is used in Submariner for: A. clustercidr and servicecidr auto-discovery (which you workaround by providing --clustercidr “10.10.0.0/16” --servicecidr “10.11.0.0/16” to subctl join) Are “10.10.0.0/16” and “10.11.0.0/16” for domain1 and “10.12.0.0/16” and “10.13.0.0/16” for domain2 the clusters CIDRS? B. Run some specific routing configuration if needed for this CNI , for example check this for Calico CNI.

In your case Submariner failed to detect Cilium and used “generic” CNI, Submariner refers to generic CNI as a kube-proxy/iptables based CNI.

  1. I checked Submariner logs from both clusters, found no errors. Submariner route-agent detects ‘cilium_host’ interface as the CNI interface (interface with IP address from clusterCIDR) and updates its rp_filter to ‘2’.

  2. From subctl verify output, it appears that all the connectivity tests in which one of client or server pods is running on non GW nodes are failing, further datapath investigation is needed here.

  3. I think I understand why the TCP SYN/ACK packet is wrongly handled by cilium (it should be handled by Submariner), x-wing pod IP 10.10.2.95, and checking domain-2/hermione node ip-routing table I can see:

10.10.0.0/16 via 240.19.112.84 dev vx-submariner proto static 10.10.1.0/24 via 10.12.1.86 dev cilium_host proto kernel src 10.12.1.86 mtu 1370 10.10.2.0/24 via 10.12.1.86 dev cilium_host proto kernel src 10.12.1.86 mtu 1370

So, packet should be routed by Submariner according to 10.10.0.0/16 via 240.19.112.84 dev vx-submariner proto static
but some component added some routes that includes remote cluster CIDR , like: 10.10.2.0/24 via 10.12.1.86 dev cilium_host proto kernel src 10.12.1.86 mtu 1370

in this case kernel (Longest prefix match) will choose to route the packet wrongly using 10.10.2.0/24 via 10.12.1.86 dev cilium_host proto kernel src 10.12.1.86 mtu 1370 and not via vx-submariner.

I can see similar routes also on domain1 cluster, for example : domain-1/aramis node:

ip route show default via 172.16.110.1 dev enp2s0 10.10.0.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 10.10.0.46 dev cilium_host proto kernel scope link 10.10.1.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1450 10.10.2.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1450 10.12.0.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1370 10.12.0.0/16 via 240.19.112.81 dev vx-submariner proto static 10.12.1.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1370 10.12.2.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1370 10.13.0.0/16 via 240.19.112.81 dev vx-submariner proto static 10.14.0.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1370 10.14.1.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1370 10.14.2.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1370 172.16.100.0/22 dev enp1s0 proto kernel scope link src 172.16.100.82 172.16.110.0/22 dev enp2s0 proto kernel scope link src 172.16.110.82 240.0.0.0/8 dev vx-submariner proto kernel scope link src 240.19.116.82

You should first address these routes issue

Hello @yboaron, yes I agree that the combination of the three is suspicious. I will execute the tests you ask and I was thinking about making the same tests with Cilium instead of Flannel, just in case.

Unfortunately, due to the Kubecon EU, I will not be able to test these, this week. I will keep you in touch as soon as I can.

Thanks again !

Ok so I made some tests following your recommendations :

  • I installed wireguard on the gateway nodes for both of the clusters
  • I installed K3S again to modify the cluster CIDR and the service CIDR :
    • pods in 10.44.0.0/16 and services in 10.45.0.0/16 for cluster domain-2 (--cluster-cidr=10.44.0.0/16 --service-cidr=10.45.0.0/16)
    • pods in 10.46.0.0/16 and services in 10.47.0.0/16 for cluster domain-3 (--cluster-cidr=10.46.0.0/16 --service-cidr=10.47.0.0/16)
  • next, I provided the former cidrs to the join commands and I removed every --globalnet option from them. Something like :
subctl deploy-broker --context e2e-mgmt
subctl join broker-info.subm --clustercidr "10.44.0.0/16" --servicecidr "10.45.0.0/16" --cable-driver wireguard --clusterid domain-2 --context domain-2
subctl join broker-info.subm --clustercidr "10.46.0.0/16" --servicecidr "10.47.0.0/16" --cable-driver wireguard --clusterid domain-3 --context domain-3

But again, the tests fail.

Here are the results of subctl gather and subctl diagnose all :

diagnose-202403121620.log submariner-gather-202403121620.zip

Since you labelled all 3 nodes as GWs on both clusters , tests that need to run one of the test pods on non-GW node will fail [1] while tests that need to run client pod and listener pod on GW will succeed [2] .

Can you label(submariner.io/gateway=true) only a single node as GW on each cluster and rerun subctl verify ?

[1] `Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is not on a gateway [It] should have sent the expected data from the pod to the other pod [dataplane, globalnet] github.com/submariner-io/submariner@v0.17.0/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53

[FAILED] Failed to await pod ready. Pod “tcp-check-listenermxz66” is still pending: status: { “phase”: “Pending”, “conditions”: [ { “type”: “PodScheduled”, “status”: “False”, “lastProbeTime”: null, “lastTransitionTime”: “2024-03-11T09:05:50Z”, “reason”: “Unschedulable”, “message”: “0/3 nodes are available: 3 node(s) didn’t match Pod’s node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling…” } ], “qosClass”: “BestEffort” } `

[2] `Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is on a gateway should have sent the expected data from the pod to the other pod [dataplane, globalnet, basic] github.com/submariner-io/submariner@v0.17.0/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53 STEP: Creating namespace objects with basename “dataplane-gn-conn-nd” @ 03/11/24 09:09:51.484 STEP: Generated namespace “e2e-tests-dataplane-gn-conn-nd-zz54k” in cluster “domain-2” to execute the tests in @ 03/11/24 09:09:51.737 STEP: Creating namespace “e2e-tests-dataplane-gn-conn-nd-zz54k” in cluster “domain-3” @ 03/11/24 09:09:51.737 STEP: Creating a listener pod in cluster “domain-3”, which will wait for a handshake over TCP @ 03/11/24 09:09:52.507 STEP: Pointing a ClusterIP service to the listener pod in cluster “domain-3” @ 03/11/24 09:10:04.21 STEP: Creating a connector pod in cluster “domain-2”, which will attempt the specific UUID handshake over TCP @ 03/11/24 09:10:07.866 Mar 11 09:10:21.482: INFO: ExecWithOptions &{Command:[sh -c for j in $(seq 1 50); do echo [dataplane] connector says 67dbefdb-e262-4c91-a4bd-ac2c20eb151a; done | for i in $(seq 2); do if nc -v 242.1.255.253 1234 -w 60; then break; else sleep 30; fi; done] Namespace:e2e-tests-dataplane-gn-conn-nd-zz54k PodName:customl55cl ContainerName:connector-pod Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true} STEP: Connector pod is scheduled on node “porthos” @ 03/11/24 09:10:22.741 STEP: Waiting for the listener pod “tcp-check-listenerrbzc9” on node “hermione” to exit, returning what listener sent @ 03/11/24 09:10:22.741 Mar 11 09:10:29.806: INFO: Pod “tcp-check-listenerrbzc9” on node “hermione” output: listening on 0.0.0.0:1234 … connect to 10.42.1.37:1234 from 242.0.0.8:33109 (242.0.0.8:33109) [dataplane] connector says 67dbefdb-e262-4c91-a4bd-ac2c20eb151a

STEP: Verifying that the listener got the connector’s data and the connector got the listener’s data @ 03/11/24 09:10:31.115 STEP: Verifying the output of the listener pod contains a cluster-scoped global IP [242.0.0.1 242.0.0.2 242.0.0.3 242.0.0.4 242.0.0.5 242.0.0.6 242.0.0.7 242.0.0.8] of the connector Pod @ 03/11/24 09:10:31.115 STEP: Deleting service “test-svc-tcp-check-listener” on “domain-3” @ 03/11/24 09:10:31.873 STEP: Deleting namespace “e2e-tests-dataplane-gn-conn-nd-zz54k” on cluster “domain-2” @ 03/11/24 09:10:32.83 STEP: Deleting namespace “e2e-tests-dataplane-gn-conn-nd-zz54k” on cluster “domain-3” @ 03/11/24 09:10:33.62 • [43.343 seconds] `

@sridhargaddam hello ! Thanks for your answer.

What is the CNI that is used in your clusters?

I use K3S with Flannel :

$ /var/lib/rancher/k3s/data/current/bin/flannel
CNI Plugin flannel version v0.22.2 (linux/amd64) commit HEAD built on 2024-02-06T01:58:54Z

What is the Submariner version?

$ subctl show versions
Cluster "domain-2"
 ✓ Showing versions 
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING                     ARCH    
submariner-gateway              quay.io/submariner   0.17.0       release-0.17-72c0e6dd56c8   amd64   
submariner-routeagent           quay.io/submariner   0.17.0       release-0.17-72c0e6dd56c8   amd64   
submariner-globalnet            quay.io/submariner   0.17.0       release-0.17-72c0e6dd56c8   amd64   
submariner-metrics-proxy        quay.io/submariner   0.17.0       release-0.17-81b7e55f5306   amd64   
submariner-operator             quay.io/submariner   0.17.0       release-0.17-d750fbdcb610   amd64   
submariner-lighthouse-agent     quay.io/submariner   0.17.0       release-0.17-7ad4dd387b0b   amd64   
submariner-lighthouse-coredns   quay.io/submariner   0.17.0       release-0.17-7ad4dd387b0b   amd64   

Cluster "domain-3"
 ✓ Showing versions 
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING                     ARCH    
submariner-gateway              quay.io/submariner   0.17.0       release-0.17-72c0e6dd56c8   amd64   
submariner-routeagent           quay.io/submariner   0.17.0       release-0.17-72c0e6dd56c8   amd64   
submariner-globalnet            quay.io/submariner   0.17.0       release-0.17-72c0e6dd56c8   amd64   
submariner-metrics-proxy        quay.io/submariner   0.17.0       release-0.17-81b7e55f5306   amd64   
submariner-operator             quay.io/submariner   0.17.0       release-0.17-d750fbdcb610   amd64   
submariner-lighthouse-agent     quay.io/submariner   0.17.0       release-0.17-7ad4dd387b0b   amd64   
submariner-lighthouse-coredns   quay.io/submariner   0.17.0       release-0.17-7ad4dd387b0b   amd64   

Cluster "e2e-mgmt"
 ✓ Showing versions 
COMPONENT             REPOSITORY           CONFIGURED   RUNNING                     ARCH    
submariner-operator   quay.io/submariner   0.17.0       release-0.17-d750fbdcb610   amd64

Also, can you run subctl verify --context <kubeContext1> --tocontext <kubeContext2> --only connectivity --verbose ?

It seems that I have failing tests on domain-2 and domain-3 because the pods that must run the tests cannot be deployed.
The events mention :

"message": "0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.."

What affinity/selector should have the nodes to be able to deploy the pods for tests ?

See attachment for the entire file.

verifiy.log

Please attach the tar of subctl gather from domain-2 and domain-3 clusters.

See attachment (sorry github only supports zip files, not tar).

submariner-gather.zip

Also, did you check if your use-case (i.e., curl to remote cluster service) is working fine when both control-plane and data-plane were using the same interface?

To verify this, I installed HTTP servers on every domain listening on :

  • enp1s0 on port 8080
  • enp2s0 on port 8081

Test for control-plane (enp1s0) :

# from domain-3 to domain-2
# control-plane
curl 172.16.100.81:8080 -sSLI
HTTP/1.0 200 OK
# data plane
curl 172.16.110.81:8081 -sSLI
HTTP/1.0 200 OK

# from domain-2 to domain-3
# control plane
curl 172.16.100.84:8080 -sSLI
HTTP/1.0 200 OK
# data plane
curl 172.16.110.84:8081 -sSLI
HTTP/1.0 200 OK

Everything works fine for simple curl commands.

@IceManGreen we would need some additional info to narrow down the problem. Can you clarify the following.

  1. What is the CNI that is used in your clusters?
  2. What is the Submariner version?
  3. Also, can you run subctl verify --context <kubeContext1> --tocontext <kubeContext2> --only connectivity --verbose. This will help us to know if the connectivity is working between the Gateway nodes but failing when client/server is on a non-Gateway node. Note: Since subctl show connections is looking good, I think Gateway to Gateway communication is probably working in your setup, but when one of them is on non-Gateway, its failing.
  4. Please attach the tar of subctl gather from domain-2 and domain-3 clusters.
  5. Also, did you check if your use-case (i.e., curl to remote cluster service) is working fine when both control-plane and data-plane were using the same interface?