submariner: Exported service results in timeout from consuming onPremise cluster
Hello everyone,
I have an issue with a Submariner usecase where I want to separate the control-plane network (used by the broker to communicate with the participating clusters) and the data plane network (used by participating clusters to connect my applications through the gateways).
I have deployed 3 clusters :
- 1 Broker cluster called
e2e-mgmt - 2 deployment clusters called
domain-2anddomain-3
Note that each deployment clusters have 3 nodes. Each node has 2 interfaces :
- 1 for the control plane : 172.16.100.0/22 dev enp1s0
- 1 for the data plane : 172.16.110.0/22 dev enp2s0
The nodes are actually virtual machines. Some tests showed that each VM can reach the other
communicating through the control-plane or the data-plane (ex: from 172.16.100.10 to 172.16.100.11 but not from 172.16.100.10 to 172.16.110.11).
I labeled and annotated all my nodes in domain-2 and domain-3 like :
$ kubectl annotate --list node --all --context domain-3 | grep submariner.io/public-ip
gateway.submariner.io/public-ip=ipv4:172.16.110.84
gateway.submariner.io/public-ip=ipv4:172.16.110.85
gateway.submariner.io/public-ip=ipv4:172.16.110.86
$ kubectl label --list nodes --all --context domain-3 | grep submariner
submariner.io/gateway=true
submariner.io/gateway=true
submariner.io/gateway=true
Because I want the VPN tunnels to communicate through 172.16.110.0/24 (data plane) even if the Kubernetes APIs are listening to 172.16.100.0/24 (control plane).
I created the broker and joined the clusters using :
subctl deploy-broker --globalnet --context e2e-mgmt
subctl join broker-info.subm --clustercidr "10.42.0.0/16" --globalnet --context domain-2 --clusterid domain-2
subctl join broker-info.subm --clustercidr "10.42.0.0/16" --globalnet --context domain-3 --clusterid domain-3
Indeed, the clusters joined the broker :
kubectl -n submariner-k8s-broker get clusters.submariner.io --context e2e-mgmt
NAME AGE
domain-2 6h10m
domain-3 5h38m
The connections seem good :
$ subctl show connections
subctl show connections
Cluster "domain-2"
✓ Showing Connections
GATEWAY CLUSTER REMOTE IP NAT CABLE DRIVER SUBNETS STATUS RTT avg.
hermione domain-3 172.16.100.85 no libreswan 242.1.0.0/16 connected 998.16µs
Cluster "domain-3"
✓ Showing Connections
GATEWAY CLUSTER REMOTE IP NAT CABLE DRIVER SUBNETS STATUS RTT avg.
porthos domain-2 172.16.100.81 no libreswan 242.0.0.0/16 connected 859.482µs
Cluster "e2e-mgmt"
⚠ Submariner connectivity feature is not installed
Or maybe the remote IPs should be on 172.16.110.0/24 ?
The gateways seem good :
$ subctl show gateways
Cluster "domain-3"
✓ Showing Gateways
NODE HA STATUS SUMMARY
harry passive There are no connections
hermione active All connections (1) are established
ronald passive There are no connections
Cluster "e2e-mgmt"
⚠ Submariner connectivity feature is not installed
Cluster "domain-2"
✓ Showing Gateways
NODE HA STATUS SUMMARY
aramis passive There are no connections
athos passive There are no connections
porthos active All connections (1) are established
I created a Nginx service in domain-2 and namespace hello-domain-2 called hello-world-svc.
Using netshoot, I can tell that the requests are working locally from domain-2
kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot --context domain-2
> curl hello-world-svc.hello-domain-2.svc.clusterset.local. -sSLI
> # HTTP/1.1 200 OK
I exported the service and domain-3 consumed properly the ServiceImport from the Broker :
$ kubectl get serviceimport -n hello-domain-2 --context domain-3
NAME TYPE IP AGE
hello-world-svc ClusterSetIP 10m
However, the same test with netshoot from domain-3 does not work :
kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot --context domain-3
> curl hello-world-svc.hello-domain-2.svc.clusterset.local. -sSLI
> # TIMEOUT
Even if dig resolves it properly :
$ dig hello-world-svc.hello-domain-2.svc.clusterset.local.
; <<>> DiG 9.18.21 <<>> hello-world-svc.hello-domain-2.svc.clusterset.local.
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57307
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: f53618936495597c (echoed)
;; QUESTION SECTION:
;hello-world-svc.hello-domain-2.svc.clusterset.local. IN A
;; ANSWER SECTION:
hello-world-svc.hello-domain-2.svc.clusterset.local. 5 IN A 242.0.255.253
;; Query time: 8 msec
;; SERVER: 10.43.0.10#53(10.43.0.10) (UDP)
;; WHEN: Fri Mar 08 15:41:07 UTC 2024
;; MSG SIZE rcvd: 159
What did I do wrong ?
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 27 (14 by maintainers)
@IceManGreen , I’m going to close this topic, feel free to reopen it if it’s still relevant
Hello @yboaron , sorry for the very late answer too !
Yes it is !
Regarding the rest of your answer, thank you so much for guiding me through so much details, it helps a lot. Unfortunately, I did not succeed in managing the routes and make things work in the end. Since CIlium is adding its own routing logic that comes in conflicts with Submariner, it makes the solution very hard to manage because I was planning to scale things up in terms machines and clusters … So I will probably look for another solution, like Calico + Submariner if they fit together. This is what I recommend for anyone who faces the same kind of trouble with Cilium and Submariner.
Thanks again for your help ! I think we are good with this issue !
Hi @IceManGreen , Sorry for the late response,
In your case Submariner failed to detect Cilium and used “generic” CNI, Submariner refers to generic CNI as a kube-proxy/iptables based CNI.
I checked Submariner logs from both clusters, found no errors. Submariner route-agent detects ‘cilium_host’ interface as the CNI interface (interface with IP address from clusterCIDR) and updates its rp_filter to ‘2’.
From subctl verify output, it appears that all the connectivity tests in which one of client or server pods is running on non GW nodes are failing, further datapath investigation is needed here.
I think I understand why the TCP SYN/ACK packet is wrongly handled by cilium (it should be handled by Submariner), x-wing pod IP 10.10.2.95, and checking domain-2/hermione node ip-routing table I can see:
10.10.0.0/16 via 240.19.112.84 dev vx-submariner proto static 10.10.1.0/24 via 10.12.1.86 dev cilium_host proto kernel src 10.12.1.86 mtu 1370 10.10.2.0/24 via 10.12.1.86 dev cilium_host proto kernel src 10.12.1.86 mtu 1370
So, packet should be routed by Submariner according to 10.10.0.0/16 via 240.19.112.84 dev vx-submariner proto static
but some component added some routes that includes remote cluster CIDR , like: 10.10.2.0/24 via 10.12.1.86 dev cilium_host proto kernel src 10.12.1.86 mtu 1370
in this case kernel (Longest prefix match) will choose to route the packet wrongly using 10.10.2.0/24 via 10.12.1.86 dev cilium_host proto kernel src 10.12.1.86 mtu 1370 and not via vx-submariner.
I can see similar routes also on domain1 cluster, for example : domain-1/aramis node:
ip route show default via 172.16.110.1 dev enp2s0 10.10.0.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 10.10.0.46 dev cilium_host proto kernel scope link 10.10.1.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1450 10.10.2.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1450 10.12.0.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1370 10.12.0.0/16 via 240.19.112.81 dev vx-submariner proto static 10.12.1.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1370 10.12.2.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1370 10.13.0.0/16 via 240.19.112.81 dev vx-submariner proto static 10.14.0.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1370 10.14.1.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1370 10.14.2.0/24 via 10.10.0.46 dev cilium_host proto kernel src 10.10.0.46 mtu 1370 172.16.100.0/22 dev enp1s0 proto kernel scope link src 172.16.100.82 172.16.110.0/22 dev enp2s0 proto kernel scope link src 172.16.110.82 240.0.0.0/8 dev vx-submariner proto kernel scope link src 240.19.116.82
You should first address these routes issue
Hello @yboaron, yes I agree that the combination of the three is suspicious. I will execute the tests you ask and I was thinking about making the same tests with Cilium instead of Flannel, just in case.
Unfortunately, due to the Kubecon EU, I will not be able to test these, this week. I will keep you in touch as soon as I can.
Thanks again !
Ok so I made some tests following your recommendations :
10.44.0.0/16and services in10.45.0.0/16for clusterdomain-2(--cluster-cidr=10.44.0.0/16 --service-cidr=10.45.0.0/16)10.46.0.0/16and services in10.47.0.0/16for clusterdomain-3(--cluster-cidr=10.46.0.0/16 --service-cidr=10.47.0.0/16)joincommands and I removed every--globalnetoption from them. Something like :But again, the tests fail.
Here are the results of
subctl gatherandsubctl diagnose all:diagnose-202403121620.log submariner-gather-202403121620.zip
Since you labelled all 3 nodes as GWs on both clusters , tests that need to run one of the test pods on non-GW node will fail [1] while tests that need to run client pod and listener pod on GW will succeed [2] .
Can you label(submariner.io/gateway=true) only a single node as GW on each cluster and rerun subctl verify ?
[1] `[0mBasic TCP connectivity tests across overlapping clusters without discovery [38;5;243mwhen a pod connects via TCP to the globalIP of a remote service [0mwhen the pod is on a gateway and the remote service is not on a gateway [38;5;9m[1m[It] should have sent the expected data from the pod to the other pod[0m [38;5;204m[dataplane, globalnet][0m [38;5;243mgithub.com/submariner-io/submariner@v0.17.0/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53[0m
[38;5;9m[FAILED] Failed to await pod ready. Pod “tcp-check-listenermxz66” is still pending: status: { “phase”: “Pending”, “conditions”: [ { “type”: “PodScheduled”, “status”: “False”, “lastProbeTime”: null, “lastTransitionTime”: “2024-03-11T09:05:50Z”, “reason”: “Unschedulable”, “message”: “0/3 nodes are available: 3 node(s) didn’t match Pod’s node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling…” } ], “qosClass”: “BestEffort” } `
[2] `[0mBasic TCP connectivity tests across overlapping clusters without discovery [38;5;243mwhen a pod connects via TCP to the globalIP of a remote service [0mwhen the pod is on a gateway and the remote service is on a gateway [0m[1mshould have sent the expected data from the pod to the other pod[0m [38;5;204m[dataplane, globalnet, basic][0m [38;5;243mgithub.com/submariner-io/submariner@v0.17.0/test/e2e/dataplane/tcp_gn_pod_connectivity.go:53[0m [1mSTEP:[0m Creating namespace objects with basename “dataplane-gn-conn-nd” [38;5;243m@ 03/11/24 09:09:51.484[0m [1mSTEP:[0m Generated namespace “e2e-tests-dataplane-gn-conn-nd-zz54k” in cluster “domain-2” to execute the tests in [38;5;243m@ 03/11/24 09:09:51.737[0m [1mSTEP:[0m Creating namespace “e2e-tests-dataplane-gn-conn-nd-zz54k” in cluster “domain-3” [38;5;243m@ 03/11/24 09:09:51.737[0m [1mSTEP:[0m Creating a listener pod in cluster “domain-3”, which will wait for a handshake over TCP [38;5;243m@ 03/11/24 09:09:52.507[0m [1mSTEP:[0m Pointing a ClusterIP service to the listener pod in cluster “domain-3” [38;5;243m@ 03/11/24 09:10:04.21[0m [1mSTEP:[0m Creating a connector pod in cluster “domain-2”, which will attempt the specific UUID handshake over TCP [38;5;243m@ 03/11/24 09:10:07.866[0m Mar 11 09:10:21.482: INFO: ExecWithOptions &{Command:[sh -c for j in $(seq 1 50); do echo [dataplane] connector says 67dbefdb-e262-4c91-a4bd-ac2c20eb151a; done | for i in $(seq 2); do if nc -v 242.1.255.253 1234 -w 60; then break; else sleep 30; fi; done] Namespace:e2e-tests-dataplane-gn-conn-nd-zz54k PodName:customl55cl ContainerName:connector-pod Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true} [1mSTEP:[0m Connector pod is scheduled on node “porthos” [38;5;243m@ 03/11/24 09:10:22.741[0m [1mSTEP:[0m Waiting for the listener pod “tcp-check-listenerrbzc9” on node “hermione” to exit, returning what listener sent [38;5;243m@ 03/11/24 09:10:22.741[0m Mar 11 09:10:29.806: INFO: Pod “tcp-check-listenerrbzc9” on node “hermione” output: listening on 0.0.0.0:1234 … connect to 10.42.1.37:1234 from 242.0.0.8:33109 (242.0.0.8:33109) [dataplane] connector says 67dbefdb-e262-4c91-a4bd-ac2c20eb151a
[1mSTEP:[0m Verifying that the listener got the connector’s data and the connector got the listener’s data [38;5;243m@ 03/11/24 09:10:31.115[0m [1mSTEP:[0m Verifying the output of the listener pod contains a cluster-scoped global IP [242.0.0.1 242.0.0.2 242.0.0.3 242.0.0.4 242.0.0.5 242.0.0.6 242.0.0.7 242.0.0.8] of the connector Pod [38;5;243m@ 03/11/24 09:10:31.115[0m [1mSTEP:[0m Deleting service “test-svc-tcp-check-listener” on “domain-3” [38;5;243m@ 03/11/24 09:10:31.873[0m [1mSTEP:[0m Deleting namespace “e2e-tests-dataplane-gn-conn-nd-zz54k” on cluster “domain-2” [38;5;243m@ 03/11/24 09:10:32.83[0m [1mSTEP:[0m Deleting namespace “e2e-tests-dataplane-gn-conn-nd-zz54k” on cluster “domain-3” [38;5;243m@ 03/11/24 09:10:33.62[0m [38;5;10m• [43.343 seconds][0m `
@sridhargaddam hello ! Thanks for your answer.
I use K3S with Flannel :
It seems that I have failing tests on domain-2 and domain-3 because the pods that must run the tests cannot be deployed.
The events mention :
What affinity/selector should have the nodes to be able to deploy the pods for tests ?
See attachment for the entire file.
verifiy.log
See attachment (sorry github only supports zip files, not tar).
submariner-gather.zip
To verify this, I installed HTTP servers on every domain listening on :
enp1s0on port8080enp2s0on port8081Test for control-plane (enp1s0) :
Everything works fine for simple
curlcommands.@IceManGreen we would need some additional info to narrow down the problem. Can you clarify the following.
subctl verify --context <kubeContext1> --tocontext <kubeContext2> --only connectivity --verbose. This will help us to know if the connectivity is working between the Gateway nodes but failing when client/server is on a non-Gateway node. Note: Sincesubctl show connectionsis looking good, I think Gateway to Gateway communication is probably working in your setup, but when one of them is on non-Gateway, its failing.subctl gatherfrom domain-2 and domain-3 clusters.