submariner: can't access to k8s api server by cluster IP across cluster

I successfully deployed submariner to my 3 k8s clusters(version 1.19.7) created by kubeadm, all works well include pod/pod, pod/service communication and service discovery across cluster, but I can’t access to k8s api server by cluster IP across cluster. for example, the pod on managed cluster-a can’t access cluster-b api server. BTW, the k3s cluster don’t have such issue according to doc https://submariner.io/getting-started/quickstart/k3s/

user case

We have many k8s clusters located at different IDC(only have outbound network) with k8s version less than 1.17, it is hard to manage since they don’t have a unified control panel, in addition, the pod/pod, pod/service communication and service discovery crosses cluster doesn’t work as well. To address this issue, we want to create a submariner broker cluster at public cloud, then other k8s clusters at IDC join to this broker cluster so that enable unified control panel crosses cluster, the broker cluster will access to managed cluster api server by cluster IP/service directly for life cycle management

reproduce steps

create 3 k8s clusters by kubeadm with no pod/service cidr overlap
install calico CNI and config ippool according to https://submariner.io/operations/deployment/calico/
deploy broker on cluster-a
join cluster-a , cluster-b and cluster-c to broker cluster cluster-a
try verify-manually case https://submariner.io/getting-started/quickstart/k3s/#verify-manually, all works

case success: access to `nginx` service and cluster IP

deploy nginx to cluster-b, run a test pod on cluster-a to verify

bash-5.0# curl nginx.default.svc.clusterset.local
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
bash-5.0# curl 10.145.6.100
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
bash-5.0#

case failed: access to k8s api server by cluster IP

on cluster-b

root@vm-0-127-ubuntu:~# kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.145.0.1     <none>        443/TCP   5h14m
nginx        ClusterIP   10.145.6.100   <none>        80/TCP    135m

on cluster-a

root@vm-0-80-ubuntu:~# kubectl  -n default  run tmp-shell --rm -i --tty --image quay.io/submariner/nettest -- /bin/bash
If you don't see a command prompt, try pressing enter.
bash-5.0# curl -k https://10.145.0.1:443
curl: (28) Operation timed out after 300466 milliseconds with 0 out of 0 bytes received
bash-5.0#

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (7 by maintainers)

Most upvoted comments

Ok, so for:

OpenshiftSDN to -> OVN it doesn’t work
OVN->OpenshiftSDN it works.

I suspect the issue is different in the case of OVN because the network architecture at the border is a little bit different and iptables is not used.

We will need to look at the issues individually I believe.

mangelajo on Jun 3, 2021