submariner: subctl diagnose all passes all checks but service discovery does not work

I have with the following ip ranges:

cluster 1: 
    "cluster-cidr": "10.46.0.0/16",
     "service-cidr": "10.45.0.0/16"

cluster 2:
 "cluster-cidr": "10.42.0.0/16", 
   "service-cidr": "10.43.0.0/16"

I joined the clusters using a command like this

subctl join --kubeconfig cluster1.yaml broker-info.subm --clusterid cluster-1 --context cluster-4f50834d  --natt=false --air-gapped   --health-check=false --force-udp-encaps

subctl show all for each cluster shows the connections and endpoints properly and all connected.

The nginx service exported successfully

kubectl --kubeconfig cluster2.yaml get serviceexports.multicluster.x-k8s.io nginx -o json
{
    "apiVersion": "multicluster.x-k8s.io/v1alpha1",
    "kind": "ServiceExport",
    "metadata": {
        "creationTimestamp": "2024-03-29T01:10:07Z",
        "generation": 1,
        "name": "nginx",
        "namespace": "default",
        "resourceVersion": "24281",
        "uid": "8cf288b4-5484-4288-96aa-96ad4dc6cac9"
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2024-03-29T01:43:35Z",
                "message": "",
                "reason": "",
                "status": "True",
                "type": "Valid"
            },
            {
                "lastTransitionTime": "2024-03-29T01:10:08Z",
                "message": "Service was successfully exported to the broker",
                "reason": "",
                "status": "True",
                "type": "Ready"
            }
        ]
    }
}

the service import looks normal:

kubectl --kubeconfig cluster1.yaml get serviceimports.multicluster.x-k8s.io nginx -o json
{
    "apiVersion": "multicluster.x-k8s.io/v1alpha1",
    "kind": "ServiceImport",
    "metadata": {
        "creationTimestamp": "2024-03-29T01:44:09Z",
        "generation": 1,
        "name": "nginx",
        "namespace": "default",
        "resourceVersion": "25296",
        "uid": "be999abd-1326-4585-9e9f-946940113ba1"
    },
    "spec": {
        "ports": [
            {
                "port": 80,
                "protocol": "TCP"
            }
        ],
        "type": "ClusterSetIP"
    },
    "status": {
        "clusters": [
            {
                "cluster": "cluster-2"
            }
        ]
    }
}

The result of dig shows the DNS is working fine.

dig nginx.default.svc.clusterset.local

; <<>> DiG 9.16.6 <<>> nginx.default.svc.clusterset.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29308
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 106f10ae4159f478 (echoed)
;; QUESTION SECTION:
;nginx.default.svc.clusterset.local. IN A

;; ANSWER SECTION:
nginx.default.svc.clusterset.local. 5 IN A      10.43.51.67

;; Query time: 0 msec
;; SERVER: 10.45.0.10#53(10.45.0.10)
;; WHEN: Fri Mar 29 05:10:41 UTC 2024
;; MSG SIZE  rcvd: 125

the subctl diagnose pass all checks

subctl diagnose all --kubeconfig cluster1.yaml
Cluster "cluster-4f50834d"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.27.9+rke2r1" is supported

 ✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap 
 ✓ Checking DaemonSet "submariner-gateway"
 ✓ Checking DaemonSet "submariner-routeagent"
 ✓ Checking DaemonSet "submariner-metrics-proxy"
 ✓ Checking Deployment "submariner-lighthouse-agent"
 ✓ Checking Deployment "submariner-lighthouse-coredns"
 ✓ Checking the status of all Submariner pods
 ✓ Checking that gateway metrics are accessible from non-gateway nodes
 ✓ Skipping this check as it's a single node cluster

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("calico") is supported
 ✓ Calico CNI detected, checking if the Submariner IPPool pre-requisites are configured
 ✓ Checking gateway connections
 ✓ Checking Submariner support for the kube-proxy mode 
 ✓ The kube-proxy mode is supported
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic
 ✓ Skipping this check as it's a single node cluster

 ✓ Checking that services have been exported properly

The endpointslice in the broker does exist

kubectl get endpointslices.discovery.k8s.io -n submariner-k8s-broker -o json
{
    "apiVersion": "v1",
    "items": [
        {
            "addressType": "IPv4",
            "apiVersion": "discovery.k8s.io/v1",
            "endpoints": [
                {
                    "addresses": [
                        "10.43.51.67"
                    ],
                    "conditions": {
                        "ready": true
                    }
                }
            ],
            "kind": "EndpointSlice",
            "metadata": {
                "annotations": {
                    "lighthouse.submariner.io/globalnet-enabled": "false",
                    "lighthouse.submariner.io/publish-not-ready-addresses": "false"
                },
                "creationTimestamp": "2024-03-29T01:43:38Z",
                "generateName": "nginx-",
                "generation": 1,
                "labels": {
                    "app": "nginx",
                    "endpointslice.kubernetes.io/managed-by": "lighthouse-agent.submariner.io",
                    "lighthouse.submariner.io/is-headless": "false",
                    "lighthouse.submariner.io/sourceNamespace": "default",
                    "multicluster.kubernetes.io/service-name": "nginx",
                    "multicluster.kubernetes.io/source-cluster": "cluster-2",
                    "submariner-io/clusterID": "cluster-2",
                    "submariner-io/originatingNamespace": "default"
                },
                "name": "nginx-ppdwn",
                "namespace": "submariner-k8s-broker",
                "resourceVersion": "394372",
                "uid": "e7afd998-5f98-4e8a-8935-068c16ddcf11"
            },
            "ports": [
                {
                    "name": "",
                    "port": 80,
                    "protocol": "TCP"
                }
            ]
        }
    ],
    "kind": "List",
    "metadata": {
        "resourceVersion": "",
        "selfLink": ""
    }
}

But curl does not return any response back. Anything else I am missing? I followed all the steps in this document as well https://submariner.io/operations/troubleshooting/ and everything seems normal and healthy. How can I debug from now on? any help would be great.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 20 (6 by maintainers)

Most upvoted comments

@adibrastegarnia please stop pinging individual contributors — you don’t know who the best person to take care of your issue is, and we triage bugs regularly so your issue will get attention even without pings.