submariner: subctl diagnose all passes all checks but service discovery does not work
I have with the following ip ranges:
cluster 1:
"cluster-cidr": "10.46.0.0/16",
"service-cidr": "10.45.0.0/16"
cluster 2:
"cluster-cidr": "10.42.0.0/16",
"service-cidr": "10.43.0.0/16"
I joined the clusters using a command like this
subctl join --kubeconfig cluster1.yaml broker-info.subm --clusterid cluster-1 --context cluster-4f50834d --natt=false --air-gapped --health-check=false --force-udp-encaps
subctl show all for each cluster shows the connections and endpoints properly and all connected.
The nginx service exported successfully
kubectl --kubeconfig cluster2.yaml get serviceexports.multicluster.x-k8s.io nginx -o json
{
"apiVersion": "multicluster.x-k8s.io/v1alpha1",
"kind": "ServiceExport",
"metadata": {
"creationTimestamp": "2024-03-29T01:10:07Z",
"generation": 1,
"name": "nginx",
"namespace": "default",
"resourceVersion": "24281",
"uid": "8cf288b4-5484-4288-96aa-96ad4dc6cac9"
},
"status": {
"conditions": [
{
"lastTransitionTime": "2024-03-29T01:43:35Z",
"message": "",
"reason": "",
"status": "True",
"type": "Valid"
},
{
"lastTransitionTime": "2024-03-29T01:10:08Z",
"message": "Service was successfully exported to the broker",
"reason": "",
"status": "True",
"type": "Ready"
}
]
}
}
the service import looks normal:
kubectl --kubeconfig cluster1.yaml get serviceimports.multicluster.x-k8s.io nginx -o json
{
"apiVersion": "multicluster.x-k8s.io/v1alpha1",
"kind": "ServiceImport",
"metadata": {
"creationTimestamp": "2024-03-29T01:44:09Z",
"generation": 1,
"name": "nginx",
"namespace": "default",
"resourceVersion": "25296",
"uid": "be999abd-1326-4585-9e9f-946940113ba1"
},
"spec": {
"ports": [
{
"port": 80,
"protocol": "TCP"
}
],
"type": "ClusterSetIP"
},
"status": {
"clusters": [
{
"cluster": "cluster-2"
}
]
}
}
The result of dig shows the DNS is working fine.
dig nginx.default.svc.clusterset.local
; <<>> DiG 9.16.6 <<>> nginx.default.svc.clusterset.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29308
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 106f10ae4159f478 (echoed)
;; QUESTION SECTION:
;nginx.default.svc.clusterset.local. IN A
;; ANSWER SECTION:
nginx.default.svc.clusterset.local. 5 IN A 10.43.51.67
;; Query time: 0 msec
;; SERVER: 10.45.0.10#53(10.45.0.10)
;; WHEN: Fri Mar 29 05:10:41 UTC 2024
;; MSG SIZE rcvd: 125
the subctl diagnose pass all checks
subctl diagnose all --kubeconfig cluster1.yaml
Cluster "cluster-4f50834d"
✓ Checking Submariner support for the Kubernetes version
✓ Kubernetes version "v1.27.9+rke2r1" is supported
✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
✓ Checking DaemonSet "submariner-gateway"
✓ Checking DaemonSet "submariner-routeagent"
✓ Checking DaemonSet "submariner-metrics-proxy"
✓ Checking Deployment "submariner-lighthouse-agent"
✓ Checking Deployment "submariner-lighthouse-coredns"
✓ Checking the status of all Submariner pods
✓ Checking that gateway metrics are accessible from non-gateway nodes
✓ Skipping this check as it's a single node cluster
✓ Checking Submariner support for the CNI network plugin
✓ The detected CNI network plugin ("calico") is supported
✓ Calico CNI detected, checking if the Submariner IPPool pre-requisites are configured
✓ Checking gateway connections
✓ Checking Submariner support for the kube-proxy mode
✓ The kube-proxy mode is supported
✓ Checking that firewall configuration allows intra-cluster VXLAN traffic
✓ Skipping this check as it's a single node cluster
✓ Checking that services have been exported properly
The endpointslice in the broker does exist
kubectl get endpointslices.discovery.k8s.io -n submariner-k8s-broker -o json
{
"apiVersion": "v1",
"items": [
{
"addressType": "IPv4",
"apiVersion": "discovery.k8s.io/v1",
"endpoints": [
{
"addresses": [
"10.43.51.67"
],
"conditions": {
"ready": true
}
}
],
"kind": "EndpointSlice",
"metadata": {
"annotations": {
"lighthouse.submariner.io/globalnet-enabled": "false",
"lighthouse.submariner.io/publish-not-ready-addresses": "false"
},
"creationTimestamp": "2024-03-29T01:43:38Z",
"generateName": "nginx-",
"generation": 1,
"labels": {
"app": "nginx",
"endpointslice.kubernetes.io/managed-by": "lighthouse-agent.submariner.io",
"lighthouse.submariner.io/is-headless": "false",
"lighthouse.submariner.io/sourceNamespace": "default",
"multicluster.kubernetes.io/service-name": "nginx",
"multicluster.kubernetes.io/source-cluster": "cluster-2",
"submariner-io/clusterID": "cluster-2",
"submariner-io/originatingNamespace": "default"
},
"name": "nginx-ppdwn",
"namespace": "submariner-k8s-broker",
"resourceVersion": "394372",
"uid": "e7afd998-5f98-4e8a-8935-068c16ddcf11"
},
"ports": [
{
"name": "",
"port": 80,
"protocol": "TCP"
}
]
}
],
"kind": "List",
"metadata": {
"resourceVersion": "",
"selfLink": ""
}
}
But curl does not return any response back. Anything else I am missing? I followed all the steps in this document as well https://submariner.io/operations/troubleshooting/ and everything seems normal and healthy. How can I debug from now on? any help would be great.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 20 (6 by maintainers)
@adibrastegarnia please stop pinging individual contributors — you don’t know who the best person to take care of your issue is, and we triage bugs regularly so your issue will get attention even without pings.