submariner: exported service address can't be resolved from MWC

What happened:

I have set up two openshift clusters(mcscluster1 and mcscluster2) with different CIDRs; And then use mcscluster1 ad submariner broker, and then join the mcscluster1 and mcscluster2 to the broker. I have verified the submariner is working fine by deploying and exporting nginx application in mcscluster2 and access it from mcscluster1 with FQDN(nginx.default.svc.clusterset.local:8080).

Then in mcscluster1, I deploy a service(istiod in istio-system namespace) and export it with serviceexport:

# oc --context=mcscluster1 -n istio-system get svc
NAME     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                 AGE
istiod   ClusterIP   172.30.235.61   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP   99m
# oc --context=mcscluster1  -n istio-system get serviceexport
NAME     AGE
istiod     90m

and in mcscluster2, I deploy a WMC(MutatingWebhookConfiguration) with URL pointed to exported istiod service address(istiod.istio-system.svc.clusterset.local):

# oc --context=mcscluster2 get mutatingwebhookconfiguration istio-sidecar-injector -o yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
...
webhooks:
- admissionReviewVersions:
  - v1beta1
  - v1
  clientConfig:
    caBundle: xxxx===
    url: https://istiod.istio-system.svc.clusterset.local:443/inject
  name: namespace.sidecar-injector.istio.io
  namespaceSelector:
    matchExpressions:
    - key: istio-injection
      operator: In
      values:
      - enabled
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: '*'
  sideEffects: None

The WMC(MutatingWebhookConfiguration) indicates it will call the exported service(URL: istiod.istio-system.svc.clusterset.local) from openshift’s apiserver each time a new pod is created in namespace with label istio-injection=enabled.

However, when I create pod in namespace sample with label istio-injection=enabled, the pod can’t be created because the call from openshift’s apiserver to the exported service failed with DNS lookup error:

# oc -n sample describe rs helloworld-v1-776f57d5f6
Name:           helloworld-v1-776f57d5f6
Namespace:      sample
...
Events:
  Type     Reason        Age                  From                   Message
  ----     ------        ----                 ----                   -------
  Warning  FailedCreate  3m8s (x20 over 23m)  replicaset-controller  Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc.clusterset.local:443/inject:ENV:cluster=mcscluster2:ENV:net=network1?timeout=10s": dial tcp: lookup istiod.istio-system.svc.clusterset.local on 10.0.0.2:53: no such host

If I check the logs of openshift apiserver, I can find the similar logs:

W0618 07:56:03.504351      19 dispatcher.go:182] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc.clusterset.local:443/inject:ENV:cluster=mcscluster2:ENV:net=network1?timeout=10s": dial tcp: lookup istiod.istio-system.svc.clusterset.local on 10.0.0.2:53: no such host

The oopenshift clusters are using OVN network, and the apioserver is hostNetwork, I’m not sure if this is issue of my environment, since I found the OVN and hostNetwork is support from submariner 0.8, after https://github.com/submariner-io/submariner/pull/1020

What you expected to happen:

I expected the request from hostnetwork pod with OVN network to exported service can be fulfilled.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Actually, I try to replace the URL in MWC with the ip of istiod service, the request still can’t connected to destination service. But, before we debug on that, we should resolve the DNS issue.

Environment:

• Submariner version (use subctl version): 0.9.0
• Kubernetes version (use kubectl version):
• Diagnose information (use subctl diagnose all):
• Gather information (use subctl gather)
• Cloud provider or hardware configuration: openshift on AWS
• OS (e.g cat /etc/os-release):
• Kernel (e.g uname -a):
• Install tools:
• Others: oopenshift 4.8.0-rc0