kserve: Cannot access KFServing InferenceService through Ingress IAP GCP (Error 503).

/kind bug

What steps did you take and what happened: Cannot access KFServing InferenceService through Ingress (Error 503).

I deployed an instance of Kubeflow (v1.3.0) to GCP accompanied by an KFServing InferenceService. I am unable to access the InferenceService external to the cluster.

The InferenceService is a custom model that was deployed following the custom model tutorial: https://github.com/kubeflow/kfserving/tree/master/docs/samples/v1alpha2/custom/kfserving-custom-model. The manifest:

apiVersion: "serving.kubeflow.org/v1beta1"
kind: InferenceService
metadata:
  annotations:
    sidecar.istio.io/inject: "false"
  name: { Model }
spec:
  predictor:
    containers:
      - name: { Model }
        image: { My Image }

I was able to verify the above deployment by exec-ing into the deployment and submitting a requests.

kubectl exec -it { Model }-predictor-default-00002-deployment-... -- /bin/bash

curl -X POST http://127.0.0.1:8080/v1/models/{ Model }:predict -d '{"instances": [...]}'
>>> {"predictions": "..."}

After verifying the deployment, I followed the GCP IAP guide for KFserving (https://github.com/kubeflow/kfserving/tree/master/docs/samples/gcp-iap). I applied an Istio VirtualService manifest to route external traffic from the ingress gateway to the deployment.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: vs-iap
spec:
  gateways:
  - kubeflow/kubeflow-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /kfserving/default/{ Model }
    route:
    - destination:
        ## knative-local-gateway is in the `knative-serving` namespace.
        ## Attempted to make the request with both of the following.
        # host: knative-local-gateway.knative-serving.svc.cluster.local
        host: knative-local-gateway.istio-system.svc.cluster.local
      headers:
        request:
          set:
            Host: { Model }-predictor-default.default.{ Cluster Name }.endpoints.{ Project }.cloud.goog
      weight: 100
    rewrite:
        uri: /v1/models/{ Model }
    timeout: 300s

Using the following example, I submit a request to the cluster: https://github.com/kubeflow/kubeflow/blob/master/docs/gke/iap_request.py

Request

python3 iap_request.py https://{ Cluster }.endpoints.{ Project }.cloud.goog/kfserving/default/{ Model }:predict xxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com --input=./input.json

Response

Bad response from application: 503 / {'date': 'Fri, 11 Jun 2021 04:03:34 GMT', 'server': 'istio-envoy', 'content-length': '0', 'Via': '1.1 google', 'Alt-Svc': 'clear'} / ''

What did you expect to happen:

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.] During setup I followed the prerequisites (https://github.com/kubeflow/kfserving#prerequisites).

• Istio 1.9.0 required: Istio version is 1.9.3.
• Knative is 0.22.0: No cluster-local-gateway setup needed (knative-local-gateway has replaced cluster-local-gateway)
• Cert Manager: Applied script.
• Applied the inference service config kubectl apply -f https://raw.githubusercontent.com/kubeflow/kfserving/master/config/configmap/inferenceservice.yaml

Related to:

• Issue on older deployments of kubeflow with cluster-local-gateway - https://github.com/kubeflow/kfserving/issues/1393
• Same issue on Kubeflow 1.1: https://github.com/kubeflow/kfserving/issues/1199

The solution in issue 1199 appears to be out of date due to the deprecation of the cluster-local-gateway.

Any help resolving this would be greatly appreciated.

Environment:

• Istio Version: 1.9.3
• Knative Version: v0.22.0
• KFServing Version: v0.5.1 (According to the kubeflow docs https://www.kubeflow.org/docs/components/kfserving/kfserving/)
• Kubeflow version: 1.3.0
• Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
• Minikube version:
• Kubernetes version: (use kubectl version): 1.18.17-gke.1901
• OS (e.g. from /etc/os-release):