kserve: Flowers-sample still broken for KF 1.1 (including potential reason)

/kind bug

What steps did you take and what happened: Kubeflow in EKS installed using: https://github.com/kubeflow/manifests/blob/v1.1-branch/kfdef/kfctl_istio_dex.yaml

flowers-sample deployed exactly as described here: https://github.com/kubeflow/kfserving/tree/master/docs/samples/tensorflow

What did you expect to happen: I would be able to use flowers-sample and kfserving in general

Anything else you would like to add: Just after applying https://github.com/kubeflow/kfserving/tree/master/docs/samples/tensorflow

1. istioctl analyze

Error [IST0101] (VirtualService flowers-sample.default) Referenced gateway not found: "kubeflow-gateway.kubeflow"
Error [IST0101] (VirtualService flowers-sample.default) Referenced host not found: "cluster-local-gateway.istio-system.svc.cluster.local"
Info [IST0118] (Service flowers-sample-predictor-default-6dz95-private.default) Port name queue-metrics (port: 9090, targetPort: queue-metrics) doesn't follow the naming convention of Istio port.
Error: Analyzers found issues when analyzing namespace: default.
See https://istio.io/docs/reference/config/analysis for more information about causes and resolutions.

1. Broken virtualservice configuration - notice the section “gateways: - kubeflow-gateway.kubeflow” shouldn’t it be “gateways: - kubeflow/kubeflow-gateway” ???

kubectl get virtualservices flowers-sample -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"serving.kubeflow.org/v1alpha2","kind":"InferenceService","metadata":{"annotations":{},"name":"flowers-sample","namespace":"default"},"spec":{"default":{"predictor":{"tensorflow":{"storageUri":"gs://kfserving-samples/models/tensorflow/flowers"}}}}}
  creationTimestamp: "2020-08-04T15:31:34Z"
  generation: 1
  name: flowers-sample
  namespace: default
  ownerReferences:
  - apiVersion: serving.kubeflow.org/v1alpha2
    blockOwnerDeletion: true
    controller: true
    kind: InferenceService
    name: flowers-sample
    uid: 9a2208ed-4c9c-4763-8467-d1f747577acf
  resourceVersion: "62233107"
  selfLink: /apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices/flowers-sample
  uid: 3ea1c2e6-0bef-4d79-910e-f26d2642d3b2
spec:
  gateways:
  - kubeflow-gateway.kubeflow
  - knative-serving/cluster-local-gateway
  hosts:
  - flowers-sample.default.example.com
  - flowers-sample.default.svc.cluster.local
  http:
  - match:
    - authority:
        regex: ^flowers-sample\.default\.example\.com(?::\d{1,5})?$
      gateways:
      - kubeflow-gateway.kubeflow
      uri:
        prefix: /v1/models/flowers-sample:predict
    - authority:
        regex: ^flowers-sample\.default(\.svc(\.cluster\.local)?)?(?::\d{1,5})?$
      gateways:
      - knative-serving/cluster-local-gateway
      uri:
        prefix: /v1/models/flowers-sample:predict
    retries:
      attempts: 3
      perTryTimeout: 600s
    route:
    - destination:
        host: cluster-local-gateway.istio-system.svc.cluster.local
        port:
          number: 80
      headers:
        request:
          set:
            Host: flowers-sample-predictor-default.default.svc.cluster.local
      weight: 100

1. How can I modify the virtualservice descriptor? I wanted to change the gateway section so tried to dump vs yaml and then apply it after modification:

kubectl get virtualservices flowers-sample -o yaml /tmp/x.yaml
(modify gateway)
kubectl apply -f /tmp/x.yaml

1. Side note, getting 404 (I guess as expected taking into consideration missconfiguration above):

curl -v -H "Host:flowers-sample.default.example.com" http://cluster-local-gateway.istio-system.svc.cluster.local/v1/models/flowers-sample
> GET /v1/models/flowers-sample HTTP/1.1
> User-Agent: curl/7.35.0
> Accept: */*
> Host:flowers-sample.default.example.com
> 
< HTTP/1.1 404 Not Found
< date: Tue, 04 Aug 2020 14:06:34 GMT
< server: envoy
< content-length: 0
< x-envoy-upstream-service-time: 0

But my changes are ignored and “kubectl get virtualservices flowers-sample -o yaml” still returns “old” version.

Environment:

Default unaltered KF 1.1 installation

• Istio Version: 1.3 (KF 1.1 default)
• Knative Version: 0.11.2
• KFServing Version: 0.3.0
• Kubeflow version: 1.1
• Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
• Minikube version:
• Kubernetes version: (use kubectl version): 17 (v1.17.6-eks)
• OS (e.g. from /etc/os-release):