kubeflow: [GCP] istio-ingressgateway failing health check on /healthz/ready

/kind bug

I just deployed kubeflow using kfctl_gcp_iap.yaml from master. When I checked the ingress the backend (istio-ingressready) was reported as unhealthy with 0 successful health checks.

The health check was set to /healthz/ready which looks like the intended result https://github.com/kubeflow/manifests/blob/21160c1463a46645a5b70e465988723ec1df6ccc/gcp/iap-ingress/base/config-map.yaml#L186

However when I ported forwarded to the ingress gateway I got a 401 on /healthz/ready

curl -I http://localhost:8080/healthz/ready
HTTP/1.1 401 Unauthorized
content-length: 29
content-type: text/plain
date: Thu, 10 Oct 2019 05:01:07 GMT
server: istio-envoy

/healthz gave me a 200.

curl -I http://localhost:8080/healthz
HTTP/1.1 200 OK
x-powered-by: Express
accept-ranges: bytes
cache-control: public, max-age=0
last-modified: Wed, 09 Oct 2019 22:56:09 GMT
etag: W/"599-16db2bc9f28"
content-type: text/html; charset=UTF-8
content-length: 1433
date: Thu, 10 Oct 2019 05:01:35 GMT
x-envoy-upstream-service-time: 2
server: istio-envoy

So I deleted the backend-updater and in the UI set the health check to /healthz.

The health check passed and my backend came up.

So it looks like the health check is incorrect.

Did something change?

Could this explain why the endpoint test is ready is failing kubeflow/kfctl#42

/cc @zhenghuiwang @lluunn