istio: istio-ingressgateway constant 503's - helloworld example

This may or may not be a bug, but I’ve spent close to 16 hours on this now, and I think I’ve read every post online about 503’s from Istio ingress gateways 😉. I think it’s a bug that I fail to debug this with the available Istio tools at the very least.

In short; I’ve deployed the helloworld example on GKE and istio-ingressgateway is always returning 503 Service Unavailable when doing curl http://35.228.214.214/hello -i.

When doing k port-forward svc/helloworld 5000 (in the fou namespace, and then $ curl http://localhost:5000/hello --silent, I get the expected reply: Hello version: v1, instance: helloworld-v1-7bb88866c4-t6jrr`.

I have tried to write a detailed description of the error and some of what I’ve done to debug it in this Stackoverflow question: https://stackoverflow.com/questions/58509666/istio-503s-between-public-gateway-and-service and on the forum and on GKE’s Slack and on the K8S slack (I haven’t tried twitter yet!), but no replies. istioctl reports A-OK on all things (that I can see), kiali the same.

• I’ve tried with and without CNI, with and without SDS and I’ve tried deleting all of Istio and reinstalling it again.
• I can curl http://localhost/hello from within the istio-ingressgateway and it routes successfully to helloworld.fou, giving me an OK reply with the above output (like when I port-forward to it).
• I have mTLS enabled in the cluster, via --values ./istio/helm/istio/values-istio-sds-auth.yaml when Helm-templating.
• It worked before, but then broke.
• I see no errors in the logs, even when I put all proxies to verbose logging

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure [ ] Docs [ ] Installation [x] Networking [ ] Performance and Scalability [ ] Policies and Telemetry [ ] Security [ ] Test and Release [x] User Experience [ ] Developer Infrastructure

Expected behavior

Hello world

Steps to reproduce the bug

See https://github.com/istio/istio/issues/18221 but point to GKE and use values-istio-sds-auth.yaml with the helloworld sample and gateway and a public static IP configured on the istio-ingressgateway service (as it’s then the LoadBalancer type). Curl the IP.

Version (include the output of istioctl version --remote and kubectl version)

istioctl version --remote
client version: 1.3.3
citadel version: 1.3.3
galley version: 1.3.3
ingressgateway version: 1.3.3
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
pilot version: 1.3.3
policy version: 1.3.3
sidecar-injector version: 1.3.3
telemetry version: 1.3.3

kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.7", GitCommit:"8fca2ec50a6133511b771a11559e24191b1aa2b4", GitTreeState:"clean", BuildDate:"2019-09-18T14:47:22Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6-gke.13", GitCommit:"acdb9a03a6dc0f7f62d7acdda75c9a9faca50fee", GitTreeState:"clean", BuildDate:"2019-09-20T23:13:58Z", GoVersion:"go1.12.9b4", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?

The helm-template way from the install guide. See previous issue for Makefile that does an installation.

Environment where bug was observed (cloud vendor, OS, etc) GKE, 1.14

Will do if you want me to.

I’m sorry if this is not a bug, but I have no idea how to understand why I’m getting 503’s back. It’s probably something small, but I’m really just using the samples/helloworld deployment, so I don’t see how I can make a mistake with that. And I don’t understand how it can work to curl from the ingress to the service, from the ingress itself, but not when the request comes from the outside. Perhaps I’m missing a debugging step, but then again, I’ve tried what I could find in the docs 😦

Thanks for your help and understanding