istio-operator: Significant performance degrade (ingress pods) with Istio 1.1

Describe the bug Ingress latency for HTTPS increases dramatically that it will actually cause timeouts when trying to establish a TLS handshake with the ingress pods.

Once the TLS handshake has been established with the ingress pods, the process speed to the VirtualService and the actual Service are pretty fast.

Steps to reproduce the issue: On a Kubernetes cluster in AWS (we create our cluster from kubeadm so not using EKS),

1. Install Istio through their officially supported Helm installation (v1.0.7)
2. Create a Gateway to attach to the generated ingress load balancer and update the DNS to point to the ELB
3. Create a cert as secret and attach to the Gateway created in 2 for TLS.
4. Deploy the sample nginx deployment in kubernetes docs with a VirtualService
5. Try running curl -v https://<url-from-4> a couple times or use something like lolcat

Expected behavior Networking behavior shouldn’t change including latency to the ingress pods regardless if it’s TLS or not.

Screenshots This is copied from an actual output when I was running curl. It takes like 5 ~ 10 seconds to even get the initial response from the ingress and it’s slow enough that I can actually terminate the request before the handshake even starts.

❯ curl -v https://somewebsite.com/health
*   Trying 52.27.17.16...
* TCP_NODELAY set
* Connected to somewebsite.entelo.com (52.27.17.16) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
^C

Additional context branch: release-1.0 istio: v1.0.7 kubernetes: v1.14.1 not EKS certs: generated from Let's Encrypt with cert-manager

We were trying to switch to the 1.0 Istio operator before doing an upgrade but this issue caused a cluster outage and we had to revert back to the Helm installation. I tried digging around looking for the diffs between the Helm installation and the operator and there literally aren’t much that seems to be different except for the stdio and stdiotcp rules the Operator adds to Mixer.

Our Istio installation and usage are fairly simple and the settings are very close to the default.

Also, reverting back to using the Helm installation immediately resolves all networking issues as well so there are definitely something going on when using the operator.

Trying a fresh install of Istio with the operator also seems to cause the same issues as well.