istio: 503 "upstream connect error or disconnect/reset before headers" in 1.1 with low traffic

Describe the bug Many services in our environment are experiencing a handful (~1% or so) of 503 errors with the response “upstream connect error or disconnect/reset before headers”. I have rolled istio out to two environments and they both experience similar issues.

These environments do not have any Istio gateways – it’s just service to service communication; however, I did replicate the issue using curl through k8s ingress (outside mesh) to a mesh service so I assume it’s isolated to the destination’s sidecar, which returns the 503.

Digging through some older issues, it appears this is related to a keepalive timeout, which does fit my experience. It’s usually my first curl after a long break that fails, if that makes sense with anything. In 1.1, there was a default retry added that I believe was supposed to handle this case but it doesn’t appear to be kicking in for me.

Is there anything that needs to be enabled for the default retry to take effect? Policy checks, for example. I also don’t have Virtual Services or Destination Rules added yet but I was told those are needed for it either. I did do a quick test by adding one and it didn’t help.

Here’s a curl example (through k8s ingress) that I tried this morning after being Idle all night:

 ~ curl -v https://my-service.stg.example.com/v1/available-inventory/201904091406 https://my-service.stg.example.com/v1/available-inventory/201904091406                                                            5m
*   Trying 10.20.23.98...
* TCP_NODELAY set
* Connected to my-service.stg.example.com (10.20.23.98) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=District of Columbia; L=Washington; O=MyCompany, LLC; OU=Engineering; CN=*.stg.example.com
*  start date: Jan 17 00:00:00 2017 GMT
*  expire date: Mar 25 12:00:00 2020 GMT
*  subjectAltName: host "my-service.stg.example.com" matched cert's "*.stg.example.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc42400d400)
> GET /v1/available-inventory/201904091406 HTTP/2
> Host: my-service.stg.example.com
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 503
< server: nginx/1.15.8
< date: Wed, 10 Apr 2019 13:36:44 GMT
< content-type: text/plain
< content-length: 57
< x-envoy-decorator-operation: my-service.default.svc.cluster.local:80/*
< strict-transport-security: max-age=15724800; includeSubDomains
<
* Connection #0 to host my-service.stg.example.com left intact
upstream connect error or disconnect/reset before headers* Found bundle for host my-service.stg.example.com: 0x7fc423c1d2e0 [can multiplex]
* Re-using existing connection! (#0) with host my-service.stg.example.com
* Connected to my-service.stg.example.com (10.20.23.98) port 443 (#0)
* Using Stream ID: 3 (easy handle 0x7fc42400d400)
> GET /v1/available-inventory/201904091406 HTTP/2
> Host: my-service.stg.example.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/2 200
< server: nginx/1.15.8
< date: Wed, 10 Apr 2019 13:36:44 GMT
< content-type: application/json; charset=utf-8
< content-length: 18
< x-powered-by: Express
< etag: W/"12-0TGR6gSl8vAg7WE2Lc4UXbiP8BI"
< x-envoy-upstream-service-time: 4
< x-envoy-decorator-operation: my-service.default.svc.cluster.local:80/*
< strict-transport-security: max-age=15724800; includeSubDomains
<
* Connection #0 to host my-service.stg.example.com left intact
{"inventory":null}

Expected behavior 503s with “upstream connect error or disconnect/reset before headers” are retried by default.

Steps to reproduce the bug Not completely sure, but some key information:

• No Virtual Services or Destination Rules
• 560 pods in 200 services
• Node services mostly (which I understand have some keepalive-related defaults in some Istio issues)

Version Istio 1.1.2 K8S 1.11.8

Installation Helm template / kubectl apply; nothing of note in the helm value overrides – just setting a minimum of 2 for istio pods, rewriting http probes, enabling mtls (permissive), and enabling grafana/kiali.

Environment AWS / Kops