istio: http alpn override applied to external services breaks connections

Describe the bug

Connections to external HTTP services configured with SIMPLE_TLS destination rules are aborted by the external server or envoy due to non-negotiated alpn errors.

[ ] Configuration Infrastructure [ ] Docs [ ] Installation [ x] Networking [ ] Performance and Scalability [ ] Policies and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure

Expected behavior

Envoy and the external service should connect using http 1.1. or h2. Istio/envoy should not send an ALPN list consisting entirely of unsupported protocols (ie, [“istio-http1.1”, “istio”]). Unsupported protocols are not ignored by servers.

In the event that the server supports no protocols that the client advertises, then the server SHALL respond with a fatal “no_application_protocol” alert. https://tools.ietf.org/html/rfc7301#section-3.1

Steps to Reproduce

Apply:

apiVersion: apps/v1                  
kind: Deployment
metadata:                               
  name: sleep     
spec:    
  selector:      
    matchLabels:
      app: sleep
  template:  
    metadata:
      annotations:
      labels:            
        app: sleep
    spec:
      containers:                       
      - name: sleep  
        image: pstauffer/curl
        command: ["/bin/sleep", "3650d"]
        imagePullPolicy: IfNotPresent
--- 
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: http2.pro
spec:
  hosts:
  - http2.pro
  ports:
  - name: http
    number: 443
    protocol: HTTP
  location: MESH_EXTERNAL
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: http2.pro
spec:
  host: http2.pro
  trafficPolicy:
    tls:
      mode: SIMPLE

Attempt to access external service and observe failure:

$ kubectl exec -t sleep-69bd787db7-tmvqn -n istio-monitoring -c sleep -- curl -v -s --http2  http://http2.pro:443/api/v1
*   Trying 8.9.5.7...
* TCP_NODELAY set
* Connected to http2.pro (8.9.5.7) port 443 (#0)
> GET /api/v1 HTTP/1.1
> Host: http2.pro:443
> User-Agent: curl/7.60.0
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
>
upstream connect error or disconnect/reset before headers. reset reason: connection failure< HTTP/1.1 503 Service Unavailable
< content-length: 91
< content-type: text/plain
< date: Fri, 12 Jun 2020 01:50:04 GMT
< server: envoy
<

Envoy log:

[2020-06-12T01:50:04.423Z] "GET /api/v1 HTTP/1.1" 503 "upstream_reset_before_response_started{connection failure,TLS error: 268435715:SSL routines:OPENSSL_internal:INVALID_ALPN_PROTOCOL 268435605:SSL rout
ines:OPENSSL_internal:ERROR_PARSING_EXTENSION 268435646:SSL routines:OPENSSL_internal:PARSE_TLSEXT}" UF,URX "-" "TLS error: 268435715:SSL routines:OPENSSL_internal:INVALID_ALPN_PROTOCOL 268435605:SSL rout
ines:OPENSSL_internal:ERROR_PARSING_EXTENSION 268435646:SSL routines:OPENSSL_internal:PARSE_TLSEXT" 0 91 503 - "-" "curl/7.60.0" "dc47bd2e-7827-46df-a123-c52f5733e96c" "http2.pro:443" "8.9.5.7:443" outbou
nd|443||http2.pro - 8.9.5.7:443 172.23.5.168:51367 - default 

Analysis

We first noticed connections to external services defined in ServiceEntries with envoy TLS origination no longer worked when we upgraded from 1.4.X to 1.5.4.

It appears that the outbound HTTP connection manager always overrides the alpn with istio-h2 or istio-http1.1. Could it add the non-istio variants as well? https://github.com/istio/istio/blob/1.5.4/pilot/pkg/networking/core/v1alpha3/listener.go#L1802

Response from 1.4.4:

$ kubectl exec -t sleep-69bd787db7-hb66q -n istio-monitoring -c sleep -- curl  -s --http2  http://http2.pro:443/api/v1
{"http2":0,"protocol":"HTTP\/1.1","push":0,"user_agent":"curl\/7.60.0"}