istio: Cannot call a gRPC service from ingress gateway, will be remote reset with 503

Bug Description

Our ingress gateway can not talk to gRPC service. It will be remote reset with 503 as:

* TCP_NODELAY set
* Connected to httpbin.example.com (100.127.1.175) port 31028 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: example.com.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
...
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fad1180d600)
> GET /echo HTTP/2
> Host:httpbin.example.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 503
< content-length: 85
< content-type: text/plain
< date: Mon, 13 Dec 2021 20:44:01 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 64
<
* Connection #0 to host httpbin.example.com left intact
upstream connect error or disconnect/reset before headers. reset reason: remote reset* Closing connection 0

HTTP service works fine with this gateway:

* TCP_NODELAY set
* Connected to httpbin.example.com (100.127.129.48) port 31028 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: example.com.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
...
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fb73800d600)
> GET / HTTP/2
> Host:httpbin.example.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 200
< date: Mon, 13 Dec 2021 23:43:38 GMT
< content-length: 0
< x-envoy-upstream-service-time: 0
< server: istio-envoy

I’ve already tried to set "envoy.reloadable_features.preserve_downstream_scheme": false flag in envoy_bootstrap after reading this issue, but it doesn’t work. (And our server is not a .NET service)

Here is an example of our setup gateway:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: ingressgateway
  namespace: ingressgateway
spec:
  selector:
    app: ingressgateway
  servers:
  - hosts:
    - "*.example.com"
    port:
      number: 8443
      name: https-8443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: httpbin-credential # must be the same as secret

virtual service

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin
  namespace: httpbin
spec:
  hosts:
  - httpbin.example.com
  gateways:
  - ingressgateway/ingressgateway
  http:
  - route:
    - destination:
        host: [service PQDN]
        port:
          number: 8079 // The grpc port.

Version

istiod version: 1.11.4
data plane version: 1.11.4(3 proxies)

Additional Information

No response