istio: gRPC calls fail when using external url (but not internally) in 1.10.0

(NOTE: This is used to report product bugs: To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities To ask questions about how to use Istio, please visit https://discuss.istio.io)

Bug description

After upgrading to Istio 1.10.0 from 1.9.1, all gRPC requests across the internet (via public url) to publicly exposed services fail with this in the sidecar proxy:

2021-05-27T00:45:29.328138305Z [2021-05-27T00:45:25.603Z[] "POST /REDACTEDROUTE/REDACTEDSUBROUTE HTTP/2" 200 UR upstream_reset_before_response_started{remote_reset} - "-" 8 0 1 - "10.252.40.95" "grpc-dotnet/2.29.0.0" "5743dd73-c019-4522-93e4-d77549e12830" "[REDACTED_PUBLIC_URL]" "10.252.40.205:5010" inbound|5010|| 127.0.0.6:42273 10.252.40.205:5010 10.252.40.95:0 [REDACTED_PUBLIC_URL] default

This occurs many times before the request gives up.

Below is the configuration with names/urls/keys redacted:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: api-gateway
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http-apis
      number: 80
      protocol: HTTP2
    tls:
      httpsRedirect: true
  - hosts:
    - '*'
    port:
      name: https-apis
      number: 443
      protocol: HTTPS
    tls:
      minProtocolVersion: TLSV1_2
      mode: SIMPLE
      privateKey: [key]
      serverCertificate: [cert]

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
spec:
  gateways:
  - api-gateway
  - mesh
  hosts:
  - [REDACTED PUBLIC URL]
  - [servicename].default.svc.cluster.local
  - [servicename]
  http:
  - corsPolicy:
      allowHeaders:
      - DNT
      - X-CustomHeader
      - Keep-Alive
      - User-Agent
      - X-Requested-With
      - If-Modified-Since
      - Cache-Control
      - Content-Type
      - Authorization
      - ... 
      - ...
      - ...
    match:
    - headers:
        content-type:
          exact: application/grpc
    route:
    - destination:
        host: [servicename]
        port:
          number: 5010
  - match:
    - port: 80
    - port: 443
    route:
    - destination:
        host: [servicename]
        port:
          number: 80
      headers:
        response:
          add:
            Referrer-Policy: no-referrer
            Strict-Transport-Security: max-age=31536000; includeSubDomains
            X-Content-Type-Options: nosniff
            X-DNS-Prefetch-Control: "off"
            X-Download-Options: noopen
            X-Frame-Options: SAMEORIGIN
            X-XSS-Protection: 1; mode=block
          remove:
          - server

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
...
spec:
  host: [servicename]
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

Any help would be greatly appreciated.

[ ] Docs [ ] Installation [ X ] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure [ X ] Upgrade

Expected behavior Making a grpc request from the public facing URL should successfully forward to the service. Steps to reproduce the bug Send grpc request via over 443 to grpc service

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)

istio

client version: 1.10.0
control plane version: 1.10.0
data plane version: 1.10.0 (72 proxies)

k8s

Server Version: v1.20.5

How was Istio installed? Istio-Operator Environment where the bug was observed (cloud vendor, OS, etc) Azure AKS

Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue. Refer cluster state archive for more details.