istio: gRPC calls fail when using external url (but not internally) in 1.10.0

(NOTE: This is used to report product bugs: To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities To ask questions about how to use Istio, please visit https://discuss.istio.io)

Bug description

After upgrading to Istio 1.10.0 from 1.9.1, all gRPC requests across the internet (via public url) to publicly exposed services fail with this in the sidecar proxy:

2021-05-27T00:45:29.328138305Z [2021-05-27T00:45:25.603Z[] "POST /REDACTEDROUTE/REDACTEDSUBROUTE HTTP/2" 200 UR upstream_reset_before_response_started{remote_reset} - "-" 8 0 1 - "10.252.40.95" "grpc-dotnet/2.29.0.0" "5743dd73-c019-4522-93e4-d77549e12830" "[REDACTED_PUBLIC_URL]" "10.252.40.205:5010" inbound|5010|| 127.0.0.6:42273 10.252.40.205:5010 10.252.40.95:0 [REDACTED_PUBLIC_URL] default

This occurs many times before the request gives up.

Below is the configuration with names/urls/keys redacted:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: api-gateway
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http-apis
      number: 80
      protocol: HTTP2
    tls:
      httpsRedirect: true
  - hosts:
    - '*'
    port:
      name: https-apis
      number: 443
      protocol: HTTPS
    tls:
      minProtocolVersion: TLSV1_2
      mode: SIMPLE
      privateKey: [key]
      serverCertificate: [cert]

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
spec:
  gateways:
  - api-gateway
  - mesh
  hosts:
  - [REDACTED PUBLIC URL]
  - [servicename].default.svc.cluster.local
  - [servicename]
  http:
  - corsPolicy:
      allowHeaders:
      - DNT
      - X-CustomHeader
      - Keep-Alive
      - User-Agent
      - X-Requested-With
      - If-Modified-Since
      - Cache-Control
      - Content-Type
      - Authorization
      - ... 
      - ...
      - ...
    match:
    - headers:
        content-type:
          exact: application/grpc
    route:
    - destination:
        host: [servicename]
        port:
          number: 5010
  - match:
    - port: 80
    - port: 443
    route:
    - destination:
        host: [servicename]
        port:
          number: 80
      headers:
        response:
          add:
            Referrer-Policy: no-referrer
            Strict-Transport-Security: max-age=31536000; includeSubDomains
            X-Content-Type-Options: nosniff
            X-DNS-Prefetch-Control: "off"
            X-Download-Options: noopen
            X-Frame-Options: SAMEORIGIN
            X-XSS-Protection: 1; mode=block
          remove:
          - server
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
...
spec:
  host: [servicename]
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

Any help would be greatly appreciated.

[ ] Docs [ ] Installation [ X ] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure [ X ] Upgrade

Expected behavior Making a grpc request from the public facing URL should successfully forward to the service. Steps to reproduce the bug Send grpc request via over 443 to grpc service

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)

istio

client version: 1.10.0
control plane version: 1.10.0
data plane version: 1.10.0 (72 proxies)

k8s

Server Version: v1.20.5

How was Istio installed? Istio-Operator Environment where the bug was observed (cloud vendor, OS, etc) Azure AKS

Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue. Refer cluster state archive for more details.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 18 (7 by maintainers)

Most upvoted comments

@heatonmatthew For the complete noob, where is envoy.reloadable_features.preserve_downstream_scheme set in the Istio config? Does it need to be added as a new EnvoyFilter?

Hi all. Based on the original issue description, it looks like the poster is describing a situation where the connection from outside the K8s cluster is encrypted and the Service Mesh (i.e. Envoy proxy instance) is doing a downgrade from encrypted to unencrypted for the cluster internal gRPC traffic.

The change in envoyproxy/envoy#14587 is 99% likely to be the issue. I’ve confirmed that it’s an issue with the .NET implementation of the gRPC stack personally and the dotnet/aspnetcore#30532 issue referenced above shows that Microsoft is aware of the issue and that their stack appears to be more restrictive that (some) others regarding scheme matching.

The envoy.reloadable_features.preserve_downstream_scheme runtime flag does work as I mentioned in my comment, but the Envoy team is looking to see what long-term solution might be suitable for supporting this scenario. I guess the questions I have of the Istio project team that might help the Envoy team is:

  • Do you intend to continue to support scheme downgrade of gRPC traffic from HTTPS to HTTP in the Istio Service Mesh?
  • If so, would Allysa’s proposal to support a specific scheme re-write option be a helpful long-term solution to this issue?