istio: gRPC calls fail when using external url (but not internally) in 1.10.0
(NOTE: This is used to report product bugs: To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities To ask questions about how to use Istio, please visit https://discuss.istio.io)
Bug description
After upgrading to Istio 1.10.0 from 1.9.1, all gRPC requests across the internet (via public url) to publicly exposed services fail with this in the sidecar proxy:
2021-05-27T00:45:29.328138305Z [2021-05-27T00:45:25.603Z[] "POST /REDACTEDROUTE/REDACTEDSUBROUTE HTTP/2" 200 UR upstream_reset_before_response_started{remote_reset} - "-" 8 0 1 - "10.252.40.95" "grpc-dotnet/2.29.0.0" "5743dd73-c019-4522-93e4-d77549e12830" "[REDACTED_PUBLIC_URL]" "10.252.40.205:5010" inbound|5010|| 127.0.0.6:42273 10.252.40.205:5010 10.252.40.95:0 [REDACTED_PUBLIC_URL] default
This occurs many times before the request gives up.
Below is the configuration with names/urls/keys redacted:
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: api-gateway
namespace: default
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- '*'
port:
name: http-apis
number: 80
protocol: HTTP2
tls:
httpsRedirect: true
- hosts:
- '*'
port:
name: https-apis
number: 443
protocol: HTTPS
tls:
minProtocolVersion: TLSV1_2
mode: SIMPLE
privateKey: [key]
serverCertificate: [cert]
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
spec:
gateways:
- api-gateway
- mesh
hosts:
- [REDACTED PUBLIC URL]
- [servicename].default.svc.cluster.local
- [servicename]
http:
- corsPolicy:
allowHeaders:
- DNT
- X-CustomHeader
- Keep-Alive
- User-Agent
- X-Requested-With
- If-Modified-Since
- Cache-Control
- Content-Type
- Authorization
- ...
- ...
- ...
match:
- headers:
content-type:
exact: application/grpc
route:
- destination:
host: [servicename]
port:
number: 5010
- match:
- port: 80
- port: 443
route:
- destination:
host: [servicename]
port:
number: 80
headers:
response:
add:
Referrer-Policy: no-referrer
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-DNS-Prefetch-Control: "off"
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
remove:
- server
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
...
spec:
host: [servicename]
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
Any help would be greatly appreciated.
[ ] Docs [ ] Installation [ X ] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure [ X ] Upgrade
Expected behavior Making a grpc request from the public facing URL should successfully forward to the service. Steps to reproduce the bug Send grpc request via over 443 to grpc service
Version (include the output of istioctl version --remote
and kubectl version --short
and helm version --short
if you used Helm)
istio
client version: 1.10.0
control plane version: 1.10.0
data plane version: 1.10.0 (72 proxies)
k8s
Server Version: v1.20.5
How was Istio installed?
Istio-Operator
Environment where the bug was observed (cloud vendor, OS, etc)
Azure AKS
Additionally, please consider running istioctl bug-report
and attach the generated cluster-state tarball to this issue.
Refer cluster state archive for more details.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 18 (7 by maintainers)
@heatonmatthew For the complete noob, where is
envoy.reloadable_features.preserve_downstream_scheme
set in the Istio config? Does it need to be added as a new EnvoyFilter?Hi all. Based on the original issue description, it looks like the poster is describing a situation where the connection from outside the K8s cluster is encrypted and the Service Mesh (i.e. Envoy proxy instance) is doing a downgrade from encrypted to unencrypted for the cluster internal gRPC traffic.
The change in envoyproxy/envoy#14587 is 99% likely to be the issue. I’ve confirmed that it’s an issue with the .NET implementation of the gRPC stack personally and the dotnet/aspnetcore#30532 issue referenced above shows that Microsoft is aware of the issue and that their stack appears to be more restrictive that (some) others regarding scheme matching.
The
envoy.reloadable_features.preserve_downstream_scheme runtime
flag does work as I mentioned in my comment, but the Envoy team is looking to see what long-term solution might be suitable for supporting this scenario. I guess the questions I have of the Istio project team that might help the Envoy team is: