aspnetcore: GRPC :scheme pseudo-header passed from proxy/loadbalancer causes ConnectionAbortedException
Describe the bug
Grpc kestrel .net 5 sample app greeter is running in k8s under http transport, an AWS network load balancer terminates SSL and sends request to the app with :scheme pseudo-header set to “https”, but the actual transport is “http” results in this error -
Trace id "0HM6QO8HEK1N9:00000001": HTTP/2 stream error "PROTOCOL_ERROR". A Reset is being sent to the stream. Microsoft.AspNetCore.Connections.ConnectionAbortedException: The request :scheme header 'https' does not match the transport scheme 'http'.
The ask here is to be less stringent on validating on the :schema pseudo header, Go lang sample apps works fine.
To Reproduce
Deploy any sample grpc app from here - https://github.com/grpc/grpc-dotnet/tree/master/examples to k8s, setup the service type loadbalancer using AWS NLB. Test with grpcurl client.
Exceptions (if any)
Microsoft.AspNetCore.Connections.ConnectionAbortedException: The request :scheme header 'https' does not match the transport scheme 'http'.
Further technical details
- ASP.NET Core version
- Include the output of
dotnet --info - The IDE (VS / VS Code/ VS4Mac) you’re running on, and its version
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 41 (21 by maintainers)
Commits related to this issue
- Allow alternate schemes in Kestrel requests #30532 — committed to Tratcher/aspnetcore by Tratcher 3 years ago
- Allow alternate schemes in Kestrel requests #30532 (#34013) — committed to dotnet/aspnetcore by Tratcher 3 years ago
PSA for google cloud run users, google actually fixed this issue on their side
https://issuetracker.google.com/issues/187470251?pli=1
This recently broke me in Cloud Run, I guess GCP updated envoy because it was working fine a couple of weeks ago. Here’s a workaround if anyone needs it:
git clone https://github.com/dotnet/aspnetcore.git && cd aspnetcore && git checkout tags/v5.0.5git submodule update --init --recursive/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Stream.csand comment out lines 244-250. If not version 5.0.5, it’s the block of code that starts withif (!ReferenceEquals(headerScheme, Scheme) &&, comment that whole if block./build.sh --configuration Releasein the root of the project. It had a few errors but the file I needed was built.aspnetcore/artifacts/bin/Microsoft.AspNetCore.Server.Kestrel.Core/Release/net5.0/Microsoft.AspNetCore.Server.Kestrel.Core.dlland copy it to a folder with the following Dockerfile:A bit of a pain, but we’re back up on Cloud Run with this …
I grudgingly approve, with the proviso there’s a warning about the security in the xml docs. Maybe an aka.ms links to a doc explaining the issue in detail as well?
Envoy is just one case, there are other cases that already show the same obscure problem (AWS LB, GCP Cloud Run). It would not be realistic to change all the cases to fit .NET enforcement, for example, if I use a managed proxy in front of .NET server, I just have no way to unblock myself. If .NET provides an config option, life would be much better.
This problem also affects reverse proxy (such as Envoy) that terminates TLS and then forward gRPC over h2c.
Is it possible to add an option to disable the server schema check? .NET is the only language that has this behavior. It’a arguably a correct implementation, but it causes a lot of user pains.
The original poster indicated that the issue was with .NET 5, but I have confirm that it exists for .NET Core 3.1 also. My hunch is that it’s in the underlying gRPC framework implementation (but I’m sure you’re aware of that).
I notice that @Tratcher 's original comment refers to the RFC spec being vague. I agree. I’m having a brief discussion with the Envoy team regarding their interpretation and can see both sides of the argument.
Since the Envoy team, and the Istio team, are needing to deal with this issue (possibly specifically for .NET), I’m wondering if there is any indication from the .NET team about where their deliberations are heading (i.e. especially given that this issue has been marked with the
Next sprint planningmilestone)? It would probably be helpful to those discussions to know whether this is something that needs to be dealt with in a long-term capacity.Thanks. 😃
Well that’s what we want to find out 😄.
Thanks. To add an additional data point: IIRC, all other languages with gRPC support don’t (at least by default) have this enforcement on the h2c gRPC channel.