aspnetcore: Kestrel HTTP/2 errors with multiple gRPC connections

Kestrel starts to error when receiving requests over multiple gRPC connections. 99.9% of requests complete successfully. 0.1% cause an error to be reported in the .NET gRPC client library.

Requests over a single gRPC connection to Kestrel are always successful. Requests over multiple gRPC connections to the native gRPC server are always successful.

To Reproduce

  1. Get source from https://github.com/aspnet/grpc-dotnet/pull/2
  2. Start src/perf/benchmarkapps/BenchmarkServer (dotnet run -c Release)
  3. Start src/perf/benchmarkapps/BenchmarkClient (dotnet run -c Release)
  4. Observe errors in client from gRPC library:
[02:52:22.794] Target server: 127.0.0.1:50051
[02:52:22.822] Creating channels: 2
[02:52:22.872] Connecting channel '0'
[02:52:22.887] Connecting channel '1'
[02:52:22.888] Starting benchmark
[02:52:22.891] 0: Starting
[02:52:22.891] 1: Starting
[02:52:24.128] 0: Error message: Status(StatusCode=Unknown, Detail="Stream removed")
[02:52:24.128] 1: Error message: Status(StatusCode=Unavailable, Detail="Transport closed")
[02:52:25.298] 1: Error message: Status(StatusCode=Unknown, Detail="Stream removed")
[02:52:25.298] 0: Error message: Status(StatusCode=Unknown, Detail="Stream removed")
[02:52:26.483] 0: Error message: Status(StatusCode=Unknown, Detail="Stream removed")
[02:52:26.483] 1: Error message: Status(StatusCode=Unavailable, Detail="Transport closed")
[02:52:27.072] 1: Error message: Status(StatusCode=Unknown, Detail="Stream removed")
[02:52:27.072] 0: Error message: Status(StatusCode=Unavailable, Detail="Transport closed")
[02:52:36.484] 0: Error message: Status(StatusCode=Unknown, Detail="Stream removed")
[02:52:36.484] 1: Error message: Status(StatusCode=Unknown, Detail="Stream removed")
[02:52:38.244] 0: Error message: Status(StatusCode=Unknown, Detail="Stream removed")
[02:52:38.244] 1: Error message: Status(StatusCode=Unknown, Detail="Stream removed")
[02:52:39.416] 0: Error message: Status(StatusCode=Unknown, Detail="Stream removed")
[02:52:39.416] 1: Error message: Status(StatusCode=Unknown, Detail="Stream removed")
[02:52:41.766] 1: Error message: Status(StatusCode=Unknown, Detail="Stream removed")
[02:52:41.766] 0: Error message: Status(StatusCode=Unknown, Detail="Stream removed")
[02:52:42.890] 1: Finished
[02:52:42.890] 0: Finished
[02:52:42.890] Benchmark duration complete
Requests per second: 8937
Shutting down

Expected behavior

All requests should complete successfully. The native server - src/perf/benchmarkapps/NativeServer (dotnet run -c Release) - does not error when called from the client.

Additional context

Run on ASP.NET Core 3.0 Preview 1

@Tratcher @davidfowl @JunTaoLuo

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 26 (25 by maintainers)

Most upvoted comments

Clients are suppose to ignore the bit. It takes a bug on both sides for it to show up.