grpc: Grpc.Core.Internal.AsyncCall<>.UnaryCall hangs (malformed call status?)
Ver 1.45 (1.46 code is the same) The symptom is hanging in Grpc.Core.Internal.AsyncCall<>.UnaryCall on return unaryResponseTcs.Task.GetAwaiter().GetResult(); line.
The stack trace example is:
[Waiting on Async Operation, double-click or press enter to view Async Call Stacks] Grpc.Core.dll!Grpc.Core.Internal.AsyncCall<GenericRequest, GenericResponse>.UnaryCall(GenericRequest msg) Line 116 C# Grpc.Core.dll!Grpc.Core.Calls.BlockingUnaryCall<GenericRequest, GenericResponse>(Grpc.Core.CallInvocationDetails<GenericRequest, GenericResponse> call, GenericRequest req) Line 25 C# Grpc.Core.dll!Grpc.Core.DefaultCallInvoker.BlockingUnaryCall<GenericRequest, GenericResponse>(Grpc.Core.Method<GenericRequest, GenericResponse> method, string host, Grpc.Core.CallOptions options, GenericRequest request) Line 16 C# Grpc.Core.Api.dll!Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall.AnonymousMethod__3_0<GenericRequest, MGenericResponse>(GenericRequest req, Grpc.Core.Interceptors.ClientInterceptorContext<GenericRequest, GenericResponse> ctx) Line 51 C# Grpc.Core.Api.dll!Grpc.Core.ClientBase.ClientBaseConfiguration.ClientBaseConfigurationInterceptor.BlockingUnaryCall<GenericRequest, GenericResponse>(GenericRequest request, Grpc.Core.Interceptors.ClientInterceptorContext<GenericRequest, GenericResponse> context, Grpc.Core.Interceptors.Interceptor.BlockingUnaryCallContinuation<GenericRequest, GenericResponse> continuation) Line 174 C# Grpc.Core.Api.dll!Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall<GenericRequest, GenericResponse>(Grpc.Core.Method<GenericRequest, GenericResponse> method, string host, Grpc.Core.CallOptions options, GenericRequest request) Line 48 C#
It is happening periodically, the problem how I think is in the next code in of the UnaryCall method:
try
{
using (profiler.NewScope("AsyncCall.UnaryCall.HandleBatch"))
{
HandleUnaryResponse(success, ctx.GetReceivedStatusOnClient(), ctx.GetReceivedMessageReader(), ctx.GetReceivedInitialMetadata());
}
}
catch (Exception e)
{
Logger.Error(e, "Exception occurred while invoking completion delegate.");
**!!! here should be "throw;" !!!**
}
Without that line, in case of HandleUnaryResponse error, unaryResponseTcs will never get neither error, nor value and will wait infinitely.
Also, a test could be added to ensure, the unaryResponseTcs state is not WaitingForActivationcan prior to call GetResult().
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 24 (13 by maintainers)
Commits related to this issue
- Fix hang in AsyncCall.UnaryCall Issue https://github.com/grpc/grpc/issues/29854 An exception thrown by decoding status code was not being handled correctly and this caused the call to hang. — committed to tonydnewell/grpc by tonydnewell 2 years ago
- Fix hang in AsyncCall.UnaryCall Issue https://github.com/grpc/grpc/issues/29854 An exception thrown by decoding status code was not being handled correctly and this caused the call to hang. — committed to tonydnewell/grpc by tonydnewell 2 years ago
- Fix hanging UnaryCall with corrupt status This doesn't address why the status could not be decoded, but it does wrap reading the status and metadata so that exceptions are not thrown so that the call... — committed to tonydnewell/grpc by tonydnewell 2 years ago
- workaround #29854 — committed to jtattermusch/grpc by jtattermusch 2 years ago
- Work around UnaryCall hang with corrupt status detail string #29854 (#31853) * workaround #29854 * Update BatchContextSafeHandle.cs — committed to grpc/grpc by jtattermusch 2 years ago
@tonydnewell, the production testing showed the fix is working. Guys, thank you again.