grpc: C# client throws StatusCode=Unknown Detail="Stream removed"

What version of gRPC and what language are you using?

1.12 C# both client and server

What operating system (Linux, Windows, …) and version?

Windows Server 2014

What runtime / compiler are you using (e.g. python version or version of gcc)

C# .Net Framework 4.7.2

What did you expect to see?

Not crashing with Grpc.Core.RpcException

What did you see instead?

Type:
Grpc.Core.RpcException

Message:
Status(StatusCode=Unknown, Detail="Stream removed")

StackTrace:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Grpc.Core.Internal.AsyncCall`2.UnaryCall(TRequest msg)
   at Grpc.Core.DefaultCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
   at Grpc.Core.Interceptors.InterceptingCallInvoker.<BlockingUnaryCall>b__3_0[TRequest,TResponse](TRequest req, ClientInterceptorContext`2 ctx)
   at Grpc.Core.ClientBase.ClientBaseConfiguration.ClientBaseConfigurationInterceptor.BlockingUnaryCall[TRequest,TResponse](TRequest request, ClientInterceptorContext`2 context, BlockingUnaryCallContinuation`2 continuation)
   at Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
   at Removed.Rpc.FileServer.FileServerClient.DirectoryCreate(DirectoryCreateRequest request, CallOptions options)

Anything else we should know about your project / environment?

We created file server with gRPC and are getting these errors. This error might happen because there is a firewall between client and server. I think it will disconnect connection after it has been idle for a while and then first call from client to server will throw this exception client side. First call to gRPC in the morning is the one that fails. No errors are logged in server side. Second call works.

We are using ChannelCredentials.Insecure.

Is there some sort of retry system in gRPC that we could use to fix this problem?

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 1
Comments: 45 (16 by maintainers)

Most upvoted comments

@Wisgary this is how I’m using interceptors to retry AsyncUnaryCall (but also waiting on retries in grpc core):

public override AsyncUnaryCall<TResponse> AsyncUnaryCall<TRequest, TResponse>(
    TRequest request,
    ClientInterceptorContext<TRequest, TResponse> context,
    AsyncUnaryCallContinuation<TRequest, TResponse> continuation)
{
    var retryCount = 0;

    async Task<TResponse> RetryCallback(Task<TResponse> responseTask)
    {
        var response = responseTask;
        if (!response.IsFaulted)
        {
            return response.Result;
        }

        retryCount++;

        if (retryCount == RetryCount)
        {
            return response.Result;
        }

        await Task.Delay(_retryTimeout.NextRetryTimeout).ConfigureAwait(false);
        var result = continuation(request, context).ResponseAsync.ContinueWith(RetryCallback).Unwrap();
        return result.Result;
    }

    var responseContinuation = continuation(request, context);
    var responseAsync = responseContinuation.ResponseAsync.ContinueWith(RetryCallback);

    return new AsyncUnaryCall<TResponse>(
        responseAsync.Result,
        responseContinuation.ResponseHeadersAsync,
        responseContinuation.GetStatus,
        responseContinuation.GetTrailers,
        responseContinuation.Dispose);
}

Hope this helps.

vadimi on Jun 19, 2018

I was having the same problem. In my case my server was listening to localhost instead of 0.0.0.0. Not really related to this issue I think.

ricardopieper on Sep 23, 2018