grpc: Python future raises deadline exceeded after a disconnect

I discovered an interesting edge case when using python futures. If a future is used successfully but the remote server connection terminates, the next future call can raise a deadline exceeded exception (or hang forever without a deadline) instead of raising unavailable. I was able to write a small test using the helloworld service.

from concurrent import futures                                                                       
import grpc                                                                                          

import helloworld_pb2                                                                                


class Greeter(helloworld_pb2.GreeterServicer):                                                       

    def SayHello(self, request, context):                                                            
        return helloworld_pb2.HelloReply(message='Hello, %s!' % request.name)                        


def main1():                                                                                         
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))                                 
    helloworld_pb2.add_GreeterServicer_to_server(Greeter(), server)                                  
    server.add_insecure_port('[::]:50051')                                                           
    server.start()                                                                                   

    channel = grpc.insecure_channel('127.0.0.1:50051')                                               
    client = helloworld_pb2.GreeterStub(channel)                                                     
    print client.SayHello.future(                                                                    
        helloworld_pb2.HelloRequest(name='this works'), 10).result()                                 

    server.stop(None)                                                                                

    try:                                                                                             
        client.SayHello.future(                                                                      
            helloworld_pb2.HelloRequest(name='deadline exceeded'), 2).result()                       
    except grpc.RpcError as e:                                                                       
        print e.code()                                                                               


def main2():                                                                                         
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))                                 
    helloworld_pb2.add_GreeterServicer_to_server(Greeter(), server)                                  
    server.add_insecure_port('[::]:50052')                                                           
    server.start()                                                                                   

    channel = grpc.insecure_channel('127.0.0.1:50052')                                               
    channel.subscribe(lambda s: s)                                                                   
    client = helloworld_pb2.GreeterStub(channel)                                                     
    print client.SayHello.future(                                                                    
        helloworld_pb2.HelloRequest(name='this works'), 10).result()                                 

    server.stop(None)                                                                                

    try:                                                                                             
        client.SayHello.future(                                                                      
            helloworld_pb2.HelloRequest(name='unavailable'), 2).result()                             
    except grpc.RpcError as e:                                                                       
        print e.code()                                                                               


if __name__ == '__main__':                                                                           
    main1()                                                                                          
    main2()

main1 demonstrates the bug. The server disconnects after a successful call to SayHello.future. The next call times out after 2 seconds. If I do not supply a timeout, the method never returns.

main2 demonstrates a weird “fix”. If I subscribe to channel state changes, the second call to SayHello.future immediately notices the connection has died and raises unavailable. My guess is that watching the channel state causes some change in timing.

This was tested against the v1.0.0 release.