grpc: Graceful shutdown is not working for async server

What version of gRPC and what language are you using?

1.37

What operating system (Linux, Windows,…) and version?

macOS Cataline 10.15.7

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.9.0

What did you do?

I am running the following code

import logging
import asyncio
import grpc


async def serve() -> None:
    server = grpc.aio.server()
    listen_addr = "[::]:50051"
    server.add_insecure_port(listen_addr)
    logging.info("Starting server on %s", listen_addr)
    await server.start()
    try:
        await server.wait_for_termination()
    except KeyboardInterrupt:
        # Shuts down the server with 0 seconds of grace period. During the
        # grace period, the server won't accept new connections and allow
        # existing RPCs to continue within the grace period.
        await server.stop(0)


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    asyncio.run(serve())

What did you expect to see?

A graceful shutdown

What did you see instead?

/Users/ilian/.virtualenvs/usersettings-service/bin/python /Users/ilian/workspace/lifesum/usersettings-service/tmp.py
INFO:root:Starting server on [::]:50051
Traceback (most recent call last):
  File "/Users/ilian/workspace/lifesum/usersettings-service/tmp.py", line 38, in <module>
    asyncio.run(serve())
  File "/Users/ilian/.pyenv/versions/3.9.0/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/ilian/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete
    self.run_forever()
  File "/Users/ilian/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
    self._run_once()
  File "/Users/ilian/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 1854, in _run_once
    event_list = self._selector.select(timeout)
  File "/Users/ilian/.pyenv/versions/3.9.0/lib/python3.9/selectors.py", line 562, in select
    kev_list = self._selector.control(None, max_ev, timeout)
KeyboardInterrupt
Exception ignored in: <function Server.__del__ at 0x10b05c5e0>
Traceback (most recent call last):
  File "/Users/ilian/.virtualenvs/usersettings-service/lib/python3.9/site-packages/grpc/aio/_server.py", line 169, in __del__
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 116, in grpc._cython.cygrpc.schedule_coro_threadsafe
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 108, in grpc._cython.cygrpc.schedule_coro_threadsafe
  File "/Users/ilian/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 431, in create_task
  File "/Users/ilian/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 510, in _check_closed
RuntimeError: Event loop is closed
sys:1: RuntimeWarning: coroutine 'AioServer.shutdown' was never awaited

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 11
Comments: 18 (8 by maintainers)

Most upvoted comments

I created PR https://github.com/grpc/grpc/pull/26622 to add a proper graceful shutdown example. @IlianIliev Please let me know if it works or not.

@damarvin The 2 CTRL+C problem is different than the graceful shutdown described in this issue. The graceful shutdown issue is about where can KeyInterrupt be propagated to. Can you share your reproduce snippet?

lidizheng on Jul 7, 2021

I have opened a ticket there -> https://bugs.python.org/msg398400 Let’s see if they can help in any way. And thank you for your help to look at this issue )

IlianIliev on Jul 28, 2021

Good point. I reproduced your result with a busy loop in the server handler. In normal Python, KeyboardInterrupt will be populated to the main thread, so it’s deterministic. But with asyncio, the main thread might be occupied by the request handler or the event loop, hence the KeyboardInterrupt problem exists.

This sounds like a design problem for asyncio. I don’t have an answer. Maybe we should open an issue to CPython? https://bugs.python.org/

lidizheng on Jul 26, 2021

I tried your change. If we use time.sleep the KeyboardInterrupt exception will be populated to the time.sleep, hence it will not be caught by our new try-catch clause at all.

lidizheng on Jul 23, 2021

I see exactly the output as the case is about, with the 1st CTRL+C already. I assume @IlianIliev needs a 2nd too, to get back on the console, so I’d wait for #26123 is fixed.

damarvin on Jul 2, 2021