uvicorn: [BUG] Exit at startup failure on multiple workers

Checklist

The bug is reproducible against the latest release and/or master.
There are no similar issues or pull requests to fix it yet.

Describe the bug

When the startup fails, either on reload mode or multiple workers, the main process is not terminated.

To reproduce

Application:

# test.py
from fastapi import FastAPI

app = FastAPI()

@app.on_event("startup")
def startup():
    raise Exception("Hi")

uvicorn test:app --workers 2
# or
uvicorn test:app --reload

Expected behavior

All the processes (parent and children) should be terminated.

Actual behavior

❯ uvicorn test:app --reload
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [40026] using watchgod
INFO:     Started server process [40028]
INFO:     Waiting for application startup.
ERROR:    Traceback (most recent call last):
  File "/home/marcelo/anaconda3/envs/uvicorn/lib/python3.8/site-packages/starlette/routing.py", line 540, in lifespan
    async for item in self.lifespan_context(app):
  File "/home/marcelo/anaconda3/envs/uvicorn/lib/python3.8/site-packages/starlette/routing.py", line 481, in default_lifespan
    await self.startup()
  File "/home/marcelo/anaconda3/envs/uvicorn/lib/python3.8/site-packages/starlette/routing.py", line 518, in startup
    handler()
  File "/home/marcelo/Development/./test.py", line 7, in startup
    raise Exception("Hi")
Exception: Hi

ERROR:    Application startup failed. Exiting.

After this log, it hangs forever. On the reload, it doesn’t prevent the failure from happening multiple times if you modify a watched file. See:

❯ uvicorn test:app --reload            
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [40674] using watchgod
INFO:     Started server process [40676]
INFO:     Waiting for application startup.
ERROR:    Traceback (most recent call last):
  File "/home/marcelo/anaconda3/envs/uvicorn/lib/python3.8/site-packages/starlette/routing.py", line 540, in lifespan
    async for item in self.lifespan_context(app):
  File "/home/marcelo/anaconda3/envs/uvicorn/lib/python3.8/site-packages/starlette/routing.py", line 481, in default_lifespan
    await self.startup()
  File "/home/marcelo/anaconda3/envs/uvicorn/lib/python3.8/site-packages/starlette/routing.py", line 518, in startup
    handler()
  File "/home/marcelo/Development/./test.py", line 7, in startup
    raise Exception("Hi")
Exception: Hi

ERROR:    Application startup failed. Exiting.
WARNING:  WatchGodReload detected file change in '['/home/marcelo/Development/test.py']'. Reloading...
INFO:     Started server process [40734]
INFO:     Waiting for application startup.
ERROR:    Traceback (most recent call last):
  File "/home/marcelo/anaconda3/envs/uvicorn/lib/python3.8/site-packages/starlette/routing.py", line 540, in lifespan
    async for item in self.lifespan_context(app):
  File "/home/marcelo/anaconda3/envs/uvicorn/lib/python3.8/site-packages/starlette/routing.py", line 481, in default_lifespan
    await self.startup()
  File "/home/marcelo/anaconda3/envs/uvicorn/lib/python3.8/site-packages/starlette/routing.py", line 518, in startup
    handler()
  File "/home/marcelo/Development/./test.py", line 7, in startup
    raise Exception("Hi")
Exception: Hi

ERROR:    Application startup failed. Exiting.

In the case of multiple workers, it just hangs forever.

Environment

OS / Python / Uvicorn version: Running uvicorn 0.14.0 with CPython 3.8.10 on Linux

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 17 (11 by maintainers)

Most upvoted comments

Just want to point out that this is still an issue on 0.17.0.post1.

Korijn on Feb 2, 2022

Yes.

Kludex on Feb 3, 2022

Thanks for that hint

danieldaeschle on Dec 1, 2021

You shouldn’t be using uvicorn’s --workers feature in production. The recommendation is to use standalone uvicorn on k8s without workers, and let the management on a higher level, but if you still want to have multiple workers, you should be using gunicorn.

In any case, a workaround can be found on the first commit of #1177.

Kludex on Dec 1, 2021

I would check how gunicorn and hypercorn are doing it.

Kludex on Aug 21, 2021

Oh, nice to know, I bumped into this issue while writing some tests to improve the coverage, I wasn’t sure it was something expected or a bug, but I added a note to check it later when I have more time.

This change will probably require a minor bump instead of a patch, since someone can be relying on this hanging behaviour at their daemons. Even though the app isn’t really up, careless configured daemons can peak processing by infinity loops of app reboots.

In my tests, I noticed it hangs even sooner at the startup process, for example, if you miss-configure the app.

It fails to load the app and hangs instead of exiting:

➜ uvicorn test --workers 2 
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started parent process [15068]
ERROR:    Error loading ASGI app. Import string "test" must be in format "<module>:<attribute>".
ERROR:    Error loading ASGI app. Import string "test" must be in format "<module>:<attribute>".

The same for the reload case:

➜ uvicorn test --reload   
INFO:     Will watch for changes in these directories: ['/home/user/tests/server']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [15234] using watchgod
ERROR:    Error loading ASGI app. Import string "test" must be in format "<module>:<attribute>".

humrochagf on Aug 17, 2021