uvicorn-gunicorn-fastapi-docker: Workers go into restarting/crash cycle (WORKER TIMEOUT / signal 6)

Managed to resolve this issue, sharing in case this helps.

Our issue originated from making external API calls from within an async endpoint. These API calls did not support async, which introduced blocking calls to the event loop, resulting in the uvicorn worker timing out. Our reliance on FastAPI Cache decorators for these async endpoints prevented us from simply redefining these endpoints as sync (async def -> def).

To resolve, we made use of the run_in_threadpool() utility function to ensure these sync calls are run in a separate threadpool, outside the event loop. Alongside this, we updated our gunicorn config so the workers and threads count was equal - setting these to 4.

from fastapi.concurrency import run_in_threadpool

@api.get('/handler')
async def handler():
    ...
    # Slow async function
    await my_async_function()
    ....
    # Slow running sync function
    await run_in_threadpool(sync_function)

We released this update over 2 weeks ago and haven’t seen any worker timeouts. Hopefully this helps 🙂

i resolved this issue by adding worker timeout while initiating my gunicorn application.

gunicorn -k uvicorn.workers.UvicornWorker ${APP_MODULE} --bind 0.0.0.0:80 --timeout ${WORKER_TIMEOUT}

@nicholarmccrea your solution seems like a good solution, but i imagined removing the “async” from the endpoint functions allowed the requests to be handled in a different threadpool. Do you need to explicitly run it in a seperate function?

I am taking the knowledge from the comment on this post : https://stackoverflow.com/questions/71516140/fastapi-runs-api-calls-in-serial-instead-of-parallel-fashion

@mcazim98 in our case we were not able to redefine our endpoints as sync due to our reliance on FastAPI cache decorators. The FastAPI cache version we were using was 0.1.8, which did not support sync functions, therefore we needed to use the run_in_threadpool utility function as a workaround.

Thankfully, FastAPI cache now support sync functions as of version 0.2.0, which means we can now redefine our endpoints as sync and move away from using the run_in_threadpool function 🙂

Facing this issue while using docker. Working perfectly fine if run directly with gunicorn -w 1 -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8080 main:app.

None of the following suggested solutions worked:

1. Assigning more memory
2. Changing worker class to gevent
3. Changing python version to 3.7 from 3.9
4. Adding timeout
5. Running directly with uvicorn without gunicorn

Can someone please point me in the right direction to resolve this issue?

Facing the same issue. Whenever following code is executed with incorrect smtp_url, port, my worker crashes:

def validate_smtp(smtp_url: str):
     try:
         smtp = SMTP()
         smtp.connect(smtp_url)
         smtp.quit()
         return True
     except:
         return False

There is no crash if smtp_url or port is valid. Dependencies:

1. gunicorn: 20.1.0
2. uvicorn: 0.17.1
3. fastapi: 0.73.0

@nicholasmccrea wow, should have been hard to identify it! Great news!

yup i tried with 1 worker, still no luck.

Facing the same issue when I use the haystack. I modifed the docker-compose.yml as following: command: “/bin/bash -c ‘sleep 10 && gunicorn rest_api.application:app -b 0.0.0.0 -k uvicorn.workers.UvicornWorker – workers 1 --timeout 600’” It can work.

Facing the same issue when running long processes on websockets and it ends up terminating the websocket connection. Any fixes?