fastapi: FastAPI/Uvicorn crashes every day around the same time, different APIs, different providers

First Check

  • I added a very descriptive title to this issue.
  • I used the GitHub search to find a similar issue and didn’t find it.
  • I searched the FastAPI documentation, with the integrated search.
  • I already searched in Google “How to X in FastAPI” and didn’t find any information.
  • I already read and followed all the tutorial in the docs and didn’t find an answer.
  • I already checked if it is not related to FastAPI but to Pydantic.
  • I already checked if it is not related to FastAPI but to Swagger UI.
  • I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

  • I commit to help with one of those options 👆

Example Code

app = FastAPI()


@app.get("/", response_class=HTMLResponse)
async def read_item(request: Request, pac_id: str = None, _oauth2_proxy: str = Cookie(default="")):  
   pass

Description

Three different FastAPI’s are crashing every day around the same time (different times per API), running Dockerized on Kubernetes with Uvicorn.

It is not DDOS or anything, as the APIs are using Basic Auth and are behind an nginx proxy which handles authentication.

It is not DB dependent, as one uses MongoDB, one PostgreSQL, one no DB.

Providers are Hetzner, AWS and Advin (Frankfurt, Nürnberg, Amsterdam)

The docker container has a huge CPU and Disk usage before Kubernetes “resets” the container, which causes a downtime of ~5 minutes every day.

I read in some tickets that people experience something similar with uvicorn - is that a problem? I’m running the app in the docker image with: CMD ["uvicorn", "main:app", "--host", "0.0.0.0"] and am using python:3.10-slim

Any recommendations? I would say either a FastAPI problem or Uvicorn

Operating System

Linux

Operating System Details

Kubernetes 1.24.9 Docker Image 3.10-slim Linux Host (Ubuntu Server 22.04)

FastAPI Version

0.89.0

Python Version

3.10.9

Additional Context

Screenshot 2023-01-24 100520

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 1
  • Comments: 15

Most upvoted comments

Any news?

yes indeed @yallxe’s solution of using gunicorn instead of uvicorn solves the problem! So it’s a uvicorn bug, weird, but okay! Thanks!!

What probably happens now is that gunicorn will restart the workers, but the problem should still exist.

You are running app directly using uvicorn command. According to https://www.uvicorn.org/deployment/, when deploying to production you should use gunicorn -k uvicorn.workers.UvicornWorker. I also looked up a few examples of docker combined with fastapi, most of them are using gunicorn with uvicorn as a worker class for production.

Do you have the access logs from 9:41 to the time it shut down?

Can you show me the signature of those endpoints?