fastapi: Gunicorn on Google Cloud Run get an 504 error status (Upstream Request Timeout)

Glad to read that, good job 👍

@frankie567 Hello frankie, I’m so sorry for my late update. So, this is my update.

Load your model lazily in a dependency I’ve done this, so in here my script load the model when only needed, it was very useful. I can request HTTP more faster than before, and it can avoid error 504 too (surprisingly 😄 ).

Use joblib instead of pickle I’ve done this, and the result unexpected, my model now can load it faster than using pickle. (It’s crazy 🤣 )

About maximum_instances I’ve set my maximum instance to 50 instances, from the story that you give, that was so crazy lol $72000, that soooo damnnnnn tooo muchhh, especially, I can’t imagine that huge of money.

Thank you so much @frankie567

I’m not a specialist, but here is two things worth to try:

Load your model lazily in a dependency

Instead of loading your model at startup, try to load it on the first prediction query. You can wrap this in a FastAPI dependency:

import time

from fastapi import FastAPI, Depends


class MyModelPrediction:
  model_loaded: bool = False

  def __call__(self, query: str):
    if not self.model_loaded:
      self._load_model()
    print(f"Make prediction for query {query}")
    time.sleep(1)
    return query

  def _load_model(self):
    print("Load model...")
    time.sleep(10)
    self.model_loaded = True


app = FastAPI()

@app.get("/predict")
def predict(prediction = Depends(MyModelPrediction())):
  return prediction

I’ve replaced your actual logic with time.sleep to simulate loading times. The first prediction will be slow because it’ll have to load the model for the first time. But subsequent predictions will be faster because it’ll be loaded in memory (until the container turns idle). The container startup is now instantaneous, because it has nothing to do.

Notice that I’ve defined the dependency as a synchronous method (not async). Since your loading logic performs blocking I/O, it’s better to define it like this because FastAPI will then run it in the external threadpool (doc: https://fastapi.tiangolo.com/async/?h=techn#path-operation-functions). It means that it won’t block your main loop while the model is loading.

Use joblib instead of pickle

joblib is a very good library for persisting objects on the hard-disk. It may prove more efficient that standard pickle.

About maximum_instances

Cloud Run has an autoscaling feature. It means that if you experience very high traffic for various reasons (or that there is a bug in your code that cause the server to loop), Google will create new containers until maximum_instances is reached. With a limit of 1000, you can have a bill of thousands of dollars without even noticing!

Relevant story about this: https://www.theregister.com/2020/12/10/google_cloud_over_run/

Could you give us your Cloud Run configuration (or the command you make to deploy your service)? By default, Cloud Run instances start with 256 Mb of RAM so, given the size of your model, I suspect you run out of memory.

@frankie567 No problem frankie, thank you so much for answering all of my questions. I really appreciate you. Thank you. Now here I will close my question here.

Sorry, I saw your question and then forgot to answer.

Basically, Cloud Run will scale by creating new instances based on the number of incoming requests. So if you experience a traffic spike, he’ll be able to handle it.

However, creating an instance takes time (we say “cold start”) ; until it can answer requests, which can induce some latency. If you set minimum_instances to 4/10/X, it means that there’ll always be 4/10/X containers ready to serve requests even if there is no request to handle. Of course, you are billed for those instances.

Unless you have a service with very high traffic, or you expect an enormous traffic spike because a TV show will talk about your company, I don’t think this option will be helpful for you.

Official docs: https://cloud.google.com/run/docs/configuring/min-instances

@rudi101101 So, how did it went? 🙂

Gunicorn uvicorn, with the first prediction.

Be careful with timeout if you choose this approach.

@frankie567 God thank you Frankie, I will try it first. Soon, I will give you the result.

Well, theoretically even after being shut down because it didn’t receive traffic for a certain period of time, it should cold-start and run the startup event again without any issue.

Now, I don’t really understand why you get a timeout error on subsequent start. Random thought: make sure you don’t have any open resources or background tasks pending in your router that could prevent a proper shutdown of the container.

Yes, you could set a minimum instance, but it obviously incurs costs (BTW, you really should set the maximum instances parameter if don’t want to have very bad surprises – the default is at 1000 🤯 –).

Yes, the default is 5 minutes. Should be sufficient for you but, you know, computers 😅

Just to be sure we understand on what’s happening:

1. You make a first HTTP request
2. The service makes a cold start (first request received) and executes your startup event ; which takes around 15 seconds
3. Then, the route logic is executed (it could be interesting that you post it also). How much time does it take?
4. You get a response (right?)
5. You wait 30-60 minutes. At this point, it’s probable that Cloud Run shuts down the container because it doesn’t receive traffic (https://cloud.google.com/run/docs/about-instance-autoscaling#idle-instance).
6. You make a new HTTP request.
7. You get the 504 timeout error.

Is that so?

Thanks! May I suggest to increase Cloud Run timeout:

• https://cloud.google.com/sdk/gcloud/reference/run/deploy?hl=fr#--timeout
• https://cloud.google.com/run/docs/configuring/request-timeout

It clearly states that a 504 error is triggered when the timeout is reached.