python-aiplatform: GenerativeModel response hang on multithreaded (linux)

GenerativeModel reponse hang on multireaded case. Works fine on windows. Also added workaround.

Environment details

  • OS type and version: linux (any type of include GCP containerized OS, cloud run, ubuntu etc)
  • Python version: 3.10.xx(12)
  • pip version:
  • google-cloud-aiplatform version: 1.42.1

Steps to reproduce

  1. run attached sample on linux. (windows has no issues)
  2. expected response is as below Waiting to be completed Hello world! I am a large language model, trained by Google. Hello world! I am a large language model, trained by Google. Hello world! I am a large language model, trained by Google. Hello world! I am a large language model, trained by Google. Hello world! I am a large language model, trained by Google.

Code example

# example
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import as_completed

def analyze_text_gemini_pro(input):
    from vertexai.generative_models import GenerativeModel, HarmCategory, HarmBlockThreshold
    def get_model():
        return GenerativeModel("gemini-1.0-pro-001")
    responses = get_model().generate_content(
        input,
        generation_config={
            "candidate_count": 1,
            "max_output_tokens": 8192,
            "temperature": 0,
            "top_p": 0.5,
            "top_k": 1
        },
        safety_settings={
            HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,
            HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,
        },
        stream=True
    )
    #print(f"=== Number of Tokens === \n{response._raw_response.usage_metadata}\n===")
    #print(response)
    #return responses                             # Workaround return here
    txt = ""
    for response in responses:                # Hang here on linux
        txt += response.text
    return txt

async_jobs = []

for i in range(0, 5):
    async_jobs.append((analyze_text_gemini_pro, ("Hello world",), i))

futures = {ThreadPoolExecutor(8).submit(func, *param): decor for func, param, decor in async_jobs}

print("Waiting to be completed")
for idx, future in enumerate(as_completed(futures), start=1):
    responses = future.result()
    print(responses)
    #for response in responses:              # Workaround
    #    print(response.text, end="")

Stack trace

N/A

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 16 (6 by maintainers)

Most upvoted comments

I was able to get around this issue by subclassing _GenerativeModel (I’m actually using _PreviewGenerativeModel because I need context caching)

class ThreadSafeGenerativeModel(_PreviewGenerativeModel):
    # Class-level lock shared across all instances
    _prediction_client_lock = threading.Lock()

    @property
    def _prediction_client(self):
        if getattr(self, "_prediction_client_value", None):
            return self._prediction_client_value
        else:
            with self._prediction_client_lock:
                if not getattr(self, "_prediction_client_value", None):
                    # Initialize the client using the superclass property
                    self._prediction_client_value = super()._prediction_client
            return self._prediction_client_value

I was experiencing this issue with gRPC 1.60.1, both when using GenerativeModel from multiple threads and when using it within an async event loop (sending off multiple requests in quick succession). Adding a 10ms sleep (or too many debugging print statements) made the problem go away, which might explain why it was difficult to reproduce.

Bumping to grpcio>=1.71.0 fixed the issue for me!

In my case, the problem was with _prediction_client. Before using the model instance in a multithreading environment, simply accessing the property could help.

model = GenerativeModel()
_ = model._prediction_client