python-aiplatform: GenerativeModel response hang on multithreaded (linux)
GenerativeModel reponse hang on multireaded case. Works fine on windows. Also added workaround.
Environment details
- OS type and version: linux (any type of include GCP containerized OS, cloud run, ubuntu etc)
- Python version: 3.10.xx(12)
- pip version:
google-cloud-aiplatformversion: 1.42.1
Steps to reproduce
- run attached sample on linux. (windows has no issues)
- expected response is as below Waiting to be completed Hello world! I am a large language model, trained by Google. Hello world! I am a large language model, trained by Google. Hello world! I am a large language model, trained by Google. Hello world! I am a large language model, trained by Google. Hello world! I am a large language model, trained by Google.
Code example
# example
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import as_completed
def analyze_text_gemini_pro(input):
from vertexai.generative_models import GenerativeModel, HarmCategory, HarmBlockThreshold
def get_model():
return GenerativeModel("gemini-1.0-pro-001")
responses = get_model().generate_content(
input,
generation_config={
"candidate_count": 1,
"max_output_tokens": 8192,
"temperature": 0,
"top_p": 0.5,
"top_k": 1
},
safety_settings={
HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,
HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,
},
stream=True
)
#print(f"=== Number of Tokens === \n{response._raw_response.usage_metadata}\n===")
#print(response)
#return responses # Workaround return here
txt = ""
for response in responses: # Hang here on linux
txt += response.text
return txt
async_jobs = []
for i in range(0, 5):
async_jobs.append((analyze_text_gemini_pro, ("Hello world",), i))
futures = {ThreadPoolExecutor(8).submit(func, *param): decor for func, param, decor in async_jobs}
print("Waiting to be completed")
for idx, future in enumerate(as_completed(futures), start=1):
responses = future.result()
print(responses)
#for response in responses: # Workaround
# print(response.text, end="")
Stack trace
N/A
Making sure to follow these steps will guarantee the quickest resolution possible.
Thanks!
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 16 (6 by maintainers)
I was able to get around this issue by subclassing
_GenerativeModel(I’m actually using_PreviewGenerativeModelbecause I need context caching)I was experiencing this issue with gRPC
1.60.1, both when using GenerativeModel from multiple threads and when using it within an async event loop (sending off multiple requests in quick succession). Adding a 10ms sleep (or too many debugging print statements) made the problem go away, which might explain why it was difficult to reproduce.Bumping to
grpcio>=1.71.0fixed the issue for me!In my case, the problem was with
_prediction_client. Before using the model instance in a multithreading environment, simply accessing the property could help.