vllm: Key Error when handle multiple requests simultaneously
There was a previous thread but I didnt see a resolution, running into this issue:
OS: Ubuntu 20.04 CUDA Version: 11.2 CPU: 30 CPU RAM: 200 GPU: 8, Tesla V100-SXM2
vllm = LLM(model="mosaicml/mpt-7b-instruct", trust_remote_code=True,dtype="float16",tensor_parallel_size=1)
summary_prompt = """
Summarize the message below, delimited by triple backticks, using short bullet points.
```{message}```
BULLET POINT SUMMARY:
"""
generated_summary = summarize( vllm, summary_prompt, sampling_params)
File "/usr/local/lib/python3.8/dist-packages/vllm/entrypoints/llm.py", line 130, in generate
return self._run_engine(use_tqdm)
File "/usr/local/lib/python3.8/dist-packages/vllm/entrypoints/llm.py", line 150, in _run_engine
step_outputs = self.llm_engine.step()
File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 559, in step
return self._process_model_outputs(output, scheduler_outputs)
File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 518, in _process_model_outputs
self._process_sequence_group_samples(seq_group, samples)
File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 357, in _process_sequence_group_samples
parent_child_dict[sample.parent_seq_id].append(sample)
KeyError: 266
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 5
- Comments: 15 (2 by maintainers)
Commits related to this issue
- Update TESTOWNERS (#1200) — committed to yiliu30/vllm-fork by madamczyk-intel 2 years ago
You are most probably calling LLM.generate() from different threads. The LLM class is not thread-safe. Use AsyncLLMEngine and run_coroutine_threadsafe, if applicable, instead.