vllm: Key Error when handle multiple requests simultaneously

There was a previous thread but I didnt see a resolution, running into this issue:

OS: Ubuntu 20.04 CUDA Version: 11.2 CPU: 30 CPU RAM: 200 GPU: 8, Tesla V100-SXM2

vllm = LLM(model="mosaicml/mpt-7b-instruct", trust_remote_code=True,dtype="float16",tensor_parallel_size=1)
summary_prompt = """
    Summarize the message below, delimited by triple backticks, using short bullet points.
    ```{message}```
    BULLET POINT SUMMARY:
"""

generated_summary = summarize( vllm, summary_prompt, sampling_params)

  File "/usr/local/lib/python3.8/dist-packages/vllm/entrypoints/llm.py", line 130, in generate
    return self._run_engine(use_tqdm)
  File "/usr/local/lib/python3.8/dist-packages/vllm/entrypoints/llm.py", line 150, in _run_engine
    step_outputs = self.llm_engine.step()
  File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 559, in step
    return self._process_model_outputs(output, scheduler_outputs)
  File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 518, in _process_model_outputs
    self._process_sequence_group_samples(seq_group, samples)
  File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 357, in _process_sequence_group_samples
    parent_child_dict[sample.parent_seq_id].append(sample)
KeyError: 266

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 5
  • Comments: 15 (2 by maintainers)

Commits related to this issue

Most upvoted comments

You are most probably calling LLM.generate() from different threads. The LLM class is not thread-safe. Use AsyncLLMEngine and run_coroutine_threadsafe, if applicable, instead.