vllm: [Bug]: v0.5.5 crash: "AssertionError: expected running sequences"

Your current environment

Running the standard v0.5.5 docker image from your Dockerhub repo without anything additional added to it.

πŸ› Describe the bug

When using Llama 3.1 70b AWQ model running on 4 A10G 24Gb GPUs with args:

--model hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4
--tensor-parallel-size 4
--gpu-memory-utilization 0.95
--enforce-eager
--trust-remote-code
--worker-use-ray
--enable-prefix-caching
--num-scheduler-steps 8
--dtype half
--max-model-len 32768

vLLM crashes and requires a full restart. Error:

	
INFO 08-29 19:33:37 server.py:222] vLLM ZMQ RPC Server was interrupted.
Future exception was never retrieved
future: <Future finished exception=AssertionError('expected running sequences')>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/server.py", line 111, in generate
    async for request_output in results_generator:
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 1064, in generate
    async for output in await self.add_request(
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 113, in generator
    raise result
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 55, in _log_task_completion
    return_value = task.result()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 930, in run_engine_loop
    result = task.result()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 873, in engine_step
    request_outputs = await self.engine.step_async(virtual_engine)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 356, in step_async
    request_outputs = self._process_model_outputs(
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 1232, in _process_model_outputs
    self.output_processor.process_outputs(seq_group, outputs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/output_processor/multi_step.py", line 73, in process_outputs
    assert seqs, "expected running sequences"
AssertionError: expected running sequences

The issue is random, the same query does NOT reproduce it.

We have upgraded 6 hours ago and since then this happened 3 times.

We now need to downgrade and consider v0.5.5 a buggy release.

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 34

Most upvoted comments

There isn’t much to share. We are using your docker image from Dockerhub: https://hub.docker.com/r/vllm/vllm-openai/tags No modification, we run it as-is. At the top of this ticket you can see the parameters we use and the GPUs it is running on.

sounds good. thanks

Indeed it reproduced. As expected, it is an illegal memory access in flash attention due to prefix caching. I will dig in further. took about 4 hours to trigger it

  - "32768"

The source of AssertionError: expected running sequences is due to abort not yet being supported with multi-step scheduling. multi-step scheduling is a new feature we are still working on - I would not yet recommend using multi-step in production use cases until the feature is finalized. The tracking issue for development of multi-step scheduling is here:

* [[Tracking issue] [Help wanted]: Multi-step scheduling follow-ups #7528](https://github.com/vllm-project/vllm/issues/7528)

@zoltan-fedor re: the issues you are seeing with illegal memory access on v0.5.4 / v0.5.5, we have seen intermittent reports of this with --enable-prefix-caching. We have been working on trying to reproduce the issue. If possible, sharing:

* the full logs

* anything you can re: access patterns (the client code which generates the issue)

Would help us a lot of reproduce and resolve the issue

@robertgshaw2-neuralmagic , I have also seen the same error with v0.5.4 WITHOUT the --enable-prefix-caching flag!

So, with the following command?:

--model hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4
--tensor-parallel-size 4
--gpu-memory-utilization 0.95
--enforce-eager
--trust-remote-code
--worker-use-ray
--dtype half
--max-model-len 32768

That is correct. That was the command, so no --enable-prefix-caching

Thanks. Will run this in the background today and see if I can reproduce. The illegal memory access I had before seemed to occur in attention, so perhaps it is something related to chunked-prefill rather than prefix caching (since chunked-prefill is on by default is max-len >32k)

Update on AssertionError: expected running sequences:

Two minutes later the next error:

β”‚     return self._call_impl(*args, **kwargs)                                                                                                                                                                                                                                                                                                                                                           β”‚
β”‚   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl                                                                                                                                                                                                                                                                                                 β”‚
β”‚     return forward_call(*args, **kwargs)                                                                                                                                                                                                                                                                                                                                                              β”‚
β”‚   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/sampler.py", line 110, in forward                                                                                                                                                                                                                                                                                          β”‚
β”‚     self._init_sampling_tensors(logits, sampling_metadata)                                                                                                                                                                                                                                                                                                                                            β”‚
β”‚   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/sampler.py", line 87, in _init_sampling_tensors                                                                                                                                                                                                                                                                            β”‚
β”‚     do_min_p) = SamplingTensors.from_sampling_metadata(                                                                                                                                                                                                                                                                                                                                               β”‚
β”‚   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/sampling_metadata.py", line 520, in from_sampling_metadata                                                                                                                                                                                                                                                                        β”‚
β”‚     sampling_tensors = SamplingTensors.from_lists(                                                                                                                                                                                                                                                                                                                                                    β”‚
β”‚   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/sampling_metadata.py", line 564, in from_lists                                                                                                                                                                                                                                                                                    β”‚
β”‚     temperatures_t = torch.tensor(                                                                                                                                                                                                                                                                                                                                                                    β”‚
β”‚ RuntimeError: CUDA error: an illegal memory access was encountered                                                                                                                                                                                                                                                                                                                                    β”‚
β”‚ CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.                                                                                                                                                                                                                                                                               β”‚
β”‚ For debugging consider passing CUDA_LAUNCH_BLOCKING=1                                                                                                                                                                                                                                                                                                                                                 β”‚
β”‚ Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.                                                                                                                                                                                                                                                                                                                                   β”‚
β”‚                                                                                                                                                                                                                                                                                                                                                                                                       β”‚
β”‚                                                                                                                                                                                                                                                                                                                                                                                                       β”‚
β”‚ The above exception was the direct cause of the following exception:                                                                                                                                                                                                                                                                                                                                  β”‚
β”‚                                                                                                                                                                                                                                                                                                                                                                                                       β”‚
β”‚ Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                                                    β”‚
β”‚   File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run                                                                                                                                                                                                                                                                                                                                    β”‚
β”‚   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 67, in _log_task_completion                                                                                                                                                                                                                                                                                    β”‚
β”‚     raise AsyncEngineDeadError(                                                                                                                                                                                                                                                                                                                                                                       β”‚
β”‚ vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for the actual cause.                                                                                                                                                                                                                  β”‚
β”‚ ERROR 08-30 19:24:28 client.py:265] Got Unhealthy response from RPC Server                                                                                                                                                                                                                                                                                                                            β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412] AsyncEngineDeadError('Background loop is stopped.')                                                                                                                                                                                                                                                                                                               β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412] Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 409, in generate                                                                                                                                                                                                                                                     β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]     await self.check_health(socket=socket)                                                                                                                                                                                                                                                                                                                        β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 431, in check_health                                                                                                                                                                                                                                                 β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]     await self._send_one_way_rpc_request(                                                                                                                                                                                                                                                                                                                         β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 266, in _send_one_way_rpc_request                                                                                                                                                                                                                                    β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]     raise response                                                                                                                                                                                                                                                                                                                                                β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412] vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop is stopped.                                                                                                                                                                                                                                                                                    β”‚
β”‚ CRITICAL 08-30 19:24:28 launcher.py:82] AsyncLLMEngine has failed, terminating server process                                                                                                                                                                                                                                                                                                         β”‚
β”‚ INFO:     10.94.90.10:46306 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error                                                                                                                                                                                                                                                                                                               β”‚
β”‚ ERROR 08-30 19:24:28 client.py:265] Got Unhealthy response from RPC Server                                                                                                                                                                                                                                                                                                                            β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412] AsyncEngineDeadError('Background loop is stopped.')                                                                                                                                                                                                                                                                                                               β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412] Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 409, in generate                                                                                                                                                                                                                                                     β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]     await self.check_health(socket=socket)                                                                                                                                                                                                                                                                                                                        β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 431, in check_health                                                                                                                                                                                                                                                 β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]     await self._send_one_way_rpc_request(                                                                                                                                                                                                                                                                                                                         β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 266, in _send_one_way_rpc_request                                                                                                                                                                                                                                    β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]     raise response                                                                                                                                                                                                                                                                                                                                                β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412] vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop is stopped.                                                                                                                                                                                                                                                                                    β”‚
β”‚ ERROR 08-30 19:24:28 client.py:265] Got Unhealthy response from RPC Server                                                                                                                                                                                                                                                                                                                            β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412] AsyncEngineDeadError('Background loop is stopped.')                                                                                                                                                                                                                                                                                                               β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412] Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 409, in generate                                                                                                                                                                                                                                                     β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]     await self.check_health(socket=socket)                                                                                                                                                                                                                                                                                                                        β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 431, in check_health                                                                                                                                                                                                                                                 β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]     await self._send_one_way_rpc_request(                                                                                                                                                                                                                                                                                                                         β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 266, in _send_one_way_rpc_request                                                                                                                                                                                                                                    β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412]     raise response                                                                                                                                                                                                                                                                                                                                                β”‚
β”‚ ERROR 08-30 19:24:28 client.py:412] vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop is stopped.                                                                                                                                                                                                                                                                                    β”‚
β”‚ CRITICAL 08-30 19:24:28 launcher.py:82] AsyncLLMEngine has failed, terminating server process                                                                                                                                                                                                                                                                                                         β”‚
β”‚ INFO:     10.94.91.181:55960 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error                                                                                                                                                                                                                                                                                                              β”‚
β”‚ CRITICAL 08-30 19:24:28 launcher.py:82] AsyncLLMEngine has failed, terminating server process                                                                                                                                                                                                                                                                                                         β”‚
β”‚ INFO:     10.94.88.168:48038 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error                                                                                                                                                                                                                                                                                                              β”‚
β”‚ [2024-08-30 19:24:28,686 E 65 3518] logging.cc:115: Stack trace:                                                                                                                                                                                                                                                                                                                                      β”‚
β”‚  /usr/local/lib/python3.10/dist-packages/ray/_raylet.so(+0x10b96aa) [0x7f7b2e5c76aa] ray::operator<<()                                                                                                                                                                                                                                                                                                β”‚
β”‚ /usr/local/lib/python3.10/dist-packages/ray/_raylet.so(+0x10bc932) [0x7f7b2e5ca932] ray::TerminateHandler()                                                                                                                                                                                                                                                                                           β”‚
β”‚ /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa37c) [0x7f7c7bb0a37c]                                                                                                                                                                                                                                                                                                                                   β”‚
β”‚ /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa3e7) [0x7f7c7bb0a3e7]                                                                                                                                                                                                                                                                                                                                   β”‚
β”‚ /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa36f) [0x7f7c7bb0a36f]                                                                                                                                                                                                                                                                                                                                   β”‚
β”‚ /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so(+0xe5ab35) [0x7f7c2e01eb35] c10d::ProcessGroupNCCL::ncclCommWatchdog()                                                                                                                                                                                                                                                             β”‚
β”‚ /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7f7c7bb36df4]                                                                                                                                                                                                                                                                                                                                   β”‚
β”‚ /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f7c7cd07609] start_thread                                                                                                                                                                                                                                                                                                                      β”‚
β”‚ /usr/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f7c7ce41353] __clone                                                                                                                                                                                                                                                                                                                              β”‚
β”‚                                                                                                                                                                                                                                                                                                                                                                                                       β”‚
β”‚ *** SIGABRT received at time=1725071068 on cpu 16 ***                                                                                                                                                                                                                                                                                                                                                 β”‚
β”‚ PC: @     0x7f7c7cd6500b  (unknown)  raise                                                                                                                                                                                                                                                                                                                                                            β”‚
β”‚     @     0x7f7c7cd65090       3216  (unknown)                                                                                                                                                                                                                                                                                                                                                        β”‚
β”‚     @     0x7f7c7bb0a37c  (unknown)  (unknown)                                                                                                                                                                                                                                                                                                                                                        β”‚
β”‚     @     0x7f7c7bb0a090  (unknown)  (unknown)                                                                                                                                                                                                                                                                                                                                                        β”‚
β”‚ [2024-08-30 19:24:28,688 E 65 3518] logging.cc:440: *** SIGABRT received at time=1725071068 on cpu 16 ***                                                                                                                                                                                                                                                                                             β”‚
β”‚ [2024-08-30 19:24:28,688 E 65 3518] logging.cc:440: PC: @     0x7f7c7cd6500b  (unknown)  raise                                                                                                                                                                                                                                                                                                        β”‚
β”‚ [2024-08-30 19:24:28,688 E 65 3518] logging.cc:440:     @     0x7f7c7cd65090       3216  (unknown)                                                                                                                                                                                                                                                                                                    β”‚
β”‚ [2024-08-30 19:24:28,688 E 65 3518] logging.cc:440:     @     0x7f7c7bb0a37c  (unknown)  (unknown)                                                                                                                                                                                                                                                                                                    β”‚
β”‚ [2024-08-30 19:24:28,689 E 65 3518] logging.cc:440:     @     0x7f7c7bb0a090  (unknown)  (unknown)                                                                                                                                                                                                                                                                                                    β”‚
β”‚ Fatal Python error: Aborted         

This seems to be the same illegal memory address issue as #8025

Are these logs from v0.5.4 or v0.5.5?

Update: looks like v0.5.5 based on line numbers

Update: I am able to reproduce the issue sporadically

@zoltan-fedor Thanks for reporting the bug. Could you please use without --num-scheduler-steps 8? I think there were several bug fixes on it after v0.5.5.

Hi.

I had the exact same issue.

There is an obvious behavior that is causing this issue. When I place a steady load, it doesn’t matter how long the load is maintained, but if I stop the load in the middle of making a request and getting a response, it seems to cause this issue by dropping the connection.

Below are the options I started VLLM with.

    - args:
      - --model
      - /data/models/llama-3-1-70b-instruct/base
      - --tensor-parallel-size
      - "4"
      - --load-format
      - "auto"
      - --max-model-len
      - "16384"
      - --disable-log-requests
      - --uvicorn-log-level
      - "warning"
      - --gpu-memory-utilization
      - "0.9"
      - --enable-prefix-caching
      - --num-scheduler-steps
      - "8"

Below is the log when the bug occurred.

ERROR 08-30 22:11:29 async_llm_engine.py:65] Engine background task failed
ERROR 08-30 22:11:29 async_llm_engine.py:65] Traceback (most recent call last):
ERROR 08-30 22:11:29 async_llm_engine.py:65]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 55, in _log_task_completion
ERROR 08-30 22:11:29 async_llm_engine.py:65]     return_value = task.result()
ERROR 08-30 22:11:29 async_llm_engine.py:65]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 930, in run_engine_loop
ERROR 08-30 22:11:29 async_llm_engine.py:65]     result = task.result()
ERROR 08-30 22:11:29 async_llm_engine.py:65]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 873, in engine_step
ERROR 08-30 22:11:29 async_llm_engine.py:65]     request_outputs = await self.engine.step_async(virtual_engine)
ERROR 08-30 22:11:29 async_llm_engine.py:65]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 356, in step_async
ERROR 08-30 22:11:29 async_llm_engine.py:65]     request_outputs = self._process_model_outputs(
ERROR 08-30 22:11:29 async_llm_engine.py:65]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 1232, in _process_model_outputs
ERROR 08-30 22:11:29 async_llm_engine.py:65]     self.output_processor.process_outputs(seq_group, outputs)
ERROR 08-30 22:11:29 async_llm_engine.py:65]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/output_processor/multi_step.py", line 73, in process_outputs
ERROR 08-30 22:11:29 async_llm_engine.py:65]     assert seqs, "expected running sequences"
ERROR 08-30 22:11:29 async_llm_engine.py:65] AssertionError: expected running sequences
ERROR:asyncio:Exception in callback functools.partial(<function _log_task_completion at 0x7f00de437be0>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f00c643a320>>)
handle: <Handle functools.partial(<function _log_task_completion at 0x7f00de437be0>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f00c643a320>>)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 55, in _log_task_completion
    return_value = task.result()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 930, in run_engine_loop
    result = task.result()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 873, in engine_step
    request_outputs = await self.engine.step_async(virtual_engine)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 356, in step_async
    request_outputs = self._process_model_outputs(
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 1232, in _process_model_outputs
    self.output_processor.process_outputs(seq_group, outputs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/output_processor/multi_step.py", line 73, in process_outputs
    assert seqs, "expected running sequences"
AssertionError: expected running sequences

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 67, in _log_task_completion
    raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for the actual cause.
ERROR 08-30 22:11:29 client.py:265] Got Unhealthy response from RPC Server
ERROR 08-30 22:11:29 client.py:412] AsyncEngineDeadError('Background loop is stopped.')
ERROR 08-30 22:11:29 client.py:412] Traceback (most recent call last):
ERROR 08-30 22:11:29 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 409, in generate
ERROR 08-30 22:11:29 client.py:412]     await self.check_health(socket=socket)
ERROR 08-30 22:11:29 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 431, in check_health
ERROR 08-30 22:11:29 client.py:412]     await self._send_one_way_rpc_request(
ERROR 08-30 22:11:29 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 266, in _send_one_way_rpc_request
ERROR 08-30 22:11:29 client.py:412]     raise response
ERROR 08-30 22:11:29 client.py:412] vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop is stopped.
ERROR 08-30 22:11:29 client.py:265] Got Unhealthy response from RPC Server
ERROR 08-30 22:11:29 client.py:412] AsyncEngineDeadError('Background loop is stopped.')
ERROR 08-30 22:11:29 client.py:412] Traceback (most recent call last):
ERROR 08-30 22:11:29 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 409, in generate
ERROR 08-30 22:11:29 client.py:412]     await self.check_health(socket=socket)
ERROR 08-30 22:11:29 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 431, in check_health
ERROR 08-30 22:11:29 client.py:412]     await self._send_one_way_rpc_request(
ERROR 08-30 22:11:29 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 266, in _send_one_way_rpc_request
ERROR 08-30 22:11:29 client.py:412]     raise response
ERROR 08-30 22:11:29 client.py:412] vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop is stopped.
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 265, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 261, in wrap
    await func()
  File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 238, in listen_for_disconnect
    message = await receive()
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 555, in receive
    await self.message_event.wait()
  File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f55771d0ca0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 754, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 774, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 295, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 75, in app
    await response(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 258, in __call__
    async with anyio.create_task_group() as task_group:
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
    raise BaseExceptionGroup(
exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
ERROR 08-30 22:11:29 client.py:265] Got Unhealthy response from RPC Server
ERROR 08-30 22:11:29 client.py:412] AsyncEngineDeadError('Background loop is stopped.')
ERROR 08-30 22:11:29 client.py:412] Traceback (most recent call last):
ERROR 08-30 22:11:29 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 409, in generate
ERROR 08-30 22:11:29 client.py:412]     await self.check_health(socket=socket)
ERROR 08-30 22:11:29 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 431, in check_health
ERROR 08-30 22:11:29 client.py:412]     await self._send_one_way_rpc_request(
ERROR 08-30 22:11:29 client.py:412]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 266, in _send_one_way_rpc_request
ERROR 08-30 22:11:29 client.py:412]     raise response
ERROR 08-30 22:11:29 client.py:412] vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop is stopped.
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 265, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 261, in wrap
    await func()
  File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 238, in listen_for_disconnect
    message = await receive()
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 555, in receive
    await self.message_event.wait()
  File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f557ebc4580