vllm: [Bug]: vLLM v0.6.1 Instability issue under load.
Your current environment
The output of `python collect_env.py`
Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31
Python version: 3.10.14 (main, Apr 6 2024, 18:45:05) [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-25-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3
Nvidia driver version: 535.86.10
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 52 bits physical, 57 bits virtual
CPU(s): 96
On-line CPU(s) list: 0-95
Thread(s) per core: 1
Core(s) per socket: 48
Socket(s): 2
NUMA node(s): 8
Vendor ID: GenuineIntel
CPU family: 6
Model: 143
Model name: Intel(R) Xeon(R) Platinum 8468
Stepping: 8
CPU MHz: 2100.000
CPU max MHz: 2100.0000
CPU min MHz: 800.0000
BogoMIPS: 4200.00
L1d cache: 4.5 MiB
L1i cache: 3 MiB
L2 cache: 192 MiB
L3 cache: 210 MiB
NUMA node0 CPU(s): 0-11
NUMA node1 CPU(s): 12-23
NUMA node2 CPU(s): 24-35
NUMA node3 CPU(s): 36-47
NUMA node4 CPU(s): 48-59
NUMA node5 CPU(s): 60-71
NUMA node6 CPU(s): 72-83
NUMA node7 CPU(s): 84-95
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr avx512_fp16 flush_l1d arch_capabilities
Versions of relevant libraries:
[pip3] flashinfer==0.1.6+cu121torch2.4
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.6.68
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pyzmq==26.2.0
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.44.2
[pip3] triton==3.0.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.0@
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 PXB SYS SYS SYS SYS SYS SYS SYS 0-11 0 N/A
GPU1 NV18 X NV18 NV18 SYS PIX PIX PXB SYS SYS SYS SYS 24-35 2 N/A
GPU2 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS SYS PXB 72-83 6 N/A
GPU3 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS SYS PIX 72-83 6 N/A
NIC0 PXB SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS
NIC1 SYS PIX SYS SYS SYS X PIX PXB SYS SYS SYS SYS
NIC2 SYS PIX SYS SYS SYS PIX X PXB SYS SYS SYS SYS
NIC3 SYS PXB SYS SYS SYS PXB PXB X SYS SYS SYS SYS
NIC4 SYS SYS SYS SYS SYS SYS SYS SYS X PIX PXB SYS
NIC5 SYS SYS SYS SYS SYS SYS SYS SYS PIX X PXB SYS
NIC6 SYS SYS SYS SYS SYS SYS SYS SYS PXB PXB X SYS
NIC7 SYS SYS PXB PIX SYS SYS SYS SYS SYS SYS SYS X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
🐛 Describe the bug
I did a load test on vLLM v0.6.0 based on the conversation history. I’ve run the test about 3 times, and I’ve always encountered the issue. I’m not sure if this issue appears after GPU cache usage reaches 100%, but so far it’s been reproduced during load after GPU cache usage reaches 100%.
ERROR 09-05 17:22:27 async_llm_engine.py:63] Engine background task failed
ERROR 09-05 17:22:27 async_llm_engine.py:63] Traceback (most recent call last):
ERROR 09-05 17:22:27 async_llm_engine.py:63] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 53, in _log_task_completion
ERROR 09-05 17:22:27 async_llm_engine.py:63] return_value = task.result()
ERROR 09-05 17:22:27 async_llm_engine.py:63] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 939, in run_engine_loop
ERROR 09-05 17:22:27 async_llm_engine.py:63] result = task.result()
ERROR 09-05 17:22:27 async_llm_engine.py:63] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 868, in engine_step
ERROR 09-05 17:22:27 async_llm_engine.py:63] request_outputs = await self.engine.step_async(virtual_engine)
ERROR 09-05 17:22:27 async_llm_engine.py:63] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 345, in step_async
ERROR 09-05 17:22:27 async_llm_engine.py:63] output = await self.model_executor.execute_model_async(
ERROR 09-05 17:22:27 async_llm_engine.py:63] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 177, in execute_model_async
ERROR 09-05 17:22:27 async_llm_engine.py:63] return await self._driver_execute_model_async(execute_model_req)
ERROR 09-05 17:22:27 async_llm_engine.py:63] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 231, in _driver_execute_model_async
ERROR 09-05 17:22:27 async_llm_engine.py:63] return await self.driver_exec_model(execute_model_req)
ERROR 09-05 17:22:27 async_llm_engine.py:63] File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR 09-05 17:22:27 async_llm_engine.py:63] result = self.fn(*self.args, **self.kwargs)
ERROR 09-05 17:22:27 async_llm_engine.py:63] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 303, in execute_model
ERROR 09-05 17:22:27 async_llm_engine.py:63] inputs = self.prepare_input(execute_model_req)
ERROR 09-05 17:22:27 async_llm_engine.py:63] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 291, in prepare_input
ERROR 09-05 17:22:27 async_llm_engine.py:63] return self._get_driver_input_and_broadcast(execute_model_req)
ERROR 09-05 17:22:27 async_llm_engine.py:63] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 253, in _get_driver_input_and_broadcast
ERROR 09-05 17:22:27 async_llm_engine.py:63] self.model_runner.prepare_model_input(
ERROR 09-05 17:22:27 async_llm_engine.py:63] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1380, in prepare_model_input
ERROR 09-05 17:22:27 async_llm_engine.py:63] model_input = self._prepare_model_input_tensors(
ERROR 09-05 17:22:27 async_llm_engine.py:63] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1042, in _prepare_model_input_tensors
ERROR 09-05 17:22:27 async_llm_engine.py:63] return builder.build() # type: ignore
ERROR 09-05 17:22:27 async_llm_engine.py:63] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 698, in build
ERROR 09-05 17:22:27 async_llm_engine.py:63] max(inter_data.seq_lens))
ERROR 09-05 17:22:27 async_llm_engine.py:63] ValueError: max() arg is an empty sequence
ERROR:asyncio:Exception in callback functools.partial(<function _log_task_completion at 0x7f204842b640>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f20447a8970>>)
handle: <Handle functools.partial(<function _log_task_completion at 0x7f204842b640>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f20447a8970>>)>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 53, in _log_task_completion
return_value = task.result()
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 939, in run_engine_loop
result = task.result()
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 868, in engine_step
request_outputs = await self.engine.step_async(virtual_engine)
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 345, in step_async
output = await self.model_executor.execute_model_async(
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 177, in execute_model_async
return await self._driver_execute_model_async(execute_model_req)
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 231, in _driver_execute_model_async
return await self.driver_exec_model(execute_model_req)
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 303, in execute_model
inputs = self.prepare_input(execute_model_req)
File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 291, in prepare_input
return self._get_driver_input_and_broadcast(execute_model_req)
File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 253, in _get_driver_input_and_broadcast
self.model_runner.prepare_model_input(
File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1380, in prepare_model_input
model_input = self._prepare_model_input_tensors(
File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1042, in _prepare_model_input_tensors
return builder.build() # type: ignore
File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 698, in build
max(inter_data.seq_lens))
ValueError: max() arg is an empty sequence
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 65, in _log_task_completion
raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for the actual cause.
(...)
ERROR 09-05 17:22:27 client.py:412] Traceback (most recent call last):
ERROR 09-05 17:22:27 client.py:412] File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 409, in generate
ERROR 09-05 17:22:27 client.py:412] await self.check_health(socket=socket)
ERROR 09-05 17:22:27 client.py:412] File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 429, in check_health
ERROR 09-05 17:22:27 client.py:412] await self._send_one_way_rpc_request(
ERROR 09-05 17:22:27 client.py:412] File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 267, in _send_one_way_rpc_request
ERROR 09-05 17:22:27 client.py:412] raise response
(...)
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 257, in __call__
await wrap(partial(self.listen_for_disconnect, receive))
File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 253, in wrap
await func()
File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 230, in listen_for_disconnect
message = await receive()
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 555, in receive
await self.message_event.wait()
File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7fb1397c3c10
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 113, in __call__
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85, in __call__
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 62, in wrapped_app
raise exc
File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 51, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 715, in __call__
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 735, in app
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 62, in wrapped_app
raise exc
File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 51, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 74, in app
await response(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 250, in __call__
async with anyio.create_task_group() as task_group:
File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
raise BaseExceptionGroup(
exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 257, in __call__
await wrap(partial(self.listen_for_disconnect, receive))
File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 253, in wrap
await func()
File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 230, in listen_for_disconnect
message = await receive()
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 555, in receive
await self.message_event.wait()
File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7fb13b480a00
The average number of input prompt tokens in the conversation history we used for testing was 1,600 tokens, and the average number of answer tokens from the model was 150 tokens.
The following are the vLLM startup arguments.
- args:
- --model
- /data/models/llama-65b-instruct/base
- --tensor-parallel-size
- "4"
- --load-format
- "auto"
- --block-size
- "32"
- --max-seq-len-to-capture
- "8192"
- --max-model-len
- "8192"
- --disable-log-requests
- --uvicorn-log-level
- "warning"
- --gpu-memory-utilization
- "0.95"
This error did not occur in versions prior to v0.5.5. (I did load tests 3 times with exactly same arguments.)
During a load, I also got this warning message 5 minutes before the system died. Could this be related to the issue?
WARNING 09-05 17:10:02 scheduler.py:1355] Sequence group cmpl-0c49e59124fe4f9c8b8e6e0f4bae49d7-0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space. This can affect the end-to-end performance. Increase gpu_memory_utilization or tensor_parallel_size to provide more KV cache memory. total_num_cumulative_preemption=1
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 23 (10 by maintainers)
@ashgold here is the PR to fix it https://github.com/vllm-project/vllm/pull/8267 should be merged soon. Thanks for pointing this, it was a real race condition.
CC: @robertgshaw2-neuralmagic
I’m working on a fix, should be ready soon.
Update:
--disable-async-output-procdoes prevent this@ashgold Cool!
issue disappeared in v0.6.1.post1.
Thank you guys!!
@okwinds https://github.com/vllm-project/vllm/pull/8417 should fix it
@alexm-neuralmagic @youkaichao @SolitaryThinker @robertgshaw2-neuralmagic I did load test on v0.6.1 today, and engine STILL DIES.
Now the timing of the error is a little different. As soon as the warning message below is printed, the engine dies.
the same exception under load (running 32B-W4A16)
4090 RTX
–dtype auto
–gpu-memory-utilization 0.95
–block-size 16
–max-model-len 5200
–kv-cache-dtype auto
–max-num-batched-tokens 5200
–max-seq-len-to-capture 5200
–enable-prefix-caching
–use-v2-block-manager
–num-lookahead-slots 1
–gpu-memory-utilization 0.95
–max-num-seqs 30
–num-scheduler-steps 2
–scheduler-delay-factor 0.3
vLLM version 0.6.0
CRITICAL 09-07 01:49:38 launcher.py:98] AsyncLLMEngine is already dead, terminating server process INFO: 127.0.0.1:50572 - “POST /v1/chat/completions HTTP/1.1” 500 Internal Server Error INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [3071562] INFO 09-07 01:49:38 server.py:228] vLLM ZMQ RPC Server was interrupted. ERROR:asyncio:Future exception was never retrieved future: <Future finished exception=AssertionError()> Traceback (most recent call last): File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py”, line 115, in generate async for request_output in results_generator: File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 1073, in generate async for output in await self.add_request( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 111, in generator raise result File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 53, in _log_task_completion return_value = task.result() File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 939, in run_engine_loop result = task.result() File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 868, in engine_step request_outputs = await self.engine.step_async(virtual_engine) File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py”, line 296, in step_async ) = self.scheduler[virtual_engine].schedule() File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/core/scheduler.py”, line 1109, in schedule scheduler_outputs = self._schedule() File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/core/scheduler.py”, line 1074, in _schedule return self._schedule_default() File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/core/scheduler.py”, line 921, in _schedule_default running_scheduled = self._schedule_running(budget, File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/core/scheduler.py”, line 551, in _schedule_running num_running_tokens = self._get_num_new_tokens( File “/home/gavin/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/core/scheduler.py”, line 1462, in _get_num_new_tokens assert num_new_tokens > 0 AssertionError
Let me try to reproduce on my end and take a look. Meanwhile, @ashgold, @br3no could you please trying
--disable-async-output-procand see if that changes anything?I have now observed another exception:
Both exceptions have in common, that they expect content, but instead there is nothing there. It seems that some sequences get lost on high load.
I have observed the same issue while load testing 0.6.0.
I also have observed the error when the GPU KV cache usage was close to 100%. I’m not sure there is a causal relation, though.
This issue leads to no graceful degradation, vLLM needs to be restarted.