vllm: 0.4.3 error CUDA error: an illegal memory access was encountered

Your current environment

vllm 0.4.3 CUDA Driver Version: 555.42.02 4060Ti Super * 2

VLLM_ATTENTION_BACKEND=FLASH_ATTN CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server –gpu-memory-utilization 0.85 –quantization gptq --host 0.0.0.0 --port 1234 -tp 1 –max-model-len 32768 --served-model-name qwen2 –trust-remote-code –enable-prefix-caching

🐛 Describe the bug

** desc: ** I have tried running it both with a single and dual GPU, but after running for a period of time, it starts to report errors; the issue occurs 100% of the time. The commands used are as follows:

VLLM_ATTENTION_BACKEND=FLASH_ATTN CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server –gpu-memory-utilization 0.85 –quantization gptq --host 0.0.0.0 --port 1234 -tp 1 –max-model-len 32768 --served-model-name qwen2 –trust-remote-code –enable-prefix-caching

** error: ** 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: RuntimeError: CUDA error: an illegal memory access was encountered 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

** logs: ** 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: INFO: 192.168.1.161:38834 - “POST /v1/chat/completions HTTP/1.1” 500 Internal Server Error 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: ERROR: Exception in ASGI application 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: Traceback (most recent call last): 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py”, line 419, in run_asgi 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: result = await app( # type: ignore[func-returns-value] 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py”, line 84, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.app(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/applications.py”, line 1054, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await super().call(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/applications.py”, line 123, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 186, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 164, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, _send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/cors.py”, line 83, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/exceptions.py”, line 62, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 762, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 782, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await route.handle(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 297, in handle 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 77, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(app, request)(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 72, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: response = await func(request) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 299, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise e 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 294, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raw_response = await run_endpoint_function( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 191, in run_endpoint_function 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await dependant.call(**values) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/api_server.py”, line 103, in create_chat_completion 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: generator = await openai_serving_chat.create_chat_completion( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 198, in create_chat_completion 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.chat_completion_full_generator( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 360, in chat_completion_full_generator 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for res in result_generator: 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 662, in generate 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for output in self._process_request( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 769, in _process_request 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise e 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 765, in _process_request 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for request_output in stream: 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 80, in anext 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise result 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py”, line 419, in run_asgi 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: result = await app( # type: ignore[func-returns-value] 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py”, line 84, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.app(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/applications.py”, line 1054, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await super().call(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/applications.py”, line 123, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 186, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 164, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, _send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/cors.py”, line 83, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/exceptions.py”, line 62, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 762, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 782, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await route.handle(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 297, in handle 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 77, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(app, request)(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 72, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: response = await func(request) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 299, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise e 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 294, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raw_response = await run_endpoint_function( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 191, in run_endpoint_function 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await dependant.call(**values) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/api_server.py”, line 103, in create_chat_completion 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: generator = await openai_serving_chat.create_chat_completion( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 198, in create_chat_completion 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.chat_completion_full_generator( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 360, in chat_completion_full_generator 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for res in result_generator: 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 662, in generate 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for output in self._process_request( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 769, in _process_request 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise e 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 765, in _process_request 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for request_output in stream: 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 80, in anext 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise result 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py”, line 419, in run_asgi 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: result = await app( # type: ignore[func-returns-value] 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py”, line 84, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.app(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/applications.py”, line 1054, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await super().call(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/applications.py”, line 123, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 186, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 164, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, _send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/cors.py”, line 83, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/exceptions.py”, line 62, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 762, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 782, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await route.handle(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 297, in handle 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 77, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(app, request)(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 72, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: response = await func(request) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 299, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise e 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 294, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raw_response = await run_endpoint_function( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 191, in run_endpoint_function 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await dependant.call(**values) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/api_server.py”, line 103, in create_chat_completion 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: generator = await openai_serving_chat.create_chat_completion( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 198, in create_chat_completion 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.chat_completion_full_generator( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 360, in chat_completion_full_generator 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for res in result_generator: 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 662, in generate 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for output in self._process_request( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 769, in _process_request 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise e 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 765, in _process_request 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for request_output in stream: 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 80, in anext 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise result 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 40, in _raise_exception_on_finish 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: task.result() 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 521, in run_engine_loop 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: has_requests_in_progress = await asyncio.wait_for( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/asyncio/tasks.py”, line 479, in wait_for 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return fut.result() 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 495, in engine_step 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: request_outputs = await self.engine.step_async() 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 226, in step_async 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: output = await self.model_executor.execute_model_async( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/executor/gpu_executor.py”, line 117, in execute_model_async 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: output = await make_async(self.driver_worker.execute_model 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/concurrent/futures/thread.py”, line 58, in run 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: result = self.fn(*self.args, **self.kwargs) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/utils/_contextlib.py”, line 115, in decorate_context 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return func(*args, **kwargs) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/worker/worker.py”, line 272, in execute_model 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: output = self.model_runner.execute_model(seq_group_metadata_list, 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/utils/_contextlib.py”, line 115, in decorate_context 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return func(*args, **kwargs) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/worker/model_runner.py”, line 738, in execute_model 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: output = self.model.sample( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/model_executor/models/chatglm.py”, line 379, in sample 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: next_tokens = self.sampler(logits, sampling_metadata) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1532, in _wrapped_call_impl 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return self._call_impl(*args, **kwargs) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1541, in _call_impl 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return forward_call(*args, **kwargs) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/model_executor/layers/sampler.py”, line 96, in forward 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: sample_results, maybe_sampled_tokens_tensor = _sample( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/model_executor/layers/sampler.py”, line 655, in _sample 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return _sample_with_torch( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/model_executor/layers/sampler.py”, line 544, in _sample_with_torch 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: sample_results = _random_sample(seq_groups, 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/model_executor/layers/sampler.py”, line 324, in _random_sample 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: random_samples = random_samples.cpu() 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: RuntimeError: CUDA error: an illegal memory access was encountered 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1. 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: The above exception was the direct cause of the following exception: 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: Traceback (most recent call last): 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py”, line 419, in run_asgi 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: result = await app( # type: ignore[func-returns-value] 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py”, line 84, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.app(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/applications.py”, line 1054, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await super().call(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/applications.py”, line 123, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 186, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 164, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, _send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/cors.py”, line 83, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/exceptions.py”, line 62, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 762, in call 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 782, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await route.handle(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 297, in handle 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 77, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(app, request)(scope, receive, send) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 72, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: response = await func(request) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 299, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise e 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 294, in app 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raw_response = await run_endpoint_function( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 191, in run_endpoint_function 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await dependant.call(**values) 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/api_server.py”, line 103, in create_chat_completion 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: generator = await openai_serving_chat.create_chat_completion( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 198, in create_chat_completion 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.chat_completion_full_generator( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 360, in chat_completion_full_generator 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for res in result_generator: 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 662, in generate 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for output in self._process_request( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 756, in _process_request 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: stream = await self.add_request( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 561, in add_request 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: self.start_background_loop() 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 431, in start_background_loop 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise AsyncEngineDeadError( 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop has errored already. 6月 10 16:45:06 ma-MS-TZZ-Z690M bash[107468]: INFO 06-10 16:45:06 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 3 reqs, Swapped: 0 reqs, Pending: 0 reqs> 6月 10 16:45:09 ma-MS-TZZ-Z690M systemd[1]: Stopping Uvicorn server for my app…

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 36 (3 by maintainers)

Most upvoted comments

Still happen to me in v0.6.4-v0.6.6post1/2… Anyone has a fix yet ?

EDIT: This is specific to flash attention. Set export VLLM_ATTENTION_BACKEND=XFORMERS to fix.

Still occurs with --enforce-eager here are my settings:

    llm = LLM(model=model_name,
              quantization="fp8",
              enable_chunked_prefill=True,
              enable_prefix_caching=True,
              gpu_memory_utilization=0.5,
              max_num_batched_tokens=16384,
              max_model_len=2048,
              enforce_eager=True,
              )

Thanks~

Either enforce_eager=True or os.environ["VLLM_ATTENTION_BACKEND"] = "XFORMERS" can solve my problem, and in my task, enforce_eager=True is faster than XFORMERS,However,it’s slower than enforce_eager=False。

Still happened in version 0.6.1.post2!!!

Hi @maxin9966 , have you resolved the issue? If so, what steps did you take to do so? If not, can you please provide steps to reproduce it using the following template (replacing the italicized values with your values)?嗨@maxin9966,你解决了这个问题吗?如果是这样,您采取了哪些步骤来做到这一点?如果不是,您能否提供使用以下模板重现它的步骤(将斜体值替换为您的值)?

  • GPUs

    • 8x A6000
  • Client code

    • python3 benchmarks/benchmark_prefix_caching.py --model meta-llama/Meta-Llama-3-70B-Instruct --dataset-path ShareGPT.json --enable-prefix-caching --num-prompts 20 --repeat-count 5 --input-length-range 128:256
  • Server code

    • python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3-70B-Instruct --gpu-memory-utilization 0.40 --tensor-parallel-size 8 --max-model-len 2048 --trust-remote-code --enable-prefix-caching --max-num-seqs 128
  • Env Vars

    • export VLLM_ATTENTION_BACKEND=FLASH_ATTN
    • export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7,8
  • Version(s) Tested

    • v0.5.2

just change to export VLLM_ATTENTION_BACKEND=XFORMERS

try it.

Was running into the very same issue. Woosuk (@WoosukKwon) Setting VLLM_ATTENTION_BACKEND=XFORMERS seems to have resolved it for our runs.

+1 in 0.6.0

@maxin9966 你好,想问一下这个问题在最近的版本上解决了吗?我在0.5.3上运行再次出现了这个问题。

Hi, can you follow the tips so that we can get more information on which operation caused the problem?

In addition, does it work with --enforce-eager ?