vllm: 0.4.3 error CUDA error: an illegal memory access was encountered
Your current environment
vllm 0.4.3 CUDA Driver Version: 555.42.02 4060Ti Super * 2
VLLM_ATTENTION_BACKEND=FLASH_ATTN CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server –gpu-memory-utilization 0.85 –quantization gptq --host 0.0.0.0 --port 1234 -tp 1 –max-model-len 32768 --served-model-name qwen2 –trust-remote-code –enable-prefix-caching
🐛 Describe the bug
** desc: ** I have tried running it both with a single and dual GPU, but after running for a period of time, it starts to report errors; the issue occurs 100% of the time. The commands used are as follows:
VLLM_ATTENTION_BACKEND=FLASH_ATTN CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server –gpu-memory-utilization 0.85 –quantization gptq --host 0.0.0.0 --port 1234 -tp 1 –max-model-len 32768 --served-model-name qwen2 –trust-remote-code –enable-prefix-caching
** error: ** 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: RuntimeError: CUDA error: an illegal memory access was encountered 6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
** logs: **
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: INFO: 192.168.1.161:38834 - “POST /v1/chat/completions HTTP/1.1” 500 Internal Server Error
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: ERROR: Exception in ASGI application
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: Traceback (most recent call last):
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py”, line 419, in run_asgi
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: result = await app( # type: ignore[func-returns-value]
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py”, line 84, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.app(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/applications.py”, line 1054, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await super().call(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/applications.py”, line 123, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 186, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 164, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, _send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/cors.py”, line 83, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/exceptions.py”, line 62, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 762, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 782, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await route.handle(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 297, in handle
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 77, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(app, request)(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 72, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: response = await func(request)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 299, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise e
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 294, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raw_response = await run_endpoint_function(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 191, in run_endpoint_function
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await dependant.call(**values)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/api_server.py”, line 103, in create_chat_completion
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: generator = await openai_serving_chat.create_chat_completion(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 198, in create_chat_completion
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.chat_completion_full_generator(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 360, in chat_completion_full_generator
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for res in result_generator:
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 662, in generate
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for output in self._process_request(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 769, in _process_request
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise e
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 765, in _process_request
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for request_output in stream:
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 80, in anext
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise result
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py”, line 419, in run_asgi
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: result = await app( # type: ignore[func-returns-value]
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py”, line 84, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.app(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/applications.py”, line 1054, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await super().call(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/applications.py”, line 123, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 186, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 164, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, _send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/cors.py”, line 83, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/exceptions.py”, line 62, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 762, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 782, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await route.handle(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 297, in handle
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 77, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(app, request)(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 72, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: response = await func(request)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 299, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise e
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 294, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raw_response = await run_endpoint_function(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 191, in run_endpoint_function
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await dependant.call(**values)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/api_server.py”, line 103, in create_chat_completion
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: generator = await openai_serving_chat.create_chat_completion(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 198, in create_chat_completion
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.chat_completion_full_generator(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 360, in chat_completion_full_generator
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for res in result_generator:
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 662, in generate
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for output in self._process_request(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 769, in _process_request
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise e
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 765, in _process_request
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for request_output in stream:
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 80, in anext
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise result
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py”, line 419, in run_asgi
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: result = await app( # type: ignore[func-returns-value]
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py”, line 84, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.app(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/applications.py”, line 1054, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await super().call(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/applications.py”, line 123, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 186, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 164, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, _send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/cors.py”, line 83, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/exceptions.py”, line 62, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 762, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 782, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await route.handle(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 297, in handle
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 77, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(app, request)(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 72, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: response = await func(request)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 299, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise e
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 294, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raw_response = await run_endpoint_function(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 191, in run_endpoint_function
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await dependant.call(**values)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/api_server.py”, line 103, in create_chat_completion
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: generator = await openai_serving_chat.create_chat_completion(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 198, in create_chat_completion
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.chat_completion_full_generator(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 360, in chat_completion_full_generator
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for res in result_generator:
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 662, in generate
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for output in self._process_request(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 769, in _process_request
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise e
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 765, in _process_request
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for request_output in stream:
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 80, in anext
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise result
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 40, in _raise_exception_on_finish
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: task.result()
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 521, in run_engine_loop
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: has_requests_in_progress = await asyncio.wait_for(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/asyncio/tasks.py”, line 479, in wait_for
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return fut.result()
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 495, in engine_step
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: request_outputs = await self.engine.step_async()
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 226, in step_async
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: output = await self.model_executor.execute_model_async(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/executor/gpu_executor.py”, line 117, in execute_model_async
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: output = await make_async(self.driver_worker.execute_model
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/concurrent/futures/thread.py”, line 58, in run
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: result = self.fn(*self.args, **self.kwargs)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/utils/_contextlib.py”, line 115, in decorate_context
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return func(*args, **kwargs)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/worker/worker.py”, line 272, in execute_model
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: output = self.model_runner.execute_model(seq_group_metadata_list,
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/utils/_contextlib.py”, line 115, in decorate_context
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return func(*args, **kwargs)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/worker/model_runner.py”, line 738, in execute_model
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: output = self.model.sample(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/model_executor/models/chatglm.py”, line 379, in sample
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: next_tokens = self.sampler(logits, sampling_metadata)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1532, in _wrapped_call_impl
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return self._call_impl(*args, **kwargs)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1541, in _call_impl
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return forward_call(*args, **kwargs)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/model_executor/layers/sampler.py”, line 96, in forward
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: sample_results, maybe_sampled_tokens_tensor = _sample(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/model_executor/layers/sampler.py”, line 655, in _sample
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return _sample_with_torch(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/model_executor/layers/sampler.py”, line 544, in _sample_with_torch
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: sample_results = _random_sample(seq_groups,
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/model_executor/layers/sampler.py”, line 324, in _random_sample
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: random_samples = random_samples.cpu()
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: RuntimeError: CUDA error: an illegal memory access was encountered
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: The above exception was the direct cause of the following exception:
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: Traceback (most recent call last):
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py”, line 419, in run_asgi
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: result = await app( # type: ignore[func-returns-value]
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py”, line 84, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.app(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/applications.py”, line 1054, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await super().call(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/applications.py”, line 123, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 186, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 164, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, _send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/cors.py”, line 83, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/middleware/exceptions.py”, line 62, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 762, in call
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.middleware_stack(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 782, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await route.handle(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 297, in handle
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await self.app(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 77, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await wrap_app_handling_exceptions(app, request)(scope, receive, send)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 64, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise exc
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/_exception_handler.py”, line 53, in wrapped_app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: await app(scope, receive, sender)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/starlette/routing.py”, line 72, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: response = await func(request)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 299, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise e
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 294, in app
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raw_response = await run_endpoint_function(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/fastapi/routing.py”, line 191, in run_endpoint_function
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await dependant.call(**values)
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/api_server.py”, line 103, in create_chat_completion
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: generator = await openai_serving_chat.create_chat_completion(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 198, in create_chat_completion
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: return await self.chat_completion_full_generator(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/entrypoints/openai/serving_chat.py”, line 360, in chat_completion_full_generator
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for res in result_generator:
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 662, in generate
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: async for output in self._process_request(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 756, in _process_request
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: stream = await self.add_request(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 561, in add_request
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: self.start_background_loop()
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: File “/home/ma/miniconda3/envs/myenv/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py”, line 431, in start_background_loop
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: raise AsyncEngineDeadError(
6月 10 16:45:05 ma-MS-TZZ-Z690M bash[107468]: vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop has errored already.
6月 10 16:45:06 ma-MS-TZZ-Z690M bash[107468]: INFO 06-10 16:45:06 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 3 reqs, Swapped: 0 reqs, Pending: 0 reqs>
6月 10 16:45:09 ma-MS-TZZ-Z690M systemd[1]: Stopping Uvicorn server for my app…
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 36 (3 by maintainers)
Still happen to me in v0.6.4-v0.6.6post1/2…
Anyone has a fix yet ?EDIT: This is specific to flash attention. Set
export VLLM_ATTENTION_BACKEND=XFORMERSto fix.Still occurs with --enforce-eager here are my settings:
Thanks~
Either
enforce_eager=Trueoros.environ["VLLM_ATTENTION_BACKEND"] = "XFORMERS"can solve my problem, and in my task,enforce_eager=Trueis faster than XFORMERS,However,it’s slower thanenforce_eager=False。Still happened in version 0.6.1.post2!!!
just change to
export VLLM_ATTENTION_BACKEND=XFORMERStry it.
Was running into the very same issue. Woosuk (@WoosukKwon) Setting
VLLM_ATTENTION_BACKEND=XFORMERSseems to have resolved it for our runs.+1 in 0.6.0
@maxin9966 你好,想问一下这个问题在最近的版本上解决了吗?我在0.5.3上运行再次出现了这个问题。
Hi, can you follow the tips so that we can get more information on which operation caused the problem?
In addition, does it work with
--enforce-eager?