vllm: [Bug]: 0.8.5 post1 cuda error
Your current environment
env: 5070ti 16G+5080TI 16G model:Qwen3-32B-AWQ command:PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_TRITON_FLASH_ATTN=1 vllm serve xunlei_model/Qwen3-32B-AWQ --port 8099 --max-model-len 25000 --served-model-name vllm --enforce_eager --tensor-parallel-size 2 --gpu_memory_utilization 0.95
A few days ago I installed a relatively new vllm, which allowed me to run tensors in parallel on two graphics cards. A lot of concurrent tests were done at that time and they were successful. But today, I cannot run tensors in parallel. I tried to reinstall vllm but it still doesn’t work.
(vllmnv) root@epyc:~# pip list |grep torch pytorch-triton 3.3.0+git96316ce5 torch 2.8.0.dev20250507+cu128 torchaudio 2.6.0.dev20250507+cu128 torchvision 0.22.0.dev20250507+cu128 (vllmnv) root@epyc:~# pip list |grep vllm vllm 0.8.5.post2.dev0+g3015d5634.d20250507.cu128 /root/vllmnv (vllmnv) root@epyc:~# pip list |grep triton pytorch-triton 3.3.0+git96316ce5 triton 3.3.0 (vllmnv) root@epyc:~#
🐛 Describe the bug
(vllmnv) root@epyc:~# PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_TRITON_FLASH_ATTN=1 vllm serve xunlei_model/Qwen3-32B-AWQ --port 8099 --max-model-len 25000 --served-model-name vllm --enforce_eager --tensor-parallel-size 2 --gpu_memory_utilization 0.95
INFO 05-07 17:25:49 [importing.py:53] Triton module has been replaced with a placeholder.
INFO 05-07 17:25:49 [init.py:239] Automatically detected platform cuda.
WARNING 05-07 17:25:49 [cuda.py:409] Detected different devices in the system: NVIDIA GeForce RTX 5070 Ti, NVIDIA GeForce RTX 5080. Please make sure to set CUDA_DEVICE_ORDER=PCI_BUS_ID to avoid unexpected behavior.
INFO 05-07 17:25:55 [api_server.py:1043] vLLM API server version 0.8.5.post2.dev0+g3015d5634.d20250507
INFO 05-07 17:25:55 [api_server.py:1044] args: Namespace(subparser=‘serve’, model_tag=‘xunlei_model/Qwen3-32B-AWQ’, config=‘’, host=None, port=8099, uvicorn_log_level=‘info’, disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=[‘‘], allowed_methods=[’’], allowed_headers=[‘*’], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format=‘auto’, response_role=‘assistant’, ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin=‘’, model=‘xunlei_model/Qwen3-32B-AWQ’, task=‘auto’, tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode=‘auto’, trust_remote_code=False, allowed_local_media_path=None, load_format=‘auto’, download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: ‘auto’>, dtype=‘auto’, max_model_len=25000, guided_decoding_backend=‘auto’, reasoning_parser=None, logits_processor_pattern=None, model_impl=‘auto’, distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=2, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.95, swap_space=4, kv_cache_dtype=‘auto’, num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo=‘builtin’, cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=True, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type=‘ray’, tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype=‘auto’, long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device=‘auto’, speculative_config=None, ignore_patterns=[], served_model_name=[‘vllm’], qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy=‘fcfs’, enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls=‘vllm.core.scheduler.Scheduler’, override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls=‘auto’, worker_extension_cls=‘’, generation_config=‘auto’, override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x7f35899649d0>)
INFO 05-07 17:26:04 [config.py:717] This model supports multiple tasks: {‘reward’, ‘generate’, ‘embed’, ‘score’, ‘classify’}. Defaulting to ‘generate’.
INFO 05-07 17:26:04 [awq_marlin.py:113] The model is convertible to awq_marlin during runtime. Using awq_marlin kernel.
INFO 05-07 17:26:04 [config.py:1770] Defaulting to use mp for distributed inference
INFO 05-07 17:26:04 [config.py:2003] Chunked prefill is enabled with max_num_batched_tokens=2048.
WARNING 05-07 17:26:04 [cuda.py:93] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 05-07 17:26:06 [importing.py:53] Triton module has been replaced with a placeholder.
INFO 05-07 17:26:07 [init.py:239] Automatically detected platform cuda.
WARNING 05-07 17:26:07 [cuda.py:409] Detected different devices in the system: NVIDIA GeForce RTX 5070 Ti, NVIDIA GeForce RTX 5080. Please make sure to set CUDA_DEVICE_ORDER=PCI_BUS_ID to avoid unexpected behavior.
INFO 05-07 17:26:10 [core.py:58] Initializing a V1 LLM engine (v0.8.5.post2.dev0+g3015d5634.d20250507) with config: model=‘xunlei_model/Qwen3-32B-AWQ’, speculative_config=None, tokenizer=‘xunlei_model/Qwen3-32B-AWQ’, skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=25000, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=awq_marlin, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend=‘auto’, reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=vllm, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={“splitting_ops”:[],“compile_sizes”:[],“cudagraph_capture_sizes”:[],“max_capture_size”:0}
WARNING 05-07 17:26:10 [multiproc_worker_utils.py:306] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 05-07 17:26:10 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1], buffer_handle=(2, 10485760, 10, ‘psm_2079fba2’), local_subscribe_addr=‘ipc:///tmp/07b09b72-231b-415c-946e-24f6964ba32a’, remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 05-07 17:26:12 [importing.py:53] Triton module has been replaced with a placeholder.
INFO 05-07 17:26:12 [importing.py:53] Triton module has been replaced with a placeholder.
INFO 05-07 17:26:12 [init.py:239] Automatically detected platform cuda.
INFO 05-07 17:26:12 [init.py:239] Automatically detected platform cuda.
WARNING 05-07 17:26:12 [cuda.py:409] Detected different devices in the system: NVIDIA GeForce RTX 5070 Ti, NVIDIA GeForce RTX 5080. Please make sure to set CUDA_DEVICE_ORDER=PCI_BUS_ID to avoid unexpected behavior.
WARNING 05-07 17:26:12 [cuda.py:409] Detected different devices in the system: NVIDIA GeForce RTX 5070 Ti, NVIDIA GeForce RTX 5080. Please make sure to set CUDA_DEVICE_ORDER=PCI_BUS_ID to avoid unexpected behavior.
WARNING 05-07 17:26:15 [utils.py:2522] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fe2e1efc9a0>
WARNING 05-07 17:26:15 [utils.py:2522] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7f84bdc7c880>
(VllmWorker rank=1 pid=361841) INFO 05-07 17:26:15 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, ‘psm_fe2f07c0’), local_subscribe_addr=‘ipc:///tmp/54a564d7-3635-49bb-a6c8-d5916fdc682e’, remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=0 pid=361840) INFO 05-07 17:26:15 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, ‘psm_2d704d99’), local_subscribe_addr=‘ipc:///tmp/71592e17-2d66-4927-bbd5-101877bb407b’, remote_subscribe_addr=None, remote_addr_ipv6=False)
[W507 17:26:16.180558180 socket.cpp:755] [c10d] The client socket cannot be initialized to connect to [::ffff:127.0.0.1]:40307 (errno: 97 - Address family not supported by protocol).
[W507 17:26:16.195914705 socket.cpp:755] [c10d] The client socket cannot be initialized to connect to [::ffff:127.0.0.1]:40307 (errno: 97 - Address family not supported by protocol).
[W507 17:26:16.196550423 ProcessGroupNCCL.cpp:1027] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
[W507 17:26:16.201489509 ProcessGroupNCCL.cpp:1027] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
[Gloo] Rank 1 is connected to 1 peer ranks. Expected number of connected peer ranks is : 1
[Gloo] Rank 0 is connected to 1 peer ranks. Expected number of connected peer ranks is : 1
[Gloo] Rank 1 is connected to 1 peer ranks. Expected number of connected peer ranks is : 1
[Gloo] Rank 0 is connected to 1 peer ranks. Expected number of connected peer ranks is : 1
(VllmWorker rank=0 pid=361840) INFO 05-07 17:26:16 [utils.py:1055] Found nccl from library libnccl.so.2
(VllmWorker rank=0 pid=361840) INFO 05-07 17:26:16 [pynccl.py:69] vLLM is using nccl==2.26.2
(VllmWorker rank=1 pid=361841) INFO 05-07 17:26:16 [utils.py:1055] Found nccl from library libnccl.so.2
(VllmWorker rank=1 pid=361841) INFO 05-07 17:26:16 [pynccl.py:69] vLLM is using nccl==2.26.2
(VllmWorker rank=1 pid=361841) INFO 05-07 17:26:47 [custom_all_reduce_utils.py:244] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
(VllmWorker rank=0 pid=361840) INFO 05-07 17:26:47 [custom_all_reduce_utils.py:244] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
(VllmWorker rank=1 pid=361841) WARNING 05-07 17:26:47 [custom_all_reduce.py:146] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorker rank=0 pid=361840) WARNING 05-07 17:26:47 [custom_all_reduce.py:146] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorker rank=0 pid=361840) INFO 05-07 17:26:47 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, ‘psm_727734f0’), local_subscribe_addr=‘ipc:///tmp/45a0040c-6109-4d9d-8ed7-67106199776a’, remote_subscribe_addr=None, remote_addr_ipv6=False)
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(VllmWorker rank=0 pid=361840) INFO 05-07 17:26:47 [parallel_state.py:1004] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0
(VllmWorker rank=0 pid=361840) INFO 05-07 17:26:47 [cuda.py:221] Using Flash Attention backend on V1 engine.
(VllmWorker rank=0 pid=361840) WARNING 05-07 17:26:47 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=0 pid=361840) INFO 05-07 17:26:47 [gpu_model_runner.py:1329] Starting to load model xunlei_model/Qwen3-32B-AWQ…
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(VllmWorker rank=1 pid=361841) INFO 05-07 17:26:47 [parallel_state.py:1004] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 1
(VllmWorker rank=1 pid=361841) INFO 05-07 17:26:47 [cuda.py:221] Using Flash Attention backend on V1 engine.
(VllmWorker rank=1 pid=361841) WARNING 05-07 17:26:47 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=1 pid=361841) INFO 05-07 17:26:47 [gpu_model_runner.py:1329] Starting to load model xunlei_model/Qwen3-32B-AWQ…
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:05<00:15, 5.00s/it]
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:09<00:09, 4.97s/it]
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:15<00:05, 5.06s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:19<00:00, 4.76s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:19<00:00, 4.85s/it]
(VllmWorker rank=0 pid=361840)
(VllmWorker rank=1 pid=361841) INFO 05-07 17:27:07 [loader.py:458] Loading weights took 19.49 seconds
(VllmWorker rank=0 pid=361840) INFO 05-07 17:27:07 [loader.py:458] Loading weights took 19.64 seconds
(VllmWorker rank=1 pid=361841) INFO 05-07 17:27:08 [gpu_model_runner.py:1347] Model loading took 9.0121 GiB and 20.197659 seconds
(VllmWorker rank=0 pid=361840) INFO 05-07 17:27:08 [gpu_model_runner.py:1347] Model loading took 9.0121 GiB and 20.314751 seconds
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] WorkerProc hit an exception.
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] Traceback (most recent call last):
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/v1/executor/multiproc_executor.py”, line 465, in worker_busy_loop
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] output = func(*args, **kwargs)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/utils/_contextlib.py”, line 116, in decorate_context
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return func(*args, **kwargs)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/v1/worker/gpu_worker.py”, line 183, in determine_available_memory
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] self.model_runner.profile_run()
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/v1/worker/gpu_model_runner.py”, line 1651, in profile_run
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/utils/_contextlib.py”, line 116, in decorate_context
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return func(*args, **kwargs)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/v1/worker/gpu_model_runner.py”, line 1497, in _dummy_run
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] outputs = model(
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1755, in _wrapped_call_impl
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1766, in _call_impl
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/model_executor/models/qwen3.py”, line 299, in forward
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] hidden_states = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/compilation/decorators.py”, line 172, in call
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return self.forward(*args, **kwargs)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/model_executor/models/qwen2.py”, line 343, in forward
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] hidden_states, residual = layer(
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1755, in _wrapped_call_impl
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1766, in _call_impl
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/model_executor/models/qwen3.py”, line 212, in forward
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] hidden_states = self.self_attn(
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1755, in _wrapped_call_impl
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1766, in _call_impl
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/model_executor/models/qwen3.py”, line 131, in forward
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] qkv, _ = self.qkv_proj(hidden_states)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1755, in _wrapped_call_impl
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1766, in call_impl
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/model_executor/layers/linear.py”, line 485, in forward
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] output_parallel = self.quant_method.apply(self, input, bias)
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/model_executor/layers/quantization/awq_marlin.py”, line 305, in apply
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return apply_awq_marlin_linear(
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/model_executor/layers/quantization/utils/marlin_utils.py”, line 393, in apply_awq_marlin_linear
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] output = ops.gptq_marlin_gemm(reshaped_x,
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/_custom_ops.py”, line 822, in gptq_marlin_gemm
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return torch.ops._C.gptq_marlin_gemm(a, b_q_weight, b_scales, b_zeros,
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/_ops.py”, line 1208, in call
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return self._op(*args, **(kwargs or {}))
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] RuntimeError: CUDA error: an illegal memory access was encountered
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
(VllmWorker rank=1 pid=361841) ERROR 05-07 17:27:08 [multiproc_executor.py:470]
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] WorkerProc hit an exception.
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] Traceback (most recent call last):
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/v1/executor/multiproc_executor.py”, line 465, in worker_busy_loop
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] output = func(*args, **kwargs)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/utils/_contextlib.py”, line 116, in decorate_context
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return func(*args, **kwargs)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/v1/worker/gpu_worker.py”, line 183, in determine_available_memory
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] self.model_runner.profile_run()
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/v1/worker/gpu_model_runner.py”, line 1651, in profile_run
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/utils/_contextlib.py”, line 116, in decorate_context
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return func(*args, **kwargs)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/v1/worker/gpu_model_runner.py”, line 1497, in _dummy_run
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] outputs = model(
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1755, in _wrapped_call_impl
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1766, in _call_impl
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/model_executor/models/qwen3.py”, line 299, in forward
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] hidden_states = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/compilation/decorators.py”, line 172, in call
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return self.forward(*args, **kwargs)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/model_executor/models/qwen2.py”, line 343, in forward
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] hidden_states, residual = layer(
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1755, in _wrapped_call_impl
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1766, in _call_impl
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/model_executor/models/qwen3.py”, line 212, in forward
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] hidden_states = self.self_attn(
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1755, in _wrapped_call_impl
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1766, in _call_impl
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/model_executor/models/qwen3.py”, line 131, in forward
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] qkv, _ = self.qkv_proj(hidden_states)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1755, in _wrapped_call_impl
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1766, in call_impl
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/model_executor/layers/linear.py”, line 485, in forward
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] output_parallel = self.quant_method.apply(self, input, bias)
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/model_executor/layers/quantization/awq_marlin.py”, line 305, in apply
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return apply_awq_marlin_linear(
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/model_executor/layers/quantization/utils/marlin_utils.py”, line 393, in apply_awq_marlin_linear
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] output = ops.gptq_marlin_gemm(reshaped_x,
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/vllmnv/vllm/_custom_ops.py”, line 822, in gptq_marlin_gemm
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return torch.ops._C.gptq_marlin_gemm(a, b_q_weight, b_scales, b_zeros,
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/torch/_ops.py”, line 1208, in call
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] return self._op(*args, **(kwargs or {}))
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] RuntimeError: CUDA error: an illegal memory access was encountered
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
(VllmWorker rank=0 pid=361840) ERROR 05-07 17:27:08 [multiproc_executor.py:470]
ERROR 05-07 17:27:08 [core.py:396] EngineCore failed to start.
ERROR 05-07 17:27:08 [core.py:396] Traceback (most recent call last):
ERROR 05-07 17:27:08 [core.py:396] File “/root/vllmnv/vllm/v1/engine/core.py”, line 387, in run_engine_core
ERROR 05-07 17:27:08 [core.py:396] engine_core = EngineCoreProc(*args, **kwargs)
ERROR 05-07 17:27:08 [core.py:396] File “/root/vllmnv/vllm/v1/engine/core.py”, line 329, in init
ERROR 05-07 17:27:08 [core.py:396] super().init(vllm_config, executor_class, log_stats,
ERROR 05-07 17:27:08 [core.py:396] File “/root/vllmnv/vllm/v1/engine/core.py”, line 71, in init
ERROR 05-07 17:27:08 [core.py:396] self._initialize_kv_caches(vllm_config)
ERROR 05-07 17:27:08 [core.py:396] File “/root/vllmnv/vllm/v1/engine/core.py”, line 129, in _initialize_kv_caches
ERROR 05-07 17:27:08 [core.py:396] available_gpu_memory = self.model_executor.determine_available_memory()
ERROR 05-07 17:27:08 [core.py:396] File “/root/vllmnv/vllm/v1/executor/abstract.py”, line 75, in determine_available_memory
ERROR 05-07 17:27:08 [core.py:396] output = self.collective_rpc(“determine_available_memory”)
ERROR 05-07 17:27:08 [core.py:396] File “/root/vllmnv/vllm/v1/executor/multiproc_executor.py”, line 185, in collective_rpc
ERROR 05-07 17:27:08 [core.py:396] raise RuntimeError(
ERROR 05-07 17:27:08 [core.py:396] RuntimeError: Worker failed with error 'CUDA error: an illegal memory access was encountered
ERROR 05-07 17:27:08 [core.py:396] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
ERROR 05-07 17:27:08 [core.py:396] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
ERROR 05-07 17:27:08 [core.py:396] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
ERROR 05-07 17:27:08 [core.py:396] ', please check the stack trace above for the root cause
ERROR 05-07 17:27:10 [multiproc_executor.py:123] Worker proc VllmWorker-0 died unexpectedly, shutting down executor.
Process EngineCore_0:
Traceback (most recent call last):
File “/root/miniconda3/envs/vllmnv/lib/python3.10/multiprocessing/process.py”, line 314, in _bootstrap
self.run()
File “/root/miniconda3/envs/vllmnv/lib/python3.10/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “/root/vllmnv/vllm/v1/engine/core.py”, line 400, in run_engine_core
raise e
File “/root/vllmnv/vllm/v1/engine/core.py”, line 387, in run_engine_core
engine_core = EngineCoreProc(*args, **kwargs)
File “/root/vllmnv/vllm/v1/engine/core.py”, line 329, in init
super().init(vllm_config, executor_class, log_stats,
File “/root/vllmnv/vllm/v1/engine/core.py”, line 71, in init
self._initialize_kv_caches(vllm_config)
File “/root/vllmnv/vllm/v1/engine/core.py”, line 129, in _initialize_kv_caches
available_gpu_memory = self.model_executor.determine_available_memory()
File “/root/vllmnv/vllm/v1/executor/abstract.py”, line 75, in determine_available_memory
output = self.collective_rpc(“determine_available_memory”)
File “/root/vllmnv/vllm/v1/executor/multiproc_executor.py”, line 185, in collective_rpc
raise RuntimeError(
RuntimeError: Worker failed with error 'CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
', please check the stack trace above for the root cause
Traceback (most recent call last):
File “/root/miniconda3/envs/vllmnv/bin/vllm”, line 8, in <module>
sys.exit(main())
File “/root/vllmnv/vllm/entrypoints/cli/main.py”, line 53, in main
args.dispatch_function(args)
File “/root/vllmnv/vllm/entrypoints/cli/serve.py”, line 27, in cmd
uvloop.run(run_server(args))
File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/uvloop/init.py”, line 82, in run
return loop.run_until_complete(wrapper())
File “uvloop/loop.pyx”, line 1518, in uvloop.loop.Loop.run_until_complete
File “/root/miniconda3/envs/vllmnv/lib/python3.10/site-packages/uvloop/init.py”, line 61, in wrapper
return await main
File “/root/vllmnv/vllm/entrypoints/openai/api_server.py”, line 1078, in run_server
async with build_async_engine_client(args) as engine_client:
File “/root/miniconda3/envs/vllmnv/lib/python3.10/contextlib.py”, line 199, in aenter
return await anext(self.gen)
File “/root/vllmnv/vllm/entrypoints/openai/api_server.py”, line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File “/root/miniconda3/envs/vllmnv/lib/python3.10/contextlib.py”, line 199, in aenter
return await anext(self.gen)
File “/root/vllmnv/vllm/entrypoints/openai/api_server.py”, line 178, in build_async_engine_client_from_engine_args
async_llm = AsyncLLM.from_vllm_config(
File “/root/vllmnv/vllm/v1/engine/async_llm.py”, line 150, in from_vllm_config
return cls(
File “/root/vllmnv/vllm/v1/engine/async_llm.py”, line 118, in init
self.engine_core = core_client_class(
File “/root/vllmnv/vllm/v1/engine/core_client.py”, line 642, in init
super().init(
File “/root/vllmnv/vllm/v1/engine/core_client.py”, line 398, in init
self._wait_for_engine_startup()
File “/root/vllmnv/vllm/v1/engine/core_client.py”, line 430, in _wait_for_engine_startup
raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above.
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 15 (1 by maintainers)
I change nvidia-nccl-cu12 from 2.26.2 to 2.26.5 but still not work.
change nvidia-nccl-cu12 from 2.26.2 to 2.26.5, fixed this issue
pip install nvidia-nccl-cu12==2.26.5