vllm: [Usage]: tensor-parallel-size=2,The program just kept hanging

parameters

node=1
gpu=2 tensor-parallel-size=2

The program just kept hanging

How would you like to use vllm

CUDA_VISIBLE_DEVICES=0,1 python3 -m vllm.entrypoints.openai.api_server --tensor-parallel-size 2 --model Qwen/Qwen2-VL-7B

log info as follow

Image

INFO 02-14 08:59:06 __init__.py:186] Automatically detected platform cuda. INFO 02-14 08:59:08 api_server.py:841] vLLM API server version 0.1.dev4487+gbbd2f98.d20250213 INFO 02-14 08:59:08 api_server.py:842] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template='/app/qwen25-vl-chat_template.jinja', chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='Qwen/Qwen2.5-VL-7B-autoglm-android-wechat-test-250211/', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=16384, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=True, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt={'image': 2}, mm_processor_kwargs={'max_pixels': 1000000}, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['qwen2p5-vl-7b-autoglm'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False) INFO 02-14 08:59:08 api_server.py:206] Started engine process with PID 13227 INFO 02-14 08:59:11 __init__.py:186] Automatically detected platform cuda. INFO 02-14 08:59:15 config.py:542] This model supports multiple tasks: {'embed', 'score', 'generate', 'reward', 'classify'}. Defaulting to 'generate'. INFO 02-14 08:59:15 config.py:1402] Defaulting to use mp for distributed inference WARNING 02-14 08:59:15 cuda.py:95] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used WARNING 02-14 08:59:15 config.py:678] Async output processing is not supported on the current platform type cuda. INFO 02-14 08:59:18 config.py:542] This model supports multiple tasks: {'embed', 'reward', 'generate', 'score', 'classify'}. Defaulting to 'generate'. INFO 02-14 08:59:18 config.py:1402] Defaulting to use mp for distributed inference WARNING 02-14 08:59:18 cuda.py:95] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used WARNING 02-14 08:59:18 config.py:678] Async output processing is not supported on the current platform type cuda. INFO 02-14 08:59:18 llm_engine.py:234] Initializing a V0 LLM engine (v0.1.dev4487+gbbd2f98.d20250213) with config: model='Qwen/Qwen2.5-VL-7B-autoglm-android-wechat-test-250211/', speculative_config=None, tokenizer='Qwen/Qwen2.5-VL-7B-autoglm-android-wechat-test-250211/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=qwen2p5-vl-7b-autoglm, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs={'max_pixels': 1000000}, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[],"max_capture_size":0}, use_cached_outputs=True, WARNING 02-14 08:59:19 multiproc_worker_utils.py:300] Reducing Torch parallelism from 62 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. INFO 02-14 08:59:19 custom_cache_manager.py:19] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager (VllmWorkerProcess pid=13501) INFO 02-14 08:59:19 multiproc_worker_utils.py:229] Worker ready; awaiting tasks INFO 02-14 08:59:20 cuda.py:230] Using Flash Attention backend. (VllmWorkerProcess pid=13501) INFO 02-14 08:59:20 cuda.py:230] Using Flash Attention backend. [W214 08:59:20.113971975 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator()) [W214 08:59:20.120284563 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator()) INFO 02-14 08:59:20 utils.py:950] Found nccl from library libnccl.so.2 INFO 02-14 08:59:20 pynccl.py:69] vLLM is using nccl==2.21.5 (VllmWorkerProcess pid=13501) INFO 02-14 08:59:20 utils.py:950] Found nccl from library libnccl.so.2 (VllmWorkerProcess pid=13501) INFO 02-14 08:59:20 pynccl.py:69] vLLM is using nccl==2.21.5

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 25

Most upvoted comments

环境如下: Rocky 8.10 Driver Version: 560.35.03 CUDA Driver Version: 12.6 python 3.12 vllm v0.7.3

(vllm-v0.7.3) [root@Fii-vLLM-T1 vllm]# nvidia-smi topo -p2p n
 	GPU0	GPU1	GPU2	GPU3	
 GPU0	X	NS	NS	NS	
 GPU1	NS	X	NS	NS	
 GPU2	NS	NS	X	NS	
 GPU3	NS	NS	NS	X	

运行脚本如下:

#!/bin/bash

# Define variables
export VLLM_LOGGING_LEVEL=DEBUG
export NCCL_P2P_DISABLE=1
export TENSOR_PARALLEL_SIZE=2
export MAX_NUM_BATCHED_TOKENS=1081344
export MAX_NUM_SEQS=4
export HOST="10.157.168.166"
export PORT=8444

# Run the command
python -m vllm.entrypoints.openai.api_server \
    --model /data/modelsfiles/DeepSeek-R1-Distill-Qwen-7B \
    --served-model-name DeepSeek-R1-Distill-Qwen-7B \
    --tensor-parallel-size $TENSOR_PARALLEL_SIZE \
    --max-num-batched-tokens $MAX_NUM_BATCHED_TOKENS \
    --max-num-seqs $MAX_NUM_SEQS \
    --device cuda \
    --host $HOST \
    --port $PORT

export NCCL_P2P_DISABLE=1 已经设置,但是依然hanging…

Image

关闭vt-d或者iommu,vt-d和iommu会导致nccl hang。

Image

lspci -vvv|grep ACSCtl结果中有“SrcValid+”说明iommu没有关闭,进bios关闭下就行。

Image

如何解决

import os os.environ['NCCL_P2P_DISABLE'] = '1'