vllm: [Usage]: tensor-parallel-size=2,The program just kept hanging
parameters
node=1
gpu=2
tensor-parallel-size=2
The program just kept hanging
How would you like to use vllm
CUDA_VISIBLE_DEVICES=0,1 python3 -m vllm.entrypoints.openai.api_server --tensor-parallel-size 2 --model Qwen/Qwen2-VL-7B
log info as follow
INFO 02-14 08:59:06 __init__.py:186] Automatically detected platform cuda. INFO 02-14 08:59:08 api_server.py:841] vLLM API server version 0.1.dev4487+gbbd2f98.d20250213 INFO 02-14 08:59:08 api_server.py:842] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template='/app/qwen25-vl-chat_template.jinja', chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='Qwen/Qwen2.5-VL-7B-autoglm-android-wechat-test-250211/', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=16384, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=True, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt={'image': 2}, mm_processor_kwargs={'max_pixels': 1000000}, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['qwen2p5-vl-7b-autoglm'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False) INFO 02-14 08:59:08 api_server.py:206] Started engine process with PID 13227 INFO 02-14 08:59:11 __init__.py:186] Automatically detected platform cuda. INFO 02-14 08:59:15 config.py:542] This model supports multiple tasks: {'embed', 'score', 'generate', 'reward', 'classify'}. Defaulting to 'generate'. INFO 02-14 08:59:15 config.py:1402] Defaulting to use mp for distributed inference WARNING 02-14 08:59:15 cuda.py:95] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used WARNING 02-14 08:59:15 config.py:678] Async output processing is not supported on the current platform type cuda. INFO 02-14 08:59:18 config.py:542] This model supports multiple tasks: {'embed', 'reward', 'generate', 'score', 'classify'}. Defaulting to 'generate'. INFO 02-14 08:59:18 config.py:1402] Defaulting to use mp for distributed inference WARNING 02-14 08:59:18 cuda.py:95] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used WARNING 02-14 08:59:18 config.py:678] Async output processing is not supported on the current platform type cuda. INFO 02-14 08:59:18 llm_engine.py:234] Initializing a V0 LLM engine (v0.1.dev4487+gbbd2f98.d20250213) with config: model='Qwen/Qwen2.5-VL-7B-autoglm-android-wechat-test-250211/', speculative_config=None, tokenizer='Qwen/Qwen2.5-VL-7B-autoglm-android-wechat-test-250211/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=qwen2p5-vl-7b-autoglm, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs={'max_pixels': 1000000}, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[],"max_capture_size":0}, use_cached_outputs=True, WARNING 02-14 08:59:19 multiproc_worker_utils.py:300] Reducing Torch parallelism from 62 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. INFO 02-14 08:59:19 custom_cache_manager.py:19] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager (VllmWorkerProcess pid=13501) INFO 02-14 08:59:19 multiproc_worker_utils.py:229] Worker ready; awaiting tasks INFO 02-14 08:59:20 cuda.py:230] Using Flash Attention backend. (VllmWorkerProcess pid=13501) INFO 02-14 08:59:20 cuda.py:230] Using Flash Attention backend. [W214 08:59:20.113971975 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator()) [W214 08:59:20.120284563 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator()) INFO 02-14 08:59:20 utils.py:950] Found nccl from library libnccl.so.2 INFO 02-14 08:59:20 pynccl.py:69] vLLM is using nccl==2.21.5 (VllmWorkerProcess pid=13501) INFO 02-14 08:59:20 utils.py:950] Found nccl from library libnccl.so.2 (VllmWorkerProcess pid=13501) INFO 02-14 08:59:20 pynccl.py:69] vLLM is using nccl==2.21.5
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 25
关闭vt-d或者iommu,vt-d和iommu会导致nccl hang。
lspci -vvv|grep ACSCtl结果中有“SrcValid+”说明iommu没有关闭,进bios关闭下就行。
import os os.environ['NCCL_P2P_DISABLE'] = '1'