vllm: [Bug]: vLLM 0.5.1 tensor parallel 2 stuck with Mixtral-8x7B-Instruct-v0.1
Your current environment
PyTorch version: 2.3.0+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A
OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)
Clang version: Could not collect
CMake version: version 3.29.0
Libc version: glibc-2.17
Python version: 3.9.16 (main, Aug 15 2023, 19:38:56) [GCC 8.3.1 20190311 (Red Hat 8.3.1-3)] (64-bit runtime)
Python platform: Linux-4.18.0-147.mt20200626.413.el8_1.x86_64-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
Nvidia driver version: 470.103.01
cuDNN version: Probably one of the following:
/usr/lib64/libcudnn.so.8.9.2
/usr/lib64/libcudnn_adv_infer.so.8.9.2
/usr/lib64/libcudnn_adv_train.so.8.9.2
/usr/lib64/libcudnn_cnn_infer.so.8.9.2
/usr/lib64/libcudnn_cnn_train.so.8.9.2
/usr/lib64/libcudnn_ops_infer.so.8.9.2
/usr/lib64/libcudnn_ops_train.so.8.9.2
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 43 bits physical, 48 bits virtual
CPU(s): 192
On-line CPU(s) list: 0-7
Off-line CPU(s) list: 8-191
Thread(s) per core: 0
Core(s) per socket: 48
Socket(s): 2
NUMA node(s): 2
Vendor ID: AuthenticAMD
CPU family: 23
Model: 49
Model name: AMD EPYC 7642 48-Core Processor
Stepping: 0
Frequency boost: enabled
CPU MHz: 3287.557
CPU max MHz: 2300.0000
CPU min MHz: 1500.0000
BogoMIPS: 4571.13
Virtualization: AMD-V
L1d cache: 1.5 MiB
L1i cache: 1.5 MiB
L2 cache: 24 MiB
L3 cache: 256 MiB
NUMA node0 CPU(s): 0-47,96-143
NUMA node1 CPU(s): 48-95,144-191
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2: Vulnerable, IBPB: disabled, STIBP: disabled
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca
Versions of relevant libraries:
[pip3] flake8==7.1.0
[pip3] mt-tritonclient==1.0.4
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.26.3
[pip3] nvidia-nccl-cu11==2.20.5
[pip3] nvidia-nccl-cu12==2.19.3
[pip3] onnx==1.12.0
[pip3] onnx-graphsurgeon==0.3.12
[pip3] onnxruntime==1.15.1
[pip3] scalellm==0.1.5+cu118torch2.3
[pip3] torch==2.3.0+cu118
[pip3] torchvision==0.18.1
[pip3] transformers==4.42.3
[pip3] transformers-stream-generator==0.0.4
[pip3] triton==2.3.0
[pip3] tritonclient==2.42.0
[pip3] vllm-nccl-cu11==2.18.1.0.4.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.1
vLLM Build Flags:
CUDA Archs: ; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 mlx5_0 mlx5_1 mlx5_2 mlx5_3 mlx5_4 mlx5_5 mlx5_6 mlx5_7 CPU Affinity NUMA Affinity
GPU0 X NV12 SYS SYS SYS SYS NODE NODE PXB PXB 48-95,144-191 1
GPU1 NV12 X SYS SYS SYS SYS NODE NODE PXB PXB 48-95,144-191 1
mlx5_0 SYS SYS X PIX NODE NODE SYS SYS SYS SYS
mlx5_1 SYS SYS PIX X NODE NODE SYS SYS SYS SYS
mlx5_2 SYS SYS NODE NODE X PIX SYS SYS SYS SYS
mlx5_3 SYS SYS NODE NODE PIX X SYS SYS SYS SYS
mlx5_4 NODE NODE SYS SYS SYS SYS X PIX NODE NODE
mlx5_5 NODE NODE SYS SYS SYS SYS PIX X NODE NODE
mlx5_6 PXB PXB SYS SYS SYS SYS NODE NODE X PIX
mlx5_7 PXB PXB SYS SYS SYS SYS NODE NODE PIX X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
🐛 Describe the bug
ref https://github.com/vllm-project/vllm/issues/6187#issuecomment-2212874496
python3 -m vllm.entrypoints.openai.api_server --model /workdir/Mixtral-8x7B-Instruct-v0.1 --tensor-parallel-size 2
INFO 07-08 10:51:48 api_server.py:206] vLLM API server version 0.5.1
INFO 07-08 10:51:48 api_server.py:207] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='/workdir/Mixtral-8x7B-Instruct-v0.1', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=False, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, model_loader_extra_config=None, preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None)
INFO 07-08 10:51:48 config.py:698] Defaulting to use mp for distributed inference
INFO 07-08 10:51:48 llm_engine.py:169] Initializing an LLM engine (v0.5.1) with config: model='/workdir/Mixtral-8x7B-Instruct-v0.1', speculative_config=None, tokenizer='/workdir/Mixtral-8x7B-Instruct-v0.1', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=/workdir/Mixtral-8x7B-Instruct-v0.1, use_v2_block_manager=False, enable_prefix_caching=False)
INFO 07-08 10:51:49 selector.py:191] Cannot use FlashAttention-2 backend because the vllm_flash_attn package is not found. `pip install vllm-flash-attn` for better performance.
INFO 07-08 10:51:49 selector.py:53] Using XFormers backend.
(VllmWorkerProcess pid=96623) INFO 07-08 10:51:49 selector.py:191] Cannot use FlashAttention-2 backend because the vllm_flash_attn package is not found. `pip install vllm-flash-attn` for better performance.
(VllmWorkerProcess pid=96623) INFO 07-08 10:51:49 selector.py:53] Using XFormers backend.
(VllmWorkerProcess pid=96623) INFO 07-08 10:51:50 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
INFO 07-08 10:51:51 utils.py:741] Found nccl from library libnccl.so.2
INFO 07-08 10:51:51 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=96623) INFO 07-08 10:51:51 utils.py:741] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=96623) INFO 07-08 10:51:51 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 07-08 10:51:51 custom_all_reduce_utils.py:202] generating GPU P2P access cache in /root/.config/vllm/gpu_p2p_access_cache_for_0,1.json
The startup is stuck here. cc @youkaichao
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (15 by maintainers)
@zhyncs
The server can start normally, but the performance is quite poor, nearly in an unavailable stateIt may be caused by test serving withexport VLLM_LOGGING_LEVEL=DEBUG export CUDA_LAUNCH_BLOCKING=1 export NCCL_DEBUG=TRACE export VLLM_TRACE_FUNCTION=1, you can try start server without these env.I understand that you currently have no plans to work on compatibility. I completely understand, especially since the team is very small. I have closed the issue for now. If more people encounter this problem in the future, we can consider reopening it or finding a better solution. Thanks anyway.