vllm: [Bug]: vllm stall on llama3-70b warmup with 0.4.1
Your current environment
I’m running a minimally modified image of nvcr.io/nvidia/pytorch:23.10-py3 with Python 3.11.
PyTorch version: 2.2.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.29.2
Libc version: glibc-2.35
Python version: 3.11.5 (main, Aug 26 2023, 07:22:50) [Clang 16.0.3 ] (64-bit runtime)
Python platform: Linux-4.4.0-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
Nvidia driver version: 535.129.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 5
On-line CPU(s) list: 0-4
Vendor ID: GenuineIntel
Model name: unknown
CPU family: 6
Model: 85
Thread(s) per core: 0
Core(s) per socket: 0
Socket(s): 0
Stepping: unknown
BogoMIPS: 2200.19
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_vnni md_clear arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.19.3
[pip3] torch==2.2.1
[pip3] triton==2.2.0
[pip3] vllm_nccl_cu11==2.18.1.0.3.0
[conda] Could not collectROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.4.1
vLLM Build Flags:
CUDA Archs: 5.2 6.0 6.1 7.0 7.2 7.5 8.0 8.6 8.7 9.0+PTX; ROCm: Disabled; Neuron: Disabled
GPU Topology:
Could not collect
🐛 Describe the bug
I am trying to boot a llama-3 70b model with llvm, but it times out while trying to instantiate. So far I’ve let it rest for upwards of 10min with minimal observed GPU activity. It allocates a maximum of 7GB despite having access to the full 160GB (A100 80x2).
MODEL_DIR = "/model"
MODEL_NAME = "NousResearch/Meta-Llama-3-70B-Instruct"
Before this run, I’m downloading the safetensor files through huggingface-cli and placing them in the local /model volume. No downloading seems to be happening when the entrypoint boots.
engine_args = AsyncEngineArgs(
model=MODEL_DIR,
tensor_parallel_size=2,
gpu_memory_utilization=0.90,
enforce_eager=False,
disable_log_requests=True,
)
engine = AsyncLLMEngine.from_engine_args(engine_args)
Starting cold inference...
2024-04-22 22:29:35,056 INFO worker.py:1749 -- Started a local Ray instance.
INFO 04-22 22:29:40 llm_engine.py:98] Initializing an LLM engine (v0.4.1) with config: model='/model', speculative_config=None, tokenizer='/model', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 04-22 22:29:56 utils.py:570] Found nccl from environment variable VLLM_NCCL_SO_PATH=/usr/lib/x86_64-linux-gnu/libnccl.so
(RayWorkerWrapper pid=419) INFO 04-22 22:29:56 utils.py:570] Found nccl from environment variable VLLM_NCCL_SO_PATH=/usr/lib/x86_64-linux-gnu/libnccl.so
INFO 04-22 22:29:58 selector.py:28] Using FlashAttention backend.
(RayWorkerWrapper pid=419) INFO 04-22 22:29:58 selector.py:28] Using FlashAttention backend.
INFO 04-22 22:29:59 pynccl_utils.py:45] vLLM is using nccl==2.19.3
(RayWorkerWrapper pid=419) INFO 04-22 22:29:59 pynccl_utils.py:45] vLLM is using nccl==2.19.3
Note that before getting here, my env was throwing an exception related to https://github.com/vllm-project/vllm/issues/4257, so I’m providing the NCCL path manually:
export VLLM_NCCL_SO_PATH="/usr/lib/x86_64-linux-gnu/libnccl.so"
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 19 (9 by maintainers)
export NCCL_DEBUG=TRACE, which will give you more information on nccl.export NCCL_P2P_DISABLE=1, which might resolve the problem?I think the problem might come from nccl side.
If you are patient, you can also try to throw https://github.com/vllm-project/vllm/pull/4248 into your environment, and see if it helps.