vllm: [Bug]: Model load on 2 or 4-gpu A100 setup may cause default text encoding to be ascii, unless enforce_eager=True

Your current environment

The output of `python collect_env.py`
Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.30.2
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.5.0-1025-azure-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100 80GB PCIe
GPU 1: NVIDIA A100 80GB PCIe

Nvidia driver version: 550.90.07
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.3
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.3
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.3
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.3
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.3
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.3
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.3
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             48
On-line CPU(s) list:                0-47
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 7V13 64-Core Processor
CPU family:                         25
Model:                              1
Thread(s) per core:                 1
Core(s) per socket:                 48
Socket(s):                          1
Stepping:                           1
BogoMIPS:                           4890.89
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm
Hypervisor vendor:                  Microsoft
Virtualization type:                full
L1d cache:                          1.5 MiB (48 instances)
L1i cache:                          1.5 MiB (48 instances)
L2 cache:                           24 MiB (48 instances)
L3 cache:                           192 MiB (6 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-23
NUMA node1 CPU(s):                  24-47
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] flake8==7.1.0
[pip3] mypy==1.11.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu11==11.11.3.6
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu11==11.8.87
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu11==11.8.89
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu11==11.8.89
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu11==8.7.0.84
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu11==10.9.0.58
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu11==10.3.0.86
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu11==11.4.1.48
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu11==11.7.5.86
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu11==2.20.5
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.6.20
[pip3] nvidia-nvtx-cu11==11.8.86
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pyzmq==26.1.1
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.44.1
[pip3] triton==3.0.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.1@7de49aa86c7f169eb0962b6db29ad53fff519ffb
vLLM Build Flags:
CUDA Archs: 8.0; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    NIC0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV12    NODE    0-23    0               N/A
GPU1    NV12     X      SYS     24-47   1               N/A
NIC0    NODE    SYS      X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0

Model Input Dumps

No response

🐛 Describe the bug

Loading a model with vllm.LLM on a 2-gpu A100 machine appears to cause the python default file encoding to change from utf-8 to ascii, at least for the Phi-3 and Phi-3.5 MoE models I am loading. This results in a number of strange behaviors, including a requirement for the encoding to be manually specified for python file opens after the model load. In particular, it is easy to see it in action with readline.add_history(…utfcharacters…). I do not see this problem on single-gpu setups. This problem does not occur 100% of the time—rarely everything is OK after model load. In all cases my locale appears to remain a utf-8 locale and sys.getdefaultlocale() reports “utf-8”. Enclosed is a program that shows this bug for me:

import sys
import readline
import torch
from vllm import LLM

readline.add_history("😊")
model = LLM(
    'Phi-3-medium-128k-instruct', trust_remote_code=True,
    tensor_parallel_size=torch.cuda.device_count(),
)
print(">>> My Current encoding is:", sys.getdefaultencoding())
readline.add_history("😊")  # Crashes

Here’s what the output looks like when I run this:

INFO 09-17 12:45:36 config.py:904] Defaulting to use mp for distributed inference
WARNING 09-17 12:45:36 arg_utils.py:900] The model has a long context length (131072). This may cause OOM errors during the initial memory profiling phase, or result in low performance due to small KV cache space. Consider setting --max-model-len to a smaller value.
INFO 09-17 12:45:36 llm_engine.py:232] Initializing an LLM engine (v0.6.1) with config: model='Phi-3-medium-128k-instruct', speculative_config=None, tokenizer='Phi-3-medium-128k-instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Phi-3-medium-128k-instruct, use_v2_block_manager=False, num_scheduler_steps=1, enable_prefix_caching=False, use_async_output_proc=True)
WARNING 09-17 12:45:36 multiproc_gpu_executor.py:56] Reducing Torch parallelism from 48 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 09-17 12:45:36 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 09-17 12:45:36 selector.py:240] Cannot use FlashAttention-2 backend due to sliding window.
INFO 09-17 12:45:36 selector.py:116] Using XFormers backend.
(VllmWorkerProcess pid=12521) INFO 09-17 12:45:36 selector.py:240] Cannot use FlashAttention-2 backend due to sliding window.
(VllmWorkerProcess pid=12521) INFO 09-17 12:45:36 selector.py:116] Using XFormers backend.
/home/carl/.local/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
  @torch.library.impl_abstract("xformers_flash::flash_fwd")
(VllmWorkerProcess pid=12521) /home/carl/.local/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=12521)   @torch.library.impl_abstract("xformers_flash::flash_fwd")
/home/carl/.local/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
  @torch.library.impl_abstract("xformers_flash::flash_bwd")
(VllmWorkerProcess pid=12521) /home/carl/.local/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=12521)   @torch.library.impl_abstract("xformers_flash::flash_bwd")
(VllmWorkerProcess pid=12521) INFO 09-17 12:45:37 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=12521) INFO 09-17 12:45:38 utils.py:981] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=12521) INFO 09-17 12:45:38 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 09-17 12:45:38 utils.py:981] Found nccl from library libnccl.so.2
INFO 09-17 12:45:38 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 09-17 12:45:38 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/carl/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
(VllmWorkerProcess pid=12521) INFO 09-17 12:45:38 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/carl/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
INFO 09-17 12:45:38 shm_broadcast.py:235] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x758623b268f0>, local_subscribe_port=53769, remote_subscribe_port=None)
INFO 09-17 12:45:38 model_runner.py:997] Starting to load model Phi-3-medium-128k-instruct...
(VllmWorkerProcess pid=12521) INFO 09-17 12:45:38 model_runner.py:997] Starting to load model Phi-3-medium-128k-instruct...
(VllmWorkerProcess pid=12521) INFO 09-17 12:45:38 selector.py:240] Cannot use FlashAttention-2 backend due to sliding window.
(VllmWorkerProcess pid=12521) INFO 09-17 12:45:38 selector.py:116] Using XFormers backend.
INFO 09-17 12:45:38 selector.py:240] Cannot use FlashAttention-2 backend due to sliding window.
INFO 09-17 12:45:38 selector.py:116] Using XFormers backend.
Loading safetensors checkpoint shards:   0% Completed | 0/6 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  17% Completed | 1/6 [00:00<00:02,  1.92it/s]
Loading safetensors checkpoint shards:  33% Completed | 2/6 [00:01<00:02,  1.93it/s]
Loading safetensors checkpoint shards:  50% Completed | 3/6 [00:01<00:01,  1.90it/s]
Loading safetensors checkpoint shards:  67% Completed | 4/6 [00:02<00:01,  1.90it/s]
Loading safetensors checkpoint shards:  83% Completed | 5/6 [00:02<00:00,  2.08it/s]
Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:02<00:00,  2.06it/s]
Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:02<00:00,  2.00it/s]

INFO 09-17 12:45:41 model_runner.py:1008] Loading model weights took 13.1796 GB
(VllmWorkerProcess pid=12521) INFO 09-17 12:45:41 model_runner.py:1008] Loading model weights took 13.1796 GB
INFO 09-17 12:45:51 distributed_gpu_executor.py:57] # GPU blocks: 26896, # CPU blocks: 2621
INFO 09-17 12:45:54 model_runner.py:1309] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 09-17 12:45:54 model_runner.py:1313] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=12521) INFO 09-17 12:45:54 model_runner.py:1309] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=12521) INFO 09-17 12:45:54 model_runner.py:1313] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=12521) INFO 09-17 12:46:05 custom_all_reduce.py:223] Registering 2835 cuda graph addresses
INFO 09-17 12:46:05 custom_all_reduce.py:223] Registering 2835 cuda graph addresses
(VllmWorkerProcess pid=12521) INFO 09-17 12:46:05 model_runner.py:1428] Graph capturing finished in 11 secs.
INFO 09-17 12:46:05 model_runner.py:1428] Graph capturing finished in 11 secs.
>>> My Current encoding is: utf-8
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/carl/gutenllm/./vllm_bug.py", line 12, in <module>
[rank0]:     readline.add_history("😊")  # Crashes
[rank0]: UnicodeEncodeError: 'locale' codec can't encode character '\U0001f60a' in position 0: encoding error
INFO 09-17 12:46:07 multiproc_worker_utils.py:123] Killing local vLLM worker processes
[rank0]:[W917 12:46:08.474815406 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
/usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (5 by maintainers)

Most upvoted comments

The bug seems to be that libnvrtc.so does a duplocale() or a newlocale(), and then never reverts to the original one. Python uses nl_langinfo() to get the charset encoding which always uses the canonical global locale LC_GLOBAL_LOCALE (the pointer with long value == -1). So the fix is the following horrific hack after model load:

import ctypes
ctypes.CDLL(None).uselocale(-1)