vllm: [Bug]: vllm doesn't support multi-instance GPU
Your current environment
Collecting environment information...
/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/usage/usage_lib.py:19: RuntimeWarning: Failed to read commit hash:
No module named 'vllm.commit_id'
from vllm.version import __version__ as VLLM_VERSION
Traceback (most recent call last):
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 306, in _lazy_init
queued_call()
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 174, in _check_capability
capability = get_device_capability(d)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 430, in get_device_capability
prop = get_device_properties(device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 448, in get_device_properties
return _get_device_properties(device) # type: ignore[name-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: device >= 0 && device < num_gpus INTERNAL ASSERT FAILED at "/build/source/aten/src/ATen/cuda/CUDAContext.cpp":50, please report a bug to PyTorch. device=, num_gpus=
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/mnt/data/home/HammillC/documents/llm-platform/collect_env.py", line 728, in <module>
main()
File "/mnt/data/home/HammillC/documents/llm-platform/collect_env.py", line 707, in main
output = get_pretty_env_info()
^^^^^^^^^^^^^^^^^^^^^
File "/mnt/data/home/HammillC/documents/llm-platform/collect_env.py", line 702, in get_pretty_env_info
return pretty_str(get_env_info())
^^^^^^^^^^^^^^
File "/mnt/data/home/HammillC/documents/llm-platform/collect_env.py", line 548, in get_env_info
cuda_module_loading=get_cuda_module_loading_config(),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/data/home/HammillC/documents/llm-platform/collect_env.py", line 485, in get_cuda_module_loading_config
torch.cuda.init()
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 260, in init
_lazy_init()
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 312, in _lazy_init
raise DeferredCudaCallError(msg) from e
torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: device >= 0 && device < num_gpus INTERNAL ASSERT FAILED at "/build/source/aten/src/ATen/cuda/CUDAContext.cpp":50, please report a bug to PyTorch. device=, num_gpus=
CUDA call was originally invoked at:
File "/mnt/data/home/HammillC/documents/llm-platform/collect_env.py", line 16, in <module>
import torch
File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/__init__.py", line 1480, in <module>
_C._initExtension(manager_path())
File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 238, in <module>
_lazy_call(_check_capability)
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 235, in _lazy_call
_queued_calls.append((callable, traceback.format_stack()))
đ Describe the bug
Fortunately, the collect environment output clearly illustrates the problem. Multi-instance GPUs require you to set a single MIG slice or full GPU in order to work correctly. To do this you specify
CUDA_VISIBLE_DEVICES=GPU-<hash>
#or
CUDA_VISIBLE_DEVICES=MIG-<hash>
but if you try
CUDA_VISIBLE_DEVICES=MIG-<hash> nixGL vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123
you get
Traceback (most recent call last):
File "/nix/store/91r5lqqzw1bv7xvvyn8kc8b3bmlyyimv-python3.11-vllm-0.5.2/bin/.vllm-wrapped", line 9, in <module>
sys.exit(main())
^^^^^^
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/scripts.py", line 148, in main
args.dispatch_function(args)
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/scripts.py", line 28, in serve
run_server(args)
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 224, in run_server
if llm_engine is not None else AsyncLLMEngine.from_engine_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/engine/async_llm_engine.py", line 444, in from_engine_args
engine = cls(
^^^^
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/engine/async_llm_engine.py", line 373, in __init__
self.engine = self._init_engine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/engine/async_llm_engine.py", line 520, in _init_engine
return engine_class(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 249, in __init__
self.model_executor = executor_class(
^^^^^^^^^^^^^^^
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 150, in __init__
super().__init__(model_config, cache_config, parallel_config,
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 46, in __init__
self._init_executor()
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/executor/gpu_executor.py", line 24, in _init_executor
self.driver_worker.init_device()
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/worker/worker.py", line 125, in init_device
_check_if_gpu_supports_dtype(self.model_config.dtype)
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/worker/worker.py", line 353, in _check_if_gpu_supports_dtype
compute_capability = current_platform.get_device_capability()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/platforms/cuda.py", line 48, in get_device_capability
physical_device_id = device_id_to_physical_device_id(device_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/platforms/cuda.py", line 38, in device_id_to_physical_device_id
return int(physical_device_id)
^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: 'MIG-52b4198c-7365-5c41-8e3b-358b2c032d63'
I tried to work around this by setting NVIDIA_VISIBLE_DEVICES to the MIG slice with CUDA_VISIBLE_DEVICES=0 but that didnât work.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (6 by maintainers)
Should I open a separate issue for not supporting using device UUIDs? If vLLM doesnât wish to properly support CUDA_VISIBLE_DEVICES adding a legible error for users if they try would be helpful. Even going so far as to suggest the docker work around, or to use my patch.
edit: happy to PR such an error message
Confirming that the fix I proposed in https://github.com/vllm-project/vllm/pull/6582 works. Itâs actually not that hard to test this.
CUDA_VISIBLE_DEVICESsupports device UUIDs, you donât even need MIG to exercise this code path. Just use the UUID for device 0 instead. But understand if MIG is too niche to support.That being said assuming CUDA_VISIBLE_DEVICES is either an int or a list of ints is a bug, regardless of whether you want to support MIG or not.
sure, you can add it here:
https://github.com/vllm-project/vllm/blob/30efe41532b7cd7202d128ffedfc703c4039767f/vllm/envs.py#L186
this is where all vLLM code reads the
CUDA_VISIBLE_DEVICESenvironment variable. error here if you find non-integers.