vllm: [Bug]: vllm doesn't support multi-instance GPU

Your current environment

Collecting environment information...
/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/usage/usage_lib.py:19: RuntimeWarning: Failed to read commit hash:
No module named 'vllm.commit_id'
  from vllm.version import __version__ as VLLM_VERSION
Traceback (most recent call last):
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 306, in _lazy_init
    queued_call()
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 174, in _check_capability
    capability = get_device_capability(d)
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 430, in get_device_capability
    prop = get_device_properties(device)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 448, in get_device_properties
    return _get_device_properties(device)  # type: ignore[name-defined]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: device >= 0 && device < num_gpus INTERNAL ASSERT FAILED at "/build/source/aten/src/ATen/cuda/CUDAContext.cpp":50, please report a bug to PyTorch. device=, num_gpus=

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/data/home/HammillC/documents/llm-platform/collect_env.py", line 728, in <module>
    main()
  File "/mnt/data/home/HammillC/documents/llm-platform/collect_env.py", line 707, in main
    output = get_pretty_env_info()
             ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/home/HammillC/documents/llm-platform/collect_env.py", line 702, in get_pretty_env_info
    return pretty_str(get_env_info())
                      ^^^^^^^^^^^^^^
  File "/mnt/data/home/HammillC/documents/llm-platform/collect_env.py", line 548, in get_env_info
    cuda_module_loading=get_cuda_module_loading_config(),
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/home/HammillC/documents/llm-platform/collect_env.py", line 485, in get_cuda_module_loading_config
    torch.cuda.init()
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 260, in init
    _lazy_init()
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 312, in _lazy_init
    raise DeferredCudaCallError(msg) from e
torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: device >= 0 && device < num_gpus INTERNAL ASSERT FAILED at "/build/source/aten/src/ATen/cuda/CUDAContext.cpp":50, please report a bug to PyTorch. device=, num_gpus=

CUDA call was originally invoked at:

  File "/mnt/data/home/HammillC/documents/llm-platform/collect_env.py", line 16, in <module>
    import torch
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/__init__.py", line 1480, in <module>
    _C._initExtension(manager_path())
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 238, in <module>
    _lazy_call(_check_capability)
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 235, in _lazy_call
    _queued_calls.append((callable, traceback.format_stack()))

🐛 Describe the bug

Fortunately, the collect environment output clearly illustrates the problem. Multi-instance GPUs require you to set a single MIG slice or full GPU in order to work correctly. To do this you specify

CUDA_VISIBLE_DEVICES=GPU-<hash>
#or
CUDA_VISIBLE_DEVICES=MIG-<hash>

but if you try

CUDA_VISIBLE_DEVICES=MIG-<hash> nixGL vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123

you get

Traceback (most recent call last):
  File "/nix/store/91r5lqqzw1bv7xvvyn8kc8b3bmlyyimv-python3.11-vllm-0.5.2/bin/.vllm-wrapped", line 9, in <module>
    sys.exit(main())
             ^^^^^^
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/scripts.py", line 148, in main
    args.dispatch_function(args)
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/scripts.py", line 28, in serve
    run_server(args)
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 224, in run_server
    if llm_engine is not None else AsyncLLMEngine.from_engine_args(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/engine/async_llm_engine.py", line 444, in from_engine_args
    engine = cls(
             ^^^^
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/engine/async_llm_engine.py", line 373, in __init__
    self.engine = self._init_engine(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/engine/async_llm_engine.py", line 520, in _init_engine
    return engine_class(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 249, in __init__
    self.model_executor = executor_class(
                          ^^^^^^^^^^^^^^^
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 150, in __init__
    super().__init__(model_config, cache_config, parallel_config,
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 46, in __init__
    self._init_executor()
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/executor/gpu_executor.py", line 24, in _init_executor
    self.driver_worker.init_device()
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/worker/worker.py", line 125, in init_device
    _check_if_gpu_supports_dtype(self.model_config.dtype)
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/worker/worker.py", line 353, in _check_if_gpu_supports_dtype
    compute_capability = current_platform.get_device_capability()
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/platforms/cuda.py", line 48, in get_device_capability
    physical_device_id = device_id_to_physical_device_id(device_id)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ar0r42s7ygg1h1clhr5i3k50zbqyrf8c-python3-3.11.9-env/lib/python3.11/site-packages/vllm/platforms/cuda.py", line 38, in device_id_to_physical_device_id
    return int(physical_device_id)
           ^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: 'MIG-52b4198c-7365-5c41-8e3b-358b2c032d63'

I tried to work around this by setting NVIDIA_VISIBLE_DEVICES to the MIG slice with CUDA_VISIBLE_DEVICES=0 but that didn’t work.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (6 by maintainers)

Most upvoted comments

Should I open a separate issue for not supporting using device UUIDs? If vLLM doesn’t wish to properly support CUDA_VISIBLE_DEVICES adding a legible error for users if they try would be helpful. Even going so far as to suggest the docker work around, or to use my patch.

edit: happy to PR such an error message

We can only maintain code path for common usecases. Future maintainers, highly likely, don’t know anything about MIG. This code can be broken at any time, because we don’t have any test against it.

Confirming that the fix I proposed in https://github.com/vllm-project/vllm/pull/6582 works. It’s actually not that hard to test this. CUDA_VISIBLE_DEVICES supports device UUIDs, you don’t even need MIG to exercise this code path. Just use the UUID for device 0 instead. But understand if MIG is too niche to support.

That being said assuming CUDA_VISIBLE_DEVICES is either an int or a list of ints is a bug, regardless of whether you want to support MIG or not.

sure, you can add it here:

https://github.com/vllm-project/vllm/blob/30efe41532b7cd7202d128ffedfc703c4039767f/vllm/envs.py#L186

this is where all vLLM code reads the CUDA_VISIBLE_DEVICES environment variable. error here if you find non-integers.