vllm: [Bug]: 8xV100 gpus: Failed to infer device type

Your current environment

I’m trying to run inference in docker-compose, host: ubuntu 22.04

uname -r
5.19.0-1010-nvidia-lowlatency
version: '3.8'

services:
  mixtral7x8b:
    image: vllm/vllm-openai:v0.5.5
    restart: unless-stopped
    ports:
      - "100.73.239.219:8000:8000"
    volumes:
      - /opt/ai_models:/models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
    environment:
      - VLLM_API_KEY=<token here>
      # parameters for uvicorn (seconds)
      - TIMEOUT_KEEP_ALIVE=120
      - TIMEOUT_GRACEFUL_SHUTDOWN=30
    runtime: nvidia
    shm_size: 100gb
    command: >
      --host 0.0.0.0
      --dtype=half
      --served-model-name Mixtral-8x7b
      --model /models/mistralai/Mixtral-8x7B-Instruct-v0.1
nvidia-smi
Fri Sep 27 04:43:53 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.12              Driver Version: 550.90.12      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla V100-SXM3-32GB           Off |   00000000:1A:00.0 Off |                    0 |
| N/A   35C    P0             53W /  350W |       1MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Tesla V100-SXM3-32GB           Off |   00000000:1C:00.0 Off |                    0 |
| N/A   34C    P0             49W /  350W |       1MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  Tesla V100-SXM3-32GB           Off |   00000000:1D:00.0 Off |                    0 |
| N/A   36C    P0             56W /  350W |       1MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  Tesla V100-SXM3-32GB           Off |   00000000:1E:00.0 Off |                    0 |
| N/A   37C    P0             53W /  350W |       1MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  Tesla V100-SXM3-32GB           Off |   00000000:3E:00.0 Off |                    0 |
| N/A   38C    P0             57W /  350W |       1MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  Tesla V100-SXM3-32GB           Off |   00000000:3F:00.0 Off |                    0 |
| N/A   35C    P0             57W /  350W |       1MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  Tesla V100-SXM3-32GB           Off |   00000000:40:00.0 Off |                    0 |
| N/A   37C    P0             53W /  350W |       1MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  Tesla V100-SXM3-32GB           Off |   00000000:41:00.0 Off |                    0 |
| N/A   35C    P0             59W /  350W |       1MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Model Input Dumps

No response

🐛 Describe the bug

mixtral7x8b-1  | INFO 09-26 21:36:47 api_server.py:144] Multiprocessing frontend to use ipc:///tmp/32318298-afda-4100-99be-34a8f1cfb8ea for RPC Path.
mixtral7x8b-1  | INFO 09-26 21:36:47 api_server.py:161] Started engine process with PID 79
mixtral7x8b-1  | WARNING 09-26 21:36:56 config.py:1563] Casting torch.bfloat16 to torch.float16.
mixtral7x8b-1  | INFO 09-26 21:36:56 llm_engine.py:184] Initializing an LLM engine (v0.5.5) with config: model='/models/mistralai/Mixtral-8x7B-Instruct-v0.1', speculative_config=None, tokenizer='/models/mistralai/Mixtral-8x7B-Instruct-v0.1', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Mixtral-8x7b, use_v2_block_manager=False, enable_prefix_caching=False)
mixtral7x8b-1  | Process SpawnProcess-1:
mixtral7x8b-1  | Traceback (most recent call last):
mixtral7x8b-1  |   File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
mixtral7x8b-1  |     self.run()
mixtral7x8b-1  |   File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
mixtral7x8b-1  |     self._target(*self._args, **self._kwargs)
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/server.py", line 230, in run_rpc_server
mixtral7x8b-1  |     server = AsyncEngineRPCServer(async_engine_args, usage_context, rpc_path)
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/server.py", line 31, in __init__
mixtral7x8b-1  |     self.engine = AsyncLLMEngine.from_engine_args(
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 740, in from_engine_args
mixtral7x8b-1  |     engine = cls(
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 636, in __init__
mixtral7x8b-1  |     self.engine = self._init_engine(*args, **kwargs)
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 840, in _init_engine
mixtral7x8b-1  |     return engine_class(*args, **kwargs)
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 272, in __init__
mixtral7x8b-1  |     super().__init__(*args, **kwargs)
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 270, in __init__
mixtral7x8b-1  |     self.model_executor = executor_class(
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 46, in __init__
mixtral7x8b-1  |     self._init_executor()
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/gpu_executor.py", line 37, in _init_executor
mixtral7x8b-1  |     self.driver_worker = self._create_worker()
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/gpu_executor.py", line 104, in _create_worker
mixtral7x8b-1  |     return create_worker(**self._get_create_worker_kwargs(
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/gpu_executor.py", line 23, in create_worker
mixtral7x8b-1  |     wrapper.init_worker(**kwargs)
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 444, in init_worker
mixtral7x8b-1  |     self.worker = worker_class(*args, **kwargs)
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 99, in __init__
mixtral7x8b-1  |     self.model_runner: GPUModelRunnerBase = ModelRunnerClass(
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 842, in __init__
mixtral7x8b-1  |     self.attn_backend = get_attn_backend(
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/attention/selector.py", line 108, in get_attn_backend
mixtral7x8b-1  |     backend = which_attn_to_use(num_heads, head_size, num_kv_heads,
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/attention/selector.py", line 215, in which_attn_to_use
mixtral7x8b-1  |     if current_platform.get_device_capability()[0] < 8:
mixtral7x8b-1  |   File "/usr/local/lib/python3.10/dist-packages/vllm/platforms/interface.py", line 28, in get_device_capability
mixtral7x8b-1  |     raise NotImplementedError
mixtral7x8b-1  | NotImplementedError
mixtral7x8b-1  | ERROR 09-26 21:36:57 api_server.py:171] RPCServer process died before responding to readiness probe

It used to work just few hours ago, we simply restarted docker compose, same image, same tag, same mode. Not sure what might have happened

Why did it start checking compute capability <8 if it used to run on 7 just a day ago? I appreciate your help, I almost broke my head already.

on v0.6.2 getting a different error:

llama-3.1-8b-1  | Process SpawnProcess-1:
llama-3.1-8b-1  | Traceback (most recent call last):
llama-3.1-8b-1  |   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
llama-3.1-8b-1  |     self.run()
llama-3.1-8b-1  |   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
llama-3.1-8b-1  |     self._target(*self._args, **self._kwargs)
llama-3.1-8b-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 388, in run_mp_engine
llama-3.1-8b-1  |     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
llama-3.1-8b-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llama-3.1-8b-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 134, in from_engine_args
llama-3.1-8b-1  |     engine_config = engine_args.create_engine_config()
llama-3.1-8b-1  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llama-3.1-8b-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 873, in create_engine_config
llama-3.1-8b-1  |     device_config = DeviceConfig(device=self.device)
llama-3.1-8b-1  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llama-3.1-8b-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 1081, in __init__
llama-3.1-8b-1  |     raise RuntimeError("Failed to infer device type")
llama-3.1-8b-1  | RuntimeError: Failed to infer device type

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (7 by maintainers)

Most upvoted comments

I solved it through export LD_LIBRARY_PATH=/usr/lib64:$LD_LIBRARY_PATH

@DarkLight1337

Absolutely. I’m not sure if this is the good way to test it, but this is what poped up instantly:

import torch

def check_pytorch_gpu():
    try:
        if torch.cuda.is_available():
            print(f"PyTorch can access {torch.cuda.device_count()} GPU(s).")
            for i in range(torch.cuda.device_count()):
                print(f"GPU {i}: {torch.cuda.get_device_name(i)}")
        else:
            print("PyTorch cannot access any GPUs.")
    except Exception as e:
        print(f"An error occurred: {e}")

if __name__ == "__main__":
    check_pytorch_gpu()

pyproject.toml

[tool.poetry]
name = "try-gpus"
version = "0.1.0"
description = ""
authors = ["Gleb Erokhin"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"
torch = "^2.4.1"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Execution output:

PyTorch can access 8 GPU(s).
GPU 0: Tesla V100-SXM3-32GB
GPU 1: Tesla V100-SXM3-32GB
GPU 2: Tesla V100-SXM3-32GB
GPU 3: Tesla V100-SXM3-32GB
GPU 4: Tesla V100-SXM3-32GB
GPU 5: Tesla V100-SXM3-32GB
GPU 6: Tesla V100-SXM3-32GB
GPU 7: Tesla V100-SXM3-32GB