Your current environment
Traceback (most recent call last):
File "/mnt/MSAI/home/cephdon/sources/vllm/collect_env.py", line 15, in <module>
from vllm.envs import environment_variables
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/__init__.py", line 3, in <module>
from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/engine/arg_utils.py", line 11, in <module>
from vllm.config import (CacheConfig, CompilationConfig, ConfigFormat,
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/config.py", line 21, in <module>
from vllm.model_executor.layers.quantization import (QUANTIZATION_METHODS,
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/model_executor/__init__.py", line 1, in <module>
from vllm.model_executor.parameter import (BasevLLMParameter,
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/model_executor/parameter.py", line 7, in <module>
from vllm.distributed import get_tensor_model_parallel_rank
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/distributed/__init__.py", line 1, in <module>
from .communication_op import *
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/distributed/communication_op.py", line 6, in <module>
from .parallel_state import get_tp_group
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/distributed/parallel_state.py", line 38, in <module>
import vllm.distributed.kv_transfer.kv_transfer_agent as kv_transfer
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/distributed/kv_transfer/kv_transfer_agent.py", line 15, in <module>
from vllm.distributed.kv_transfer.kv_connector.factory import (
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/distributed/kv_transfer/kv_connector/factory.py", line 3, in <module>
from .base import KVConnectorBase
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/distributed/kv_transfer/kv_connector/base.py", line 14, in <module>
from vllm.sequence import IntermediateTensors
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/sequence.py", line 16, in <module>
from vllm.inputs import SingletonInputs, SingletonInputsAdapter
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/inputs/__init__.py", line 7, in <module>
from .registry import (DummyData, InputContext, InputProcessingContext,
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/inputs/registry.py", line 13, in <module>
from vllm.transformers_utils.tokenizer import AnyTokenizer
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/transformers_utils/tokenizer.py", line 16, in <module>
from vllm.utils import make_async
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/utils.py", line 44, in <module>
from vllm.platforms import current_platform
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/platforms/__init__.py", line 100, in <module>
from .cuda import CudaPlatform
File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/platforms/cuda.py", line 14, in <module>
import vllm._C # noqa
ModuleNotFoundError: No module named 'vllm._C'
How you are installing vllm
pip install -vvv vllm
pip install vllm==0.6.4
pip install -e .
Before submitting a new issue…
I should note that this is on an NVIDIA AGX Orin 64gb machine, so aarch64. I have cuda 12.6, but building from source has this same issue that I cannot seem to get past. Installing numpy (any version) does not resolve the issue. I am using miniconda to manage virtual environments. pip list and conda list both show numpy.
Note that the error message above is trying to use the CPU device, even if I have the cuda version of torch installed. Using the CPU version of torch does not resolve the issue.
@youkaichao
That’s not how supercomputing centers work. They tend to compile everything to their platform, and try to extract every bit of performance they can. Running a linux kernel and a whole toolchain on top of a a linux kernel and a whole toolchain (repeat as needed) is not acceptable in many places. The overhead is not worth the convenience.
Plus, the security nightmare that docker makes.
I have read your documentation multiple times. I have managed to compile it, but never managed to make it work.
It is got so bad that the market has been offering gh200 under cost because literally no one can run vllm on them properly because you can’t bother to offer a precompiled wheel.
Do you need links?
we cannot provide wheels, because pytorch does not have official release wheels for gh200 (arm gpus). we have to wait for pytorch 2.6 release.
technically, you can get wheels by copying from the docker container, but the wheel does not specify pytorch as dependency, and you need to install pytorch before installing vllm using that wheel.
can you elaborate?
@cephdon
this is a warning, you can ignore.
this is the real error.
see the great note from @dtrifiro
@youkaichao @dtrifiro any idea about this?