vllm: ARM aarch-64 server build failed (host OS: Ubuntu22.04.3)

do as: https://docs.vllm.ai/en/latest/getting_started/installation.html

  1. docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
  2. git clone https://github.com/vllm-project/vllm.git
  3. cd vllm
  4. pip install -e .

here is the details in side the docker instance: root@f8c2e06fbf8b:/mnt/vllm# pip install -e . Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com Obtaining file:///mnt/vllm Installing build dependencies … done Checking if build backend supports build_editable … done Getting requirements to build editable … error error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully. │ exit code: 1 ╰─> [22 lines of output] /tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: No module named ‘numpy’ (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:84.) device: torch.device = torch.device(torch._C._get_default_device()), # torch.device(‘cpu’), No CUDA runtime is found, using CUDA_HOME=‘/usr/local/cuda’ <string>:142: UserWarning: Unsupported CUDA/ROCM architectures ({‘6.1’, ‘7.2’, ‘8.7’, ‘5.2’, ‘6.0’}) are excluded from the TORCH_CUDA_ARCH_LIST env variable (5.2 6.0 6.1 7.0 7.2 7.5 8.0 8.6 8.7 9.0+PTX). Supported CUDA/ROCM architectures are: {‘7.5’, ‘8.0’, ‘9.0’, ‘7.0’, ‘8.6+PTX’, ‘9.0+PTX’, ‘8.6’, ‘8.0+PTX’, ‘8.9+PTX’, ‘8.9’, ‘7.0+PTX’, ‘7.5+PTX’}. Traceback (most recent call last): File “/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py”, line 353, in <module> main() File “/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py”, line 335, in main json_out[‘return_val’] = hook(**hook_input[‘kwargs’]) File “/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py”, line 132, in get_requires_for_build_editable return hook(config_settings) File “/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py”, line 441, in get_requires_for_build_editable return self.get_requires_for_build_wheel(config_settings) File “/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py”, line 325, in get_requires_for_build_wheel return self._get_build_requires(config_settings, requirements=[‘wheel’]) File “/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py”, line 295, in _get_build_requires self.run_setup() File “/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py”, line 311, in run_setup exec(code, locals()) File “<string>”, line 297, in <module> File “<string>”, line 267, in get_vllm_version NameError: name ‘nvcc_cuda_version’ is not defined. Did you mean: ‘cuda_version’? [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[notice] A new release of pip is available: 23.2.1 -> 23.3.1 [notice] To update, run: python -m pip install --upgrade pip

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 4
  • Comments: 54 (21 by maintainers)

Commits related to this issue

Most upvoted comments

If anyone comes across this and is trying to get Llama-3.1 to work with the GH200 (or aarch64 + H100), I have the latest working container (v0.5.3-post1 with a couple more commits) image up at https://hub.docker.com/r/drikster80/vllm-gh200-openai Pull it with docker pull drikster80/vllm-gh200-openai:latest

Codes is still in the https://github.com/drikster80/vllm/tree/gh200-docker branch.

Validated Llama-3.1-8b-Instruct works, and trying to working to test 405B-FP8 now (with cpu-offload)

Got it working with the changes in this branch: https://github.com/haileyschoelkopf/vllm/tree/aarm64-dockerfile , with built dockerfiles here: https://hub.docker.com/r/haileysch/vllm-aarch64-base https://hub.docker.com/r/haileysch/vllm-aarch64-openai hopefully this’ll be helpful to others!

Updated the aarch64 remote branch to v0.5.2: https://github.com/drikster80/vllm/tree/gh200-docker

Pushed up a GH200 specific version (built for SM 9.0+PTX) to https://hub.docker.com/r/drikster80/vllm-gh200-openai

Building a more generic version now and will update this comment when complete.

@cyc00518 You can see the list of full changes here: https://github.com/vllm-project/vllm/compare/main...drikster80:vllm:gh200-docker

Effectively, xformers and vllm-flash-attention don’t release ARM64 wheels, so those need to be built from source. Also, since Nvidia’s PyTorch container already containers torch, torchvision, and some other stuff, those need to be commented out in the requirements file. The only 3 files that are changes are Dockerfile, requirements-build.txt, and requirements-cuda.txt.

As a side note, if you’re using a GH200 bare metal, you might also want to checkout my auto-install for GH200s. Getting it setup with optimizations, NCCL, OFED, for high-speed distributed training/inference was a pain, so automated it for people to use or reference: https://github.com/drikster80/gh200-Ubuntu-22.04-autoinstall

Main issues that needed to be overcome:

  • Use Nvidia’s Pytorch container due to PyTorch not supporting ARM64. (specifically nvcr.io/nvidia/pytorch:24.04-py3 to ensure PyTorch 2.3 build and latest optimizations (e.g. Lightning-Thunder). Release Notes for 24.04-py3
  • xformers hangs on pip install. Not sure why (maybe just taking forever to compile?)
  • Triton needs to be installed from source
  • vllm-flash-attn needs to be built from source
  • Comment out “torch”, “xformers”, and “vllm-flash-attn” in requirements files (handling that in the Dockerfile directly).

add

nvcc_cuda_version = get_nvcc_cuda_version(CUDA_HOME) 

to setup.py at line 268

Had a similar problem on the GH200 (aarch64 Grace CPU). Similar to @haileyschoelkopf, I updated the Dockerfile and requirements to work with v0.5.1. Here is the forked version: https://github.com/drikster80/vllm/tree/gh200-docker

Main issues that needed to be overcome:

  • Use Nvidia’s Pytorch container due to PyTorch not supporting ARM64. (specifically nvcr.io/nvidia/pytorch:24.04-py3 to ensure PyTorch 2.3 build and latest optimizations (e.g. Lightning-Thunder). Release Notes for 24.04-py3
  • xformers hangs on pip install. Not sure why (maybe just taking forever to compile?)
  • Triton needs to be installed from source
  • vllm-flash-attn needs to be built from source
  • Comment out “torch”, “xformers”, and “vllm-flash-attn” in requirements files (handling that in the Dockerfile directly).

For future updating, you can see the changes here: https://github.com/drikster80/vllm/commit/359fd4fa3d21b3033acd288de56b25a241a36a3c

Thank you all.

I have built the image using the script provided by @drikster80, and it takes about 12 hours (most of the time is spent for mamba builder and xformers). So to save time for others, I have made the image public at https://hub.docker.com/r/zihaokevinzhou/vllm-aarch64-openai . I have validated it works well for my personal hosting of fp8 quantized version of llama-3-70b.

If anyone with a MGX or GH200 wants to test the latest version with different models, let me know if any supported models fail.

Uploaded the latest (v0.6.4.post1) to dockerhub: https://hub.docker.com/r/drikster80/vllm-gh200-openai/tags docker pull drikster80/vllm-gh200-openai:latest

The new version ( PR #10499 ) is unique that it doesn’t use the Nvidia Pytorch container anymore and matches closer to standard VLLM container. It uses the nightly version of Pytorch (now supports aarch64) and compiles the modules that don’t release a aarch64 wheel (e.g. bitsandbytes, flashinfer, triton, mamba, causal-conv1d, etc.). So it should be considered experimental, but I’ve run some tests on a couple of models and haven’t had any issues.

can you please try out #8713 ? @drikster80 @gongchengli

I spare some time to investigate the issue, and it looks the most complicated part is to bring your own pytorch ( @drikster80 does this by using ngc pytorch container). other than that, it is pretty straight-forward.

on that branch, I can easily build vllm from scratch with nightly pytorch, in a fresh new environment:

$ pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu124
$ git clone https://github.com/vllm-project/vllm.git
$ cd vllm
$ python use_existing_torch.py
$ pip install -r requirements-build.txt
$ pip install -vvv -e . --no-build-isolation

@youkaichao @samos123 I created a new PR for building Docker with the updates. Still some testing to do on it to confirm it doesn’t break anything on x86_64 systems, but confirmed that it’s working on the GH200. Also, haven’t tested with exhaustive models, but it does work with a couple of Llama’s that I tested.

https://github.com/vllm-project/vllm/pull/10499

I am experiencing the same issue as @Jerrrrykun on a GH200 Node with architecture 5.14.0-362.24.1.el9_3.aarch64. I have the same cuda version 12.4. I tried PyTorch wheels from the official release (https://download.pytorch.org/whl/cu124) and nightly (https://download.pytorch.org/whl/nightly/cu124) and then following the code above exactly.

The cmake command fails. I run without parallelism as suggested with -j1 and observe that the build fails on the first step:

[ 16%] Building CXX object CMakeFiles/_moe_C.dir/csrc/moe/torch_bindings.cpp.o
"<conda_environ>/lib/python3.10/site-packages/torch/include/c10/util/Half.h", line 334: error: identifier "float16_t" is undefined
  inline float16_t fp16_from_bits(uint16_t h) {
         ^

"<conda_environ>/lib/python3.10/site-packages/torch/include/c10/util/Half.h", line 335: error: identifier "float16_t" is undefined
    return c10::bit_cast<float16_t>(h);
                         ^

"<conda_environ>/lib/python3.10/site-packages/torch/include/c10/util/Half.h", line 338: error: identifier "float16_t" is undefined
  inline uint16_t fp16_to_bits(float16_t f) {
                               ^

"<conda_environ>/lib/python3.10/site-packages/torch/include/c10/util/Half.h", line 339: error: no instance of function template "c10::bit_cast" matches the argument list
            argument types are: (<error-type>)
    return c10::bit_cast<uint16_t>(f);
...
78 errors detected in the compilation of "<WORK_DIR>/vllm/csrc/moe/torch_bindings.cpp".
gmake[3]: *** [CMakeFiles/_moe_C.dir/build.make:76: CMakeFiles/_moe_C.dir/csrc/moe/torch_bindings.cpp.o] Error 2
gmake[2]: *** [CMakeFiles/Makefile2:205: CMakeFiles/_moe_C.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:212: CMakeFiles/_moe_C.dir/rule] Error 2
gmake: *** [Makefile:202: _moe_C] Error 2

A bunch of errors occurs in Half.h and Half-inl.h, a few in torch/include/torch/csrc/api/include/torch/nn/functional/loss.h and torch/include/c10/util/StringUtil.h. Does this mean we are not somehow installing torch correctly?

I can verify that my Pytorch installation is working:

$ python -c "import torch; print(torch.__version__); print(torch.cuda.is_available()); print(torch.version.cuda)"
2.4.0
True
12.4

@tuanhe Still face same problem, Anyone know vllm support aarch-64 now?