vllm: ARM aarch-64 server build failed (host OS: Ubuntu22.04.3)
do as: https://docs.vllm.ai/en/latest/getting_started/installation.html
- docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
- git clone https://github.com/vllm-project/vllm.git
- cd vllm
- pip install -e .
here is the details in side the docker instance: root@f8c2e06fbf8b:/mnt/vllm# pip install -e . Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com Obtaining file:///mnt/vllm Installing build dependencies … done Checking if build backend supports build_editable … done Getting requirements to build editable … error error: subprocess-exited-with-error
× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> [22 lines of output]
/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: No module named ‘numpy’ (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:84.)
device: torch.device = torch.device(torch._C._get_default_device()), # torch.device(‘cpu’),
No CUDA runtime is found, using CUDA_HOME=‘/usr/local/cuda’
<string>:142: UserWarning: Unsupported CUDA/ROCM architectures ({‘6.1’, ‘7.2’, ‘8.7’, ‘5.2’, ‘6.0’}) are excluded from the TORCH_CUDA_ARCH_LIST env variable (5.2 6.0 6.1 7.0 7.2 7.5 8.0 8.6 8.7 9.0+PTX). Supported CUDA/ROCM architectures are: {‘7.5’, ‘8.0’, ‘9.0’, ‘7.0’, ‘8.6+PTX’, ‘9.0+PTX’, ‘8.6’, ‘8.0+PTX’, ‘8.9+PTX’, ‘8.9’, ‘7.0+PTX’, ‘7.5+PTX’}.
Traceback (most recent call last):
File “/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py”, line 353, in <module>
main()
File “/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py”, line 335, in main
json_out[‘return_val’] = hook(**hook_input[‘kwargs’])
File “/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py”, line 132, in get_requires_for_build_editable
return hook(config_settings)
File “/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py”, line 441, in get_requires_for_build_editable
return self.get_requires_for_build_wheel(config_settings)
File “/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py”, line 325, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[‘wheel’])
File “/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py”, line 295, in _get_build_requires
self.run_setup()
File “/tmp/pip-build-env-4xoxai9j/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py”, line 311, in run_setup
exec(code, locals())
File “<string>”, line 297, in <module>
File “<string>”, line 267, in get_vllm_version
NameError: name ‘nvcc_cuda_version’ is not defined. Did you mean: ‘cuda_version’?
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error
× Getting requirements to build editable did not run successfully. │ exit code: 1 ╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
[notice] A new release of pip is available: 23.2.1 -> 23.3.1 [notice] To update, run: python -m pip install --upgrade pip
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 4
- Comments: 54 (21 by maintainers)
If anyone comes across this and is trying to get Llama-3.1 to work with the GH200 (or aarch64 + H100), I have the latest working container (v0.5.3-post1 with a couple more commits) image up at https://hub.docker.com/r/drikster80/vllm-gh200-openai Pull it with
docker pull drikster80/vllm-gh200-openai:latestCodes is still in the https://github.com/drikster80/vllm/tree/gh200-docker branch.
Validated Llama-3.1-8b-Instruct works, and trying to working to test 405B-FP8 now (with cpu-offload)
Got it working with the changes in this branch: https://github.com/haileyschoelkopf/vllm/tree/aarm64-dockerfile , with built dockerfiles here: https://hub.docker.com/r/haileysch/vllm-aarch64-base https://hub.docker.com/r/haileysch/vllm-aarch64-openai hopefully this’ll be helpful to others!
Updated the aarch64 remote branch to v0.5.2: https://github.com/drikster80/vllm/tree/gh200-docker
Pushed up a GH200 specific version (built for SM 9.0+PTX) to https://hub.docker.com/r/drikster80/vllm-gh200-openai
Building a more generic version now and will update this comment when complete.
@cyc00518 You can see the list of full changes here: https://github.com/vllm-project/vllm/compare/main...drikster80:vllm:gh200-docker
Effectively,
xformersandvllm-flash-attentiondon’t release ARM64 wheels, so those need to be built from source. Also, since Nvidia’s PyTorch container already containers torch, torchvision, and some other stuff, those need to be commented out in the requirements file. The only 3 files that are changes areDockerfile,requirements-build.txt, andrequirements-cuda.txt.As a side note, if you’re using a GH200 bare metal, you might also want to checkout my auto-install for GH200s. Getting it setup with optimizations, NCCL, OFED, for high-speed distributed training/inference was a pain, so automated it for people to use or reference: https://github.com/drikster80/gh200-Ubuntu-22.04-autoinstall
add
to setup.py at line 268
Had a similar problem on the GH200 (aarch64 Grace CPU). Similar to @haileyschoelkopf, I updated the Dockerfile and requirements to work with v0.5.1. Here is the forked version: https://github.com/drikster80/vllm/tree/gh200-docker
Main issues that needed to be overcome:
For future updating, you can see the changes here: https://github.com/drikster80/vllm/commit/359fd4fa3d21b3033acd288de56b25a241a36a3c
Thank you all.
I have built the image using the script provided by @drikster80, and it takes about 12 hours (most of the time is spent for mamba builder and xformers). So to save time for others, I have made the image public at https://hub.docker.com/r/zihaokevinzhou/vllm-aarch64-openai . I have validated it works well for my personal hosting of fp8 quantized version of llama-3-70b.
If anyone with a MGX or GH200 wants to test the latest version with different models, let me know if any supported models fail.
Uploaded the latest (v0.6.4.post1) to dockerhub: https://hub.docker.com/r/drikster80/vllm-gh200-openai/tags
docker pull drikster80/vllm-gh200-openai:latestThe new version ( PR #10499 ) is unique that it doesn’t use the Nvidia Pytorch container anymore and matches closer to standard VLLM container. It uses the nightly version of Pytorch (now supports aarch64) and compiles the modules that don’t release a aarch64 wheel (e.g. bitsandbytes, flashinfer, triton, mamba, causal-conv1d, etc.). So it should be considered experimental, but I’ve run some tests on a couple of models and haven’t had any issues.
@youkaichao @samos123 I created a new PR for building Docker with the updates. Still some testing to do on it to confirm it doesn’t break anything on x86_64 systems, but confirmed that it’s working on the GH200. Also, haven’t tested with exhaustive models, but it does work with a couple of Llama’s that I tested.
https://github.com/vllm-project/vllm/pull/10499
I am experiencing the same issue as @Jerrrrykun on a GH200 Node with architecture
5.14.0-362.24.1.el9_3.aarch64. I have the same cuda version 12.4. I tried PyTorch wheels from the official release (https://download.pytorch.org/whl/cu124) and nightly (https://download.pytorch.org/whl/nightly/cu124) and then following the code above exactly.The
cmakecommand fails. I run without parallelism as suggested with-j1and observe that the build fails on the first step:A bunch of errors occurs in
Half.handHalf-inl.h, a few intorch/include/torch/csrc/api/include/torch/nn/functional/loss.handtorch/include/c10/util/StringUtil.h. Does this mean we are not somehow installing torch correctly?I can verify that my Pytorch installation is working:
@tuanhe Still face same problem, Anyone know vllm support aarch-64 now?