vllm: ImportError: /ramyapra/vllm/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol:
I’m trying to run vllm and lm-eval-harness. I’m using vllm 0.2.5. After I’m done installing both, if I try importing vllm I get the following error:
File "/ramyapra/lm-evaluation-harness/lm_eval/models/__init__.py", line 7, in <module> from . import vllm_causallms File "/ramyapra/lm-evaluation-harness/lm_eval/models/vllm_causallms.py", line 16, in <module> from vllm import LLM, SamplingParams File "/ramyapra/vllm/vllm/__init__.py", line 3, in <module> from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs File "/ramyapra/vllm/vllm/engine/arg_utils.py", line 6, in <module> from vllm.config import (CacheConfig, ModelConfig, ParallelConfig, File "/ramyapra/vllm/vllm/config.py", line 9, in <module> from vllm.utils import get_cpu_memory, is_hip File "/ramyapra/vllm/vllm/utils.py", line 8, in <module> from vllm._C import cuda_utils ImportError: /ramyapra/vllm/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops19empty_memory_format4callEN3c108ArrayRefINS2_6SymIntEEESt8optionalINS2_10ScalarTypeEES6_INS2_6LayoutEES6_INS2_6DeviceEES6_IbES6_INS2_12MemoryFormatEE
I’m using the NGC docker container 23:10-py3.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 4
- Comments: 64 (12 by maintainers)
same issue. anyone able to fix?
cuda: 12.0.1 torch : 2.2.1 transformers: 4.38.2 vllm: 0.3.2 accelerate: 0.22.0
UPDATE: solved by downgrading torch to 2.1.2
for a version that works with pytorch 2.2.1 and python 3.11 do this https://stackoverflow.com/a/78394535/1601580
when will the pytorch 2.2 will be supported ? or how to fix this error ? @youkaichao
@youkaichao since the versions of python I am using are fragile due to the current vllm (or one I use) only works with pytorch 2.1, I was wondering, what hugging face and accelerate version do we need without breaking vllm?
Need to debug this but I think this should work:
and fyi:
For flash attention I have these comments
pt 2.2 is not supported yet.
for torch 2.1.2 use the above.
pip install vllmdirectly, still uses pytorch 2.1 . stay tuned when we make the next release.ok it seems this is sensitive to python version. You have to do 3.9. Then the link by rylan works.
Code:
ref: https://docs.vllm.ai/en/latest/getting_started/installation.html
@youkaichao I followed @RylanSchaeffer advice and I still get an eror:
Can you let us know precisely the commands you recommend to run? I started new conda env and it threw the above error anyway https://docs.vllm.ai/en/latest/getting_started/installation.html
error still:
when build from source, pytorch 2.2 cuda 12.1 :
this is miss the ref :
at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>)so is there somewhere to use this ops?I found a solution!!! I just followed https://docs.vllm.ai/en/latest/getting_started/installation.html
pip install vllmworked right out of the box!!Uninstall the package called transformer-engine by using the command
pip uninstall transformer-engine. Installtransformersandaccelerateif not already installed. Then it should work.