vllm: ImportError: /ramyapra/vllm/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol:

I’m trying to run vllm and lm-eval-harness. I’m using vllm 0.2.5. After I’m done installing both, if I try importing vllm I get the following error: File "/ramyapra/lm-evaluation-harness/lm_eval/models/__init__.py", line 7, in <module> from . import vllm_causallms File "/ramyapra/lm-evaluation-harness/lm_eval/models/vllm_causallms.py", line 16, in <module> from vllm import LLM, SamplingParams File "/ramyapra/vllm/vllm/__init__.py", line 3, in <module> from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs File "/ramyapra/vllm/vllm/engine/arg_utils.py", line 6, in <module> from vllm.config import (CacheConfig, ModelConfig, ParallelConfig, File "/ramyapra/vllm/vllm/config.py", line 9, in <module> from vllm.utils import get_cpu_memory, is_hip File "/ramyapra/vllm/vllm/utils.py", line 8, in <module> from vllm._C import cuda_utils ImportError: /ramyapra/vllm/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops19empty_memory_format4callEN3c108ArrayRefINS2_6SymIntEEESt8optionalINS2_10ScalarTypeEES6_INS2_6LayoutEES6_INS2_6DeviceEES6_IbES6_INS2_12MemoryFormatEE

I’m using the NGC docker container 23:10-py3.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 4
  • Comments: 64 (12 by maintainers)

Most upvoted comments

same issue. anyone able to fix?

cuda: 12.0.1 torch : 2.2.1 transformers: 4.38.2 vllm: 0.3.2 accelerate: 0.22.0

UPDATE: solved by downgrading torch to 2.1.2

@youkaichao sorry for the spam. Where do I follow when the release for vllm + pytorch 2.2.2 will work? A need it for a special machine I’m using sadly as partially documented here

for a version that works with pytorch 2.2.1 and python 3.11 do this https://stackoverflow.com/a/78394535/1601580

conda create -n vllm_test python=3.11 
conda activate vllm_test
pip install torch==2.2.1
pip install vllm==0.4.1
# pip install vllm

pytorch 2.2 cuda 12.1

pt 2.2 is not supported yet.

when will the pytorch 2.2 will be supported ? or how to fix this error ? @youkaichao

    from vllm._C import ops
ImportError: /workspace/vllm/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops15to_dtype_layout4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEEbbNS6_INS5_12MemoryFormatEEE

@youkaichao since the versions of python I am using are fragile due to the current vllm (or one I use) only works with pytorch 2.1, I was wondering, what hugging face and accelerate version do we need without breaking vllm?

Need to debug this but I think this should work:

  # for pytorch see doc string at the top of file
    install_requires=[
        'dill',
        'networkx>=2.5',
        'scipy',
        'scikit-learn',
        'lark-parser',
        'tensorboard',
        'pandas',
        'progressbar2',
        'requests',
        'aiohttp',
        'numpy',
        'plotly',
        'wandb',
        'matplotlib',
        # 'statsmodels'
        # 'statsmodels==0.12.2'
        # 'statsmodels==0.13.5'
        # - later check why we are not installing it...
        # 'seaborn'
        # 'nltk'
        'twine',

        'torch==2.1.2',  # 2.2 not supported due to vllm see: https://github.com/vllm-project/vllm/issues/2747
        # 'torchvision',
        # 'torchaudio',

        # 'fairseq',

        # 'trl',
        'transformers==4.39.2', # my gold-ai-olympiad project uses 4.39.2
        'accelerate==0.29.2',
        # 'peft',

        'datasets==2.18.0',  # 2.18.0
        'bitsandbytes== 0.43.0',
        # 'einops',

        'vllm==0.4.0.post1', # my gold-ai-olympiad project uses 0.4.0.post1 ref: https://github.com/vllm-project/vllm/issues/2747
    ]
)

and fyi:

(snap_cluster_setup) brando9@skampere1~/lean4ai $ pip list
Package                   Version
------------------------- ------------
absl-py                   2.1.0
accelerate                0.29.2
aiohttp                   3.9.3
aiosignal                 1.3.1
annotated-types           0.6.0
anyio                     4.3.0
appdirs                   1.4.4
async-timeout             4.0.3
attrs                     23.2.0
backports.tarfile         1.0.0
bitsandbytes              0.43.0
certifi                   2024.2.2
cffi                      1.16.0
charset-normalizer        3.3.2
click                     8.1.7
cloudpickle               3.0.0
cmake                     3.29.0.1
contourpy                 1.2.1
cryptography              42.0.5
cycler                    0.12.1
datasets                  2.18.0
dill                      0.3.8
diskcache                 5.6.3
docker-pycreds            0.4.0
docutils                  0.20.1
exceptiongroup            1.2.0
fastapi                   0.110.1
filelock                  3.13.3
fonttools                 4.51.0
frozenlist                1.4.1
fsspec                    2024.2.0
gitdb                     4.0.11
GitPython                 3.1.43
grpcio                    1.62.1
h11                       0.14.0
httptools                 0.6.1
huggingface-hub           0.22.2
idna                      3.6
importlib_metadata        7.1.0
importlib_resources       6.4.0
interegular               0.3.3
jaraco.classes            3.4.0
jaraco.context            5.3.0
jaraco.functools          4.0.0
jeepney                   0.8.0
Jinja2                    3.1.3
joblib                    1.4.0
jsonschema                4.21.1
jsonschema-specifications 2023.12.1
keyring                   25.1.0
kiwisolver                1.4.5
lark                      1.1.9
lark-parser               0.12.0
llvmlite                  0.42.0
Markdown                  3.6
markdown-it-py            3.0.0
MarkupSafe                2.1.5
matplotlib                3.8.4
mdurl                     0.1.2
more-itertools            10.2.0
mpmath                    1.3.0
msgpack                   1.0.8
multidict                 6.0.5
multiprocess              0.70.16
nest-asyncio              1.6.0
networkx                  3.2.1
nh3                       0.2.17
ninja                     1.11.1.1
numba                     0.59.1
numpy                     1.26.4
nvidia-cublas-cu12        12.1.3.1
nvidia-cuda-cupti-cu12    12.1.105
nvidia-cuda-nvrtc-cu12    12.1.105
nvidia-cuda-runtime-cu12  12.1.105
nvidia-cudnn-cu12         8.9.2.26
nvidia-cufft-cu12         11.0.2.54
nvidia-curand-cu12        10.3.2.106
nvidia-cusolver-cu12      11.4.5.107
nvidia-cusparse-cu12      12.1.0.106
nvidia-nccl-cu12          2.18.1
nvidia-nvjitlink-cu12     12.4.127
nvidia-nvtx-cu12          12.1.105
outlines                  0.0.34
packaging                 24.0
pandas                    2.2.1
pillow                    10.3.0
pip                       24.0
pkginfo                   1.10.0
plotly                    5.20.0
progressbar2              4.4.2
prometheus_client         0.20.0
protobuf                  4.25.3
psutil                    5.9.8
py-cpuinfo                9.0.0
pyarrow                   15.0.2
pyarrow-hotfix            0.6
pycparser                 2.22
pydantic                  2.6.4
pydantic_core             2.16.3
Pygments                  2.17.2
pynvml                    11.5.0
pyparsing                 3.1.2
python-dateutil           2.9.0.post0
python-dotenv             1.0.1
python-utils              3.8.2
pytz                      2024.1
PyYAML                    6.0.1
ray                       2.10.0
readme_renderer           43.0
referencing               0.34.0
regex                     2023.12.25
requests                  2.31.0
requests-toolbelt         1.0.0
rfc3986                   2.0.0
rich                      13.7.1
rpds-py                   0.18.0
safetensors               0.4.2
scikit-learn              1.4.1.post1
scipy                     1.13.0
SecretStorage             3.3.3
sentencepiece             0.2.0
sentry-sdk                1.44.1
setproctitle              1.3.3
setuptools                68.2.2
six                       1.16.0
smmap                     5.0.1
sniffio                   1.3.1
starlette                 0.37.2
sympy                     1.12
tenacity                  8.2.3
tensorboard               2.16.2
tensorboard-data-server   0.7.2
threadpoolctl             3.4.0
tiktoken                  0.6.0
tokenizers                0.15.2
torch                     2.1.2
tqdm                      4.66.2
transformers              4.39.2
triton                    2.1.0
twine                     5.0.0
typing_extensions         4.11.0
tzdata                    2024.1
urllib3                   2.2.1
uvicorn                   0.29.0
uvloop                    0.19.0
vllm                      0.4.0.post1
wandb                     0.16.6
watchfiles                0.21.0
websockets                12.0
Werkzeug                  3.0.2
wheel                     0.41.2
xformers                  0.0.23.post1
xxhash                    3.4.1
yarl                      1.9.4
zipp                      3.18.1

For flash attention I have these comments

       # # ampere
        # 'dspy-ai',
        # # 'torch==2.1.2+cu118',  # 2.2 not supported due to vllm see: https://github.com/vllm-project/vllm/issues/2747
        # # 'torch==2.1.2',  # 2.2 not supported due to vllm see: https://github.com/vllm-project/vllm/issues/2747
        # # 'torch==2.2.1',  # 2.2 not supported due to vllm see: https://github.com/vllm-project/vllm/issues/2747
        # 'torch==2.2.1',  # 2.2 not supported due to vllm see: https://github.com/vllm-project/vllm/issues/2747
        # # 'torchvision',
        # # 'torchaudio',
        # # 'trl',
        # # 'transformers==4.39.2',
        # 'transformers>=4.40',
        # 'accelerate==0.29.2',
        # # 'peft',
        # # 'datasets==2.18.0', 
        # 'datasets==2.14.7',  
        # 'evaluate==0.4.1', 
        # 'bitsandbytes== 0.43.0',
        # 'einops',
        # 'flash-attn>=2.5.8',
        # 'vllm==0.4.1', # my gold-ai-olympiad project uses 0.4.0.post1 ref: https://github.com/vllm-project/vllm/issues/2747
        # # pip install -q -U google-generativeai

pytorch 2.2 cuda 12.1

pt 2.2 is not supported yet.

for a version that works with pytorch 2.2.1 and python 3.11 do this https://stackoverflow.com/a/78394535/1601580

conda create -n vllm_test python=3.11 
conda activate vllm_test
pip install torch==2.2.1
pip install vllm==0.4.1
# pip install vllm

for torch 2.1.2 use the above.

pip install vllm directly, still uses pytorch 2.1 . stay tuned when we make the next release.

@youkaichao I followed @RylanSchaeffer advice and I still get an eror:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
outlines 0.0.37 requires pydantic>=2.0, but you have pydantic 1.10.13 which is incompatible.
Successfully installed GitPython-3.1.42 Pygments-2.17.2 SecretStorage-3.3.3 absl-py-2.1.0 accelerate-0.28.0 aiohttp-3.9.3 aioprometheus-23.12.0 appdirs-1.4.4 async-timeout-4.0.3 bitsandbytes-0.43.0 cffi-1.16.0 contourpy-1.2.0 cryptography-42.0.5 cycler-0.12.1 datasets-2.18.0 dill-0.3.8 docker-pycreds-0.4.0 docstring-parser-0.16 docutils-0.20.1 einops-0.7.0 fonttools-4.50.0 fsspec-2024.2.0 gitdb-4.0.11 gold-ai-olmpiad-0.1.0 grpcio-1.62.1 importlib-metadata-7.1.0 jaraco.classes-3.3.1 jaraco.context-4.3.0 jaraco.functools-4.0.0 jeepney-0.8.0 keyring-25.0.0 kiwisolver-1.4.5 lark-parser-0.12.0 markdown-3.6 markdown-it-py-3.0.0 matplotlib-3.8.3 mdurl-0.1.2 more-itertools-10.2.0 multidict-6.0.5 multiprocess-0.70.16 nh3-0.2.17 nvidia-nccl-cu12-2.19.3 orjson-3.10.0 pandas-2.2.1 peft-0.10.0 pillow-10.2.0 pkginfo-1.10.0 plotly-5.20.0 progressbar2-4.4.2 protobuf-4.25.3 pyarrow-15.0.2 pyarrow-hotfix-0.6 pycparser-2.21 pydantic-1.10.13 pyparsing-3.1.2 python-dateutil-2.9.0.post0 python-utils-3.8.2 pytz-2024.1 quantile-python-1.1 readme-renderer-43.0 requests-toolbelt-1.0.0 rfc3986-2.0.0 rich-13.7.1 scikit-learn-1.4.1.post1 sentry-sdk-1.44.0 setproctitle-1.3.3 shtab-1.7.1 six-1.16.0 smmap-5.0.1 tenacity-8.2.3 tensorboard-2.16.2 tensorboard-data-server-0.7.2 threadpoolctl-3.4.0 torch-2.2.2 torchaudio-2.2.2 torchvision-0.17.2 triton-2.2.0 trl-0.8.1 twine-5.0.0 tyro-0.7.3 tzdata-2024.1 vllm-0.2.5 wandb-0.16.5 werkzeug-3.0.1 xformers-0.0.25.post1 xxhash-3.4.1 yarl-1.9.4 zipp-3.18.1

Can you let us know precisely the commands you recommend to run? I started new conda env and it threw the above error anyway https://docs.vllm.ai/en/latest/getting_started/installation.html

error still:

_ZN2at4_ops15to_dtype_layout4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEEbbNS6_INS5_12MemoryFormatEEE

ok it seems this is sensitive to python version. You have to do 3.9. Then the link by rylan works.

Code:

conda create -n myenv python=3.9 -y
conda activate myenv
# needed in this order for vllm to work
pip install vllm 
# optional
pip install -e ~/your_library/setup.py

ref: https://docs.vllm.ai/en/latest/getting_started/installation.html

@youkaichao I followed @RylanSchaeffer advice and I still get an eror:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
outlines 0.0.37 requires pydantic>=2.0, but you have pydantic 1.10.13 which is incompatible.
Successfully installed GitPython-3.1.42 Pygments-2.17.2 SecretStorage-3.3.3 absl-py-2.1.0 accelerate-0.28.0 aiohttp-3.9.3 aioprometheus-23.12.0 appdirs-1.4.4 async-timeout-4.0.3 bitsandbytes-0.43.0 cffi-1.16.0 contourpy-1.2.0 cryptography-42.0.5 cycler-0.12.1 datasets-2.18.0 dill-0.3.8 docker-pycreds-0.4.0 docstring-parser-0.16 docutils-0.20.1 einops-0.7.0 fonttools-4.50.0 fsspec-2024.2.0 gitdb-4.0.11 gold-ai-olmpiad-0.1.0 grpcio-1.62.1 importlib-metadata-7.1.0 jaraco.classes-3.3.1 jaraco.context-4.3.0 jaraco.functools-4.0.0 jeepney-0.8.0 keyring-25.0.0 kiwisolver-1.4.5 lark-parser-0.12.0 markdown-3.6 markdown-it-py-3.0.0 matplotlib-3.8.3 mdurl-0.1.2 more-itertools-10.2.0 multidict-6.0.5 multiprocess-0.70.16 nh3-0.2.17 nvidia-nccl-cu12-2.19.3 orjson-3.10.0 pandas-2.2.1 peft-0.10.0 pillow-10.2.0 pkginfo-1.10.0 plotly-5.20.0 progressbar2-4.4.2 protobuf-4.25.3 pyarrow-15.0.2 pyarrow-hotfix-0.6 pycparser-2.21 pydantic-1.10.13 pyparsing-3.1.2 python-dateutil-2.9.0.post0 python-utils-3.8.2 pytz-2024.1 quantile-python-1.1 readme-renderer-43.0 requests-toolbelt-1.0.0 rfc3986-2.0.0 rich-13.7.1 scikit-learn-1.4.1.post1 sentry-sdk-1.44.0 setproctitle-1.3.3 shtab-1.7.1 six-1.16.0 smmap-5.0.1 tenacity-8.2.3 tensorboard-2.16.2 tensorboard-data-server-0.7.2 threadpoolctl-3.4.0 torch-2.2.2 torchaudio-2.2.2 torchvision-0.17.2 triton-2.2.0 trl-0.8.1 twine-5.0.0 tyro-0.7.3 tzdata-2024.1 vllm-0.2.5 wandb-0.16.5 werkzeug-3.0.1 xformers-0.0.25.post1 xxhash-3.4.1 yarl-1.9.4 zipp-3.18.1

Can you let us know precisely the commands you recommend to run? I started new conda env and it threw the above error anyway https://docs.vllm.ai/en/latest/getting_started/installation.html

error still:

_ZN2at4_ops15to_dtype_layout4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEEbbNS6_INS5_12MemoryFormatEEE

when build from source, pytorch 2.2 cuda 12.1 :

    from vllm._C import ops
ImportError: /workspace/vllm/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops15to_dtype_layout4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEEbbNS6_INS5_12MemoryFormatEEE

this is miss the ref : at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) so is there somewhere to use this ops?

I found a solution!!! I just followed https://docs.vllm.ai/en/latest/getting_started/installation.html

pip install vllm worked right out of the box!!

Uninstall the package called transformer-engine by using the command pip uninstall transformer-engine. Install transformers and accelerate if not already installed. Then it should work.