vllm: [Bug]: ImportError: /workspace/vllm-abo/vllm/_C.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb

Your current environment

The output of `pip show torch torchvision `
Name: torch
Version: 2.5.1+gita8d6afb
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3-Clause
Location: /usr/local/lib/python3.12/dist-packages
Requires: filelock, fsspec, jinja2, networkx, setuptools, sympy, typing-extensions
Required-by: accelerate, compressed-tensors, flash-attn, outlines, torchaudio, torchvision, vllm, xformers, xgrammar
---
Name: torchvision
Version: 0.20.1+git3ac97aa
Summary: image and video datasets and models for torch deep learning
Home-page: https://github.com/pytorch/vision
Author: PyTorch Core Team
Author-email: soumith@pytorch.org
License: BSD
Location: /usr/local/lib/python3.12/dist-packages/torchvision-0.20.1-py3.12-linux-x86_64.egg
Requires: numpy, pillow, torch
Required-by: vllm
Name: vllm
Version: 0.6.5.dev398+gff06ff57.cu124
import torch
print(torch._C._GLIBCXX_USE_CXX11_ABI)
TRUE
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

🐛 Describe the bug

In my environment, there are some third-party packages that require compiling PyTorch from source, and during the compilation of PyTorch, the compilation option _GLIBCXX_USE_CXX11_ABI=1 is used. Correspondingly, I also need to compile the matching version of torchvision from source.

When compiling PyTorch and torchvision, I switched the branches to 2.5.1 and 0.20.1 respectively to meet the version requirements specified in requirements-cuda.txt of vllm.

However, after compiling vllm and attempting to import it, I encountered the following error. Could you please help me understand the reason for this issue?

import vllm
Traceback (most recent call last):  File "<string>", line 1, in <module>
  File "/workspace/vllm/vllm/__init__.py", line 3, in <module>
    from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
  File "/workspace/vllm/vllm/engine/arg_utils.py", line 11, in <module>
    from vllm.config import (CacheConfig, CompilationConfig, ConfigFormat,
  File "/workspace/vllm/vllm/config.py", line 21, in <module>
    from vllm.model_executor.layers.quantization import (QUANTIZATION_METHODS,
  File "/workspace/vllm/vllm/model_executor/__init__.py", line 1, in <module>
    from vllm.model_executor.parameter import (BasevLLMParameter,
  File "/workspace/vllm/vllm/model_executor/parameter.py", line 7, in <module>
    from vllm.distributed import get_tensor_model_parallel_rank
  File "/workspace/vllm/vllm/distributed/__init__.py", line 1, in <module>
    from .communication_op import *
  File "/workspace/vllm/vllm/distributed/communication_op.py", line 6, in <module>
    from .parallel_state import get_tp_group
  File "/workspace/vllm/vllm/distributed/parallel_state.py", line 39, in <module>
    import vllm.distributed.kv_transfer.kv_transfer_agent as kv_transfer
  File "/workspace/vllm/vllm/distributed/kv_transfer/kv_transfer_agent.py", line 15, in <module>
    from vllm.distributed.kv_transfer.kv_connector.factory import (
  File "/workspace/vllm/vllm/distributed/kv_transfer/kv_connector/factory.py", line 3, in <module>
    from .base import KVConnectorBase
  File "/workspace/vllm/vllm/distributed/kv_transfer/kv_connector/base.py", line 14, in <module>
    from vllm.sequence import IntermediateTensors
  File "/workspace/vllm/vllm/sequence.py", line 16, in <module>
    from vllm.inputs import SingletonInputs, SingletonInputsAdapter
  File "/workspace/vllm/vllm/inputs/__init__.py", line 7, in <module>
    from .registry import (DummyData, InputContext, InputProcessingContext,
  File "/workspace/vllm/vllm/inputs/registry.py", line 13, in <module>
    from vllm.transformers_utils.tokenizer import AnyTokenizer
  File "/workspace/vllm/vllm/transformers_utils/tokenizer.py", line 16, in <module>
    from vllm.utils import make_async
  File "/workspace/vllm/vllm/utils.py", line 44, in <module>
    from vllm.platforms import current_platform
  File "/workspace/vllm/vllm/platforms/__init__.py", line 107, in <module>
    from .cuda import CudaPlatform
  File "/workspace/vllm/vllm/platforms/cuda.py", line 15, in <module>
    import vllm._C  # noqa
    ^^^^^^^^^^^^^^
ImportError: /workspace/vllm/vllm/_C.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 6
  • Comments: 43 (3 by maintainers)

Most upvoted comments

same A100 torch2.6+cu12.6

well I met the same error And I found my torch is compiled with cuda 12.6

After I changed to torch with cuda 12.4 the error disappeared.

Folks, downgrading to torch==2.6.0+cu124 resolved the issue for me.

same torch 2.6 + cu 12.8

torch==2.6.0+cu126 im having this im getting the same error, downgrading didnt worked for me ,any more solutions?

try pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124 instead of pip3 install torch torchvision torchaudio

I had the same problem on a GH200 cluster, I think it is caused by a mismatch of torch versions used by some libraries. during their installation. To avoid this, try to make whatever library you are installing to use the existing torch version already installed (e.g., with FLASH_ATTENTION_SKIP_CUDA_BUILD=1, --no-build-isolation). For example, the following is how I was able to finally run VLLM without this issue:

pip3 install packaging
pip3 install wheel

# install torch if you haven't yet
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
pip3 install accelerate bitsandbytes datasets evaluate huggingface-hub peft tokenizers transformers trl
pip3 install ray[default,client]

pip3 install triton
pip3 install nvidia-cutlass

# flash attention, using existing torch
NVCC_APPEND_FLAGS='-allow-unsupported-compiler' MAX_JOBS=4 FLASH_ATTENTION_SKIP_CUDA_BUILD=1 pip3 install flash-attn --no-build-isolation --use-pep517

# VLLM, using existing torch
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout ed6e9075d31e32c8548b480a47d1ffb77da1f54c
python3 use_existing_torch.py
pip3 install -r requirements-build.txt
pip3 install --editable . --no-build-isolation
cd ../

# I had some isssues with these libraries, maybe you can skip right away to xformers installation
pip3 uninstall -y $(pip3 list --format=freeze | grep opencv)
pip3 install opencv-python-headless
pip3 uninstall -y pynvml
pip3 install nvidia-ml-py
pip3 install compressed-tensors

# xformers, using existing torch
git clone https://github.com/facebookresearch/xformers.git
cd xformers
git submodule update --init --recursive
MAX_JOBS=4 pip3 install --use-pep517 . --no-build-isolation
cd../

torch==2.6.0+cu126 im having this im getting the same error, downgrading didnt worked for me ,any more solutions?

Same bug, using torch 2.6 with cuda 12.6

While I cannot explain the issue, I was able to work around it by removing all instances of file _C.abi3.so from my virtual environment, building and installing vllm’s _C, and then copying my newly built _C.abi3.s0 file to my virtual env’s lib/python3.12/site-packages/vllm directory

In my working directory vllm/build:

cmake --build . --target _C
cmake --install . --component C
cp _C.abi.so _/path/to/my/virtual/environment_**/lib/python3.12/site-packages/vllm**

And now my vllm CLI seems to work. It responds without erroring, telling me about usage (chat,complete,serve). vllm -v responds with 0.7.3.

and possibly cmake --install might take a parameter for where to install whatever’s being installed? and you wouldn’t need to manually copy the file like i did? i dunno. it’s been a LONG time since i did any of this! But 11 days later, it’s looking like I can finally use my RTX 5070 ti.

Note to self: Never again run out and buy a GPU on release day for any reason other than video games to be played on Windows computers. lol

同志们,我遇到了和你们相同的报错,希望能给你们提供一下思路:

我去到vllm的docker hub镜像下载了0.8.5.post1的镜像:https://hub.docker.com/r/vllm/vllm-openai

docker中的环境如下: cuda12.4 python 3.12.10 torch 2.6.0 torchaudio 2.6.0 torchvision 0.21.0 vllm 0.8.5.post1

可以正常找到那个二进制文件,并且调用cuda: root@50c4091a5214:/vllm-workspace# nm -u /usr/local/lib/python3.12/dist-packages/vllm/_C.abi3.so | grep _ZN5torch3jit17parseSchemaOrNameERKSsb U _ZN5torch3jit17parseSchemaOrNameERKSsb root@50c4091a5214:/vllm-workspace# # 检查 libtorch.so nm -D /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch.so | grep _ZN5torch3jit17parseSchemaOrNameERKSsb

检查 libtorch_cpu.so

nm -D /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so | grep _ZN5torch3jit17parseSchemaOrNameERKSsb

检查 libtorch_cuda.so

nm -D /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so | grep _ZN5torch3jit17parseSchemaOrNameERKSsb 0000000001b68c30 T _ZN5torch3jit17parseSchemaOrNameERKSsb root@50c4091a5214:/vllm-workspace# python3 -c “import torch; print(torch.cuda.is_available()); print(torch.cuda.device_count()); print(torch.cuda.current_device()); print(torch.cuda.get_device_name(0))” True 1 0 NVIDIA TITAN RTX

可以参考这个配置来配置自己的环境,或者和我一样从docker hub中直接拉取成熟的镜像

我很期待有资深工程师老哥解决了这个问题,也会持续关注这个帖子

Looks like some incompatibility between torch and vllm, which is weird because When upgrading my torch, it says vllm 0.8.5.post1 requires torch==2.6.0, but you have torch 2.7.0+cu128 which is incompatible. But in the requirements.txt file, it says torch==2.7.0. It is a very recent change though.

So are we expecting to get a new vllm update soon?

I guess the problem is that vLLM make use of some compiled library from its dependency that doesn’t work with the latest pytorch. What I did to resolve this

  1. install latest pytorch
  2. download and compile from source these libraries: xformers, flashinfer, flash-attention
  3. recompile vLLM with full build
 pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128
 pip install nvidia-cutlass


     git clone https://github.com/Dao-AILab/flash-attention.git
     cd flash-attention
     python setup.py install

     git clone https://github.com/flashinfer-ai/flashinfer.git --recursive
     cd flashinfer
     FLASHINFER_ENABLE_AOT=1 pip install -e . -v --no-deps --no-build-isolati. .  

     git clone https://github.com/facebookresearch/xformers.git
     cd xformers
     git submodule update --init --recursive
     MAX_JOBS=4 pip3 install --use-pep517 . --no-build-isolation

cd vllm
python3 use_existing_torch.py
pip3 install -r requirements/build.txt
pip3 install --editable . --no-build-isolation

However, for me, I have performance issues with pytorch 2.8 and triton, so I switch to using Pytorch 2.7 with cuda 12.6, and everything is normal now.

Folks, downgrading to torch==2.6.0+cu124 resolved the issue for me.

Thanks, I previously had torch==2.6.0+cu126 and downgrading to this version solve the issue.

same A6000 torch2.6+cu12.6 L20 torch2.6+cu12.6

same h200 torch2.6 cu12.8/12.6

same torch 2.5.1 + cu 12.8