vllm: [Bug]: ImportError: /workspace/vllm-abo/vllm/_C.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb
Your current environment
The output of `pip show torch torchvision `
Name: torch
Version: 2.5.1+gita8d6afb
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3-Clause
Location: /usr/local/lib/python3.12/dist-packages
Requires: filelock, fsspec, jinja2, networkx, setuptools, sympy, typing-extensions
Required-by: accelerate, compressed-tensors, flash-attn, outlines, torchaudio, torchvision, vllm, xformers, xgrammar
---
Name: torchvision
Version: 0.20.1+git3ac97aa
Summary: image and video datasets and models for torch deep learning
Home-page: https://github.com/pytorch/vision
Author: PyTorch Core Team
Author-email: soumith@pytorch.org
License: BSD
Location: /usr/local/lib/python3.12/dist-packages/torchvision-0.20.1-py3.12-linux-x86_64.egg
Requires: numpy, pillow, torch
Required-by: vllm
Name: vllm
Version: 0.6.5.dev398+gff06ff57.cu124
import torch
print(torch._C._GLIBCXX_USE_CXX11_ABI)
TRUE
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0
🐛 Describe the bug
In my environment, there are some third-party packages that require compiling PyTorch from source, and during the compilation of PyTorch, the compilation option _GLIBCXX_USE_CXX11_ABI=1 is used. Correspondingly, I also need to compile the matching version of torchvision from source.
When compiling PyTorch and torchvision, I switched the branches to 2.5.1 and 0.20.1 respectively to meet the version requirements specified in requirements-cuda.txt of vllm.
However, after compiling vllm and attempting to import it, I encountered the following error. Could you please help me understand the reason for this issue?
import vllm
Traceback (most recent call last): File "<string>", line 1, in <module>
File "/workspace/vllm/vllm/__init__.py", line 3, in <module>
from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
File "/workspace/vllm/vllm/engine/arg_utils.py", line 11, in <module>
from vllm.config import (CacheConfig, CompilationConfig, ConfigFormat,
File "/workspace/vllm/vllm/config.py", line 21, in <module>
from vllm.model_executor.layers.quantization import (QUANTIZATION_METHODS,
File "/workspace/vllm/vllm/model_executor/__init__.py", line 1, in <module>
from vllm.model_executor.parameter import (BasevLLMParameter,
File "/workspace/vllm/vllm/model_executor/parameter.py", line 7, in <module>
from vllm.distributed import get_tensor_model_parallel_rank
File "/workspace/vllm/vllm/distributed/__init__.py", line 1, in <module>
from .communication_op import *
File "/workspace/vllm/vllm/distributed/communication_op.py", line 6, in <module>
from .parallel_state import get_tp_group
File "/workspace/vllm/vllm/distributed/parallel_state.py", line 39, in <module>
import vllm.distributed.kv_transfer.kv_transfer_agent as kv_transfer
File "/workspace/vllm/vllm/distributed/kv_transfer/kv_transfer_agent.py", line 15, in <module>
from vllm.distributed.kv_transfer.kv_connector.factory import (
File "/workspace/vllm/vllm/distributed/kv_transfer/kv_connector/factory.py", line 3, in <module>
from .base import KVConnectorBase
File "/workspace/vllm/vllm/distributed/kv_transfer/kv_connector/base.py", line 14, in <module>
from vllm.sequence import IntermediateTensors
File "/workspace/vllm/vllm/sequence.py", line 16, in <module>
from vllm.inputs import SingletonInputs, SingletonInputsAdapter
File "/workspace/vllm/vllm/inputs/__init__.py", line 7, in <module>
from .registry import (DummyData, InputContext, InputProcessingContext,
File "/workspace/vllm/vllm/inputs/registry.py", line 13, in <module>
from vllm.transformers_utils.tokenizer import AnyTokenizer
File "/workspace/vllm/vllm/transformers_utils/tokenizer.py", line 16, in <module>
from vllm.utils import make_async
File "/workspace/vllm/vllm/utils.py", line 44, in <module>
from vllm.platforms import current_platform
File "/workspace/vllm/vllm/platforms/__init__.py", line 107, in <module>
from .cuda import CudaPlatform
File "/workspace/vllm/vllm/platforms/cuda.py", line 15, in <module>
import vllm._C # noqa
^^^^^^^^^^^^^^
ImportError: /workspace/vllm/vllm/_C.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 6
- Comments: 43 (3 by maintainers)
same A100 torch2.6+cu12.6
well I met the same error And I found my torch is compiled with cuda 12.6
After I changed to torch with cuda 12.4 the error disappeared.
Folks, downgrading to
torch==2.6.0+cu124resolved the issue for me.same torch 2.6 + cu 12.8
try
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124instead ofpip3 install torch torchvision torchaudioI had the same problem on a GH200 cluster, I think it is caused by a mismatch of torch versions used by some libraries. during their installation. To avoid this, try to make whatever library you are installing to use the existing torch version already installed (e.g., with
FLASH_ATTENTION_SKIP_CUDA_BUILD=1,--no-build-isolation). For example, the following is how I was able to finally run VLLM without this issue:torch==2.6.0+cu126im having this im getting the same error, downgrading didnt worked for me ,any more solutions?Same bug, using torch 2.6 with cuda 12.6
While I cannot explain the issue, I was able to work around it by removing all instances of file _C.abi3.so from my virtual environment, building and installing vllm’s _C, and then copying my newly built _C.abi3.s0 file to my virtual env’s lib/python3.12/site-packages/vllm directory
In my working directory vllm/build:
And now my vllm CLI seems to work. It responds without erroring, telling me about usage (chat,complete,serve). vllm -v responds with 0.7.3.
and possibly cmake --install might take a parameter for where to install whatever’s being installed? and you wouldn’t need to manually copy the file like i did? i dunno. it’s been a LONG time since i did any of this! But 11 days later, it’s looking like I can finally use my RTX 5070 ti.
Note to self: Never again run out and buy a GPU on release day for any reason other than video games to be played on Windows computers. lol
同志们,我遇到了和你们相同的报错,希望能给你们提供一下思路:
我去到vllm的docker hub镜像下载了0.8.5.post1的镜像:https://hub.docker.com/r/vllm/vllm-openai
docker中的环境如下: cuda12.4 python 3.12.10 torch 2.6.0 torchaudio 2.6.0 torchvision 0.21.0 vllm 0.8.5.post1
可以正常找到那个二进制文件,并且调用cuda: root@50c4091a5214:/vllm-workspace# nm -u /usr/local/lib/python3.12/dist-packages/vllm/_C.abi3.so | grep _ZN5torch3jit17parseSchemaOrNameERKSsb U _ZN5torch3jit17parseSchemaOrNameERKSsb root@50c4091a5214:/vllm-workspace# # 检查 libtorch.so nm -D /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch.so | grep _ZN5torch3jit17parseSchemaOrNameERKSsb
检查 libtorch_cpu.so
nm -D /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so | grep _ZN5torch3jit17parseSchemaOrNameERKSsb
检查 libtorch_cuda.so
nm -D /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so | grep _ZN5torch3jit17parseSchemaOrNameERKSsb 0000000001b68c30 T _ZN5torch3jit17parseSchemaOrNameERKSsb root@50c4091a5214:/vllm-workspace# python3 -c “import torch; print(torch.cuda.is_available()); print(torch.cuda.device_count()); print(torch.cuda.current_device()); print(torch.cuda.get_device_name(0))” True 1 0 NVIDIA TITAN RTX
可以参考这个配置来配置自己的环境,或者和我一样从docker hub中直接拉取成熟的镜像
我很期待有资深工程师老哥解决了这个问题,也会持续关注这个帖子
Looks like some incompatibility between torch and vllm, which is weird because When upgrading my torch, it says
vllm 0.8.5.post1 requires torch==2.6.0, but you have torch 2.7.0+cu128 which is incompatible.But in the requirements.txt file, it saystorch==2.7.0. It is a very recent change though.So are we expecting to get a new vllm update soon?
I guess the problem is that vLLM make use of some compiled library from its dependency that doesn’t work with the latest pytorch. What I did to resolve this
However, for me, I have performance issues with pytorch 2.8 and triton, so I switch to using Pytorch 2.7 with cuda 12.6, and everything is normal now.
Thanks, I previously had
torch==2.6.0+cu126and downgrading to this version solve the issue.same A6000 torch2.6+cu12.6 L20 torch2.6+cu12.6
same h200 torch2.6 cu12.8/12.6
same torch 2.5.1 + cu 12.8