vllm: [Bug]: torch.ops._C.silu_and_mul does not exist

Your current environment

macOS cpu torch.ops._C.silu_and_mul does not exist

Image

stack trace:

[rank0]: Traceback (most recent call last): [rank0]: File “/Users/xuji/programs/vllm_projects_2/example.py”, line 10, in <module> [rank0]: llm = LLM(model=“deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B”,trust_remote_code=True) [rank0]: File “/Users/xuji/programs_2/vllm/vllm/utils.py”, line 1096, in inner [rank0]: return fn(*args, **kwargs) [rank0]: File “/Users/xuji/programs_2/vllm/vllm/entrypoints/llm.py”, line 243, in init [rank0]: self.llm_engine = LLMEngine.from_engine_args( [rank0]: File “/Users/xuji/programs_2/vllm/vllm/engine/llm_engine.py”, line 521, in from_engine_args [rank0]: return engine_cls.from_vllm_config( [rank0]: File “/Users/xuji/programs_2/vllm/vllm/engine/llm_engine.py”, line 497, in from_vllm_config [rank0]: return cls( [rank0]: File “/Users/xuji/programs_2/vllm/vllm/engine/llm_engine.py”, line 281, in init [rank0]: self.model_executor = executor_class(vllm_config=vllm_config, ) [rank0]: File “/Users/xuji/programs_2/vllm/vllm/executor/executor_base.py”, line 286, in init [rank0]: super().init(*args, **kwargs) [rank0]: File “/Users/xuji/programs_2/vllm/vllm/executor/executor_base.py”, line 52, in init [rank0]: self._init_executor() [rank0]: File “/Users/xuji/programs_2/vllm/vllm/executor/mp_distributed_executor.py”, line 125, in _init_executor [rank0]: self._run_workers(“load_model”, [rank0]: File “/Users/xuji/programs_2/vllm/vllm/executor/mp_distributed_executor.py”, line 185, in _run_workers [rank0]: driver_worker_output = run_method(self.driver_worker, sent_method, [rank0]: File “/Users/xuji/programs_2/vllm/vllm/utils.py”, line 2347, in run_method [rank0]: return func(*args, **kwargs) [rank0]: File “/Users/xuji/programs_2/vllm/vllm/worker/cpu_worker.py”, line 226, in load_model [rank0]: self.model_runner.load_model() [rank0]: File “/Users/xuji/programs_2/vllm/vllm/worker/cpu_model_runner.py”, line 491, in load_model [rank0]: self.model = get_model(vllm_config=self.vllm_config) [rank0]: File “/Users/xuji/programs_2/vllm/vllm/model_executor/model_loader/init.py”, line 14, in get_model [rank0]: return loader.load_model(vllm_config=vllm_config) [rank0]: File “/Users/xuji/programs_2/vllm/vllm/model_executor/model_loader/loader.py”, line 441, in load_model [rank0]: model = _initialize_model(vllm_config=vllm_config) [rank0]: File “/Users/xuji/programs_2/vllm/vllm/model_executor/model_loader/loader.py”, line 127, in _initialize_model [rank0]: return model_class(vllm_config=vllm_config, prefix=prefix) [rank0]: File “/Users/xuji/programs_2/vllm/vllm/model_executor/models/qwen2.py”, line 431, in init [rank0]: self.model = Qwen2Model(vllm_config=vllm_config, [rank0]: File “/Users/xuji/programs_2/vllm/vllm/compilation/decorators.py”, line 151, in init [rank0]: old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs) [rank0]: File “/Users/xuji/programs_2/vllm/vllm/model_executor/models/qwen2.py”, line 300, in init [rank0]: self.start_layer, self.end_layer, self.layers = make_layers( [rank0]: File “/Users/xuji/programs_2/vllm/vllm/model_executor/models/utils.py”, line 609, in make_layers [rank0]: [PPMissingLayer() for _ in range(start_layer)] + [ [rank0]: File “/Users/xuji/programs_2/vllm/vllm/model_executor/models/utils.py”, line 610, in <listcomp> [rank0]: maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}")) [rank0]: File “/Users/xuji/programs_2/vllm/vllm/model_executor/models/qwen2.py”, line 302, in <lambda> [rank0]: lambda prefix: Qwen2DecoderLayer(config=config, [rank0]: File “/Users/xuji/programs_2/vllm/vllm/model_executor/models/qwen2.py”, line 218, in init [rank0]: self.mlp = Qwen2MLP( [rank0]: File “/Users/xuji/programs_2/vllm/vllm/model_executor/models/qwen2.py”, line 92, in init [rank0]: self.act_fn = SiluAndMul() [rank0]: File “/Users/xuji/programs_2/vllm/vllm/model_executor/layers/activation.py”, line 68, in init [rank0]: self.op = torch.ops._C.silu_and_mul [rank0]: File “/Users/xuji/.pyenv/versions/3.9.6/lib/python3.9/site-packages/torch/_ops.py”, line 1232, in getattr [rank0]: raise AttributeError( [rank0]: AttributeError: ‘_OpNamespace’ ‘_C’ object has no attribute ‘silu_and_mul’

Your output of `python collect_env.py` here

INFO 04-02 17:38:05 [init.py:239] Automatically detected platform cpu. Collecting environment information… PyTorch version: 2.6.0 Is debug build: False CUDA used to build PyTorch: None ROCM used to build PyTorch: N/A

OS: macOS 15.3.2 (arm64) GCC version: Could not collect Clang version: 17.0.0 (clang-1700.0.13.3) CMake version: version 4.0.0 Libc version: N/A

Python version: 3.9.6 (default, Apr 1 2025, 20:27:55) [Clang 17.0.0 (clang-1700.0.13.3)] (64-bit runtime) Python platform: macOS-15.3.2-arm64-arm-64bit Is CUDA available: False CUDA runtime version: No CUDA CUDA_MODULE_LOADING set to: N/A GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Apple M1 Pro

Versions of relevant libraries: [pip3] numpy==2.0.2 [pip3] optree==0.14.1 [pip3] pyzmq==26.3.0 [pip3] torch==2.6.0 [pip3] torchaudio==2.6.0 [pip3] torchvision==0.21.0 [pip3] transformers==4.50.3 [conda] Could not collect ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: 0.8.3.dev205+g25293780 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: Could not collect

VLLM_USE_MODELSCOPE=True VLLM_TARGET_DEVICE=cpu LD_LIBRARY_PATH=/Users/xuji/.pyenv/versions/3.9.6/lib/python3.9/site-packages/cv2/…/…/lib: NCCL_CUMEM_ENABLE=0 TORCHINDUCTOR_COMPILE_THREADS=1

🐛 Describe the bug

Image

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 30 (15 by maintainers)

Most upvoted comments

confirm the problem, its clang problem, for macOS it needs to rool back clang version to 16 or 15

17 will cause the problem,

the main reason is from here ImportError(“dlopen(/Users/aiserver/AI/vllm/vllm/_C.abi3.so, 0x0002): symbol not found in flat namespace ‘___kmpc_dispatch_deinit’”)

and for clang17 in mac there can not find ___kmpc_dispatch_deinit symbol

it had been fixed in llvm and other version

more can check:

issue: https://trac.macports.org/ticket/70877 https://github.com/yugabyte/yugabyte-db-thirdparty/pull/298 https://github.com/llvm/llvm-project/issues/98046

will open a doc patch to info it

and for the solutin

setps

clang --version # make sure it is not 17
# if 17

sudo rm -rf /Library/Developer/CommandLineTools

# then go to https://developer.apple.com/download/all/
# download the  `Command Line Tools for Xcode 16.1`
# insatll it
# then 
clang --version # make sure its 16

cc @hmellor @youkaichao