vllm: [Bug]: AttributeError: '_OpNamespace' '_C' object has no attribute 'rotary_embedding' / gemma-2-9b with vllm=0.5.2
Your current environment
Versions of relevant libraries:
[pip3] flashinfer==0.0.9+cu121torch2.3
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] sentence-transformers==3.0.1
[pip3] torch==2.3.1
[pip3] torchvision==0.18.1
[pip3] transformers==4.42.4
[pip3] triton==2.3.1
[conda] flashinfer 0.0.9+cu121torch2.3 pypi_0 pypi
[conda] numpy 1.26.4 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.20.5 pypi_0 pypi
[conda] sentence-transformers 3.0.1 pypi_0 pypi
[conda] torch 2.3.1 pypi_0 pypi
[conda] torchvision 0.18.1 pypi_0 pypi
[conda] transformers 4.42.4 pypi_0 pypi
[conda] triton 2.3.1 pypi_0 pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.2
🐛 Describe the bug
I encountered the following error when running Gemma-2-9b. Even after deleting and reinstalling the virtual environment, the same error repeats.
INFO 07-17 00:14:06 selector.py:79] Using Flashinfer backend.
INFO 07-17 00:14:07 selector.py:79] Using Flashinfer backend.
INFO 07-17 00:14:10 model_runner.py:266] Loading model weights took 17.3781 GB
ERROR 07-17 00:14:10 _custom_ops.py:42] Error in calling custom op rotary_embedding: '_OpNamespace' '_C' object has no attribute 'rotary_embedding'
ERROR 07-17 00:14:10 _custom_ops.py:42] Possibly you have built or installed an obsolete version of vllm.
ERROR 07-17 00:14:10 _custom_ops.py:42] Please try a clean build and install of vllm,or remove old built files such as vllm/*cpython*.so and build/ .
[rank0]: Traceback (most recent call last):
[rank0]: llm = LLM(model=args.model_path,
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 150, in __init__
[rank0]: self.llm_engine = LLMEngine.from_engine_args(
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 421, in from_engine_args
[rank0]: engine = cls(
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 263, in __init__
[rank0]: self._initialize_kv_caches()
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 362, in _initialize_kv_caches
[rank0]: self.model_executor.determine_num_available_blocks())
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 78, in determine_num_available_blocks
[rank0]: return self.driver_worker.determine_num_available_blocks()
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/worker/worker.py", line 179, in determine_num_available_blocks
[rank0]: self.model_runner.profile_run()
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 923, in profile_run
[rank0]: self.execute_model(model_input, kv_caches, intermediate_tensors)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1341, in execute_model
[rank0]: hidden_or_intermediate_states = model_executable(
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py", line 336, in forward
[rank0]: hidden_states = self.model(input_ids, positions, kv_caches,
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py", line 277, in forward
[rank0]: hidden_states, residual = layer(
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py", line 221, in forward
[rank0]: hidden_states = self.self_attn(
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py", line 161, in forward
[rank0]: q, k = self.rotary_emb(positions, q, k)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/model_executor/custom_op.py", line 13, in forward
[rank0]: return self._forward_method(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/model_executor/layers/rotary_embedding.py", line 220, in forward_cuda
[rank0]: ops.rotary_embedding(positions, query, key, self.head_size,
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/_custom_ops.py", line 43, in wrapper
[rank0]: raise e
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/_custom_ops.py", line 34, in wrapper
[rank0]: return fn(*args, **kwargs)
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/vllm/_custom_ops.py", line 141, in rotary_embedding
[rank0]: torch.ops._C.rotary_embedding(positions, query, key, head_size,
[rank0]: File "/home/choco_9966/miniconda3/envs/gemma/lib/python3.10/site-packages/torch/_ops.py", line 921, in __getattr__
[rank0]: raise AttributeError(
[rank0]: AttributeError: '_OpNamespace' '_C' object has no attribute 'rotary_embedding'
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 4
- Comments: 17 (2 by maintainers)
Assuming that most people are having glibc versioning problems here, this issue should be fixed for most people in 0.5.3 and later, now that we are building on Ubuntu 20.04. I think we can go ahead and close this one.
I moved the vllm (inside vllm) subdirectory. Then installed 0.4.2 and that solved my issues. Rms and rotary.
@qgallouedec that’s the same problem as https://github.com/vllm-project/vllm/issues/6462. I think people are generally having glibc versioning issues with 0.5.2.
Working on it here: https://github.com/vllm-project/vllm/pull/6517
Downgrading to 0.5.1 solved the issue