vllm: [Bug]: ValueError: Model architectures ['LlamaForCausalLM'] are not supported for now.

Your current environment

The output of `python collect_env.py`

WARNING 10-23 23:26:52 _custom_ops.py:19] Failed to import from vllm._C with ModuleNotFoundError(“No module named ‘vllm._C’”) /home/dandiago/vllm/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash: No module named ‘vllm._version’ from vllm.version import version as VLLM_VERSION PyTorch version: 2.4.0+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: Could not collect Libc version: glibc-2.35

Python version: 3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, 15:12:24) [GCC 11.2.0] (64-bit runtime) Python platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 12.1.66 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090 GPU 1: NVIDIA GeForce RTX 3090

Nvidia driver version: 560.94 cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.9.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.5.0 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Vendor ID: GenuineIntel Model name: Genuine Intel® 0000 CPU family: 6 Model: 183 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 Stepping: 1 BogoMIPS: 3379.21 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq vmx ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves avx_vnni umip waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize flush_l1d arch_capabilities Virtualization: VT-x Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 768 KiB (16 instances) L1i cache: 512 KiB (16 instances) L2 cache: 32 MiB (16 instances) L3 cache: 36 MiB (1 instance) Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Unknown: No mitigations Vulnerability Retbleed: Mitigation; Enhanced IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected

Versions of relevant libraries: [pip3] numpy==2.1.2 [pip3] torch==2.5.0 [pip3] torchaudio==2.5.0 [pip3] torchvision==0.20.0 [pip3] triton==3.1.0 [conda] blas 1.0 mkl [conda] cuda-cudart 12.1.105 0 nvidia [conda] cuda-cupti 12.1.105 0 nvidia [conda] cuda-libraries 12.1.0 0 nvidia [conda] cuda-nvrtc 12.1.105 0 nvidia [conda] cuda-nvtx 12.1.105 0 nvidia [conda] cuda-opencl 12.6.77 0 nvidia [conda] cuda-runtime 12.1.0 0 nvidia [conda] cuda-version 12.6 3 nvidia [conda] ffmpeg 4.3 hf484d3e_0 pytorch [conda] libcublas 12.1.0.26 0 nvidia [conda] libcufft 11.0.2.4 0 nvidia [conda] libcufile 1.11.1.6 0 nvidia [conda] libcurand 10.3.7.77 0 nvidia [conda] libcusolver 11.4.4.55 0 nvidia [conda] libcusparse 12.0.2.55 0 nvidia [conda] libjpeg-turbo 2.0.0 h9bf148f_0 pytorch [conda] libnpp 12.0.2.50 0 nvidia [conda] libnvjitlink 12.1.105 0 nvidia [conda] libnvjpeg 12.1.1.14 0 nvidia [conda] mkl 2023.1.0 h213fc3f_46344 [conda] mkl-service 2.4.0 py310h5eee18b_1 [conda] mkl_fft 1.3.10 py310h5eee18b_0 [conda] mkl_random 1.2.7 py310h1128e8f_0 [conda] numpy 2.0.1 py310h5f9d8c6_1 [conda] numpy-base 2.0.1 py310hb5e798b_1 [conda] pytorch 2.5.0 py3.10_cuda12.1_cudnn9.1.0_0 pytorch [conda] pytorch-cuda 12.1 ha16c6d3_6 pytorch [conda] pytorch-mutex 1.0 cuda pytorch [conda] torchaudio 2.5.0 py310_cu121 pytorch [conda] torchtriton 3.1.0 py310 pytorch [conda] torchvision 0.20.0 py310_cu121 pytorch ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: N/A (dev) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X SYS N/A GPU1 SYS X N/A

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

Model Input Dumps

python -O -u -m vllm.entrypoints.api_server
–host=0.0.0.0
–port=8000
–model=/home/dandiago/models/Llama-2-13b-hf
–served-model-name "Llama-2-13b-hf "
–tensor-parallel-size=2
–distributed-executor-backend=ray

🐛 Describe the bug

I use ray+vllm to serve Llama-2-13-hf. But failed.

ERROR 10-23 23:20:26 registry.py:272] Error in inspecting model architecture ‘LlamaForCausalLM’ ERROR 10-23 23:20:26 registry.py:272] Traceback (most recent call last): ERROR 10-23 23:20:26 registry.py:272] File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/model_executor/models/registry.py”, line 434, in _run_in_subprocess ERROR 10-23 23:20:26 registry.py:272] returned.check_returncode() ERROR 10-23 23:20:26 registry.py:272] File “/home/dandiago/anaconda3/lib/python3.12/subprocess.py”, line 502, in check_returncode ERROR 10-23 23:20:26 registry.py:272] raise CalledProcessError(self.returncode, self.args, self.stdout, ERROR 10-23 23:20:26 registry.py:272] subprocess.CalledProcessError: Command ‘[’/home/dandiago/anaconda3/envs/myenv/bin/python’, ‘-m’, ‘vllm.model_executor.models.registry’]’ returned non-zero exit status 1. ERROR 10-23 23:20:26 registry.py:272] ERROR 10-23 23:20:26 registry.py:272] The above exception was the direct cause of the following exception: ERROR 10-23 23:20:26 registry.py:272] ERROR 10-23 23:20:26 registry.py:272] Traceback (most recent call last): ERROR 10-23 23:20:26 registry.py:272] File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/model_executor/models/registry.py”, line 270, in _try_inspect_model_cls ERROR 10-23 23:20:26 registry.py:272] return model.inspect_model_cls() ERROR 10-23 23:20:26 registry.py:272] ^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 10-23 23:20:26 registry.py:272] File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/model_executor/models/registry.py”, line 232, in inspect_model_cls ERROR 10-23 23:20:26 registry.py:272] return _run_in_subprocess( ERROR 10-23 23:20:26 registry.py:272] ^^^^^^^^^^^^^^^^^^^ ERROR 10-23 23:20:26 registry.py:272] File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/model_executor/models/registry.py”, line 437, in _run_in_subprocess ERROR 10-23 23:20:26 registry.py:272] raise RuntimeError(f"Error raised in subprocess:\n" ERROR 10-23 23:20:26 registry.py:272] RuntimeError: Error raised in subprocess: ERROR 10-23 23:20:26 registry.py:272] /home/dandiago/anaconda3/envs/myenv/bin/python: Error while finding module specification for ‘vllm.model_executor.models.registry’ (ModuleNotFoundError: No module named ‘vllm.model_executor’) ERROR 10-23 23:20:26 registry.py:272] Traceback (most recent call last): File “<frozen runpy>”, line 198, in _run_module_as_main File “<frozen runpy>”, line 88, in _run_code File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/entrypoints/api_server.py”, line 163, in <module> asyncio.run(run_server(args)) File “/home/dandiago/anaconda3/lib/python3.12/asyncio/runners.py”, line 194, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File “/home/dandiago/anaconda3/lib/python3.12/asyncio/runners.py”, line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/home/dandiago/anaconda3/lib/python3.12/asyncio/base_events.py”, line 687, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/entrypoints/api_server.py”, line 119, in run_server app = await init_app(args, llm_engine) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/entrypoints/api_server.py”, line 107, in init_app if llm_engine is not None else AsyncLLMEngine.from_engine_args( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py”, line 664, in from_engine_args engine_config = engine_args.create_engine_config() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/engine/arg_utils.py”, line 920, in create_engine_config model_config = self.create_model_config() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/engine/arg_utils.py”, line 855, in create_model_config return ModelConfig( ^^^^^^^^^^^^ File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/config.py”, line 209, in init self.multimodal_config = self._init_multimodal_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/config.py”, line 234, in _init_multimodal_config if ModelRegistry.is_multimodal_model(architectures): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/model_executor/models/registry.py”, line 392, in is_multimodal_model return self.inspect_model_cls(architectures).supports_multimodal ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/model_executor/models/registry.py”, line 361, in inspect_model_cls return self._raise_for_unsupported(architectures) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/home/dandiago/anaconda3/lib/python3.12/site-packages/vllm/model_executor/models/registry.py”, line 322, in _raise_for_unsupported raise ValueError( ValueError: Model architectures [‘LlamaForCausalLM’] are not supported for now. Supported architectures: [‘AquilaModel’, ‘AquilaForCausalLM’, ‘ArcticForCausalLM’, ‘BaiChuanForCausalLM’, ‘BaichuanForCausalLM’, ‘BloomForCausalLM’, ‘CohereForCausalLM’, ‘DbrxForCausalLM’, ‘DeciLMForCausalLM’, ‘DeepseekForCausalLM’, ‘DeepseekV2ForCausalLM’, ‘ExaoneForCausalLM’, ‘FalconForCausalLM’, ‘GemmaForCausalLM’, ‘Gemma2ForCausalLM’, ‘GPT2LMHeadModel’, ‘GPTBigCodeForCausalLM’, ‘GPTJForCausalLM’, ‘GPTNeoXForCausalLM’, ‘GraniteForCausalLM’, ‘GraniteMoeForCausalLM’, ‘InternLMForCausalLM’, ‘InternLM2ForCausalLM’, ‘InternLM2VEForCausalLM’, ‘JAISLMHeadModel’, ‘JambaForCausalLM’, ‘LlamaForCausalLM’, ‘LLaMAForCausalLM’, ‘MambaForCausalLM’, ‘FalconMambaForCausalLM’, ‘MistralForCausalLM’, ‘MixtralForCausalLM’, ‘QuantMixtralForCausalLM’, ‘MptForCausalLM’, ‘MPTForCausalLM’, ‘MiniCPMForCausalLM’, ‘MiniCPM3ForCausalLM’, ‘NemotronForCausalLM’, ‘OlmoForCausalLM’, ‘OlmoeForCausalLM’, ‘OPTForCausalLM’, ‘OrionForCausalLM’, ‘PersimmonForCausalLM’, ‘PhiForCausalLM’, ‘Phi3ForCausalLM’, ‘Phi3SmallForCausalLM’, ‘PhiMoEForCausalLM’, ‘Qwen2ForCausalLM’, ‘Qwen2MoeForCausalLM’, ‘RWForCausalLM’, ‘StableLMEpochForCausalLM’, ‘StableLmForCausalLM’, ‘Starcoder2ForCausalLM’, ‘SolarForCausalLM’, ‘XverseForCausalLM’, ‘BartModel’, ‘BartForConditionalGeneration’, ‘Florence2ForConditionalGeneration’, ‘BertModel’, ‘Gemma2Model’, ‘MistralModel’, ‘Qwen2ForRewardModel’, ‘LlavaNextForConditionalGeneration’, ‘Phi3VForCausalLM’, ‘Blip2ForConditionalGeneration’, ‘ChameleonForConditionalGeneration’, ‘ChatGLMModel’, ‘ChatGLMForConditionalGeneration’, ‘FuyuForCausalLM’, ‘InternVLChatModel’, ‘LlavaForConditionalGeneration’, ‘LlavaNextVideoForConditionalGeneration’, ‘LlavaOnevisionForConditionalGeneration’, ‘MiniCPMV’, ‘MolmoForCausalLM’, ‘NVLM_D’, ‘PaliGemmaForConditionalGeneration’, ‘PixtralForConditionalGeneration’, ‘QWenLMHeadModel’, ‘Qwen2VLForConditionalGeneration’, ‘UltravoxModel’, ‘MllamaForConditionalGeneration’, ‘EAGLEModel’, ‘MedusaModel’, ‘MLPSpeculatorPreTrainedModel’]

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 30 (14 by maintainers)

Most upvoted comments

What happened? i am running into the same issue

How did you install vLLM? There seems to be an import error

ERROR 10-23 23:20:26 registry.py:272] /home/dandiago/anaconda3/envs/myenv/bin/python: Error while finding module specification for 'vllm.model_executor.models.registry' (ModuleNotFoundError: No module named 'vllm.model_executor')