vllm: [Usage]: ValueError: Model architectures ['LlamaForCausalLM'] are not supported for now.

Your current environment

Error in using vllm

llm = LLM(model=os.path.join(cache_dir, finetuned_model_name), tensor_parallel_size=tensor_parallel_size)

The error details are

ERROR 10-26 11:35:32 registry.py:267] Error in inspecting model architecture 'LlamaForCausalLM'
ERROR 10-26 11:35:32 registry.py:267] Traceback (most recent call last):
ERROR 10-26 11:35:32 registry.py:267]   File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\model_executor\models\registry.py", line 429, in _run_in_subprocess
ERROR 10-26 11:35:32 registry.py:267]     returned.check_returncode()
ERROR 10-26 11:35:32 registry.py:267]   File "D:\soft_install\anaconda3\envs\fuseAI\lib\subprocess.py", line 448, in check_returncode
ERROR 10-26 11:35:32 registry.py:267]     raise CalledProcessError(self.returncode, self.args, self.stdout,
ERROR 10-26 11:35:32 registry.py:267] subprocess.CalledProcessError: Command '['D:\\soft_install\\anaconda3\\envs\\fuseAI\\python.exe', '-m', 'vllm.model_executor.models.registry']' returned non-zero exit status 1.
ERROR 10-26 11:35:32 registry.py:267] 
ERROR 10-26 11:35:32 registry.py:267] The above exception was the direct cause of the following exception:
ERROR 10-26 11:35:32 registry.py:267] 
ERROR 10-26 11:35:32 registry.py:267] Traceback (most recent call last):
ERROR 10-26 11:35:32 registry.py:267]   File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\model_executor\models\registry.py", line 265, in _try_inspect_model_cls
ERROR 10-26 11:35:32 registry.py:267]     return model.inspect_model_cls()
ERROR 10-26 11:35:32 registry.py:267]   File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\model_executor\models\registry.py", line 227, in inspect_model_cls
ERROR 10-26 11:35:32 registry.py:267]     return _run_in_subprocess(
ERROR 10-26 11:35:32 registry.py:267]   File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\model_executor\models\registry.py", line 432, in _run_in_subprocess
ERROR 10-26 11:35:32 registry.py:267]     raise RuntimeError(f"Error raised in subprocess:\n"
ERROR 10-26 11:35:32 registry.py:267] RuntimeError: Error raised in subprocess:
Traceback (most recent call last):
  File "D:\soft_install\anaconda3\envs\fuseAI\lib\code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "D:\soft_install\PyCharm 2020.3.5\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "D:\soft_install\PyCharm 2020.3.5\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "J:/LM/inference_llms_instruct_math_code.py", line 633, in <module>
    llm = create_llm(finetuned_model_name=args.finetuned_model_name,
  File "J:/LM/inference_llms_instruct_math_code.py", line 86, in create_llm
    llm = LLM(model=os.path.join(cache_dir, finetuned_model_name), tensor_parallel_size=tensor_parallel_size)
  File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\entrypoints\llm.py", line 177, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
  File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\engine\llm_engine.py", line 570, in from_engine_args
    engine_config = engine_args.create_engine_config()
  File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\engine\arg_utils.py", line 903, in create_engine_config
    model_config = self.create_model_config()
  File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\engine\arg_utils.py", line 839, in create_model_config
    return ModelConfig(
  File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\config.py", line 200, in __init__
    self.multimodal_config = self._init_multimodal_config(
  File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\config.py", line 219, in _init_multimodal_config
    if ModelRegistry.is_multimodal_model(architectures):
  File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\model_executor\models\registry.py", line 387, in is_multimodal_model
    return self.inspect_model_cls(architectures).supports_multimodal
  File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\model_executor\models\registry.py", line 356, in inspect_model_cls
    return self._raise_for_unsupported(architectures)
  File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\model_executor\models\registry.py", line 317, in _raise_for_unsupported
    raise ValueError(

ValueError: Model architectures ['LlamaForCausalLM'] are not supported for now. Supported architectures: ['AquilaModel', 'AquilaForCausalLM', 'ArcticForCausalLM', 'BaiChuanForCausalLM', 'BaichuanForCausalLM', 'BloomForCausalLM', 'CohereForCausalLM', 'DbrxForCausalLM', 'DeciLMForCausalLM', 'DeepseekForCausalLM', 'DeepseekV2ForCausalLM', 'ExaoneForCausalLM', 'FalconForCausalLM', 'GemmaForCausalLM', 'Gemma2ForCausalLM', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTJForCausalLM', 'GPTNeoXForCausalLM', 'GraniteForCausalLM', 'GraniteMoeForCausalLM', 'InternLMForCausalLM', 'InternLM2ForCausalLM', 'JAISLMHeadModel', 'JambaForCausalLM', 'LlamaForCausalLM', 'LLaMAForCausalLM', 'MambaForCausalLM', 'MistralForCausalLM', 'MixtralForCausalLM', 'QuantMixtralForCausalLM', 'MptForCausalLM', 'MPTForCausalLM', 'MiniCPMForCausalLM', 'MiniCPM3ForCausalLM', 'NemotronForCausalLM', 'OlmoForCausalLM', 'OlmoeForCausalLM', 'OPTForCausalLM', 'OrionForCausalLM', 'PersimmonForCausalLM', 'PhiForCausalLM', 'Phi3ForCausalLM', 'Phi3SmallForCausalLM', 'PhiMoEForCausalLM', 'Qwen2ForCausalLM', 'Qwen2MoeForCausalLM', 'RWForCausalLM', 'StableLMEpochForCausalLM', 'StableLmForCausalLM', 'Starcoder2ForCausalLM', 'SolarForCausalLM', 'XverseForCausalLM', 'BartModel', 'BartForConditionalGeneration', 'Gemma2Model', 'MistralModel', 'Qwen2ForRewardModel', 'Phi3VForCausalLM', 'Blip2ForConditionalGeneration', 'ChameleonForConditionalGeneration', 'ChatGLMModel', 'ChatGLMForConditionalGeneration', 'FuyuForCausalLM', 'InternVLChatModel', 'LlavaForConditionalGeneration', 'LlavaNextForConditionalGeneration', 'LlavaNextVideoForConditionalGeneration', 'LlavaOnevisionForConditionalGeneration', 'MiniCPMV', 'MolmoForCausalLM', 'NVLM_D', 'PaliGemmaForConditionalGeneration', 'PixtralForConditionalGeneration', 'QWenLMHeadModel', 'Qwen2VLForConditionalGeneration', 'UltravoxModel', 'MllamaForConditionalGeneration', 'EAGLEModel', 'MedusaModel', 'MLPSpeculatorPreTrainedModel']

python=3.10 vllm=0.6.3.post1

How would you like to use vllm

I want to run inference of a [specific model](put link here). I don’t know how to integrate it with vllm.

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 21 (9 by maintainers)

Most upvoted comments

Quick update. I uninstalled current torch, torchaudio and torchvision and installed latest nightly version of torch 2.6.0 and it worked!

pip3 install --no-cache-dir --pre torch>=2.6 torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm6.2

Based on the error before the stack trace, you need to update your numpy version.

Thank you very much for your reply. Updating the numpy version from 1.26.3 to 1.26.4 solved my problem.