vllm: [Usage]: ValueError: Model architectures ['LlamaForCausalLM'] are not supported for now.
Your current environment
Error in using vllm
llm = LLM(model=os.path.join(cache_dir, finetuned_model_name), tensor_parallel_size=tensor_parallel_size)
The error details are
ERROR 10-26 11:35:32 registry.py:267] Error in inspecting model architecture 'LlamaForCausalLM'
ERROR 10-26 11:35:32 registry.py:267] Traceback (most recent call last):
ERROR 10-26 11:35:32 registry.py:267] File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\model_executor\models\registry.py", line 429, in _run_in_subprocess
ERROR 10-26 11:35:32 registry.py:267] returned.check_returncode()
ERROR 10-26 11:35:32 registry.py:267] File "D:\soft_install\anaconda3\envs\fuseAI\lib\subprocess.py", line 448, in check_returncode
ERROR 10-26 11:35:32 registry.py:267] raise CalledProcessError(self.returncode, self.args, self.stdout,
ERROR 10-26 11:35:32 registry.py:267] subprocess.CalledProcessError: Command '['D:\\soft_install\\anaconda3\\envs\\fuseAI\\python.exe', '-m', 'vllm.model_executor.models.registry']' returned non-zero exit status 1.
ERROR 10-26 11:35:32 registry.py:267]
ERROR 10-26 11:35:32 registry.py:267] The above exception was the direct cause of the following exception:
ERROR 10-26 11:35:32 registry.py:267]
ERROR 10-26 11:35:32 registry.py:267] Traceback (most recent call last):
ERROR 10-26 11:35:32 registry.py:267] File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\model_executor\models\registry.py", line 265, in _try_inspect_model_cls
ERROR 10-26 11:35:32 registry.py:267] return model.inspect_model_cls()
ERROR 10-26 11:35:32 registry.py:267] File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\model_executor\models\registry.py", line 227, in inspect_model_cls
ERROR 10-26 11:35:32 registry.py:267] return _run_in_subprocess(
ERROR 10-26 11:35:32 registry.py:267] File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\model_executor\models\registry.py", line 432, in _run_in_subprocess
ERROR 10-26 11:35:32 registry.py:267] raise RuntimeError(f"Error raised in subprocess:\n"
ERROR 10-26 11:35:32 registry.py:267] RuntimeError: Error raised in subprocess:
Traceback (most recent call last):
File "D:\soft_install\anaconda3\envs\fuseAI\lib\code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "D:\soft_install\PyCharm 2020.3.5\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "D:\soft_install\PyCharm 2020.3.5\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "J:/LM/inference_llms_instruct_math_code.py", line 633, in <module>
llm = create_llm(finetuned_model_name=args.finetuned_model_name,
File "J:/LM/inference_llms_instruct_math_code.py", line 86, in create_llm
llm = LLM(model=os.path.join(cache_dir, finetuned_model_name), tensor_parallel_size=tensor_parallel_size)
File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\entrypoints\llm.py", line 177, in __init__
self.llm_engine = LLMEngine.from_engine_args(
File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\engine\llm_engine.py", line 570, in from_engine_args
engine_config = engine_args.create_engine_config()
File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\engine\arg_utils.py", line 903, in create_engine_config
model_config = self.create_model_config()
File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\engine\arg_utils.py", line 839, in create_model_config
return ModelConfig(
File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\config.py", line 200, in __init__
self.multimodal_config = self._init_multimodal_config(
File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\config.py", line 219, in _init_multimodal_config
if ModelRegistry.is_multimodal_model(architectures):
File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\model_executor\models\registry.py", line 387, in is_multimodal_model
return self.inspect_model_cls(architectures).supports_multimodal
File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\model_executor\models\registry.py", line 356, in inspect_model_cls
return self._raise_for_unsupported(architectures)
File "D:\soft_install\anaconda3\envs\fuseAI\lib\site-packages\vllm\model_executor\models\registry.py", line 317, in _raise_for_unsupported
raise ValueError(
ValueError: Model architectures ['LlamaForCausalLM'] are not supported for now. Supported architectures: ['AquilaModel', 'AquilaForCausalLM', 'ArcticForCausalLM', 'BaiChuanForCausalLM', 'BaichuanForCausalLM', 'BloomForCausalLM', 'CohereForCausalLM', 'DbrxForCausalLM', 'DeciLMForCausalLM', 'DeepseekForCausalLM', 'DeepseekV2ForCausalLM', 'ExaoneForCausalLM', 'FalconForCausalLM', 'GemmaForCausalLM', 'Gemma2ForCausalLM', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTJForCausalLM', 'GPTNeoXForCausalLM', 'GraniteForCausalLM', 'GraniteMoeForCausalLM', 'InternLMForCausalLM', 'InternLM2ForCausalLM', 'JAISLMHeadModel', 'JambaForCausalLM', 'LlamaForCausalLM', 'LLaMAForCausalLM', 'MambaForCausalLM', 'MistralForCausalLM', 'MixtralForCausalLM', 'QuantMixtralForCausalLM', 'MptForCausalLM', 'MPTForCausalLM', 'MiniCPMForCausalLM', 'MiniCPM3ForCausalLM', 'NemotronForCausalLM', 'OlmoForCausalLM', 'OlmoeForCausalLM', 'OPTForCausalLM', 'OrionForCausalLM', 'PersimmonForCausalLM', 'PhiForCausalLM', 'Phi3ForCausalLM', 'Phi3SmallForCausalLM', 'PhiMoEForCausalLM', 'Qwen2ForCausalLM', 'Qwen2MoeForCausalLM', 'RWForCausalLM', 'StableLMEpochForCausalLM', 'StableLmForCausalLM', 'Starcoder2ForCausalLM', 'SolarForCausalLM', 'XverseForCausalLM', 'BartModel', 'BartForConditionalGeneration', 'Gemma2Model', 'MistralModel', 'Qwen2ForRewardModel', 'Phi3VForCausalLM', 'Blip2ForConditionalGeneration', 'ChameleonForConditionalGeneration', 'ChatGLMModel', 'ChatGLMForConditionalGeneration', 'FuyuForCausalLM', 'InternVLChatModel', 'LlavaForConditionalGeneration', 'LlavaNextForConditionalGeneration', 'LlavaNextVideoForConditionalGeneration', 'LlavaOnevisionForConditionalGeneration', 'MiniCPMV', 'MolmoForCausalLM', 'NVLM_D', 'PaliGemmaForConditionalGeneration', 'PixtralForConditionalGeneration', 'QWenLMHeadModel', 'Qwen2VLForConditionalGeneration', 'UltravoxModel', 'MllamaForConditionalGeneration', 'EAGLEModel', 'MedusaModel', 'MLPSpeculatorPreTrainedModel']
python=3.10 vllm=0.6.3.post1
How would you like to use vllm
I want to run inference of a [specific model](put link here). I don’t know how to integrate it with vllm.
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 21 (9 by maintainers)
Quick update. I uninstalled current torch, torchaudio and torchvision and installed latest nightly version of torch 2.6.0 and it worked!
pip3 install --no-cache-dir --pre torch>=2.6 torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm6.2
Thank you very much for your reply. Updating the numpy version from 1.26.3 to 1.26.4 solved my problem.