vllm: [Usage]: Run OpenAI api_server.py on Neuron Device Failed, Model architectures ['OPTForCausalLM'] are not supported on Neuron for now. Supported architectures: ['LlamaForCausalLM', 'MistralForCausalLM']
Your current environment
Dear vLLM Support Team,
I hope this message finds you well. I am reaching out to report an issue encountered while executing the python api_server.py command within the vllm/vllm/entrypoints/openai directory on our server. Unfortunately, this attempt resulted in an error
Here is the Pip lis:
Package Version
--------------------------------- --------------------------------------
absl-py 2.1.0
accelerate 1.2.0
aiohappyeyeballs 2.4.4
aiohttp 3.11.10
aiosignal 1.3.1
annotated-types 0.7.0
anyio 4.7.0
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
asttokens 3.0.0
async-lru 2.0.4
async-timeout 5.0.1
attrs 24.2.0
aws-neuronx-runtime-discovery 2.9
awscli 1.36.18
babel 2.16.0
beautifulsoup4 4.12.3
bleach 6.2.0
boto3 1.35.77
botocore 1.35.77
cachetools 5.5.0
certifi 2024.8.30
cffi 1.17.1
charset-normalizer 3.4.0
click 8.1.7
cloud-tpu-client 0.10
cloudpickle 3.1.0
colorama 0.4.6
comm 0.2.2
compressed-tensors 0.8.0
datasets 2.19.1
debugpy 1.8.9
decorator 5.1.1
defusedxml 0.7.1
dill 0.3.8
diskcache 5.6.3
distro 1.9.0
docutils 0.16
ec2-metadata 2.14.0
einops 0.8.0
environment-kernels 1.2.0
exceptiongroup 1.2.2
executing 2.1.0
fastapi 0.115.6
fastjsonschema 2.21.1
filelock 3.16.1
fqdn 1.5.1
frozenlist 1.5.0
fsspec 2024.3.1
gguf 0.10.0
google-api-core 1.34.1
google-api-python-client 1.8.0
google-auth 2.36.0
google-auth-httplib2 0.2.0
googleapis-common-protos 1.66.0
h11 0.14.0
httpcore 1.0.7
httplib2 0.22.0
httptools 0.6.4
httpx 0.28.1
huggingface-hub 0.26.5
idna 3.10
importlib_metadata 8.5.0
iniconfig 2.0.0
interegular 0.3.3
ipykernel 6.29.5
ipython 8.30.0
ipywidgets 8.1.5
islpy 2023.2.5
isoduration 20.11.0
jedi 0.19.2
Jinja2 3.1.4
jiter 0.8.2
jmespath 1.0.1
json5 0.10.0
jsonpointer 3.0.0
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
jupyter 1.1.1
jupyter_client 8.6.3
jupyter-console 6.6.3
jupyter_core 5.7.2
jupyter-events 0.10.0
jupyter-lsp 2.2.5
jupyter_server 2.14.2
jupyter_server_terminals 0.5.3
jupyterlab 4.3.3
jupyterlab_pygments 0.3.0
jupyterlab_server 2.27.3
jupyterlab_widgets 3.0.13
lark 1.2.2
libneuronxla 2.0.5347.0
llvmlite 0.43.0
lm-format-enforcer 0.10.9
lockfile 0.12.2
MarkupSafe 3.0.2
matplotlib-inline 0.1.7
mistral_common 1.5.1
mistune 3.0.2
ml-dtypes 0.2.0
mpmath 1.3.0
msgspec 0.18.6
multidict 6.1.0
multiprocess 0.70.16
nbclient 0.10.1
nbconvert 7.16.4
nbformat 5.10.4
nest-asyncio 1.6.0
networkx 2.8.8
neuronx-cc 2.15.143.0+e39249ad
notebook 7.3.1
notebook_shim 0.2.4
numba 0.60.0
numpy 1.25.2
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.18.1
nvidia-nvjitlink-cu12 12.6.85
nvidia-nvtx-cu12 12.1.105
oauth2client 4.1.3
openai 1.57.2
opencv-python-headless 4.10.0.84
outlines 0.0.46
overrides 7.7.0
packaging 24.2
pandas 2.2.3
pandocfilters 1.5.1
parso 0.8.4
partial-json-parser 0.2.1.1.post4
pexpect 4.9.0
pgzip 0.3.5
pillow 10.4.0
pip 22.0.2
platformdirs 4.3.6
pluggy 1.5.0
prometheus_client 0.21.1
prometheus-fastapi-instrumentator 7.0.0
prompt_toolkit 3.0.48
propcache 0.2.1
protobuf 3.20.3
psutil 6.1.0
ptyprocess 0.7.0
pure_eval 0.2.3
py-cpuinfo 9.0.0
pyairports 2.1.1
pyarrow 18.1.0
pyarrow-hotfix 0.6
pyasn1 0.6.1
pyasn1_modules 0.4.1
pybind11 2.13.6
pycountry 24.6.1
pycparser 2.22
pydantic 2.10.3
pydantic_core 2.27.1
Pygments 2.18.0
pyparsing 3.2.0
pytest 8.3.4
python-daemon 3.1.2
python-dateutil 2.9.0.post0
python-dotenv 1.0.1
python-json-logger 2.0.7
pytz 2024.2
PyYAML 6.0.2
pyzmq 26.2.0
referencing 0.35.1
regex 2024.11.6
requests 2.31.0
requests-unixsocket 0.3.0
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rpds-py 0.22.3
rsa 4.7.2
s3transfer 0.10.4
safetensors 0.4.6.dev0
scipy 1.11.2
Send2Trash 1.8.3
sentencepiece 0.2.0
setuptools 59.6.0
six 1.17.0
sniffio 1.3.1
soupsieve 2.6
stack-data 0.6.3
starlette 0.41.3
sympy 1.13.3
terminado 0.18.1
tiktoken 0.7.0
tinycss2 1.4.0
tokenizers 0.21.0
tomli 2.2.1
torch 2.1.2
torch-neuronx 2.1.2.2.3.2
torch-xla 2.1.5
torchvision 0.16.2
tornado 6.4.2
tqdm 4.67.1
traitlets 5.14.3
transformers 4.47.0
transformers-neuronx 0.12.313
triton 2.1.0
types-python-dateutil 2.9.0.20241206
typing_extensions 4.12.2
tzdata 2024.2
uri-template 1.3.0
uritemplate 3.0.1
urllib3 2.2.3
uvicorn 0.32.1
uvloop 0.21.0
vllm 0.6.4.post2.dev301+g250ee65d.neuron215
watchfiles 1.0.1
wcwidth 0.2.13
webcolors 24.11.1
webencodings 0.5.1
websocket-client 1.8.0
websockets 14.1
wget 3.2
widgetsnbextension 4.0.13
xgrammar 0.1.6
xxhash 3.5.0
yarl 1.18.3
zipp 3.21.0
My infra is: Ubuntu Server 22.04 on AWS inf2 instance.
Thank you!
Model Input Dumps
No response
🐛 Describe the bug
The specifics of which are included below for your review.
INFO 12-10 20:13:37 api_server.py:625] vLLM API server version 0.6.4.post2.dev301+g250ee65d
INFO 12-10 20:13:37 api_server.py:626] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='facebook/opt-125m', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='xgrammar', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 12-10 20:13:37 api_server.py:197] Started engine process with PID 5791
INFO 12-10 20:13:42 config.py:405] This model supports multiple tasks: {'embedding', 'generate'}. Defaulting to 'generate'.
WARNING 12-10 20:13:42 config.py:517] Async output processing is not supported on the current platform type neuron.
INFO 12-10 20:13:46 config.py:405] This model supports multiple tasks: {'embedding', 'generate'}. Defaulting to 'generate'.
WARNING 12-10 20:13:46 config.py:517] Async output processing is not supported on the current platform type neuron.
INFO 12-10 20:13:46 llm_engine.py:250] Initializing an LLM engine (v0.6.4.post2.dev301+g250ee65d) with config: VllmConfig(model_config=<vllm.config.ModelConfig object at 0x71a666b0b430>, cache_config=<vllm.config.CacheConfig object at 0x71a666b09cf0>, parallel_config=ParallelConfig(pipeline_parallel_size=1, tensor_parallel_size=1, worker_use_ray=False, max_parallel_loading_workers=None, disable_custom_all_reduce=False, tokenizer_pool_config=None, ray_workers_use_nsight=False, placement_group=None, distributed_executor_backend=None, worker_cls='vllm.worker.neuron_worker.NeuronWorker', sd_worker_cls='auto', world_size=1, rank=0), scheduler_config=SchedulerConfig(task='generate', max_num_batched_tokens=2048, max_num_seqs=256, max_model_len=2048, num_lookahead_slots=0, delay_factor=0.0, enable_chunked_prefill=False, is_multimodal_model=False, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, send_delta_data=False, policy='fcfs', chunked_prefill_enabled=False), device_config=<vllm.config.DeviceConfig object at 0x71a7e3f24f40>, load_config=LoadConfig(load_format=<LoadFormat.AUTO: 'auto'>, download_dir=None, model_loader_extra_config=None, ignore_patterns=['original/**/*']), lora_config=None, speculative_config=None, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), prompt_adapter_config=None, quant_config=None, compilation_config=CompilationConfig(level=0, backend='', custom_ops=[], splitting_ops=['vllm.unified_attention', 'vllm.unified_attention_with_output'], use_inductor=True, candidate_compile_sizes=[], inductor_compile_config={}, inductor_passes={}, use_cudagraph=False, cudagraph_num_of_warmups=0, cudagraph_capture_sizes=None, cudagraph_copy_inputs=False, pass_config=PassConfig(dump_graph_stages=[], dump_graph_dir=PosixPath('.'), enable_fusion=True, enable_reshape=True), compile_sizes=[], capture_sizes=[256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], enabled_custom_ops=Counter(), disabled_custom_ops=Counter(), compilation_time=0.0, static_forward_context={}), kv_transfer_config=None, instance_id='86df0'),use_cached_outputs=True,
WARNING 12-10 20:13:47 utils.py:738] Pin memory is not supported on Neuron.
WARNING 12-10 20:13:47 neuron_model_runner.py:88] On-device sampling is turned on in Neuron by default, only top_k, top_p, and temperature are current supported sampling parameters. To turn off the on-device sampling, please set the environment variable NEURON_ON_DEVICE_SAMPLING_DISABLED=1.
WARNING 12-10 20:13:47 _custom_ops.py:20] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
ERROR 12-10 20:13:47 engine.py:366] Model architectures ['OPTForCausalLM'] are not supported on Neuron for now. Supported architectures: ['LlamaForCausalLM', 'MistralForCausalLM']
ERROR 12-10 20:13:47 engine.py:366] Traceback (most recent call last):
ERROR 12-10 20:13:47 engine.py:366] File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
ERROR 12-10 20:13:47 engine.py:366] engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 12-10 20:13:47 engine.py:366] File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 119, in from_engine_args
ERROR 12-10 20:13:47 engine.py:366] return cls(ipc_path=ipc_path,
ERROR 12-10 20:13:47 engine.py:366] File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 71, in __init__
ERROR 12-10 20:13:47 engine.py:366] self.engine = LLMEngine(*args, **kwargs)
ERROR 12-10 20:13:47 engine.py:366] File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 289, in __init__
ERROR 12-10 20:13:47 engine.py:366] self.model_executor = executor_class(vllm_config=vllm_config, )
ERROR 12-10 20:13:47 engine.py:366] File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 36, in __init__
ERROR 12-10 20:13:47 engine.py:366] self._init_executor()
ERROR 12-10 20:13:47 engine.py:366] File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/executor/neuron_executor.py", line 26, in _init_executor
ERROR 12-10 20:13:47 engine.py:366] self._init_worker()
ERROR 12-10 20:13:47 engine.py:366] File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/executor/neuron_executor.py", line 40, in _init_worker
ERROR 12-10 20:13:47 engine.py:366] self.driver_worker.load_model()
ERROR 12-10 20:13:47 engine.py:366] File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/worker/neuron_worker.py", line 49, in load_model
ERROR 12-10 20:13:47 engine.py:366] self.model_runner.load_model()
ERROR 12-10 20:13:47 engine.py:366] File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/worker/neuron_model_runner.py", line 107, in load_model
ERROR 12-10 20:13:47 engine.py:366] self.model = get_neuron_model(
ERROR 12-10 20:13:47 engine.py:366] File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/model_loader/neuron.py", line 203, in get_neuron_model
ERROR 12-10 20:13:47 engine.py:366] model.load_weights(model_config.model,
ERROR 12-10 20:13:47 engine.py:366] File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/model_loader/neuron.py", line 104, in load_weights
ERROR 12-10 20:13:47 engine.py:366] arch = _get_model_architecture(self.config)
ERROR 12-10 20:13:47 engine.py:366] File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/model_loader/neuron.py", line 120, in _get_model_architecture
ERROR 12-10 20:13:47 engine.py:366] raise ValueError(
ERROR 12-10 20:13:47 engine.py:366] ValueError: Model architectures ['OPTForCausalLM'] are not supported on Neuron for now. Supported architectures: ['LlamaForCausalLM', 'MistralForCausalLM']
Process SpawnProcess-1:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 368, in run_mp_engine
raise e
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 119, in from_engine_args
return cls(ipc_path=ipc_path,
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 71, in __init__
self.engine = LLMEngine(*args, **kwargs)
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 289, in __init__
self.model_executor = executor_class(vllm_config=vllm_config, )
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 36, in __init__
self._init_executor()
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/executor/neuron_executor.py", line 26, in _init_executor
self._init_worker()
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/executor/neuron_executor.py", line 40, in _init_worker
self.driver_worker.load_model()
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/worker/neuron_worker.py", line 49, in load_model
self.model_runner.load_model()
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/worker/neuron_model_runner.py", line 107, in load_model
self.model = get_neuron_model(
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/model_loader/neuron.py", line 203, in get_neuron_model
model.load_weights(model_config.model,
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/model_loader/neuron.py", line 104, in load_weights
arch = _get_model_architecture(self.config)
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/model_loader/neuron.py", line 120, in _get_model_architecture
raise ValueError(
ValueError: Model architectures ['OPTForCausalLM'] are not supported on Neuron for now. Supported architectures: ['LlamaForCausalLM', 'MistralForCausalLM']
Traceback (most recent call last):
File "/home/ubuntu/vllm/vllm/entrypoints/openai/api_server.py", line 683, in <module>
uvloop.run(run_server(args))
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run
return loop.run_until_complete(wrapper())
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper
return await main
File "/home/ubuntu/vllm/vllm/entrypoints/openai/api_server.py", line 649, in run_server
async with build_async_engine_client(args) as engine_client:
File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
return await anext(self.gen)
File "/home/ubuntu/vllm/vllm/entrypoints/openai/api_server.py", line 116, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
return await anext(self.gen)
File "/home/ubuntu/vllm/vllm/entrypoints/openai/api_server.py", line 213, in build_async_engine_client_from_engine_args
raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.
Could you please provide guidance on how to resolve this error? Any assistance you can offer to help us proceed would be greatly appreciated.
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (8 by maintainers)
vllm serveis basically just a shortcut to runningpython vllm/entrypoints/openai/api_server.py. The command line arguments are the same.