vllm: [Usage]: Run OpenAI api_server.py on Neuron Device Failed, Model architectures ['OPTForCausalLM'] are not supported on Neuron for now. Supported architectures: ['LlamaForCausalLM', 'MistralForCausalLM']

Your current environment

Dear vLLM Support Team,

I hope this message finds you well. I am reaching out to report an issue encountered while executing the python api_server.py command within the vllm/vllm/entrypoints/openai directory on our server. Unfortunately, this attempt resulted in an error

Here is the Pip lis:

Package                           Version
--------------------------------- --------------------------------------
absl-py                           2.1.0
accelerate                        1.2.0
aiohappyeyeballs                  2.4.4
aiohttp                           3.11.10
aiosignal                         1.3.1
annotated-types                   0.7.0
anyio                             4.7.0
argon2-cffi                       23.1.0
argon2-cffi-bindings              21.2.0
arrow                             1.3.0
asttokens                         3.0.0
async-lru                         2.0.4
async-timeout                     5.0.1
attrs                             24.2.0
aws-neuronx-runtime-discovery     2.9
awscli                            1.36.18
babel                             2.16.0
beautifulsoup4                    4.12.3
bleach                            6.2.0
boto3                             1.35.77
botocore                          1.35.77
cachetools                        5.5.0
certifi                           2024.8.30
cffi                              1.17.1
charset-normalizer                3.4.0
click                             8.1.7
cloud-tpu-client                  0.10
cloudpickle                       3.1.0
colorama                          0.4.6
comm                              0.2.2
compressed-tensors                0.8.0
datasets                          2.19.1
debugpy                           1.8.9
decorator                         5.1.1
defusedxml                        0.7.1
dill                              0.3.8
diskcache                         5.6.3
distro                            1.9.0
docutils                          0.16
ec2-metadata                      2.14.0
einops                            0.8.0
environment-kernels               1.2.0
exceptiongroup                    1.2.2
executing                         2.1.0
fastapi                           0.115.6
fastjsonschema                    2.21.1
filelock                          3.16.1
fqdn                              1.5.1
frozenlist                        1.5.0
fsspec                            2024.3.1
gguf                              0.10.0
google-api-core                   1.34.1
google-api-python-client          1.8.0
google-auth                       2.36.0
google-auth-httplib2              0.2.0
googleapis-common-protos          1.66.0
h11                               0.14.0
httpcore                          1.0.7
httplib2                          0.22.0
httptools                         0.6.4
httpx                             0.28.1
huggingface-hub                   0.26.5
idna                              3.10
importlib_metadata                8.5.0
iniconfig                         2.0.0
interegular                       0.3.3
ipykernel                         6.29.5
ipython                           8.30.0
ipywidgets                        8.1.5
islpy                             2023.2.5
isoduration                       20.11.0
jedi                              0.19.2
Jinja2                            3.1.4
jiter                             0.8.2
jmespath                          1.0.1
json5                             0.10.0
jsonpointer                       3.0.0
jsonschema                        4.23.0
jsonschema-specifications         2024.10.1
jupyter                           1.1.1
jupyter_client                    8.6.3
jupyter-console                   6.6.3
jupyter_core                      5.7.2
jupyter-events                    0.10.0
jupyter-lsp                       2.2.5
jupyter_server                    2.14.2
jupyter_server_terminals          0.5.3
jupyterlab                        4.3.3
jupyterlab_pygments               0.3.0
jupyterlab_server                 2.27.3
jupyterlab_widgets                3.0.13
lark                              1.2.2
libneuronxla                      2.0.5347.0
llvmlite                          0.43.0
lm-format-enforcer                0.10.9
lockfile                          0.12.2
MarkupSafe                        3.0.2
matplotlib-inline                 0.1.7
mistral_common                    1.5.1
mistune                           3.0.2
ml-dtypes                         0.2.0
mpmath                            1.3.0
msgspec                           0.18.6
multidict                         6.1.0
multiprocess                      0.70.16
nbclient                          0.10.1
nbconvert                         7.16.4
nbformat                          5.10.4
nest-asyncio                      1.6.0
networkx                          2.8.8
neuronx-cc                        2.15.143.0+e39249ad
notebook                          7.3.1
notebook_shim                     0.2.4
numba                             0.60.0
numpy                             1.25.2
nvidia-cublas-cu12                12.1.3.1
nvidia-cuda-cupti-cu12            12.1.105
nvidia-cuda-nvrtc-cu12            12.1.105
nvidia-cuda-runtime-cu12          12.1.105
nvidia-cudnn-cu12                 8.9.2.26
nvidia-cufft-cu12                 11.0.2.54
nvidia-curand-cu12                10.3.2.106
nvidia-cusolver-cu12              11.4.5.107
nvidia-cusparse-cu12              12.1.0.106
nvidia-nccl-cu12                  2.18.1
nvidia-nvjitlink-cu12             12.6.85
nvidia-nvtx-cu12                  12.1.105
oauth2client                      4.1.3
openai                            1.57.2
opencv-python-headless            4.10.0.84
outlines                          0.0.46
overrides                         7.7.0
packaging                         24.2
pandas                            2.2.3
pandocfilters                     1.5.1
parso                             0.8.4
partial-json-parser               0.2.1.1.post4
pexpect                           4.9.0
pgzip                             0.3.5
pillow                            10.4.0
pip                               22.0.2
platformdirs                      4.3.6
pluggy                            1.5.0
prometheus_client                 0.21.1
prometheus-fastapi-instrumentator 7.0.0
prompt_toolkit                    3.0.48
propcache                         0.2.1
protobuf                          3.20.3
psutil                            6.1.0
ptyprocess                        0.7.0
pure_eval                         0.2.3
py-cpuinfo                        9.0.0
pyairports                        2.1.1
pyarrow                           18.1.0
pyarrow-hotfix                    0.6
pyasn1                            0.6.1
pyasn1_modules                    0.4.1
pybind11                          2.13.6
pycountry                         24.6.1
pycparser                         2.22
pydantic                          2.10.3
pydantic_core                     2.27.1
Pygments                          2.18.0
pyparsing                         3.2.0
pytest                            8.3.4
python-daemon                     3.1.2
python-dateutil                   2.9.0.post0
python-dotenv                     1.0.1
python-json-logger                2.0.7
pytz                              2024.2
PyYAML                            6.0.2
pyzmq                             26.2.0
referencing                       0.35.1
regex                             2024.11.6
requests                          2.31.0
requests-unixsocket               0.3.0
rfc3339-validator                 0.1.4
rfc3986-validator                 0.1.1
rpds-py                           0.22.3
rsa                               4.7.2
s3transfer                        0.10.4
safetensors                       0.4.6.dev0
scipy                             1.11.2
Send2Trash                        1.8.3
sentencepiece                     0.2.0
setuptools                        59.6.0
six                               1.17.0
sniffio                           1.3.1
soupsieve                         2.6
stack-data                        0.6.3
starlette                         0.41.3
sympy                             1.13.3
terminado                         0.18.1
tiktoken                          0.7.0
tinycss2                          1.4.0
tokenizers                        0.21.0
tomli                             2.2.1
torch                             2.1.2
torch-neuronx                     2.1.2.2.3.2
torch-xla                         2.1.5
torchvision                       0.16.2
tornado                           6.4.2
tqdm                              4.67.1
traitlets                         5.14.3
transformers                      4.47.0
transformers-neuronx              0.12.313
triton                            2.1.0
types-python-dateutil             2.9.0.20241206
typing_extensions                 4.12.2
tzdata                            2024.2
uri-template                      1.3.0
uritemplate                       3.0.1
urllib3                           2.2.3
uvicorn                           0.32.1
uvloop                            0.21.0
vllm                              0.6.4.post2.dev301+g250ee65d.neuron215
watchfiles                        1.0.1
wcwidth                           0.2.13
webcolors                         24.11.1
webencodings                      0.5.1
websocket-client                  1.8.0
websockets                        14.1
wget                              3.2
widgetsnbextension                4.0.13
xgrammar                          0.1.6
xxhash                            3.5.0
yarl                              1.18.3
zipp                              3.21.0

My infra is: Ubuntu Server 22.04 on AWS inf2 instance.

Thank you!

Model Input Dumps

No response

🐛 Describe the bug

The specifics of which are included below for your review.

INFO 12-10 20:13:37 api_server.py:625] vLLM API server version 0.6.4.post2.dev301+g250ee65d
INFO 12-10 20:13:37 api_server.py:626] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='facebook/opt-125m', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='xgrammar', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 12-10 20:13:37 api_server.py:197] Started engine process with PID 5791
INFO 12-10 20:13:42 config.py:405] This model supports multiple tasks: {'embedding', 'generate'}. Defaulting to 'generate'.
WARNING 12-10 20:13:42 config.py:517] Async output processing is not supported on the current platform type neuron.
INFO 12-10 20:13:46 config.py:405] This model supports multiple tasks: {'embedding', 'generate'}. Defaulting to 'generate'.
WARNING 12-10 20:13:46 config.py:517] Async output processing is not supported on the current platform type neuron.
INFO 12-10 20:13:46 llm_engine.py:250] Initializing an LLM engine (v0.6.4.post2.dev301+g250ee65d) with config: VllmConfig(model_config=<vllm.config.ModelConfig object at 0x71a666b0b430>, cache_config=<vllm.config.CacheConfig object at 0x71a666b09cf0>, parallel_config=ParallelConfig(pipeline_parallel_size=1, tensor_parallel_size=1, worker_use_ray=False, max_parallel_loading_workers=None, disable_custom_all_reduce=False, tokenizer_pool_config=None, ray_workers_use_nsight=False, placement_group=None, distributed_executor_backend=None, worker_cls='vllm.worker.neuron_worker.NeuronWorker', sd_worker_cls='auto', world_size=1, rank=0), scheduler_config=SchedulerConfig(task='generate', max_num_batched_tokens=2048, max_num_seqs=256, max_model_len=2048, num_lookahead_slots=0, delay_factor=0.0, enable_chunked_prefill=False, is_multimodal_model=False, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, send_delta_data=False, policy='fcfs', chunked_prefill_enabled=False), device_config=<vllm.config.DeviceConfig object at 0x71a7e3f24f40>, load_config=LoadConfig(load_format=<LoadFormat.AUTO: 'auto'>, download_dir=None, model_loader_extra_config=None, ignore_patterns=['original/**/*']), lora_config=None, speculative_config=None, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), prompt_adapter_config=None, quant_config=None, compilation_config=CompilationConfig(level=0, backend='', custom_ops=[], splitting_ops=['vllm.unified_attention', 'vllm.unified_attention_with_output'], use_inductor=True, candidate_compile_sizes=[], inductor_compile_config={}, inductor_passes={}, use_cudagraph=False, cudagraph_num_of_warmups=0, cudagraph_capture_sizes=None, cudagraph_copy_inputs=False, pass_config=PassConfig(dump_graph_stages=[], dump_graph_dir=PosixPath('.'), enable_fusion=True, enable_reshape=True), compile_sizes=[], capture_sizes=[256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], enabled_custom_ops=Counter(), disabled_custom_ops=Counter(), compilation_time=0.0, static_forward_context={}), kv_transfer_config=None, instance_id='86df0'),use_cached_outputs=True, 
WARNING 12-10 20:13:47 utils.py:738] Pin memory is not supported on Neuron.
WARNING 12-10 20:13:47 neuron_model_runner.py:88] On-device sampling is turned on in Neuron by default, only top_k, top_p, and temperature are current supported sampling parameters. To turn off the on-device sampling, please set the environment variable NEURON_ON_DEVICE_SAMPLING_DISABLED=1.
WARNING 12-10 20:13:47 _custom_ops.py:20] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
ERROR 12-10 20:13:47 engine.py:366] Model architectures ['OPTForCausalLM'] are not supported on Neuron for now. Supported architectures: ['LlamaForCausalLM', 'MistralForCausalLM']
ERROR 12-10 20:13:47 engine.py:366] Traceback (most recent call last):
ERROR 12-10 20:13:47 engine.py:366]   File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
ERROR 12-10 20:13:47 engine.py:366]     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 12-10 20:13:47 engine.py:366]   File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 119, in from_engine_args
ERROR 12-10 20:13:47 engine.py:366]     return cls(ipc_path=ipc_path,
ERROR 12-10 20:13:47 engine.py:366]   File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 71, in __init__
ERROR 12-10 20:13:47 engine.py:366]     self.engine = LLMEngine(*args, **kwargs)
ERROR 12-10 20:13:47 engine.py:366]   File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 289, in __init__
ERROR 12-10 20:13:47 engine.py:366]     self.model_executor = executor_class(vllm_config=vllm_config, )
ERROR 12-10 20:13:47 engine.py:366]   File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 36, in __init__
ERROR 12-10 20:13:47 engine.py:366]     self._init_executor()
ERROR 12-10 20:13:47 engine.py:366]   File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/executor/neuron_executor.py", line 26, in _init_executor
ERROR 12-10 20:13:47 engine.py:366]     self._init_worker()
ERROR 12-10 20:13:47 engine.py:366]   File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/executor/neuron_executor.py", line 40, in _init_worker
ERROR 12-10 20:13:47 engine.py:366]     self.driver_worker.load_model()
ERROR 12-10 20:13:47 engine.py:366]   File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/worker/neuron_worker.py", line 49, in load_model
ERROR 12-10 20:13:47 engine.py:366]     self.model_runner.load_model()
ERROR 12-10 20:13:47 engine.py:366]   File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/worker/neuron_model_runner.py", line 107, in load_model
ERROR 12-10 20:13:47 engine.py:366]     self.model = get_neuron_model(
ERROR 12-10 20:13:47 engine.py:366]   File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/model_loader/neuron.py", line 203, in get_neuron_model
ERROR 12-10 20:13:47 engine.py:366]     model.load_weights(model_config.model,
ERROR 12-10 20:13:47 engine.py:366]   File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/model_loader/neuron.py", line 104, in load_weights
ERROR 12-10 20:13:47 engine.py:366]     arch = _get_model_architecture(self.config)
ERROR 12-10 20:13:47 engine.py:366]   File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/model_loader/neuron.py", line 120, in _get_model_architecture
ERROR 12-10 20:13:47 engine.py:366]     raise ValueError(
ERROR 12-10 20:13:47 engine.py:366] ValueError: Model architectures ['OPTForCausalLM'] are not supported on Neuron for now. Supported architectures: ['LlamaForCausalLM', 'MistralForCausalLM']
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 368, in run_mp_engine
    raise e
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 119, in from_engine_args
    return cls(ipc_path=ipc_path,
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 71, in __init__
    self.engine = LLMEngine(*args, **kwargs)
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 289, in __init__
    self.model_executor = executor_class(vllm_config=vllm_config, )
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 36, in __init__
    self._init_executor()
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/executor/neuron_executor.py", line 26, in _init_executor
    self._init_worker()
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/executor/neuron_executor.py", line 40, in _init_worker
    self.driver_worker.load_model()
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/worker/neuron_worker.py", line 49, in load_model
    self.model_runner.load_model()
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/worker/neuron_model_runner.py", line 107, in load_model
    self.model = get_neuron_model(
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/model_loader/neuron.py", line 203, in get_neuron_model
    model.load_weights(model_config.model,
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/model_loader/neuron.py", line 104, in load_weights
    arch = _get_model_architecture(self.config)
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/model_executor/model_loader/neuron.py", line 120, in _get_model_architecture
    raise ValueError(
ValueError: Model architectures ['OPTForCausalLM'] are not supported on Neuron for now. Supported architectures: ['LlamaForCausalLM', 'MistralForCausalLM']
Traceback (most recent call last):
  File "/home/ubuntu/vllm/vllm/entrypoints/openai/api_server.py", line 683, in <module>
    uvloop.run(run_server(args))
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run
    return loop.run_until_complete(wrapper())
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
  File "/home/ubuntu/vllm/vllm/entrypoints/openai/api_server.py", line 649, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/home/ubuntu/vllm/vllm/entrypoints/openai/api_server.py", line 116, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/home/ubuntu/vllm/vllm/entrypoints/openai/api_server.py", line 213, in build_async_engine_client_from_engine_args
    raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.

Could you please provide guidance on how to resolve this error? Any assistance you can offer to help us proceed would be greatly appreciated.

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (8 by maintainers)

Most upvoted comments

@DarkLight1337 I fully got what you mean here, but I am not using the vllm serve default command line. I am running on top of the vLLM.

vllm serve is basically just a shortcut to running python vllm/entrypoints/openai/api_server.py. The command line arguments are the same.