vllm: [Bug]: request reward model report 500 Internal Server Error
Your current environment
The output of `python collect_env.py`
Your output of `python collect_env.py` here
Model Input Dumps
No response
🐛 Describe the bug
following the instruction https://github.com/vllm-project/vllm/issues/8700#issuecomment-2366884425 , the reward model https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward-HF is a LlamaForCausalLM model, so I serve it with vllm add parameter --task embedding.
when I send a request, it encounter en error:
INFO: “POST /v1/embeddings HTTP/1.1” 500 Internal Server Error
ERROR 11-19 17:55:16 engine.py:135] TypeError(“object of type ‘NoneType’ has no len()”)
and then the server terminated
the shell script:
curl http://host:port/v1/embeddings \
-H "Content-Type: application/json" \
-d '{
"model": "Llama-3.1-Nemotron-70B-Reward-HF",
"input": "Your text string goes here"
}'
or use python code same as https://github.com/vllm-project/vllm/blob/main/examples/openai_embedding_client.py
has the same error
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 17 (7 by maintainers)
yes, i have the same warning message. so I change the serving command parameter --tensor-parallel-size 4 to --pipeline-parallel-size 4, And that’s OK,you can refer to https://github.com/vllm-project/vllm/issues/6614
I’ve just solve you should set the
enable-chunked-prefill=Falsewhen loading the model