vllm: [Bug]: request reward model report 500 Internal Server Error

Your current environment

The output of `python collect_env.py`
Your output of `python collect_env.py` here

Model Input Dumps

No response

🐛 Describe the bug

following the instruction https://github.com/vllm-project/vllm/issues/8700#issuecomment-2366884425 , the reward model https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward-HF is a LlamaForCausalLM model, so I serve it with vllm add parameter --task embedding.

when I send a request, it encounter en error: INFO: “POST /v1/embeddings HTTP/1.1” 500 Internal Server Error
ERROR 11-19 17:55:16 engine.py:135] TypeError(“object of type ‘NoneType’ has no len()”) and then the server terminated

the shell script:

curl http://host:port/v1/embeddings \  
        -H "Content-Type: application/json" \
        -d '{
         "model": "Llama-3.1-Nemotron-70B-Reward-HF",
         "input": "Your text string goes here"                                                      
}'

or use python code same as https://github.com/vllm-project/vllm/blob/main/examples/openai_embedding_client.py

has the same error

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 17 (7 by maintainers)

Most upvoted comments

@hrdxwandg are you able to host Llama-3.1-Nemotron-70B-Reward-HF with this flag? I am getting a strange No available block found in 60 second. when I try to send a request to it.

yes, i have the same warning message. so I change the serving command parameter --tensor-parallel-size 4 to --pipeline-parallel-size 4, And that’s OK,you can refer to https://github.com/vllm-project/vllm/issues/6614

I’ve just solve you should set the enable-chunked-prefill=False when loading the model