vllm: [Bug]: LLaMa 3.1 8B/70B/405B all behave poorly and differently using completions API as compared to good chat API
Your current environment
Docker latest 0.5.4
docker pull vllm/vllm-openai:latest
docker run -d --restart=always \
--runtime=nvidia \
--gpus '"device=0"' \
--shm-size=10.24gb \
-p 5000:5000 \
-e NCCL_IGNORE_DISABLED_P2P=1 \
-e HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN \
-e VLLM_NCCL_SO_PATH=/usr/local/lib/python3.10/dist-packages/nvidia/nccl/lib/libnccl.so.2 \
-v /etc/passwd:/etc/passwd:ro \
-v /etc/group:/etc/group:ro \
-u `id -u`:`id -g` \
-v "${HOME}"/.cache:$HOME/.cache/ \
-v "${HOME}"/.cache/huggingface:$HOME/.cache/huggingface \
-v "${HOME}"/.cache/huggingface/hub:$HOME/.cache/huggingface/hub \
-v "${HOME}"/.config:$HOME/.config/ -v "${HOME}"/.triton:$HOME/.triton/ \
--network host \
--name llama31_8b \
vllm/vllm-openai:latest \
--port=5000 \
--host=0.0.0.0 \
--model=meta-llama/Meta-Llama-3.1-8B-Instruct \
--seed 1234 \
--tensor-parallel-size=1 \
--max-model-len=131072 \
--max-num-batched-tokens=131072 --max-log-len=100 \
--served-model-name meta-llama/Meta-Llama-3.1-8B-Instruct meta-llama/Meta-Llama-3-8B-Instruct \
--download-dir=$HOME/.cache/huggingface/hub &>> logs.vllm_server.llama3-8b.txt
š Describe the bug
I cannot reproduce HuggingFace transformers (works) or Chat API with vLLM (works) against Completions API (fails).
Both streaming and non-streaming completions API behave poorly.
code:
base_url = '' # FILL
from transformers import AutoTokenizer
model = 'meta-llama/Meta-Llama-3.1-8B-Instruct'
system_prompt = """Follow these steps in solving any problem:
1) Know: This will help students find the important information.
2) Need to Know: This will force students to reread the question and write down what they are trying to solve for.
3) Organize: I think this would be a great place for teachers to emphasize drawing a model or picture.
4) Work: Students show their calculations here.
5) Solution: This is where students will ask themselves if the answer is reasonable and whether it answered the question."""
prompt = "What is bigger, 9.9 or 9.11?"
messages = [
{
"role": "system",
"content": system_prompt,
},
{
"role": "user",
"content": prompt,
}
]
tokenizer = AutoTokenizer.from_pretrained(model)
add_generation_prompt = True
prompt_llm = tokenizer.apply_chat_template(conversation=messages, tokenize=False,
add_generation_prompt=add_generation_prompt)
print("prompt_llm: %s\n\n" % prompt_llm)
kwargs = dict(max_tokens=1024, temperature=0)
from openai import OpenAI
client = OpenAI(base_url=base_url)
kwargs['model'] = model
for i in range(1, 2):
kwargs['seed'] = i
responses = client.completions.create(**kwargs, stream=False, prompt=prompt_llm)
text = responses.choices[0].text
print("seed: %s: stream nochat:\n\n %s" % (i, text))
responses = client.completions.create(**kwargs, stream=True, prompt=prompt_llm)
text = ''
for event in responses:
delta = event.choices[0].text if event.choices else None # extract the text
if delta:
text += delta # append the text
print("seed: %s: stream nochat:\n\n %s" % (i, text))
response = client.chat.completions.create(messages=messages, **kwargs, stream=False).choices[0].message.content
print("seed: %s: nostream chat:\n\n %s" % (i, response))
responses = client.chat.completions.create(stream=True, messages=messages, **kwargs)
text = ''
for chunk in responses:
delta = chunk.choices[0].delta.content if chunk.choices else None
if delta:
text += delta
print("seed: %s: stream chat:\n\n %s" % (i, text))
This always gives something like:
prompt_llm: <|begin_of_text|><|start_header_id|>system<|end_header_id|>
Cutting Knowledge Date: December 2023
Today Date: 26 Jul 2024
Follow these steps in solving any problem:
1) Know: This will help students find the important information.
2) Need to Know: This will force students to reread the question and write down what they are trying to solve for.
3) Organize: I think this would be a great place for teachers to emphasize drawing a model or picture.
4) Work: Students show their calculations here.
5) Solution: This is where students will ask themselves if the answer is reasonable and whether it answered the question.<|eot_id|><|start_header_id|>user<|end_header_id|>
What is bigger, 9.9 or 9.11?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
seed: 1: stream nochat:
To solve this problem, we will follow the steps:
1. Know: We know that we are comparing two decimal numbers, 9.9 and 9.11.
2. Need to Know: We need to know which number is bigger.
3. Organize: We can organize this by comparing the numbers. Since they are decimals, we can compare them by looking at the tenths place and the hundredths place.
4. Work:
- The tenths place is the same in both numbers (9).
- The hundredths place in 9.9 is 0, and in 9.11, it is 1.
- Since 1 is greater than 0, 9.11 is greater than 9.9.
5. Solution: Yes, 9.11 is indeed bigger than 9.9.
seed: 1: stream nochat:
To solve this problem, we will follow the steps:
1. Know: We know that we are comparing two decimal numbers, 9.9 and 9.11.
2. Need to Know: We need to know which number is bigger.
3. Organize: We can organize this by comparing the numbers. Since they are decimals, we can compare them by looking at the tenths place and the hundredths place.
4. Work:
- The tenths place is the same in both numbers (9).
- The hundredths place in 9.9 is 0, and in 9.11, it is 1.
- Since 1 is greater than 0, 9.11 is greater than 9.9.
5. Solution: Yes, 9.11 is indeed bigger than 9.9.
seed: 1: nostream chat:
Let's follow the steps to solve this problem:
**Know**: We need to compare two decimal numbers: 9.9 and 9.11.
**Need to Know**: We are trying to determine which number is bigger.
**Organize**: Let's think about the numbers. We can compare them by looking at the decimal part. If the decimal part is greater, the number is bigger.
**Work**: We can compare the decimal parts: 0.9 is less than 0.11. Since 0.11 is greater, 9.11 is bigger than 9.9.
**Solution**: Yes, 9.11 is indeed bigger than 9.9. This makes sense because 0.11 is greater than 0.9.
seed: 1: stream chat:
Let's follow the steps to solve this problem:
**Know**: We need to compare two decimal numbers: 9.9 and 9.11.
**Need to Know**: We are trying to determine which number is bigger.
**Organize**: Let's think about the numbers. We can compare them by looking at the decimal part. If the decimal part is greater, the number is bigger.
**Work**: We can compare the decimal parts: 0.9 is less than 0.11. Since 0.11 is greater, 9.11 is bigger than 9.9.
**Solution**: Yes, 9.11 is indeed bigger than 9.9. This makes sense because 0.11 is greater than 0.9.
That is, the Chat API output disagrees with the completions output, and itās always (varying seed, penalty, etc.) is always worse than the Chat API.
I checked the code in vLLM and canāt see what Iām doing differently in using the chat template. With temperature=0, should be deterministic to good extent. One can add repeats and see that itās always this way.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 22 (2 by maintainers)
I also found that llama3.1 8B performs very poorly when used to reason through math problems (from the MATH dataset). Even with the most basic vanilla CoT (Chain of Thought) method, it often gets stuck in loops, repeating outputs, and can produce very long outputs that donāt terminate.
Can this be given some higher priority? It is an absolute blocker to this set of modelsā¦
This still is an issue?
The problem still exists on 0.7.2
@arianyambao We also suspect issues with Llama-3.1 in vllm, as the scores are far from better than Llama-3. After finetuning it performs even worseā¦