vllm: [Bug]: Different sampled output when running on different GPUs
Your current environment
The output of `python collect_env.py`
Your output of `python collect_env.py` here
Model Input Dumps
No response
π Describe the bug
from openai import OpenAI
import sys
if __name__ == "__main__":
port = sys.argv[1]
client = OpenAI(base_url=f"http://0.0.0.0:{port}/v1")
gen_func = client.chat.completions.create
transformers.set_seed(42)
np.random.seed(42)
torch.manual_seed(42)
args = {
"model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "Give a speech on abortion"}],,
"seed": 892,
"max_tokens": 250,
"temperature": 0.7
}
completion = gen_func(**args)
print(completion.choices[0].message.content)
When running on different type of GPU, i.e. a100 versus h100, I would obtain different results.
How I ran the servers:
CUDA_VISIBLE_DEVICES=$GPU python -m vllm.entrypoints.openai.api_server \
--model $model --max_model_len 6144 --guided-decoding-backend lm-format-enforcer \
--tensor-parallel-size 1 --port $PORT &
But if I set temperature = 0.0 would be fine
Before submitting a new issueβ¦
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (7 by maintainers)
This is normal behaviour, since the kernel implementation may be different based on the GPU, resulting in minor numerical differences.