vllm: [Bug]: Different sampled output when running on different GPUs

Your current environment

The output of `python collect_env.py`
Your output of `python collect_env.py` here

Model Input Dumps

No response

πŸ› Describe the bug

from openai import OpenAI
import sys
if __name__ == "__main__":
    port = sys.argv[1]
    client = OpenAI(base_url=f"http://0.0.0.0:{port}/v1")
    gen_func = client.chat.completions.create 
    transformers.set_seed(42)
    np.random.seed(42)
    torch.manual_seed(42)
    args = {
        "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
        "messages": [{"role": "user", "content": "Give a speech on abortion"}],,
        "seed": 892,
        "max_tokens": 250,
        "temperature": 0.7
    }
    completion = gen_func(**args)
    print(completion.choices[0].message.content)

When running on different type of GPU, i.e. a100 versus h100, I would obtain different results.

How I ran the servers:

CUDA_VISIBLE_DEVICES=$GPU python -m vllm.entrypoints.openai.api_server \
      --model $model --max_model_len 6144 --guided-decoding-backend lm-format-enforcer \
      --tensor-parallel-size 1 --port $PORT &

But if I set temperature = 0.0 would be fine

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (7 by maintainers)

Most upvoted comments

This is normal behaviour, since the kernel implementation may be different based on the GPU, resulting in minor numerical differences.