vllm: [Performance]: The performance of bge-rerank model on vllm and huggingface is inconsistent

Proposal to improve performance

No response

Report of performance regression

No response

Misc discussion on performance

The script loaded using vllm is:

from vllm import LLM
model_path = "/data/reranker/bge-reranker-large"
# Create an LLM.
# You should pass task="score" for cross-encoder models
model = LLM(
    model=model_path,
    task="score",
    enforce_eager=True,
    dtype="float32"
)

huggingface scripts mainly use default parameters for,float32

 CrossEncoder(
            model, max_length=512, device=device
        )

After running 200 sets of data, in which 100 paragraphs were rearranged, 19 sets were found to be in inconsistent order after rearrangement I also looked up some information online and there is the following information: https://zhuanlan.zhihu.com/p/658780653

I do not know whether the parameter of my vllm is wrong, or what causes it?

Your current environment (if you think it is necessary)

Current environment linux: ubuntu 18
NVIDIA-SMI 525.105.17 Driver Version: 525.105.17 CUDA Version: 12.0 Tesla T4 vllm==0.6.6

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 29 (10 by maintainers)

Most upvoted comments

@yumingmin88

我们一件事一件事的说:

  1. 首先这种随机性来自于使用fp16或者bf16,它的精度真的很低。

科学计算需要用双精度浮点,最差也用单精度浮点,现在使用的是半精度浮点。

这还不够,现在开始使用fp8。

通常 fp8做推理已经认为是无损的,deepseek v3 甚至使用 fp8 训练模型。

所以完全相同的结果是不可能的,只要在一个大数据集上统计意义上精度差不多就行了。

  1. 具体随机结果是怎么来的

计算attention时,将多个序列放在一个批次执行时,用 mask 来将不同序列的attention区分开。

但是乘法的精度比较低,所以不是一个序列的中间向量的值也会互相影响

具体的看 Is Flash Attention Stable?

  1. 对于解码模型 使用 check_logprobs_close 对比结果是否正确

  2. 对于 embedding 模型,对比 Cosine Similarity, 当然绝对值也算吧,1.1552008800208569e-05 已经足够精确了

  3. 对于 rerank 模型,我也发现如果不使用 softmax ,误差偶尔会超过 0.01,有时确实不可接受,你试试 加 softmax 效果会好吗