vllm: [Performance]: The performance of bge-rerank model on vllm and huggingface is inconsistent
Proposal to improve performance
No response
Report of performance regression
No response
Misc discussion on performance
The script loaded using vllm is:
from vllm import LLM
model_path = "/data/reranker/bge-reranker-large"
# Create an LLM.
# You should pass task="score" for cross-encoder models
model = LLM(
model=model_path,
task="score",
enforce_eager=True,
dtype="float32"
)
huggingface scripts mainly use default parameters for,float32
CrossEncoder(
model, max_length=512, device=device
)
After running 200 sets of data, in which 100 paragraphs were rearranged, 19 sets were found to be in inconsistent order after rearrangement I also looked up some information online and there is the following information: https://zhuanlan.zhihu.com/p/658780653
I do not know whether the parameter of my vllm is wrong, or what causes it?
Your current environment (if you think it is necessary)
Current environment
linux: ubuntu 18
NVIDIA-SMI 525.105.17 Driver Version: 525.105.17 CUDA Version: 12.0 Tesla T4
vllm==0.6.6
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 29 (10 by maintainers)
@yumingmin88
我们一件事一件事的说:
科学计算需要用双精度浮点,最差也用单精度浮点,现在使用的是半精度浮点。
这还不够,现在开始使用fp8。
通常 fp8做推理已经认为是无损的,deepseek v3 甚至使用 fp8 训练模型。
所以完全相同的结果是不可能的,只要在一个大数据集上统计意义上精度差不多就行了。
计算attention时,将多个序列放在一个批次执行时,用 mask 来将不同序列的attention区分开。
但是乘法的精度比较低,所以不是一个序列的中间向量的值也会互相影响
具体的看 Is Flash Attention Stable?
对于解码模型 使用 check_logprobs_close 对比结果是否正确
对于 embedding 模型,对比 Cosine Similarity, 当然绝对值也算吧,1.1552008800208569e-05 已经足够精确了
对于 rerank 模型,我也发现如果不使用 softmax ,误差偶尔会超过 0.01,有时确实不可接受,你试试 加 softmax 效果会好吗