vllm: [Bug]: prefix-caching: inconsistent completions
Your current environment
vLLM version 0.5.0.post1
π Describe the bug
Hi,
Seems that there is a dirty cache issue with --enable-prefix-caching. We noticed it as we saw internal eval scores significantly degrade when running with --enable-prefix-caching and here Iβll show how to reproduce it with a short snippet.
Running 2 vLLM servers with:
without prefix caching:
python -m vllm.entrypoints.openai.api_server --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --port 8001
and another with prefix caching:
python -m vllm.entrypoints.openai.api_server --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --port 8002 --enable-prefix-caching
Then running this snippet:
import string
import random
import openai
vllms = {
"no-prefix-caching": "http://localhost:8001/v1",
"with-prefix-caching": "http://localhost:8002/v1",
}
random.seed(0)
prompts = []
for i in range(16):
prompts.append(''.join(random.choices(string.ascii_lowercase + string.digits, k=512)))
runs = []
for run in range(2):
print(f"\nπ run #{run+1}")
completions = {k: [] for k in vllms.keys()}
runs.append(completions)
for name, endpoint in vllms.items():
print(f"vLLM {name=}, {endpoint=}")
client = openai.OpenAI(
base_url=endpoint,
api_key="foo"
)
for prompt in prompts:
response = client.completions.create(
model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
prompt=prompt,
temperature=0,
max_tokens=4,
)
completion = response.choices[0].text
completions[name].append(completion)
print(f"completions: {completions[name]}")
if run > 0 and runs[run][name] != runs[run-1][name]:
print(f"β completions for vLLM {name=} differs from previous run!")
if completions["with-prefix-caching"] != completions["no-prefix-caching"]:
print("π completions differ between with & without prefix")
prints:
π run #1
vLLM name='no-prefix-caching', endpoint='http://localhost:8001/v1'
completions: ['6x2w', 'zwg9v', 'xjuwf', 'hu5qw', 'jg0m', '1tzkb', '4w0q', '5zx5', 'zxqj', '7v16', '0ty57', 'vk0j', 'jjnj', 'xw95', 'vxjj', 't6x7']
vLLM name='with-prefix-caching', endpoint='http://localhost:8002/v1'
completions: ['6x2w', 'zwg9v', 'xjuwf', 'hu5qw', 'jg0m', '1tzkb', '4w0q', '5zx5', 'zxqj', '7v16', '0ty57', 'vk0j', 'jjnj', 'xw95', 'vxjj', 't6x7']
π run #2
vLLM name='no-prefix-caching', endpoint='http://localhost:8001/v1'
completions: ['6x2w', 'zwg9v', 'xjuwf', 'hu5qw', 'jg0m', '1tzkb', '4w0q', '5zx5', 'zxqj', '7v16', '0ty57', 'vk0j', 'jjnj', 'xw95', 'vxjj', 't6x7']
vLLM name='with-prefix-caching', endpoint='http://localhost:8002/v1'
completions: ['6x2w', 'zwma71', '37wk', 'hu5qw', 'jg0m', '1tzkb', '4h7a', '5zq7', 'zxqj', '7k4n', '0ty57', 'vk0j', 'jjnj', 'xw95', 'vxjj', 't6x7']
β completions for vLLM name='with-prefix-caching' differs from previous run!
π completions differ between with & without prefix
This happens also with 0.4.3. With 0.4.2 this snippet crashes the server with prefix-caching enabled.
Hopefully one of these PR resolves the issue π€ :
- https://github.com/vllm-project/vllm/pull/5188
- https://github.com/vllm-project/vllm/pull/5364 (will be able to try building these branches and reproducing only in few days, hope tagging the PRs can help till then) Edit: built and tried both PRs and they donβt resolve the issue
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 22 (2 by maintainers)
Thanks for the reply @cadedaniel. I tried now with
--use-v2-block-manager(version 0.5.0.post1) and it still happens unfortunately.Edit: Tried also building current
mainbranch (commit e2b85cf86a522e734a38b1d0314cfe9625003ef9) where https://github.com/vllm-project/vllm/pull/5364 is already merged, and the issue still happens (also with--use-v2-block-manager)Thanks @colefranks I tried and seems that the workaround doesnβt seem to help but it does change the behavior, tried several combinations (all with version 0.5.0.post1).
On first iteration, there is difference in outputs between
VLLM_ATTENTION_BACKEND=XFORMERSand without. And if we assume thatβs ok, anyway when--enable-prefix-cachingis used, than second iteration with--enable-prefix-cachingdiffers from the first one.If you still see the degradation in the real evals then it would be a true bug. It calls the same kernel with different inputs here https://github.com/vllm-project/vllm/blob/5223199e03ac3729eb60043a1ef57156c8af1bc9/vllm/attention/backends/flash_attn.py#L532 depending on how if the cache hits or not. Will update here if I find anything.
@hibukipanim @kuangdao @SaltFish11 I sloved the problem by change the triton code. in this file β¦/triton/common/build.py cc, src, f"-I{hip_include_dir}β, fβ-I{py_include_dir}β, fβ-I{srcdir}β, β-sharedβ, β-fPICβ, cc, src, β-O3β, fβ-I{cu_include_dir}β, fβ-I{py_include_dir}β, fβ-I{srcdir}β, β-sharedβ, β-fPICβ, β-lcudaβ, add the β-std=c99β, after the lines,like this if is_hip(): ret = subprocess.check_call([ cc, src, fβ-I{hip_include_dir}β, fβ-I{py_include_dir}β, fβ-I{srcdir}β, β-sharedβ, β-fPICβ,β-std=c99", f"-L{hip_lib_dir}β, β-lamdhip64β, β-oβ, so ]) else: cc_cmd = [ cc, src, β-O3β, fβ-I{cu_include_dir}β, fβ-I{py_include_dir}β, fβ-I{srcdir}β, β-sharedβ, β-fPICβ, β-lcudaβ,β-std=c99", β-oβ, so ] cc_cmd += [f"-L{dir}" for dir in cuda_lib_dirs] ret = subprocess.check_call(cc_cmd)