vllm: [Performance]: guided generation is very slow in offline mode
Proposal to improve performance
With a single request / online mode I’m getting:
- no guided 300 tok/sec
outlines150 tok/sec (2x slower)lm-format-enforcer90 tok/sec (~3x slower)
with offline mode I get:
outlinesis about 10-20x slower than no guided generationlm-format-enforceris about 4x faster thanoutlines(note that it is slower thanoutlinesfor online)
for online I was using this schema:
json_template = {
"type": "object",
"properties": {
"criteria": {"type": "array", "items": {"type": "string"}, "minItems": 1},
"response": { "type": "string" }
},
"required": ["criteria", "response"]
}
for offline I was using an even simpler schema:
{
"type":"object",
"properties":{
"name":{
"type":"string", "minLength":2, "maxLength":5
},
"age":{
"type":"integer"
}
},
"required":[ "name", "age"]
}
the huge performance hit in the offline mode is very strange for both backends.
2x slow down in the online mode is pretty bad too as it’s already a huge impact. The offline mode can actually tolerate 2x no problem as there is no human in the loop, but 10-20x is a way impractical.
vllm=0.6.0 and outlines==0.0.46
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 9
- Comments: 21 (8 by maintainers)
The summary of things so far:
one problem proved to be a repeated recomputation of FSM in
outlinesbecause the cache was being ignored in vllm integration. @Lap1n has proposed a PR to fix it here https://github.com/dottxt-ai/outlines/pull/1129the other issue is re-compiling of the regex on each request - I proposed to add it to this other PR by @Lap1n: https://github.com/vllm-project/vllm/pull/8308#issuecomment-2342517212 and opened an Issue https://github.com/vllm-project/vllm/issues/8383
the truncated JSON is being actively discussed in this Issue: https://github.com/vllm-project/vllm/issues/8350
For the bigger issue that started the current main issue in the OP I’m blocked by my colleague to give me a larger repro test case, so that I could verify it and if true reduce it to a small repro case, so please bear with me until they come through.
Hi, is this issue still relevant?
@stas00 I also got the same issue, seems like an issue with the key’s format that Outlines uses to store cached data. I opened a PR a few days ago to fix this on Outlines’ repo.
the other mismatch thing I noticed is that vllm strips spaces between elements of json:
I guess that saves a few tokens from needing to be generated w/o compromising the output structure - nice!
@Lap1n, I can’t try v0.5.2 since it didn’t support guided generated in the offline mode.
But I think the problem here is something entirely different, I’m trying to dig to the root of it.
As I’m writing the offline repro scripts w/ TinyLlama-1.1B-Chat-v0.6 vllm is only 10% slower w/ outlines than w/o them once FSM has been compiled - which is absolutely normal and expected. So something else is going on - will update once I have a more clear picture.
It’s possible that there is a multitude of issues here, and it’s possible that somehow the big overhead is model-dependent (even though in theory the type or a size of a model shouldn’t matter at all overhead-wise).
Hey @Quang-elec44 - this is actually not true as of vllm’s current release: https://blog.vllm.ai/2024/09/05/perf-update.html
We also have more performance optimizations that will come out in v0.6.0: