vllm: [Performance]: guided generation is very slow in offline mode

Proposal to improve performance

With a single request / online mode I’m getting:

  • no guided 300 tok/sec
  • outlines 150 tok/sec (2x slower)
  • lm-format-enforcer 90 tok/sec (~3x slower)

with offline mode I get:

  • outlines is about 10-20x slower than no guided generation
  • lm-format-enforcer is about 4x faster than outlines (note that it is slower than outlines for online)

for online I was using this schema:

json_template = {
    "type": "object",
    "properties": {
        "criteria": {"type": "array", "items": {"type": "string"}, "minItems": 1},
        "response": { "type": "string" }
    },
    "required": ["criteria", "response"]
}

for offline I was using an even simpler schema:


{
   "type":"object",
   "properties":{
      "name":{
         "type":"string", "minLength":2, "maxLength":5
      },
      "age":{
         "type":"integer"
      }
   },
   "required":[ "name", "age"]
}

the huge performance hit in the offline mode is very strange for both backends.

2x slow down in the online mode is pretty bad too as it’s already a huge impact. The offline mode can actually tolerate 2x no problem as there is no human in the loop, but 10-20x is a way impractical.

vllm=0.6.0 and outlines==0.0.46

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 9
  • Comments: 21 (8 by maintainers)

Most upvoted comments

QQ - are you planning to open up a PR to fix this? We would definitely appreciate a contribution if you have the bandwidth

The summary of things so far:

  1. one problem proved to be a repeated recomputation of FSM in outlines because the cache was being ignored in vllm integration. @Lap1n has proposed a PR to fix it here https://github.com/dottxt-ai/outlines/pull/1129

  2. the other issue is re-compiling of the regex on each request - I proposed to add it to this other PR by @Lap1n: https://github.com/vllm-project/vllm/pull/8308#issuecomment-2342517212 and opened an Issue https://github.com/vllm-project/vllm/issues/8383

  3. the truncated JSON is being actively discussed in this Issue: https://github.com/vllm-project/vllm/issues/8350

For the bigger issue that started the current main issue in the OP I’m blocked by my colleague to give me a larger repro test case, so that I could verify it and if true reduce it to a small repro case, so please bear with me until they come through.

Hi, is this issue still relevant?

@stas00 I also got the same issue, seems like an issue with the key’s format that Outlines uses to store cached data. I opened a PR a few days ago to fix this on Outlines’ repo.

the other mismatch thing I noticed is that vllm strips spaces between elements of json:

# outlines:
{"name": "John", "age": 29}{"name": "John", "age": 30}

# vllm:
{"name":"John","age":25}{"name":"John","age":25}

I guess that saves a few tokens from needing to be generated w/o compromising the output structure - nice!

@Lap1n, I can’t try v0.5.2 since it didn’t support guided generated in the offline mode.

But I think the problem here is something entirely different, I’m trying to dig to the root of it.

As I’m writing the offline repro scripts w/ TinyLlama-1.1B-Chat-v0.6 vllm is only 10% slower w/ outlines than w/o them once FSM has been compiled - which is absolutely normal and expected. So something else is going on - will update once I have a more clear picture.

It’s possible that there is a multitude of issues here, and it’s possible that somehow the big overhead is model-dependent (even though in theory the type or a size of a model shouldn’t matter at all overhead-wise).

@stas00 In my experience, guided generation is always slower than normal. I recommend you try sglang instead. Sglang achieves better throughput than vLLM, but the guided generation is still slower.

Hey @Quang-elec44 - this is actually not true as of vllm’s current release: https://blog.vllm.ai/2024/09/05/perf-update.html

We also have more performance optimizations that will come out in v0.6.0: