vllm: [Bug]: guided generation can't always finish generating the requested structure

So it appears that guided generation returns the requested structure like json only if the model has an infinite number of tokens it can generate, but otherwise very often it fails to close the structure, e.g. if it’s a simple { "key": "value" } json schema and the new tokens are limited, it’ll often return { "key": "value.

I understand why this is happening - it’s because the guiding can only guide when it has a subset of legal tokens to be used next, but if it’s any token it can exhaust the full max_length and by the time it discovered it’s unfinished it’s too late to wrap up the structure.

Here is how to reproduce this problem:

from vllm import LLM, SamplingParams

model_name_or_path = "TinyLlama/TinyLlama-1.1B-Chat-v0.6"

schema = '{ "type": "object", "properties": { "age": { "type": "integer"}, "description": { "type": "string"} }, "required": ["age", "description"] }'

model = LLM(
    model=model_name_or_path,
    tokenizer=model_name_or_path,
    tokenizer_mode="auto",
    tensor_parallel_size=1,
    trust_remote_code=True,
    enforce_eager=True,
    dtype="bfloat16",
    gpu_memory_utilization=0.8,
    guided_decoding_backend="outlines",
)

prompt = "Give an example of a person's profile that fits this JSON schema: {schema}"

sampling_params = SamplingParams(
    temperature=1.0,
    seed=None,
    max_tokens=25,
)
kwargs = dict(
    sampling_params=sampling_params,
    guided_options_request=dict(
        guided_json=schema,
    ),
)

outputs = model.generate([prompt]*10, **kwargs)
for output in outputs:
    response = output.outputs[0].text
    print(response)

gives:

{"age":25, "description":"often misquoted as drinking soup and
{"age":22,"description": "My candidate profile presented here wears sunglasses,
{"age": 70, "description": "Nicky is in his 70's
{"age":14,"description":"Grandparent"}
{"age": 21,"description": "Could you tailor this schema to fit my specific industry by
{"age":22,"description":"I am a toast of the city. I am right by your side, always
{"age": 30,"description": "Fast and furious fan! Willing to work long hours to ful
{"age":5,"description":"Test user description..."}
{"age":30,"description":"I tend towards neosystem-oriented design with a
{ "age": 20, "description": "An enthusiastic and hard-working individual

It’s quite obvious what the problem is here.

With lm-format-enforcer backend things are even worse.

I then went to the origin and re-wrote the same code using outlines json generator and the problem is the same:

import outlines

model_name_or_path = "TinyLlama/TinyLlama-1.1B-Chat-v0.6"

schema = '{ "type": "object", "properties": { "age": { "type": "integer"}, "description": { "type": "string"} }, "required": ["age", "description"] }'

model = outlines.models.transformers(model_name_or_path, device='cuda:0')
generator = outlines.generate.json(model, schema)

prompt = "Give an example of a person's profile that fits this JSON schema: {schema}"

for i in range(10):
    response = generator(prompt, max_tokens=25)
    print(response)

gives

$ python outlines-direct-truncation.py
Compiling FSM index for all state transitions: 100%|████████████████████████████████████████████████████| 42/42 [00:00<00:00, 162.11it/s]
{'age': 50, 'description': 'Love to doggy-couting for fun!'}
Traceback (most recent call last):
  File "/mnt/nvme0/code/contextual/core-faster-ttft/dawn/exp/infer/faster-ttft/outlines-direct-truncation.py", line 16, in <module>
    response = generator(prompt, max_tokens=25)
  File "/home/stas/anaconda3/envs/py310-pt22/lib/python3.10/site-packages/outlines/generate/api.py", line 230, in __call__
    formatted = [self.format_sequence(sequence) for sequence in stripped]
  File "/home/stas/anaconda3/envs/py310-pt22/lib/python3.10/site-packages/outlines/generate/api.py", line 230, in <listcomp>
    formatted = [self.format_sequence(sequence) for sequence in stripped]
  File "/home/stas/anaconda3/envs/py310-pt22/lib/python3.10/site-packages/outlines/generate/json.py", line 60, in <lambda>
    generator.format_sequence = lambda x: pyjson.loads(x)
  File "/home/stas/anaconda3/envs/py310-pt22/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/home/stas/anaconda3/envs/py310-pt22/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/stas/anaconda3/envs/py310-pt22/lib/python3.10/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 26 (char 25)

as you can see, it succeeds on the first request and fails to generate valid json on a second request.

Making max_tokens=50 gives more completed jsons, but some still fail.

While this example is contrived to be simple to understand, this problem can occur at any max_tokens if the greedy any character regex stage catches the model at the boundery of max_tokens.

Currently the workarounds I have been using is to make the schema more strict and define minLength and maxItems - but obviously this makes the generation far from ideal.

I also added a retry mechanism, which is very inefficient at the moment since it re-tries the whole thing. The efficient retry would be to chop off some ending and feed the now longer prompt which includes most of the generated output to the model and only get it to generate a different ending which hopefully would finish the structure.

Since the problem appears to be to close the structure, my feeling is that the correct solution is to identify the closing pattern and generate with max_tokens minus length of the closing structure and then somehow append it by changing that regex to fast-forward to the closure.

I realize the problem comes from the back-ends but perhaps we have more interested people here to discuss the correct algorithm to resolve this and then we could share the solution with the back-ends. At the very least it’d be good to document that currently guided generation in vllm is not guaranteed to work.

Using latest vllm==0.6.0 and outlines==0.0.46 here.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (14 by maintainers)

Most upvoted comments

so we ended up using json_repair to solve this issue.

That’s a great idea, Nick!

So I tried a simple logit processor that promotes the select few tokens to the top towards the end of the context window and it works!

The POC logit processor:

end_chars = list('"}')
end_token_ids = [tokenizer.encode(x)[1] for x in end_chars]
def json_wrap_up(input_ids, logits):
    cur_len = len(input_ids)
    if cur_len > 10:
        logits[end_token_ids] += 10

    return logits

end-to-end code:

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_name_or_path = "TinyLlama/TinyLlama-1.1B-Chat-v0.6"

schema = '{ "type": "object", "properties": { "age": { "type": "integer"}, "description": { "type": "string"} }, "required": ["age", "description"] }'
#schema = '{ "type": "object", "properties": { "description": { "type": "string", "maxLength": 15 }, "age": { "type": "integer"} }, "required": ["age", "description"] }'

model = LLM(
    model=model_name_or_path,
    tokenizer=model_name_or_path,
    tokenizer_mode="auto",
    tensor_parallel_size=1,
    trust_remote_code=True,
    enforce_eager=True,
    dtype="bfloat16",
    gpu_memory_utilization=0.8,
    guided_decoding_backend="outlines",
)

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
end_chars = list('"}')
end_token_ids = [tokenizer.encode(x)[1] for x in end_chars]

prompt = "Give an example of a person's profile that fits this JSON schema: {schema}"

def json_wrap_up(input_ids, logits):
    cur_len = len(input_ids)
    if cur_len > 10:
        logits[end_token_ids] += 10

    return logits

sampling_params = SamplingParams(
    temperature=1.0,
    seed=None,
    max_tokens=25,
    logits_processors=[json_wrap_up]
)
kwargs = dict(
    sampling_params=sampling_params,
    guided_options_request=dict(
        guided_json=schema,
    ),
)

outputs = model.generate([prompt]*10, **kwargs)
for output in outputs:
    response = output.outputs[0].text
    print(response)

running it:

{"age":25, "description":"o " }
{"age":22,"description": "My " }
{"age": 70, "description": "Nicky " }
{"age":14,"description":"Grand " }
{"age": 21,"description": "Could " }
{"age":22,"description":"I am a " }
{"age": 30,"description": "Fast " }
{"age":5,"description":"Test user description... } " }
{"age":30,"description":" } " }
{ "age": 20, "description": " " }

all valid json endings! albeit the algorithm needs some polishing not to have } in the string and not hardcode the boundary numbers of course. And as you said probably need to include whole tokens of combined characters like "} as a single token. And of course, auto-deriving the end of schema tokens.