vllm: [Bug]: Sampling discrepancy between ollama and vLLM for gemma-3-27b-it et al.
🐛 Describe the bug
This is what the code looks like:
from vllm import LLM, SamplingParams
from vllm.sampling_params import GuidedDecodingParams
from pydantic import BaseModel, Field, create_model
from typing import Literal, List, Optional
import json
llm = LLM(model="google/gemma-3-27b-it", quantization="bitsandbytes", trust_remote_code=True, max_model_len=2048, max_num_seqs=100)
TYPE = "review on airlines posted on TripAdvisor"
json_schema = {'properties': {'gender': {'enum': ['male', 'female', 'non-binary', 'other', 'prefer_not_to_say'], 'title': 'Gender', 'type': 'string'}, 'prename': {'title': 'Prename', 'type': 'string'}, 'surname': {'title': 'Surname', 'type': 'string'}, 'age': {'maximum': 120, 'minimum': 0, 'title': 'Age', 'type': 'integer'}, 'overall_mood': {'enum': ['joy', 'trust', 'fear', 'surprise', 'sadness', 'disgust', 'anger', 'anticipation'], 'title': 'Overall Mood', 'type': 'string'}, 'persona_description': {'title': 'Persona Description', 'type': 'string'}, 'review_title': {'title': 'Review Title', 'type': 'string'}, 'travel_class': {'title': 'Travel Class', 'type': 'string'}, 'trip_type': {'title': 'Trip Type', 'type': 'string'}, 'number_of_passengers': {'title': 'Number Of Passengers', 'type': 'string'}, 'flight_distance': {'title': 'Flight Distance', 'type': 'string'}, 'reason_for_travel': {'title': 'Reason For Travel', 'type': 'string'}, 'airline_loyalty_program_member': {'title': 'Airline Loyalty Program Member', 'type': 'string'}, 'previous_airline_experience': {'title': 'Previous Airline Experience', 'type': 'string'}, 'booking_method': {'title': 'Booking Method', 'type': 'string'}, 'meal_preference': {'title': 'Meal Preference', 'type': 'string'}, 'entertainment_preference': {'title': 'Entertainment Preference', 'type': 'string'}}, 'required': ['gender', 'prename', 'surname', 'age', 'overall_mood', 'persona_description', 'review_title', 'travel_class', 'trip_type', 'number_of_passengers', 'flight_distance', 'reason_for_travel', 'airline_loyalty_program_member', 'previous_airline_experience', 'booking_method', 'meal_preference', 'entertainment_preference'], 'title': 'ReviewMeta', 'type': 'object'}
guided_decoding_params = GuidedDecodingParams(json=json_schema)
output = llm.generate(
[f"Return the metadata of a person who writes a {TYPE}: "],
sampling_params=SamplingParams(
temperature=0.8,
max_tokens=8192,
seed=2,
guided_decoding=guided_decoding_params)
)
authors = output[0].outputs[0].text
# convert author to object
import json
authors = json.loads(authors)
Output:
---------------------------------------------------------------------------
JSONDecodeError Traceback (most recent call last)
Cell In[7], line 14
12 # convert author to object
13 import json
---> 14 authors = json.loads(authors)
File ~/miniconda3/envs/llm_tuner/lib/python3.10/json/__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
341 s = s.decode(detect_encoding(s), 'surrogatepass')
343 if (cls is None and object_hook is None and
344 parse_int is None and parse_float is None and
345 parse_constant is None and object_pairs_hook is None and not kw):
--> 346 return _default_decoder.decode(s)
347 if cls is None:
348 cls = JSONDecoder
File ~/miniconda3/envs/llm_tuner/lib/python3.10/json/decoder.py:337, in JSONDecoder.decode(self, s, _w)
332 def decode(self, s, _w=WHITESPACE.match):
333 """Return the Python representation of ``s`` (a ``str`` instance
334 containing a JSON document).
335
336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
338 end = _w(s, end).end()
339 if end != len(s):
File ~/miniconda3/envs/llm_tuner/lib/python3.10/json/decoder.py:353, in JSONDecoder.raw_decode(self, s, idx)
344 """Decode a JSON document from ``s`` (a ``str`` beginning with
345 a JSON document) and return a 2-tuple of the Python
346 representation and the index in ``s`` where the document ended.
(...)
350
351 """
352 try:
--> 353 obj, end = self.scan_once(s, idx)
354 except StopIteration as err:
355 raise JSONDecodeError("Expecting value", s, err.value) from None
JSONDecodeError: Expecting ',' delimiter: line 984 column 1 (char 5140)
Output of output[0].outputs[0].text:
'{ "gender": "female", "prename": "Laura", "surname": "Smith", "age": 35, "overall_mood": "disgust", "persona_description": "A meticulous and independent traveler who values comfort and reliability. She often travels for leisure."\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n'
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 36 (9 by maintainers)
Hmm, this might be the case with quantization support. I’m not entirely sure if structured outputs works with dynamic quantization yet. It might be the case where the mask doesn’t support quantization here.
We currently have tests for the happy path with bfloat16. fwiw I think we haven’t tested with quantization at all, so probably a case where we need to support.
edit: I’m testing this out atm.
edit 2: torch.compile is not relevant here. It only capture the graph, which doesn’t affect with the operations of structured outputs.
edit 3: possibly a different sampling bug, not structured outputs
Hello @Flink-ddd This is great, thank you so much for the help. I appreciate it.
In general, i would also like to know (only if you are aware) using guided decoding, can we generate multiple answers from the model? I.e., say my task is to generate 5 Question Answers from a given text in JSON format, can guided decoding help with it? As far as my observations, it only outputs one JSON array even if i ask it to generate more. Is it the limitation with the parameter guided decoding itself?
Perhaps try to use
torch.compiler.is_compiling()to control your logging?For this case, I would suggest to use a modern debugger with breakpoint from higher-level fucntion to debug the workflow step by step.
Also cc @aarnphm, perhaps you can provide some insights about guided decoding?
Hi @NilsHellwig , I’m working on resolving this issue, and I’m sorry that I was delayed by some things. I’ll submit a PR when I’m done.
I am facing similar issue too but for QWEN 3 30B