vllm: [Bug]: `v0.8.5`: Special tokens (``, ` `) are split during streaming with Qwen3-FP8
Your current environment
The output of `python collect_env.py`
Your output of `python collect_env.py` here
🐛 Describe the bug
When serving the Qwen3-FP8 model using vLLM v0.8.5 and enabling streaming output, special tokens like <think> and </think> (used for reasoning steps) are not treated as atomic units. Instead, they are often split across multiple streamed chunks.
This issue occurs both when running the server with the basic command:
vllm serve Qwen/Qwen3-8B-FP8
vllm serve Qwen/Qwen3-8B-FP8 --enable-reasoning --reasoning-parser deepseek_r1
Steps to Reproduce:
- Install vLLM
v0.8.5. - Start the vLLM server using either of the commands mentioned above.
- Send a request to the server’s generation endpoint (e.g.,
/v1/chat/completionsor/generate) with stream=True. Use a prompt that is likely to cause the Qwen/Qwen3-8B model to output<think>…</think>blocks.
Expected Behavior:
Special tokens like <think> and </think> should be treated as atomic units by the tokenizer/detokenizer during streaming. Each special token should be contained entirely within a single streamed chunk.
Example of expected chunk sequence:
Chunk 1: <think>
Chunk 2: \n
Chunk 3: Okay
Chunk 4: \n
Chunk 5: </think>
Chunk 6: …final answer…
Actual Behavior:
The special tokens are split across chunk boundaries.
Example of actual chunk sequence observed: Chunk 1: <th Chunk 2: i Chunk 3: nk>\n Chunk 4: Okay Chunk 5: \n< Chunk 6: / Chunk 7: t Chunk 8: hin Chunk 9: k Chunk 10: >\n Chunk 11: …final answer…
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (8 by maintainers)
FWIW, I’m having the exact same issue, with same
vllmstart params. Issue does not seem present in other reasoning models, though, so I wonder if it’s specific to Qwen3 FP8 variants.I think it is caused by the model, instead of our implementation. We get the generated text token by token. We expect the whole
<think>in one generation instead of a single char from the model.Maybe we could give it a try with huggingface transformers first.
Indeed. I also want to know. Interesting.