vllm: [Bug]: `v0.8.5`: Special tokens (``, ``) are split during streaming with Qwen3-FP8

Your current environment

The output of `python collect_env.py`
Your output of `python collect_env.py` here

🐛 Describe the bug

When serving the Qwen3-FP8 model using vLLM v0.8.5 and enabling streaming output, special tokens like <think> and </think> (used for reasoning steps) are not treated as atomic units. Instead, they are often split across multiple streamed chunks.

This issue occurs both when running the server with the basic command:

vllm serve Qwen/Qwen3-8B-FP8
vllm serve Qwen/Qwen3-8B-FP8 --enable-reasoning --reasoning-parser deepseek_r1

Steps to Reproduce:

  1. Install vLLM v0.8.5.
  2. Start the vLLM server using either of the commands mentioned above.
  3. Send a request to the server’s generation endpoint (e.g., /v1/chat/completions or /generate) with stream=True. Use a prompt that is likely to cause the Qwen/Qwen3-8B model to output <think>…</think> blocks.

Expected Behavior:

Special tokens like <think> and </think> should be treated as atomic units by the tokenizer/detokenizer during streaming. Each special token should be contained entirely within a single streamed chunk.

Example of expected chunk sequence: Chunk 1: <think> Chunk 2: \n Chunk 3: Okay Chunk 4: \n Chunk 5: </think> Chunk 6: …final answer…

Actual Behavior:

The special tokens are split across chunk boundaries.

Example of actual chunk sequence observed: Chunk 1: <th Chunk 2: i Chunk 3: nk>\n Chunk 4: Okay Chunk 5: \n< Chunk 6: / Chunk 7: t Chunk 8: hin Chunk 9: k Chunk 10: >\n Chunk 11: …final answer…

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (8 by maintainers)

Most upvoted comments

FWIW, I’m having the exact same issue, with same vllm start params. Issue does not seem present in other reasoning models, though, so I wonder if it’s specific to Qwen3 FP8 variants.

I think it is caused by the model, instead of our implementation. We get the generated text token by token. We expect the whole <think> in one generation instead of a single char from the model.

Maybe we could give it a try with huggingface transformers first.

Indeed. I also want to know. Interesting.