vllm: [Bug]: Qwen/Qwen2.5-1.5B-Instruct generates out of vocabulary tokens

Your current environment

>>> import vllm
INFO 02-12 20:27:04 __init__.py:190] Automatically detected platform cuda.
>>> vllm.__version__
'0.7.2'

🐛 Describe the bug

Hi,

It looks like Qwen models can generate tokens out of vocabulary. We can see this by feeding the generate tokens to the model which sometimes result in the following exception: Token id 151779 is out of vocabulary. Here is a minimal code to reproduce this error.

import vllm
from transformers import AutoTokenizer
import numpy as np

PROMPT = """
<|im_start|>system
Please reason step by step, and put your final answer within \\boxed{}.<|im_end|>
<|im_start|>user
The equation $a^7xy-a^6y-a^5x=a^4(b^4-1)$ is equivalent to the equation $(a^mx-a^n)(a^py-a^2)=a^4b^4$ for some integers $m$, $n$, and $p$.  Find $mnp$.<|im_end|>
<|im_start|>assistant
"""

if __name__ == '__main__':
    model_path = "Qwen/Qwen2.5-1.5B-Instruct"
    tokenizer = AutoTokenizer.from_pretrained(model_path)
    PROMPT_TOKEN_IDS = tokenizer.encode(PROMPT)

    sampling_params = vllm.SamplingParams(temperature=1.2, max_tokens=100)
    llm = vllm.LLM(model_path)

    # can we now generate tokens out of vocabulary?
    out_of_vocab = []
    out_of_vocab_tokens = []
    for i in range(100):
        out = llm.generate(prompt_token_ids=PROMPT_TOKEN_IDS, sampling_params=sampling_params)
        PROMPT_COMPLETION_TOKEN_IDS = PROMPT_TOKEN_IDS + list(out[0].outputs[0].token_ids)
        try:
            out2 = llm.generate(prompt_token_ids=PROMPT_COMPLETION_TOKEN_IDS, sampling_params=sampling_params)
            out_of_vocab.append(0)
        except Exception as e:
            print(e)
            # Extract token id from error message
            token_id = int(str(e).split("Token id ")[1].split(" ")[0])
            out_of_vocab_tokens.append(token_id)
            out_of_vocab.append(1)
    
    print(f"Proportion of out of vocabulary generations: {np.mean(out_of_vocab)}")
    print(out_of_vocab_tokens)
        

selected output

Token id 151779 is out of vocabulary
Token id 151734 is out of vocabulary
...
Proportion of out of vocabulary generations: 0.03
[151925, 151779, 151734]

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 11
  • Comments: 28 (5 by maintainers)

Most upvoted comments

Since a lot of people are commenting about this, here’s a simple explanation for why this happens:

Qwen and some other models come with a few hundred extra out-of-vocab tokens that can be used for fine-tuning in case new special domain-specific tokens are required.

There’s absolutely nothing special about these tokens in terms of the model architecture, they just represent the last 400 or so dimensions of the logits vector. They just don’t have a corresponding token in the tokenizer.

This means that these tokens do have a non-zero probability of being sampled. The probability of sampling any individual token is typically exceptionally low. Still, the probability of sampling any one of the ~400 non-vocab tokens is just high enough such that it happens once in a blue moon.

I don’t believe it’s a problem of vLLM, but an intended feature of the model. vLLM already has numerous tools to prevent these tokens from being sampled, even some very conservative sampling strategy like top_k=10000 or top_p=0.999 should be sufficient to completely filter those out.

This issue still happens in vllm 0.8.5.2.post1.

I understand this is not a problem of vLLM but severely affects my RL for LLM setup. It is because during RL tuning the probabilities of these tokens gets fluctuating and eventually they are sampled and throwing errors everywhere.

What is the best fix?

Since a lot of people are commenting about this, here’s a simple explanation for why this happens:

Qwen and some other models come with a few hundred extra out-of-vocab tokens that can be used for fine-tuning in case new special domain-specific tokens are required.

There’s absolutely nothing special about these tokens in terms of the model architecture, they just represent the last 400 or so dimensions of the logits vector. They just don’t have a corresponding token in the tokenizer.

This means that these tokens do have a non-zero probability of being sampled. The probability of sampling any individual token is typically exceptionally low. Still, the probability of sampling any one of the ~400 non-vocab tokens is just high enough such that it happens once in a blue moon.

I don’t believe it’s a problem of vLLM, but an intended feature of the model. vLLM already has numerous tools to prevent these tokens from being sampled, even some very conservative sampling strategy like top_k=10000 or top_p=0.999 should be sufficient to completely filter those out.

For me the real problem is, in 0.6.3 this error never happens when using the same model.

@AlexPiche @QPHutu在最新的 vLLM 版本 0.8.2 中这仍然是一个问题吗?

Hi! same issue happens in vllm 0.8.2:

  File "/opt/tiger/ray/session_2025-04-06_01-09-11_162097_1904/runtime_resources/working_dir_files/_ray_pkg_1e3d598f7c72e7fd/verl/workers/rollout/vllm_rollout/vllm_rollout_spmd.py", line 247, in generate_sequences
    outputs = self.inference_engine.generate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/utils.py", line 1072, in inner
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/llm.py", line 457, in generate
    self._validate_and_add_requests(
  File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/llm.py", line 1308, in _validate_and_add_requests
    self._add_request(
  File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/llm.py", line 1326, in _add_request
    self.llm_engine.add_request(
  File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/llm_engine.py", line 184, in add_request
    request = self.processor.process_inputs(request_id, prompt, params,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/processor.py", line 207, in process_inputs
    self._validate_model_inputs(processed_inputs, lora_request)
  File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/processor.py", line 318, in _validate_model_inputs
    raise ValueError(
ValueError: Token id 151726 is out of vocabulary
            max_token_id = max(self.tokenizer.get_vocab().values())
            kwargs["allowed_token_ids"] = list(range(max_token_id + 1))

Something like this, in vllm_rollout_spmd.py, before update_sampling_params.

Same issue, setting top-p to 0.99 still cannot totally avoid this problem