vllm: [Bug]: Speculative decoding inconsistency for Qwen-Coder-32B
Your current environment
The output of `python collect_env.py`
Output of `python collect_env.py` was not collected because I do not have wget within my docker container. I can attempt a docker cp command to put the collect_env.py within my container later, but haven't yet.
Model Input Dumps
Running VLLM with docker. Speculative decoding for the Qwen-coder-32B using the 0.5B model does not work. Note that all the Qwen models described are from the official Qwen AWQ repos on Huggingface.
Here is the relevant section of docker-compose.yml:
command: > –model /app/models/Qwen2.5-Coder-32B-Instruct-AWQ –tensor_parallel_size 2 –max-model-len 23568 –enable-auto-tool-choice –tool-call-parser hermes –speculative_model=“/app/models/Qwen2.5-Coder-0.5B-Instruct-AWQ” –num_speculative_tokens=5
However, curiously, the 7B model DOES work.
command: > –model /app/models/Qwen2.5-Coder-32B-Instruct-AWQ –tensor_parallel_size 2 –max-model-len 23568 –enable-auto-tool-choice –tool-call-parser hermes –speculative_model=“/app/models/Qwen2.5-Coder-7B-Instruct-AWQ” –num_speculative_tokens=5
A write-up on HuggingFace says that the error could be due to a difference in the vocabulary size between 0.5/3B and 7/32B. The 7B model works well enough on my dual-3090 setup, but I would love to save the VRAM and get even faster performance from using the 0.5B model for speculative decoding.
🐛 Describe the bug
Speculative decoding for the Qwen-coder-32B using the 0.5B model does not work. However, using the 7B model for speculative decoding DOES work. Possibly due to differing vocab-sizes.
- Note that all the Qwen models described are from the official Qwen AWQ repos on Huggingface: https://huggingface.co/collections/Qwen/qwen25-coder-66eaa22e6f99801bf65b0c2f
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 23 (6 by maintainers)
@taltoris You also need to concat a padding tensor to the
model.embed_tokens.weighttensor. See https://github.com/vllm-project/vllm/issues/5203.That is weird, 151643 it the begin_of_sequence token.I was getting the vocab_size from here: https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct/blob/main/config.json and https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B/blob/main/config.json
Thanks for your reply @ShangmingCai! I see I made a flawed assumption: I thought that the draft vocabulary here was a strict subset of that of the main model, and that the workaround applied to this case. TY for clearing that up 😃
@SlapDrone In principle, the vocabulary size of the draft model and the target model should be consistent, which is the premise for speculative decoding to work. I personally confirmed with the Qwen team that the slight difference in Qwen’s vocab_size is caused by their distributed training envs. They have different padding sizes. And the index over 151646 is meaningless, which means we can pad the draft model embedding. After all, indexes over 151646 for the target model are meaningless, so our workaround will do no harm.
Since different models’ vocabulary sizes and situations may differ, this kind of fix is not strictly correct and has to be done case by case, which is really difficult.