vllm: [Bug]: llama 4 scout instruct does not support torch.compile
Your current environment
CUDA 12.6 Python 3.12 vllm 0.8.3 transformers 4.5.1 2 x h100
π Describe the bug
Iβm running version v0.8.3 of vllm and v4.5.1 transformers.
Trying to bootstrap meta-llama/Llama-4-Scout-17B-16E-Instruct with βfp8β quant, and 128K context length on 2 x H100. I keep receiving the following error:
[dckr]: (VllmWorker rank=0 pid=1004) WARNING 04-06 10:20:04 [config.py:3785] `torch.compile` is turned on, but the model meta-llama/Llama-4-Scout-17B-16E does not support it. Please open an issue on GitHub if you want it to be supported.
[dckr]: (VllmWorker rank=0 pid=1004) WARNING 04-06 10:20:04 [config.py:3785] `torch.compile` is turned on, but the model meta-llama/Llama-4-Scout-17B-16E does not support it. Please open an issue on GitHub if you want it to be supported.
[dckr]: (VllmWorker rank=1 pid=1031) WARNING 04-06 10:20:04 [config.py:3785] `torch.compile` is turned on, but the model meta-llama/Llama-4-Scout-17B-16E does not support it. Please open an issue on GitHub if you want it to be supported.
[dckr]: (VllmWorker rank=1 pid=1031) WARNING 04-06 10:20:04 [config.py:3785] `torch.compile` is turned on, but the model meta-llama/Llama-4-Scout-17B-16E does not support it. Please open an issue on GitHub if you want it to be supported.
[dckr]: (VllmWorker rank=1 pid=1031) Process SpawnProcess-1:2:
[dckr]: CRITICAL 04-06 10:20:04 [multiproc_executor.py:49] MulitprocExecutor got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
### Before submitting a new issue...
- [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18 (4 by maintainers)
Because the weights themselves arenβt fp8 quantized yet, Scout can only run with bf16 weights. However, we do support dynamic quantization of the KV cache via
--kv-cache-dtype fp8. Team from RedHat is working on post training quantization of the models (cc @mgoin @eldarkurtic).@yeqcharlotte the issue has been resolved.
For example:
vllm serve RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic -tp NUM_GPUS βmax-model-len 16384
works fine on my end (make sure to use latest vllm)