vllm: [Bug]: llama 4 scout instruct does not support torch.compile

Your current environment

CUDA 12.6 Python 3.12 vllm 0.8.3 transformers 4.5.1 2 x h100

πŸ› Describe the bug

I’m running version v0.8.3 of vllm and v4.5.1 transformers.

Trying to bootstrap meta-llama/Llama-4-Scout-17B-16E-Instruct with β€œfp8” quant, and 128K context length on 2 x H100. I keep receiving the following error:

[dckr]: (VllmWorker rank=0 pid=1004) WARNING 04-06 10:20:04 [config.py:3785] `torch.compile` is turned on, but the model meta-llama/Llama-4-Scout-17B-16E does not support it. Please open an issue on GitHub if you want it to be supported.
[dckr]: (VllmWorker rank=0 pid=1004) WARNING 04-06 10:20:04 [config.py:3785] `torch.compile` is turned on, but the model meta-llama/Llama-4-Scout-17B-16E does not support it. Please open an issue on GitHub if you want it to be supported.
[dckr]: (VllmWorker rank=1 pid=1031) WARNING 04-06 10:20:04 [config.py:3785] `torch.compile` is turned on, but the model meta-llama/Llama-4-Scout-17B-16E does not support it. Please open an issue on GitHub if you want it to be supported.
[dckr]: (VllmWorker rank=1 pid=1031) WARNING 04-06 10:20:04 [config.py:3785] `torch.compile` is turned on, but the model meta-llama/Llama-4-Scout-17B-16E does not support it. Please open an issue on GitHub if you want it to be supported.
[dckr]: (VllmWorker rank=1 pid=1031) Process SpawnProcess-1:2:
[dckr]: CRITICAL 04-06 10:20:04 [multiproc_executor.py:49] MulitprocExecutor got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (4 by maintainers)

Most upvoted comments

Because the weights themselves aren’t fp8 quantized yet, Scout can only run with bf16 weights. However, we do support dynamic quantization of the KV cache via --kv-cache-dtype fp8. Team from RedHat is working on post training quantization of the models (cc @mgoin @eldarkurtic).

@yeqcharlotte the issue has been resolved.

For example:

vllm serve RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic -tp NUM_GPUS β€”max-model-len 16384

works fine on my end (make sure to use latest vllm)