vllm: [Bug]: Gemma-3 models output empty text.

Your current environment

Using the latest main branch self-compiled LLM engine (v0.7.4.dev418+gd47807ba) to run the Gemma-3 model, only the Gemma-3-1b-it model outputs text content normally, while the 4/12/27B models output empty text.
### Running Environment
Ubuntu 20.04
Python 3.12
CUDA 12.0
CPU - 16 cores
Memory - 64GB

### Command executed:
vllm serve google/gemma-3-4b-it --tensor-parallel-size 2 --dtype=half --gpu-memory-utilization 0.9 --max-model-len=17370

🐛 Describe the bug

curl http://localhost:8000/v1/completions \
>     -H "Content-Type: application/json" \
>     -d '{
>         "model": "google/gemma-3-4b-it",
>         "prompt": "who are you?",
>         "stream": true
> }'
data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":null}

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (8 by maintainers)

Most upvoted comments

Looks like your GPU is too old. As a last resort you can try using float32 but it will increase memory usage

  1. You should use system role, not developer
  2. Gemma3 is trained on bfloat16, so changing the dtype may result in numeric instability. Try removing that argument so that bfloat16 is used automatically.

You need to also fix the dtype problem as I mentioned