vllm: [Bug]: No output / Repeated outputs when using Gemma 3 on vLLM

Your current environment

The output of python collect_env.py
Your output of `python collect_env.py` here

šŸ› Describe the bug

I’m running the google/gemma-3-27b-it model with vLLM using the OpenAI-compatible API server.

CUDA_VISIBLE_DEVICES=0 VLLM_USE_V1=1 python /opt/VLLM/vllm/vllm/entrypoints/openai/api_server.py
–model /opt/MODELS/gemma-3-27b-it/
–max-model-len 32000
–host 10.12.112.168
–port 9005
–tensor-parallel-size 1
–gpu_memory_utilization 0.9

Then, I send a standard request to the /v1/chat/completions endpoint using Python:

import requests import json

url = ā€œhttp://10.12.112.168:9005/v1/chat/completionsā€

data = { ā€œmodelā€: ā€œ/opt/MODELS/gemma-3-27b-it/ā€, ā€œmessagesā€: [ {ā€œroleā€: ā€œuserā€, ā€œcontentā€: ā€œhelloā€} ], ā€œtemperatureā€: 0.1, ā€œmax_tokensā€: 500, ā€œenable_thinkingā€: False }

headers = { ā€œContent-Typeā€: ā€œapplication/jsonā€ }

response = requests.post(url, headers=headers, data=json.dumps(data)) result = response.json() print(result[ā€˜choices’][0][ā€˜message’][ā€˜content’])

The request is processed, but the model fails to produce meaningful responses. It either:

outputs nothing,

or keeps repeating certain tokens or parts of the input (e.g., repeating ā€œselamlar bromā€).

This issue only happens with Gemma 3 IT models. I tested the exact same code and server setup with:

Qwen models

Mistral models …and they work perfectly. No repetition, and responses are coherent and aligned with the prompt.

So this looks like a Gemma-specific compatibility issue with /chat/completions, possibly due to missing or misaligned prompt formatting (e.g., lack of a compatible chat template?).

Let me know if there’s a known workaround or proper configuration required for Gemma models.

Thanks!

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Reactions: 2
  • Comments: 16

Most upvoted comments

I solved the problem in the following way: pip install transformers==4.51.1 The default installed version is 4.53.0, and I think it may be due to incompatibility between this version of transformers and gemma-3

I also confirm: Transformers 4.52.2 works.

But I don’t know whether this is a problem only related to the recent Transformers version (need to try to generate with it) or its integration with vLLM.

The issue was caused by the transformers library version. Using transformers==4.52.2 resolved the problem.

I’ve also been observing this exact behavior on v0.9.1 with Gemma3 IT models – they have completely broken outputs. I tried using the V0 engine and dtype bfloat16, which did not fix the problem. There’s also https://github.com/vllm-project/vllm/issues/17689 and https://github.com/vllm-project/vllm/issues/20060 which are potentially related.

In my case the version of transformers is 4.52.4 (0.9.0.1 vLLM version), but still there is a problem with tokens repetition on H100 GPU