vllm: [Bug]: No output / Repeated outputs when using Gemma 3 on vLLM
Your current environment
The output of python collect_env.py
Your output of `python collect_env.py` here
š Describe the bug
Iām running the google/gemma-3-27b-it model with vLLM using the OpenAI-compatible API server.
CUDA_VISIBLE_DEVICES=0 VLLM_USE_V1=1 python /opt/VLLM/vllm/vllm/entrypoints/openai/api_server.py
āmodel /opt/MODELS/gemma-3-27b-it/
āmax-model-len 32000
āhost 10.12.112.168
āport 9005
ātensor-parallel-size 1
āgpu_memory_utilization 0.9
Then, I send a standard request to the /v1/chat/completions endpoint using Python:
import requests import json
url = āhttp://10.12.112.168:9005/v1/chat/completionsā
data = { āmodelā: ā/opt/MODELS/gemma-3-27b-it/ā, āmessagesā: [ {āroleā: āuserā, ācontentā: āhelloā} ], ātemperatureā: 0.1, āmax_tokensā: 500, āenable_thinkingā: False }
headers = { āContent-Typeā: āapplication/jsonā }
response = requests.post(url, headers=headers, data=json.dumps(data)) result = response.json() print(result[āchoicesā][0][āmessageā][ācontentā])
The request is processed, but the model fails to produce meaningful responses. It either:
outputs nothing,
or keeps repeating certain tokens or parts of the input (e.g., repeating āselamlar bromā).
This issue only happens with Gemma 3 IT models. I tested the exact same code and server setup with:
Qwen models
Mistral models ā¦and they work perfectly. No repetition, and responses are coherent and aligned with the prompt.
So this looks like a Gemma-specific compatibility issue with /chat/completions, possibly due to missing or misaligned prompt formatting (e.g., lack of a compatible chat template?).
Let me know if thereās a known workaround or proper configuration required for Gemma models.
Thanks!
Before submitting a new issueā¦
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: open
- Created a year ago
- Reactions: 2
- Comments: 16
I solved the problem in the following way: pip install transformers==4.51.1 The default installed version is 4.53.0, and I think it may be due to incompatibility between this version of transformers and gemma-3
I also confirm: Transformers 4.52.2 works.
But I donāt know whether this is a problem only related to the recent Transformers version (need to try to generate with it) or its integration with vLLM.
The issue was caused by the transformers library version. Using transformers==4.52.2 resolved the problem.
Iāve also been observing this exact behavior on v0.9.1 with Gemma3 IT models ā they have completely broken outputs. I tried using the V0 engine and dtype bfloat16, which did not fix the problem. Thereās also https://github.com/vllm-project/vllm/issues/17689 and https://github.com/vllm-project/vllm/issues/20060 which are potentially related.
In my case the version of transformers is 4.52.4 (0.9.0.1 vLLM version), but still there is a problem with tokens repetition on H100 GPU