vllm: [Bug]: Paligemma support for PNG files

Your current environment

The output of `python collect_env.py`
Collecting environment information...
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.30.0
Libc version: glibc-2.35

Python version: 3.10.14 (main, May  6 2024, 19:42:50) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.5.0-1023-azure-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100 80GB PCIe
GPU 1: NVIDIA A100 80GB PCIe
GPU 2: NVIDIA A100 80GB PCIe
GPU 3: NVIDIA A100 80GB PCIe

Nvidia driver version: 535.183.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             96
On-line CPU(s) list:                0-95
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 7V13 64-Core Processor
CPU family:                         25
Model:                              1
Thread(s) per core:                 1
Core(s) per socket:                 48
Socket(s):                          2
Stepping:                           1
BogoMIPS:                           4890.88
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm
Hypervisor vendor:                  Microsoft
Virtualization type:                full
L1d cache:                          3 MiB (96 instances)
L1i cache:                          3 MiB (96 instances)
L2 cache:                           48 MiB (96 instances)
L3 cache:                           384 MiB (12 instances)
NUMA node(s):                       4
NUMA node0 CPU(s):                  0-23
NUMA node1 CPU(s):                  24-47
NUMA node2 CPU(s):                  48-71
NUMA node3 CPU(s):                  72-95
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] torch==2.3.0
[pip3] torchvision==0.18.0
[pip3] transformers==4.42.4
[pip3] triton==2.3.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] torch                     2.3.0                    pypi_0    pypi
[conda] torchvision               0.18.0                   pypi_0    pypi
[conda] transformers              4.42.4                   pypi_0    pypi
[conda] triton                    2.3.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	GPU1	GPU2	GPU3	NIC0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV12	SYS	SYS	NODE	0-23	0		N/A
GPU1	NV12	 X 	SYS	SYS	SYS	24-47	1		N/A
GPU2	SYS	SYS	 X 	NV12	SYS	48-71	2		N/A
GPU3	SYS	SYS	NV12	 X 	SYS	72-95	3		N/A
NIC0	NODE	SYS	SYS	SYS	 X 				

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0

🐛 Describe the bug

PNG files don’t seem to work for paligemma-3b-mix-448.

To test, try the following command: python -m vllm.entrypoints.openai.api_server --model google/paligemma-3b-mix-448 --chat-template examples/template_llava.jinja" on the server.

Then, test this command using:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/paligemma-3b-mix-448",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What’s in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://placehold.co/600x400/png"
            }
          }
        ]
      }
    ],
    "max_tokens": 300
  }'

Error Traceback Output:

ERROR 07-15 00:01:58 async_llm_engine.py:54]   File "/home/lmsys/vllm/vllm/multimodal/registry.py", line 93, in map_input                                                                          [41/1414]
ERROR 07-15 00:01:58 async_llm_engine.py:54]     .map_input(model_config, data_value)
ERROR 07-15 00:01:58 async_llm_engine.py:54]   File "/home/lmsys/vllm/vllm/multimodal/base.py", line 213, in map_input
ERROR 07-15 00:01:58 async_llm_engine.py:54]     return mapper(InputContext(model_config), data)
ERROR 07-15 00:01:58 async_llm_engine.py:54]   File "/home/lmsys/vllm/vllm/multimodal/image.py", line 122, in _default_input_mapper
ERROR 07-15 00:01:58 async_llm_engine.py:54]     batch_data = image_processor \
ERROR 07-15 00:01:58 async_llm_engine.py:54]   File "/home/lmsys/miniconda3/envs/vllm-source/lib/python3.10/site-packages/transformers/models/siglip/image_processing_siglip.py", line 233, in preprocess
ERROR 07-15 00:01:58 async_llm_engine.py:54]     input_data_format = infer_channel_dimension_format(images[0])
ERROR 07-15 00:01:58 async_llm_engine.py:54]   File "/home/lmsys/miniconda3/envs/vllm-source/lib/python3.10/site-packages/transformers/image_utils.py", line 255, in infer_channel_dimension_format
ERROR 07-15 00:01:58 async_llm_engine.py:54]     raise ValueError("Unable to infer channel dimension format")
ERROR 07-15 00:01:58 async_llm_engine.py:54] ValueError: Unable to infer channel dimension format

However, if we test using a jpg image:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/paligemma-3b-mix-448",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What’s in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://placehold.co/600x400/jpg"
            }
          }
        ]
      }
    ],
    "max_tokens": 300
  }'

Output:

{"id":"cmpl-7e2f12b67ab74eaeb0afd4d72d253540","object":"chat.completion","created":1721001808,"model":"google/paligemma-3b-mix-448","choices":[{"index":0,"message":{"role":"assistant","content":"Sorry, as a base VLM I am not trained to answer this question.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":1040,"total_tokens":1057,"completion_tokens":17}}

I believe that the reason why this is the case is because SigLip has a default num_channels parameter that is set to 3. When we take in PNG images, PNG images can have 4 channels (RGBA), which can lead to this mismatch. I discovered this mismatch when I was trying to load images using Image.open(image_url).convert('RGBA') and then realized that passing these images into vllm would not work due to the above error.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (9 by maintainers)

Most upvoted comments

@JanuRam I don’t think that the model should be used for chat responses. You will not receive content that is very meaningful. Try by using the llava template. However, I would say that chat is probably not the use case that you would want to use this model for. If you are looking for chat, you should try https://huggingface.co/openbmb/MiniCPM-V-2_6

python -m vllm.entrypoints.openai.api_server \
    --model google/paligemma-3b-mix-224 \
    --chat-template template_llava.jinja

thank you! works now 😃