vllm: [V1]: Unable to serve Qwen model on V1 alpha.

Your current environment

INFO 02-14 01:40:48 __init__.py:190] Automatically detected platform cuda.
Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.31.4
Libc version: glibc-2.35

Python version: 3.12.9 (main, Feb  5 2025, 08:49:00) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-107-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA RTX 6000 Ada Generation
Nvidia driver version: 535.171.04
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             128
On-line CPU(s) list:                0-127
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 75F3 32-Core Processor
CPU family:                         25
Model:                              1
Thread(s) per core:                 2
Core(s) per socket:                 32
Socket(s):                          2
Stepping:                           1
Frequency boost:                    enabled
CPU max MHz:                        4041.8450
CPU min MHz:                        1500.0000
BogoMIPS:                           5900.54
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca
Virtualization:                     AMD-V
L1d cache:                          2 MiB (64 instances)
L1i cache:                          2 MiB (64 instances)
L2 cache:                           32 MiB (64 instances)
L3 cache:                           512 MiB (16 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-31,64-95
NUMA node1 CPU(s):                  32-63,96-127
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.570.86
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.1
[pip3] torch==2.5.1
[pip3] torchaudio==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.48.2
[pip3] triton==3.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.7.2
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	NIC0	NIC1	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	SYS	SYS	0-31,64-95	0		N/A
NIC0	SYS	 X 	PIX				
NIC1	SYS	PIX	 X 				

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1

NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
NCCL_VERSION=2.17.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
VLLM_USAGE_SOURCE=production-docker-image
NVIDIA_CUDA_END_OF_LIFE=1
CUDA_VERSION=12.1.0
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

Hello vLLM team!

First, thank you so much for all the effort you are doing with this wonderful framework.

I am trying the new V1 alpha version and it looks like it is not properly working with some models. I am facing many issues, so I will try to organize all of them:

Deploying GGUF model

I am trying this quantization and it looks like the openai compatible server is not able to serve the model.

I know this is an alpha version, but this might help you to get a stable version. The model is correctly served using V0 version, BTW.

Command used

VLLM_USE_V1=1 python3 -m vllm.entrypoints.openai.api_server --model models/Qwen2.5-14B-Instruct/Qwen2.5-14B-Instruct-Q4_K_M.gguf  --tokenizer Qwen/Qwen2.5-14B-Instruct --host "0.0.0.0" --port 5000 --gpu-memory-utilization 0.98 --served-model-name "Qwen2.5-14B-VLLM" --max-num-batched-tokens 32768 --max-num-seqs 256 --max_model_len 32768 --generation-config config

Logs

INFO 02-14 01:35:37 __init__.py:190] Automatically detected platform cuda.
INFO 02-14 01:35:38 api_server.py:840] vLLM API server version 0.7.2
INFO 02-14 01:35:38 api_server.py:841] args: Namespace(host='0.0.0.0', port=5000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='models/Qwen2.5-14B-Instruct/Qwen2.5-14B-Instruct-Q4_K_M.gguf', task='auto', tokenizer='Qwen/Qwen2.5-14B-Instruct', skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=32768, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.98, num_gpu_blocks_override=None, max_num_batched_tokens=32768, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['Qwen2.5-14B-VLLM'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config='config', override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 02-14 01:35:57 config.py:2382] Downcasting torch.float32 to torch.float16.
INFO 02-14 01:36:02 config.py:542] This model supports multiple tasks: {'score', 'generate', 'reward', 'embed', 'classify'}. Defaulting to 'generate'.
WARNING 02-14 01:36:02 config.py:621] gguf quantization is not fully optimized yet. The speed can be slower than non-quantized models.
INFO 02-14 01:36:02 config.py:1556] Chunked prefill is enabled with max_num_batched_tokens=32768.
INFO 02-14 01:36:03 core.py:47] Initializing a V1 LLM engine (v0.7.2) with config: model='models/Qwen2.5-14B-Instruct/Qwen2.5-14B-Instruct-Q4_K_M.gguf', speculative_config=None, tokenizer='Qwen/Qwen2.5-14B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.GGUF, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gguf, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen2.5-14B-VLLM, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
WARNING 02-14 01:36:03 registry.py:340] `mm_limits` has already been set for model=models/Qwen2.5-14B-Instruct/Qwen2.5-14B-Instruct-Q4_K_M.gguf, and will be overwritten by the new values.
INFO 02-14 01:36:03 gpu_model_runner.py:867] Starting to load model models/Qwen2.5-14B-Instruct/Qwen2.5-14B-Instruct-Q4_K_M.gguf...
INFO 02-14 01:36:11 cuda.py:158] Using Flash Attention backend on V1 engine.
INFO 02-14 01:36:11 topk_topp_sampler.py:36] Using FlashInfer for top-p & top-k sampling.
/usr/local/lib/python3.12/dist-packages/torch/nested/__init__.py:226: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. (Triggered internally at ../aten/src/ATen/NestedTensorImpl.cpp:178.)
  return _nested.nested_tensor(
INFO 02-14 01:36:27 gpu_model_runner.py:872] Loading model weights took 8.3987 GB
ERROR 02-14 01:36:27 core.py:210] EngineCore hit an exception: Traceback (most recent call last):
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 895, in step
ERROR 02-14 01:36:27 core.py:210]     self.dispatch_table[inst.opcode](self, inst)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 582, in wrapper
ERROR 02-14 01:36:27 core.py:210]     return inner_fn(self, inst)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 2279, in CALL
ERROR 02-14 01:36:27 core.py:210]     self._call(inst)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 2273, in _call
ERROR 02-14 01:36:27 core.py:210]     self.call_function(fn, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 830, in call_function
ERROR 02-14 01:36:27 core.py:210]     self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
ERROR 02-14 01:36:27 core.py:210]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/functions.py", line 385, in call_function
ERROR 02-14 01:36:27 core.py:210]     return super().call_function(tx, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/functions.py", line 324, in call_function
ERROR 02-14 01:36:27 core.py:210]     return super().call_function(tx, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/functions.py", line 111, in call_function
ERROR 02-14 01:36:27 core.py:210]     return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 836, in inline_user_function_return
ERROR 02-14 01:36:27 core.py:210]     return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 234, in patched_inline_call
ERROR 02-14 01:36:27 core.py:210]     return inline_call(parent, func, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 3011, in inline_call
ERROR 02-14 01:36:27 core.py:210]     return cls.inline_call_(parent, func, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 3139, in inline_call_
ERROR 02-14 01:36:27 core.py:210]     tracer.run()
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 983, in run
ERROR 02-14 01:36:27 core.py:210]     while self.step():
ERROR 02-14 01:36:27 core.py:210]           ^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 898, in step
ERROR 02-14 01:36:27 core.py:210]     self.exception_handler(e)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 1432, in exception_handler
ERROR 02-14 01:36:27 core.py:210]     raise raised_exception
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 895, in step
ERROR 02-14 01:36:27 core.py:210]     self.dispatch_table[inst.opcode](self, inst)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 582, in wrapper
ERROR 02-14 01:36:27 core.py:210]     return inner_fn(self, inst)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 2279, in CALL
ERROR 02-14 01:36:27 core.py:210]     self._call(inst)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 2273, in _call
ERROR 02-14 01:36:27 core.py:210]     self.call_function(fn, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 830, in call_function
ERROR 02-14 01:36:27 core.py:210]     self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
ERROR 02-14 01:36:27 core.py:210]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/lazy.py", line 156, in realize_and_forward
ERROR 02-14 01:36:27 core.py:210]     return getattr(self.realize(), name)(*args, **kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/nn_module.py", line 899, in call_function
ERROR 02-14 01:36:27 core.py:210]     return variables.UserFunctionVariable(fn, source=source).call_function(
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/functions.py", line 324, in call_function
ERROR 02-14 01:36:27 core.py:210]     return super().call_function(tx, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/functions.py", line 111, in call_function
ERROR 02-14 01:36:27 core.py:210]     return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 836, in inline_user_function_return
ERROR 02-14 01:36:27 core.py:210]     return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 234, in patched_inline_call
ERROR 02-14 01:36:27 core.py:210]     return inline_call(parent, func, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 3011, in inline_call
ERROR 02-14 01:36:27 core.py:210]     return cls.inline_call_(parent, func, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 3139, in inline_call_
ERROR 02-14 01:36:27 core.py:210]     tracer.run()
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 983, in run
ERROR 02-14 01:36:27 core.py:210]     while self.step():
ERROR 02-14 01:36:27 core.py:210]           ^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 898, in step
ERROR 02-14 01:36:27 core.py:210]     self.exception_handler(e)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 1432, in exception_handler
ERROR 02-14 01:36:27 core.py:210]     raise raised_exception
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 895, in step
ERROR 02-14 01:36:27 core.py:210]     self.dispatch_table[inst.opcode](self, inst)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 582, in wrapper
ERROR 02-14 01:36:27 core.py:210]     return inner_fn(self, inst)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 2279, in CALL
ERROR 02-14 01:36:27 core.py:210]     self._call(inst)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 2273, in _call
ERROR 02-14 01:36:27 core.py:210]     self.call_function(fn, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 830, in call_function
ERROR 02-14 01:36:27 core.py:210]     self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
ERROR 02-14 01:36:27 core.py:210]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/functions.py", line 385, in call_function
ERROR 02-14 01:36:27 core.py:210]     return super().call_function(tx, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/functions.py", line 324, in call_function
ERROR 02-14 01:36:27 core.py:210]     return super().call_function(tx, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/functions.py", line 111, in call_function
ERROR 02-14 01:36:27 core.py:210]     return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 836, in inline_user_function_return
ERROR 02-14 01:36:27 core.py:210]     return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 234, in patched_inline_call
ERROR 02-14 01:36:27 core.py:210]     return inline_call(parent, func, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 3011, in inline_call
ERROR 02-14 01:36:27 core.py:210]     return cls.inline_call_(parent, func, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 3139, in inline_call_
ERROR 02-14 01:36:27 core.py:210]     tracer.run()
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 983, in run
ERROR 02-14 01:36:27 core.py:210]     while self.step():
ERROR 02-14 01:36:27 core.py:210]           ^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 898, in step
ERROR 02-14 01:36:27 core.py:210]     self.exception_handler(e)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 1432, in exception_handler
ERROR 02-14 01:36:27 core.py:210]     raise raised_exception
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 895, in step
ERROR 02-14 01:36:27 core.py:210]     self.dispatch_table[inst.opcode](self, inst)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 582, in wrapper
ERROR 02-14 01:36:27 core.py:210]     return inner_fn(self, inst)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 301, in impl
ERROR 02-14 01:36:27 core.py:210]     self.push(fn_var.call_function(self, self.popn(nargs), {}))
ERROR 02-14 01:36:27 core.py:210]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/builtin.py", line 967, in call_function
ERROR 02-14 01:36:27 core.py:210]     return handler(tx, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/builtin.py", line 848, in builtin_dispatch
ERROR 02-14 01:36:27 core.py:210]     rv = fn(tx, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]          ^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/builtin.py", line 766, in call_self_handler
ERROR 02-14 01:36:27 core.py:210]     result = self_handler(tx, *args, **kwargs)
ERROR 02-14 01:36:27 core.py:210]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/builtin.py", line 1472, in call_getitem
ERROR 02-14 01:36:27 core.py:210]     return args[0].call_method(tx, "__getitem__", args[1:], kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/dicts.py", line 261, in call_method
ERROR 02-14 01:36:27 core.py:210]     return self.getitem_const_raise_exception_if_absent(tx, args[0])
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/dicts.py", line 224, in getitem_const_raise_exception_if_absent
ERROR 02-14 01:36:27 core.py:210]     raise_observed_exception(KeyError, tx, self)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/exc.py", line 234, in raise_observed_exception
ERROR 02-14 01:36:27 core.py:210]     raise observed_exception_map[e]
ERROR 02-14 01:36:27 core.py:210] torch._dynamo.exc.ObservedKeyError
ERROR 02-14 01:36:27 core.py:210] 
ERROR 02-14 01:36:27 core.py:210] During handling of the above exception, another exception occurred:
ERROR 02-14 01:36:27 core.py:210] 
ERROR 02-14 01:36:27 core.py:210] Traceback (most recent call last):
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 202, in run_engine_core
ERROR 02-14 01:36:27 core.py:210]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 02-14 01:36:27 core.py:210]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 156, in __init__
ERROR 02-14 01:36:27 core.py:210]     super().__init__(vllm_config, executor_class)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 54, in __init__
ERROR 02-14 01:36:27 core.py:210]     num_gpu_blocks, num_cpu_blocks = self._initialize_kv_caches(
ERROR 02-14 01:36:27 core.py:210]                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 79, in _initialize_kv_caches
ERROR 02-14 01:36:27 core.py:210]     availble_gpu_memory = self.model_executor.determine_available_memory()
ERROR 02-14 01:36:27 core.py:210]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 61, in determine_available_memory
ERROR 02-14 01:36:27 core.py:210]     output = self.collective_rpc("determine_available_memory")
ERROR 02-14 01:36:27 core.py:210]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 51, in collective_rpc
ERROR 02-14 01:36:27 core.py:210]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 02-14 01:36:27 core.py:210]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2220, in run_method
ERROR 02-14 01:36:27 core.py:210]     return func(*args, **kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 02-14 01:36:27 core.py:210]     return func(*args, **kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 164, in determine_available_memory
ERROR 02-14 01:36:27 core.py:210]     self.model_runner.profile_run()
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1009, in profile_run
ERROR 02-14 01:36:27 core.py:210]     hidden_states = self._dummy_run(self.max_num_tokens, dummy_kv_caches)
ERROR 02-14 01:36:27 core.py:210]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 02-14 01:36:27 core.py:210]     return func(*args, **kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 894, in _dummy_run
ERROR 02-14 01:36:27 core.py:210]     hidden_states = model(
ERROR 02-14 01:36:27 core.py:210]                     ^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 02-14 01:36:27 core.py:210]     return self._call_impl(*args, **kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 02-14 01:36:27 core.py:210]     return forward_call(*args, **kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 486, in forward
ERROR 02-14 01:36:27 core.py:210]     hidden_states = self.model(input_ids, positions, kv_caches,
ERROR 02-14 01:36:27 core.py:210]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 238, in __call__
ERROR 02-14 01:36:27 core.py:210]     output = self.compiled_callable(*args, **kwargs)
ERROR 02-14 01:36:27 core.py:210]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 465, in _fn
ERROR 02-14 01:36:27 core.py:210]     return fn(*args, **kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/convert_frame.py", line 1269, in __call__
ERROR 02-14 01:36:27 core.py:210]     return self._torchdynamo_orig_callable(
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/convert_frame.py", line 526, in __call__
ERROR 02-14 01:36:27 core.py:210]     return _compile(
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/convert_frame.py", line 924, in _compile
ERROR 02-14 01:36:27 core.py:210]     guarded_code = compile_inner(code, one_graph, hooks, transform)
ERROR 02-14 01:36:27 core.py:210]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/convert_frame.py", line 666, in compile_inner
ERROR 02-14 01:36:27 core.py:210]     return _compile_inner(code, one_graph, hooks, transform)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_utils_internal.py", line 87, in wrapper_function
ERROR 02-14 01:36:27 core.py:210]     return function(*args, **kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/convert_frame.py", line 699, in _compile_inner
ERROR 02-14 01:36:27 core.py:210]     out_code = transform_code_object(code, transform)
ERROR 02-14 01:36:27 core.py:210]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object
ERROR 02-14 01:36:27 core.py:210]     transformations(instructions, code_options)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/convert_frame.py", line 219, in _fn
ERROR 02-14 01:36:27 core.py:210]     return fn(*args, **kwargs)
ERROR 02-14 01:36:27 core.py:210]            ^^^^^^^^^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/convert_frame.py", line 634, in transform
ERROR 02-14 01:36:27 core.py:210]     tracer.run()
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 2796, in run
ERROR 02-14 01:36:27 core.py:210]     super().run()
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 983, in run
ERROR 02-14 01:36:27 core.py:210]     while self.step():
ERROR 02-14 01:36:27 core.py:210]           ^^^^^^^^^^^
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 898, in step
ERROR 02-14 01:36:27 core.py:210]     self.exception_handler(e)
ERROR 02-14 01:36:27 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/symbolic_convert.py", line 1431, in exception_handler
ERROR 02-14 01:36:27 core.py:210]     raise Unsupported("Observed exception")
ERROR 02-14 01:36:27 core.py:210] torch._dynamo.exc.Unsupported: Observed exception
ERROR 02-14 01:36:27 core.py:210] 
ERROR 02-14 01:36:27 core.py:210] from user code:
ERROR 02-14 01:36:27 core.py:210]    File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 340, in forward
ERROR 02-14 01:36:27 core.py:210]     hidden_states = self.get_input_embeddings(input_ids)
ERROR 02-14 01:36:27 core.py:210] 
ERROR 02-14 01:36:27 core.py:210] Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
ERROR 02-14 01:36:27 core.py:210] 
ERROR 02-14 01:36:27 core.py:210] 
ERROR 02-14 01:36:27 core.py:210] You can suppress this exception and fall back to eager by setting:
ERROR 02-14 01:36:27 core.py:210]     import torch._dynamo
ERROR 02-14 01:36:27 core.py:210]     torch._dynamo.config.suppress_errors = True
ERROR 02-14 01:36:27 core.py:210] 
ERROR 02-14 01:36:27 core.py:210] 
CRITICAL 02-14 01:36:27 core_client.py:158] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
Killed

Concurrence

Due to this problem, I tried the GPTQ Quantization of the same model provided by Qwen. On the other hand, I also notice recurrence (parallel requests) are having some issues also. I tried with just 20 parallel requests and I got the following logs. Meanwhile en V0 this doesn’t happen.

VLLM_USE_V1=1 python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-14B-Instruct-GPTQ-Int4 --tokenizer Qwen/Qwen2.5-14B-Instruct --host “0.0.0.0” --port 5000 --gpu-memory-utilization 0.98 --served-model-name “Qwen2.5-14B-VLLM” --max-num-batched-tokens 32768 --max-num-seqs 256 --max_model_len 32768 --generation-config config

Command used

VLLM_USE_V1=1 python3 -m vllm.entrypoints.openai.api_server --model  Qwen/Qwen2.5-14B-Instruct-GPTQ-Int4  --tokenizer Qwen/Qwen2.5-14B-Instruct --host "0.0.0.0" --port 5000 --gpu-memory-utilization 0.98 --served-model-name "Qwen2.5-14B-VLLM" --max-num-batched-tokens 32768 --max-num-seqs 256 --max_model_len 32768 --generation-config config
INFO 02-14 02:39:53 __init__.py:190] Automatically detected platform cuda.
INFO 02-14 02:39:55 api_server.py:840] vLLM API server version 0.7.2
INFO 02-14 02:39:55 api_server.py:841] args: Namespace(host='0.0.0.0', port=5000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='Qwen/Qwen2.5-14B-Instruct-GPTQ-Int4', task='auto', tokenizer='Qwen/Qwen2.5-14B-Instruct', skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=32768, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.98, num_gpu_blocks_override=None, max_num_batched_tokens=32768, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['Qwen2.5-14B-VLLM'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config='config', override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 02-14 02:40:01 config.py:542] This model supports multiple tasks: {'reward', 'generate', 'classify', 'score', 'embed'}. Defaulting to 'generate'.
INFO 02-14 02:40:01 gptq_marlin.py:111] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 02-14 02:40:01 config.py:1556] Chunked prefill is enabled with max_num_batched_tokens=32768.
INFO 02-14 02:40:02 core.py:47] Initializing a V1 LLM engine (v0.7.2) with config: model='Qwen/Qwen2.5-14B-Instruct-GPTQ-Int4', speculative_config=None, tokenizer='Qwen/Qwen2.5-14B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen2.5-14B-VLLM, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
WARNING 02-14 02:40:02 registry.py:340] `mm_limits` has already been set for model=Qwen/Qwen2.5-14B-Instruct-GPTQ-Int4, and will be overwritten by the new values.
INFO 02-14 02:40:02 gpu_model_runner.py:867] Starting to load model Qwen/Qwen2.5-14B-Instruct-GPTQ-Int4...
INFO 02-14 02:40:02 gptq_marlin.py:202] Using MarlinLinearKernel for GPTQMarlinLinearMethod
INFO 02-14 02:40:02 cuda.py:158] Using Flash Attention backend on V1 engine.
INFO 02-14 02:40:02 topk_topp_sampler.py:36] Using FlashInfer for top-p & top-k sampling.
INFO 02-14 02:40:03 weight_utils.py:252] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  33% Completed | 1/3 [00:00<00:00,  2.04it/s]
Loading safetensors checkpoint shards:  67% Completed | 2/3 [00:01<00:00,  1.99it/s]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:01<00:00,  2.39it/s]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:01<00:00,  2.27it/s]

INFO 02-14 02:40:04 gpu_model_runner.py:872] Loading model weights took 9.3324 GB
INFO 02-14 02:40:16 backends.py:579] Using cache directory: /root/.cache/vllm/torch_compile_cache/21834f7273/rank_0 for vLLM's torch.compile
INFO 02-14 02:40:16 backends.py:587] Dynamo bytecode transform time: 11.30 s
INFO 02-14 02:40:17 backends.py:209] Directly lookup the graph for shape None from the cache
INFO 02-14 02:40:22 backends.py:323] Compiling a graph for general shape takes 5.43 s
INFO 02-14 02:40:30 monitor.py:33] torch.compile takes 16.73 s in total
INFO 02-14 02:40:32 kv_cache_utils.py:407] # GPU blocks: 9116
INFO 02-14 02:40:32 kv_cache_utils.py:410] Maximum concurrency for 32768 tokens per request: 4.45x
INFO 02-14 02:40:55 gpu_model_runner.py:1043] Graph capturing finished in 22 secs, took 3.13 GiB
INFO 02-14 02:40:55 core.py:91] init engine (profile, create kv cache, warmup model) took 50.17 seconds
INFO 02-14 02:40:55 api_server.py:756] Using supplied chat template:
INFO 02-14 02:40:55 api_server.py:756] None
INFO 02-14 02:40:55 serving_chat.py:108] Overwriting default chat sampling param with: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
INFO 02-14 02:40:55 serving_completion.py:56] Overwriting default completion sampling param with: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
INFO 02-14 02:40:55 launcher.py:21] Available routes are:
INFO 02-14 02:40:55 launcher.py:29] Route: /openapi.json, Methods: HEAD, GET
INFO 02-14 02:40:55 launcher.py:29] Route: /docs, Methods: HEAD, GET
INFO 02-14 02:40:55 launcher.py:29] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 02-14 02:40:55 launcher.py:29] Route: /redoc, Methods: HEAD, GET
INFO 02-14 02:40:55 launcher.py:29] Route: /health, Methods: GET
INFO 02-14 02:40:55 launcher.py:29] Route: /ping, Methods: GET, POST
INFO 02-14 02:40:55 launcher.py:29] Route: /tokenize, Methods: POST
INFO 02-14 02:40:55 launcher.py:29] Route: /detokenize, Methods: POST
INFO 02-14 02:40:55 launcher.py:29] Route: /v1/models, Methods: GET
INFO 02-14 02:40:55 launcher.py:29] Route: /version, Methods: GET
INFO 02-14 02:40:55 launcher.py:29] Route: /v1/chat/completions, Methods: POST
INFO 02-14 02:40:55 launcher.py:29] Route: /v1/completions, Methods: POST
INFO 02-14 02:40:55 launcher.py:29] Route: /v1/embeddings, Methods: POST
INFO 02-14 02:40:55 launcher.py:29] Route: /pooling, Methods: POST
INFO 02-14 02:40:55 launcher.py:29] Route: /score, Methods: POST
INFO 02-14 02:40:55 launcher.py:29] Route: /v1/score, Methods: POST
INFO 02-14 02:40:55 launcher.py:29] Route: /rerank, Methods: POST
INFO 02-14 02:40:55 launcher.py:29] Route: /v1/rerank, Methods: POST
INFO 02-14 02:40:55 launcher.py:29] Route: /v2/rerank, Methods: POST
INFO 02-14 02:40:55 launcher.py:29] Route: /invocations, Methods: POST
INFO:     Started server process [7080]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
INFO 02-14 02:42:29 chat_utils.py:332] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
INFO 02-14 02:42:29 logger.py:39] Received request chatcmpl-517202e8672f48baa5f299aacf03a72f: prompt: '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\nTell me a story of 2000 words about programmers<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32726, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 02-14 02:42:29 async_llm.py:161] Added request chatcmpl-517202e8672f48baa5f299aacf03a72f.
INFO 02-14 02:42:29 logger.py:39] Received request chatcmpl-04a888d8bbc64778bdc358a45b715a5f: prompt: '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\nTell me a story of 2000 words about programmers<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32726, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.

....
2025-02-14 02:42:30,092 - INFO - flashinfer.jit: Loading JIT ops: sampling
/usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:1964: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
/usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:1964: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
2025-02-14 02:42:30,111 - INFO - flashinfer.jit: Finished loading JIT ops: sampling
INFO 02-14 02:42:35 loggers.py:72] Avg prompt throughput: 167.6 tokens/s, Avg generation throughput: 1169.4 tokens/s, Running: 20 reqs, Waiting: 0 reqs GPU KV cache usage: 4.4%.
INFO 02-14 02:42:40 loggers.py:72] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1107.9 tokens/s, Running: 20 reqs, Waiting: 0 reqs GPU KV cache usage: 8.8%.
INFO 02-14 02:42:45 loggers.py:72] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1037.8 tokens/s, Running: 20 reqs, Waiting: 0 reqs GPU KV cache usage: 12.1%.
INFO:     85.61.125.112:43452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
ERROR 02-14 02:42:45 core.py:210] EngineCore hit an exception: Traceback (most recent call last):
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 203, in run_engine_core
ERROR 02-14 02:42:45 core.py:210]     engine_core.run_busy_loop()
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 243, in run_busy_loop
ERROR 02-14 02:42:45 core.py:210]     outputs = self.step()
ERROR 02-14 02:42:45 core.py:210]               ^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 129, in step
ERROR 02-14 02:42:45 core.py:210]     output = self.model_executor.execute_model(scheduler_output)
ERROR 02-14 02:42:45 core.py:210]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 77, in execute_model
ERROR 02-14 02:42:45 core.py:210]     output = self.collective_rpc("execute_model",
ERROR 02-14 02:42:45 core.py:210]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 51, in collective_rpc
ERROR 02-14 02:42:45 core.py:210]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 02-14 02:42:45 core.py:210]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2220, in run_method
ERROR 02-14 02:42:45 core.py:210]     return func(*args, **kwargs)
ERROR 02-14 02:42:45 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 02-14 02:42:45 core.py:210]     return func(*args, **kwargs)
ERROR 02-14 02:42:45 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 236, in execute_model
ERROR 02-14 02:42:45 core.py:210]     output = self.model_runner.execute_model(scheduler_output)
ERROR 02-14 02:42:45 core.py:210]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 02-14 02:42:45 core.py:210]     return func(*args, **kwargs)
ERROR 02-14 02:42:45 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 804, in execute_model
ERROR 02-14 02:42:45 core.py:210]     sampler_output = self.model.sample(
ERROR 02-14 02:42:45 core.py:210]                      ^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 505, in sample
ERROR 02-14 02:42:45 core.py:210]     next_tokens = self.sampler(logits, sampling_metadata)
ERROR 02-14 02:42:45 core.py:210]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 02-14 02:42:45 core.py:210]     return self._call_impl(*args, **kwargs)
ERROR 02-14 02:42:45 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 02-14 02:42:45 core.py:210]     return forward_call(*args, **kwargs)
ERROR 02-14 02:42:45 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/sample/sampler.py", line 46, in forward
ERROR 02-14 02:42:45 core.py:210]     logits = self.apply_penalties(logits, sampling_metadata)
ERROR 02-14 02:42:45 core.py:210]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/sample/sampler.py", line 130, in apply_penalties
ERROR 02-14 02:42:45 core.py:210]     logits = apply_all_penalties(
ERROR 02-14 02:42:45 core.py:210]              ^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/sample/ops/penalties.py", line 42, in apply_all_penalties
ERROR 02-14 02:42:45 core.py:210]     return apply_penalties(logits, prompt_token_ids, output_tokens_t,
ERROR 02-14 02:42:45 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/utils.py", line 44, in apply_penalties
ERROR 02-14 02:42:45 core.py:210]     _, prompt_mask = get_token_bin_counts_and_mask(prompt_tokens_tensor,
ERROR 02-14 02:42:45 core.py:210]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-14 02:42:45 core.py:210]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/utils.py", line 18, in get_token_bin_counts_and_mask
ERROR 02-14 02:42:45 core.py:210]     bin_counts.scatter_add_(1, tokens, torch.ones_like(tokens))
ERROR 02-14 02:42:45 core.py:210] RuntimeError: Expected index [20, 42] to be smaller than self [19, 152065] apart from dimension 1 and to be smaller size than src [20, 42]
ERROR 02-14 02:42:45 core.py:210] 
CRITICAL 02-14 02:42:45 core_client.py:158] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
Killed

And finally, I experience poor performance compared with V0 version.

Performance

20 parallel requests were impossible so I tried with just one request and I could see a “degradation” in the performance. Around 5 tok/sec are lost compared to the V0 version.

For V0

...
INFO 02-14 02:54:24 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 77.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
INFO 02-14 02:54:29 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 77.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
INFO 02-14 02:54:34 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 76.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.7%, CPU KV cache usage: 0.0%.
...

For V1

...
INFO 02-14 02:52:37 loggers.py:72] Avg prompt throughput: 8.4 tokens/s, Avg generation throughput: 71.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs GPU KV cache usage: 0.3%.
INFO 02-14 02:52:42 loggers.py:72] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 72.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs GPU KV cache usage: 0.6%.
INFO 02-14 02:52:47 loggers.py:72] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 71.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs GPU KV cache usage: 0.8%.
...

I trust you can fix this and keep democratizing AI for everyone. Thanks again for all the effort and I hope this helps you.

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 2
  • Comments: 21 (6 by maintainers)

Most upvoted comments

In conclusion, V1 looks to be better than V0 but quantization might still need a bit of work for supporting all the quantizations.

Performance

In version V1 the first tokens are faster than V0 but then I can see 4 tok/sec less in V1 version: 75 tok/sec vs 71 tok/sec

With version V0

INFO 02-24 03:29:32 logger.py:39] Received request chatcmpl-3cd4d98c8990448f9488ed03199e0f09: prompt: '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\nTell me a story of 1000 words.<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.1, top_p=0.8, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=16343, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 02-24 03:29:32 engine.py:280] Added request chatcmpl-3cd4d98c8990448f9488ed03199e0f09.
INFO 02-24 03:29:32 metrics.py:455] Avg prompt throughput: 4.0 tokens/s, Avg generation throughput: 12.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 02-24 03:29:37 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 75.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
INFO 02-24 03:29:42 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 75.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.

With version V1

INFO 02-24 03:17:42 logger.py:39] Received request chatcmpl-7b443d7ce19c449fad9afe00699e928b: prompt: '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\nTell me a story of 1000 words.<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.1, top_p=0.8, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=16343, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 02-24 03:17:42 async_llm.py:164] Added request chatcmpl-7b443d7ce19c449fad9afe00699e928b.
INFO 02-24 03:17:46 loggers.py:78] Avg prompt throughput: 8.2 tokens/s, Avg generation throughput: 51.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
INFO 02-24 03:17:51 loggers.py:78] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 71.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
INFO 02-24 03:17:56 loggers.py:78] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 71.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%