vllm: [Bug]: Out of Memory error for Qwen2.5 in 0.8.0 and 0.8.1. Worked fine in the previous versions
Your current environment
The output of `python collect_env.py`
Your output of `python collect_env.py` here
š Describe the bug
I start the inference server by the following command
python -m vllm.entrypoints.openai.api_server --dtype auto --api-key token-abc123 --tensor-parallel-size 2 --host 0.0.0.0 --port 8004 --model Qwen2.5-72B-Instruct-GPTQ-Int4 --gpu-memory-utilization 0.9 --max-model-len 16000
Hereās the log during the service start-up in the version 0.7.2
INFO 03-20 16:00:54 __init__.py:190] Automatically detected platform cuda.
INFO 03-20 16:00:54 api_server.py:840] vLLM API server version 0.7.2
INFO 03-20 16:00:54 api_server.py:841] args: Namespace(host='0.0.0.0', port=8004, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key='token-abc123', lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='Qwen2.5-72B-Instruct-GPTQ-Int4', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=16000, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 03-20 16:00:54 api_server.py:206] Started engine process with PID 931
INFO 03-20 16:00:58 __init__.py:190] Automatically detected platform cuda.
INFO 03-20 16:01:00 config.py:542] This model supports multiple tasks: {'generate', 'reward', 'classify', 'score', 'embed'}. Defaulting to 'generate'.
INFO 03-20 16:01:01 gptq_marlin.py:111] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 03-20 16:01:01 config.py:1401] Defaulting to use mp for distributed inference
INFO 03-20 16:01:03 config.py:542] This model supports multiple tasks: {'embed', 'reward', 'classify', 'score', 'generate'}. Defaulting to 'generate'.
INFO 03-20 16:01:04 gptq_marlin.py:111] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 03-20 16:01:04 config.py:1401] Defaulting to use mp for distributed inference
INFO 03-20 16:01:05 llm_engine.py:234] Initializing a V0 LLM engine (v0.7.2) with config: model='Qwen2.5-72B-Instruct-GPTQ-Int4', speculative_config=None, tokenizer='Qwen2.5-72B-Instruct-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=16000, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen2.5-72B-Instruct-GPTQ-Int4, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
WARNING 03-20 16:01:05 multiproc_worker_utils.py:300] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 03-20 16:01:05 custom_cache_manager.py:19] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
(VllmWorkerProcess pid=1392) INFO 03-20 16:01:05 multiproc_worker_utils.py:229] Worker ready; awaiting tasks
INFO 03-20 16:01:05 cuda.py:230] Using Flash Attention backend.
(VllmWorkerProcess pid=1392) INFO 03-20 16:01:05 cuda.py:230] Using Flash Attention backend.
INFO 03-20 16:01:06 utils.py:950] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=1392) INFO 03-20 16:01:06 utils.py:950] Found nccl from library libnccl.so.2
INFO 03-20 16:01:06 pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorkerProcess pid=1392) INFO 03-20 16:01:06 pynccl.py:69] vLLM is using nccl==2.21.5
INFO 03-20 16:01:06 custom_all_reduce_utils.py:206] generating GPU P2P access cache in /root/.cache/vllm/gpu_p2p_access_cache_for_2,3.json
INFO 03-20 16:01:15 custom_all_reduce_utils.py:244] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_2,3.json
(VllmWorkerProcess pid=1392) INFO 03-20 16:01:15 custom_all_reduce_utils.py:244] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_2,3.json
INFO 03-20 16:01:15 shm_broadcast.py:258] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_218a6dd5'), local_subscribe_port=49961, remote_subscribe_port=None)
INFO 03-20 16:01:15 model_runner.py:1110] Starting to load model Qwen2.5-72B-Instruct-GPTQ-Int4...
(VllmWorkerProcess pid=1392) INFO 03-20 16:01:15 model_runner.py:1110] Starting to load model Qwen2.5-72B-Instruct-GPTQ-Int4...
INFO 03-20 16:01:15 gptq_marlin.py:202] Using MarlinLinearKernel for GPTQMarlinLinearMethod
(VllmWorkerProcess pid=1392) INFO 03-20 16:01:15 gptq_marlin.py:202] Using MarlinLinearKernel for GPTQMarlinLinearMethod
Loading safetensors checkpoint shards: 0% Completed | 0/11 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 9% Completed | 1/11 [00:00<00:07, 1.26it/s]
Loading safetensors checkpoint shards: 18% Completed | 2/11 [00:01<00:04, 2.21it/s]
Loading safetensors checkpoint shards: 27% Completed | 3/11 [00:01<00:04, 1.75it/s]
Loading safetensors checkpoint shards: 36% Completed | 4/11 [00:02<00:03, 1.89it/s]
Loading safetensors checkpoint shards: 45% Completed | 5/11 [00:03<00:03, 1.56it/s]
Loading safetensors checkpoint shards: 55% Completed | 6/11 [00:03<00:03, 1.43it/s]
Loading safetensors checkpoint shards: 64% Completed | 7/11 [00:04<00:03, 1.33it/s]
Loading safetensors checkpoint shards: 73% Completed | 8/11 [00:05<00:02, 1.30it/s]
Loading safetensors checkpoint shards: 82% Completed | 9/11 [00:06<00:01, 1.28it/s]
Loading safetensors checkpoint shards: 91% Completed | 10/11 [00:07<00:00, 1.26it/s]
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:07<00:00, 1.25it/s]
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:07<00:00, 1.38it/s]
INFO 03-20 16:01:24 model_runner.py:1115] Loading model weights took 19.2671 GB
(VllmWorkerProcess pid=1392) INFO 03-20 16:01:25 model_runner.py:1115] Loading model weights took 19.2671 GB
(VllmWorkerProcess pid=1392) INFO 03-20 16:01:35 worker.py:267] Memory profiling takes 10.51 seconds
(VllmWorkerProcess pid=1392) INFO 03-20 16:01:35 worker.py:267] the current vLLM instance can use total_gpu_memory (44.53GiB) x gpu_memory_utilization (0.90) = 40.07GiB
(VllmWorkerProcess pid=1392) INFO 03-20 16:01:35 worker.py:267] model weights take 19.27GiB; non_torch_memory takes 0.30GiB; PyTorch activation peak memory takes 2.78GiB; the rest of the memory reserved for KV Cache is 17.73GiB.
INFO 03-20 16:01:35 worker.py:267] Memory profiling takes 10.54 seconds
INFO 03-20 16:01:35 worker.py:267] the current vLLM instance can use total_gpu_memory (44.53GiB) x gpu_memory_utilization (0.90) = 40.07GiB
INFO 03-20 16:01:35 worker.py:267] model weights take 19.27GiB; non_torch_memory takes 0.30GiB; PyTorch activation peak memory takes 2.78GiB; the rest of the memory reserved for KV Cache is 17.73GiB.
INFO 03-20 16:01:36 executor_base.py:110] # CUDA blocks: 7262, # CPU blocks: 1638
INFO 03-20 16:01:36 executor_base.py:115] Maximum concurrency for 16000 tokens per request: 7.26x
(VllmWorkerProcess pid=1392) INFO 03-20 16:01:38 model_runner.py:1434] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 03-20 16:01:38 model_runner.py:1434] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 03-20 16:01:58 custom_all_reduce.py:226] Registering 5635 cuda graph addresses
(VllmWorkerProcess pid=1392) INFO 03-20 16:01:59 custom_all_reduce.py:226] Registering 5635 cuda graph addresses
(VllmWorkerProcess pid=1392) INFO 03-20 16:01:59 model_runner.py:1562] Graph capturing finished in 21 secs, took 2.89 GiB
INFO 03-20 16:01:59 model_runner.py:1562] Graph capturing finished in 21 secs, took 2.89 GiB
INFO 03-20 16:01:59 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 34.30 seconds
INFO 03-20 16:01:59 api_server.py:756] Using supplied chat template:
INFO 03-20 16:01:59 api_server.py:756] None
INFO 03-20 16:01:59 launcher.py:21] Available routes are:
INFO 03-20 16:01:59 launcher.py:29] Route: /openapi.json, Methods: HEAD, GET
INFO 03-20 16:01:59 launcher.py:29] Route: /docs, Methods: HEAD, GET
INFO 03-20 16:01:59 launcher.py:29] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 03-20 16:01:59 launcher.py:29] Route: /redoc, Methods: HEAD, GET
INFO 03-20 16:01:59 launcher.py:29] Route: /health, Methods: GET
INFO 03-20 16:01:59 launcher.py:29] Route: /ping, Methods: POST, GET
INFO 03-20 16:01:59 launcher.py:29] Route: /tokenize, Methods: POST
INFO 03-20 16:01:59 launcher.py:29] Route: /detokenize, Methods: POST
INFO 03-20 16:01:59 launcher.py:29] Route: /v1/models, Methods: GET
INFO 03-20 16:01:59 launcher.py:29] Route: /version, Methods: GET
INFO 03-20 16:01:59 launcher.py:29] Route: /v1/chat/completions, Methods: POST
INFO 03-20 16:01:59 launcher.py:29] Route: /v1/completions, Methods: POST
INFO 03-20 16:01:59 launcher.py:29] Route: /v1/embeddings, Methods: POST
INFO 03-20 16:01:59 launcher.py:29] Route: /pooling, Methods: POST
INFO 03-20 16:01:59 launcher.py:29] Route: /score, Methods: POST
INFO 03-20 16:01:59 launcher.py:29] Route: /v1/score, Methods: POST
INFO 03-20 16:01:59 launcher.py:29] Route: /rerank, Methods: POST
INFO 03-20 16:01:59 launcher.py:29] Route: /v1/rerank, Methods: POST
INFO 03-20 16:01:59 launcher.py:29] Route: /v2/rerank, Methods: POST
INFO 03-20 16:01:59 launcher.py:29] Route: /invocations, Methods: POST
INFO: Started server process [798]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8004 (Press CTRL+C to quit)
But when I do the same with 0.8.1, I get the CUDA Out of Memory error:
root@e00d0c989958:/workspace# python -m vllm.entrypoints.openai.api_server --dtype auto --api-key token-abc123 --tensor-parallel-size 2 --host 0.0.0.0 --port 8004 --model Qwen2.5-72B-Instruct-GPTQ-Int4 --gpu-memory-utilization 0.9 --max-model-len 16000
INFO 03-20 16:53:14 [__init__.py:256] Automatically detected platform cuda.
INFO 03-20 16:53:14 [api_server.py:977] vLLM API server version 0.8.1
INFO 03-20 16:53:14 [api_server.py:978] args: Namespace(host='0.0.0.0', port=8004, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key='token-abc123', lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='Qwen2.5-72B-Instruct-GPTQ-Int4', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=16000, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=2, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=None, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, use_tqdm_on_load=True, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, enable_reasoning=False, reasoning_parser=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False)
INFO 03-20 16:53:20 [config.py:583] This model supports multiple tasks: {'embed', 'reward', 'classify', 'generate', 'score'}. Defaulting to 'generate'.
INFO 03-20 16:53:21 [gptq_marlin.py:143] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 03-20 16:53:21 [config.py:1515] Defaulting to use mp for distributed inference
INFO 03-20 16:53:21 [config.py:1693] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 03-20 16:53:21 [core.py:53] Initializing a V1 LLM engine (v0.8.1) with config: model='Qwen2.5-72B-Instruct-GPTQ-Int4', speculative_config=None, tokenizer='Qwen2.5-72B-Instruct-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=16000, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=Qwen2.5-72B-Instruct-GPTQ-Int4, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
WARNING 03-20 16:53:21 [multiproc_worker_utils.py:306] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 03-20 16:53:21 [shm_broadcast.py:258] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1], buffer_handle=(2, 10485760, 10, 'psm_c6f9ea62'), local_subscribe_addr='ipc:///tmp/e4cde641-0b61-45c6-8043-c87f708cbb06', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 03-20 16:53:22 [utils.py:2282] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7f1bb526b7f0>
(VllmWorker rank=0 pid=2764) INFO 03-20 16:53:22 [shm_broadcast.py:258] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_f52a60bf'), local_subscribe_addr='ipc:///tmp/e0a5ee76-7adf-489a-89e4-182820b0b641', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 03-20 16:53:22 [utils.py:2282] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7f1bb526b1c0>
(VllmWorker rank=1 pid=2775) INFO 03-20 16:53:22 [shm_broadcast.py:258] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_d04b4dd2'), local_subscribe_addr='ipc:///tmp/8902b815-5d45-4f9c-a4c3-e863946aeb4d', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=1 pid=2775) INFO 03-20 16:53:23 [utils.py:925] Found nccl from library libnccl.so.2
(VllmWorker rank=1 pid=2775) INFO 03-20 16:53:23 [pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorker rank=0 pid=2764) INFO 03-20 16:53:23 [utils.py:925] Found nccl from library libnccl.so.2
(VllmWorker rank=0 pid=2764) INFO 03-20 16:53:23 [pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorker rank=1 pid=2775) INFO 03-20 16:53:23 [custom_all_reduce_utils.py:244] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_2,3.json
(VllmWorker rank=0 pid=2764) INFO 03-20 16:53:23 [custom_all_reduce_utils.py:244] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_2,3.json
(VllmWorker rank=0 pid=2764) INFO 03-20 16:53:23 [shm_broadcast.py:258] vLLM message queue communication handle: Handle(local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_4b189ac3'), local_subscribe_addr='ipc:///tmp/5f351a76-9949-4c64-b19f-d6803a05d430', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=1 pid=2775) INFO 03-20 16:53:23 [parallel_state.py:967] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 1
(VllmWorker rank=0 pid=2764) INFO 03-20 16:53:23 [parallel_state.py:967] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0
(VllmWorker rank=1 pid=2775) INFO 03-20 16:53:23 [cuda.py:215] Using Flash Attention backend on V1 engine.
(VllmWorker rank=0 pid=2764) INFO 03-20 16:53:23 [cuda.py:215] Using Flash Attention backend on V1 engine.
(VllmWorker rank=1 pid=2775) INFO 03-20 16:53:23 [gpu_model_runner.py:1164] Starting to load model Qwen2.5-72B-Instruct-GPTQ-Int4...
(VllmWorker rank=0 pid=2764) INFO 03-20 16:53:23 [gpu_model_runner.py:1164] Starting to load model Qwen2.5-72B-Instruct-GPTQ-Int4...
(VllmWorker rank=1 pid=2775) INFO 03-20 16:53:23 [gptq_marlin.py:235] Using MarlinLinearKernel for GPTQMarlinLinearMethod
(VllmWorker rank=0 pid=2764) INFO 03-20 16:53:23 [gptq_marlin.py:235] Using MarlinLinearKernel for GPTQMarlinLinearMethod
(VllmWorker rank=0 pid=2764) WARNING 03-20 16:53:23 [topk_topp_sampler.py:63] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=1 pid=2775) WARNING 03-20 16:53:23 [topk_topp_sampler.py:63] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
Loading safetensors checkpoint shards: 0% Completed | 0/11 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 9% Completed | 1/11 [00:00<00:07, 1.25it/s]
Loading safetensors checkpoint shards: 18% Completed | 2/11 [00:01<00:04, 2.10it/s]
Loading safetensors checkpoint shards: 27% Completed | 3/11 [00:01<00:04, 1.70it/s]
Loading safetensors checkpoint shards: 36% Completed | 4/11 [00:02<00:03, 1.80it/s]
Loading safetensors checkpoint shards: 45% Completed | 5/11 [00:03<00:03, 1.51it/s]
Loading safetensors checkpoint shards: 55% Completed | 6/11 [00:03<00:03, 1.37it/s]
Loading safetensors checkpoint shards: 64% Completed | 7/11 [00:04<00:03, 1.27it/s]
Loading safetensors checkpoint shards: 73% Completed | 8/11 [00:05<00:02, 1.22it/s]
Loading safetensors checkpoint shards: 82% Completed | 9/11 [00:06<00:01, 1.20it/s]
Loading safetensors checkpoint shards: 91% Completed | 10/11 [00:07<00:00, 1.18it/s]
(VllmWorker rank=1 pid=2775) INFO 03-20 16:53:31 [loader.py:429] Loading weights took 7.82 seconds
(VllmWorker rank=1 pid=2775) INFO 03-20 16:53:31 [gpu_model_runner.py:1176] Model loading took 19.2671 GB and 8.253775 seconds
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:08<00:00, 1.16it/s]
Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:08<00:00, 1.31it/s]
(VllmWorker rank=0 pid=2764)
(VllmWorker rank=0 pid=2764) INFO 03-20 16:53:32 [loader.py:429] Loading weights took 8.52 seconds
(VllmWorker rank=0 pid=2764) INFO 03-20 16:53:32 [gpu_model_runner.py:1176] Model loading took 19.2671 GB and 8.943086 seconds
(VllmWorker rank=0 pid=2764) INFO 03-20 16:53:48 [backends.py:409] Using cache directory: /root/.cache/vllm/torch_compile_cache/50a2bed080/rank_0_0 for vLLM's torch.compile
(VllmWorker rank=0 pid=2764) INFO 03-20 16:53:48 [backends.py:419] Dynamo bytecode transform time: 15.64 s
(VllmWorker rank=1 pid=2775) INFO 03-20 16:53:48 [backends.py:409] Using cache directory: /root/.cache/vllm/torch_compile_cache/50a2bed080/rank_1_0 for vLLM's torch.compile
(VllmWorker rank=1 pid=2775) INFO 03-20 16:53:48 [backends.py:419] Dynamo bytecode transform time: 15.67 s
(VllmWorker rank=0 pid=2764) INFO 03-20 16:53:49 [backends.py:115] Directly load the compiled graph for shape None from the cache
(VllmWorker rank=1 pid=2775) INFO 03-20 16:53:49 [backends.py:115] Directly load the compiled graph for shape None from the cache
(VllmWorker rank=0 pid=2764) INFO 03-20 16:54:02 [monitor.py:33] torch.compile takes 15.64 s in total
(VllmWorker rank=1 pid=2775) INFO 03-20 16:54:02 [monitor.py:33] torch.compile takes 15.67 s in total
INFO 03-20 16:54:04 [kv_cache_utils.py:537] GPU KV cache size: 128,496 tokens
INFO 03-20 16:54:04 [kv_cache_utils.py:540] Maximum concurrency for 16,000 tokens per request: 8.03x
INFO 03-20 16:54:04 [kv_cache_utils.py:537] GPU KV cache size: 128,496 tokens
INFO 03-20 16:54:04 [kv_cache_utils.py:540] Maximum concurrency for 16,000 tokens per request: 8.03x
(VllmWorker rank=0 pid=2764) INFO 03-20 16:54:37 [custom_all_reduce.py:229] Registering 8352 cuda graph addresses
(VllmWorker rank=1 pid=2775) INFO 03-20 16:54:38 [custom_all_reduce.py:229] Registering 8352 cuda graph addresses
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] WorkerProc hit an exception: %s
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] Traceback (most recent call last):
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 371, in worker_busy_loop
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] output = func(*args, **kwargs)
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 216, in compile_or_warm_up_model
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] self.model_runner.capture_model()
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1492, in capture_model
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] self._dummy_run(num_tokens)
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] return func(*args, **kwargs)
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1301, in _dummy_run
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] hidden_states = model(
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen2.py", line 462, in forward
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] hidden_states = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/decorators.py", line 245, in __call__
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] model_output = self.forward(*args, **kwargs)
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen2.py", line 320, in forward
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] def forward(
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/eval_frame.py", line 745, in _fn
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] return fn(*args, **kwargs)
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/torch/fx/graph_module.py", line 822, in call_wrapped
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] return self._wrapped_call(self, *args, **kwargs)
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/torch/fx/graph_module.py", line 400, in __call__
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] raise e
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/torch/fx/graph_module.py", line 387, in __call__
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "<eval_with_key>.162", line 2660, in forward
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] submod_140 = self.submod_140(getitem_348, s0, l_self_modules_layers_modules_69_modules_self_attn_modules_o_proj_parameters_qweight_, l_self_modules_layers_modules_69_modules_self_attn_modules_o_proj_parameters_scales_, l_self_modules_layers_modules_69_modules_self_attn_modules_o_proj_parameters_qzeros_, l_self_modules_layers_modules_69_modules_self_attn_modules_o_proj_parameters_g_idx_, l_self_modules_layers_modules_69_modules_self_attn_modules_o_proj_parameters_g_idx_sort_indices_, l_self_modules_layers_modules_69_modules_self_attn_modules_o_proj_quant_method_kernel_workspace, getitem_349, l_self_modules_layers_modules_69_modules_post_attention_layernorm_parameters_weight_, l_self_modules_layers_modules_69_modules_mlp_modules_gate_up_proj_parameters_qweight_, l_self_modules_layers_modules_69_modules_mlp_modules_gate_up_proj_parameters_scales_, l_self_modules_layers_modules_69_modules_mlp_modules_gate_up_proj_parameters_qzeros_, l_self_modules_layers_modules_69_modules_mlp_modules_gate_up_proj_parameters_g_idx_, l_self_modules_layers_modules_69_modules_mlp_modules_gate_up_proj_parameters_g_idx_sort_indices_, l_self_modules_layers_modules_69_modules_mlp_modules_gate_up_proj_quant_method_kernel_workspace, l_self_modules_layers_modules_69_modules_mlp_modules_down_proj_parameters_qweight_, l_self_modules_layers_modules_69_modules_mlp_modules_down_proj_parameters_scales_, l_self_modules_layers_modules_69_modules_mlp_modules_down_proj_parameters_qzeros_, l_self_modules_layers_modules_69_modules_mlp_modules_down_proj_parameters_g_idx_, l_self_modules_layers_modules_69_modules_mlp_modules_down_proj_parameters_g_idx_sort_indices_, l_self_modules_layers_modules_69_modules_mlp_modules_down_proj_quant_method_kernel_workspace, l_self_modules_layers_modules_70_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_70_modules_self_attn_modules_qkv_proj_parameters_qweight_, l_self_modules_layers_modules_70_modules_self_attn_modules_qkv_proj_parameters_scales_, l_self_modules_layers_modules_70_modules_self_attn_modules_qkv_proj_parameters_qzeros_, l_self_modules_layers_modules_70_modules_self_attn_modules_qkv_proj_parameters_g_idx_, l_self_modules_layers_modules_70_modules_self_attn_modules_qkv_proj_parameters_g_idx_sort_indices_, l_self_modules_layers_modules_70_modules_self_attn_modules_qkv_proj_quant_method_kernel_workspace, l_self_modules_layers_modules_70_modules_self_attn_modules_qkv_proj_parameters_bias_, l_positions_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_); getitem_348 = l_self_modules_layers_modules_69_modules_self_attn_modules_o_proj_parameters_qweight_ = l_self_modules_layers_modules_69_modules_self_attn_modules_o_proj_parameters_scales_ = l_self_modules_layers_modules_69_modules_self_attn_modules_o_proj_parameters_qzeros_ = l_self_modules_layers_modules_69_modules_self_attn_modules_o_proj_parameters_g_idx_ = l_self_modules_layers_modules_69_modules_self_attn_modules_o_proj_parameters_g_idx_sort_indices_ = l_self_modules_layers_modules_69_modules_self_attn_modules_o_proj_quant_method_kernel_workspace = getitem_349 = l_self_modules_layers_modules_69_modules_post_attention_layernorm_parameters_weight_ = l_self_modules_layers_modules_69_modules_mlp_modules_gate_up_proj_parameters_qweight_ = l_self_modules_layers_modules_69_modules_mlp_modules_gate_up_proj_parameters_scales_ = l_self_modules_layers_modules_69_modules_mlp_modules_gate_up_proj_parameters_qzeros_ = l_self_modules_layers_modules_69_modules_mlp_modules_gate_up_proj_parameters_g_idx_ = l_self_modules_layers_modules_69_modules_mlp_modules_gate_up_proj_parameters_g_idx_sort_indices_ = l_self_modules_layers_modules_69_modules_mlp_modules_gate_up_proj_quant_method_kernel_workspace = l_self_modules_layers_modules_69_modules_mlp_modules_down_proj_parameters_qweight_ = l_self_modules_layers_modules_69_modules_mlp_modules_down_proj_parameters_scales_ = l_self_modules_layers_modules_69_modules_mlp_modules_down_proj_parameters_qzeros_ = l_self_modules_layers_modules_69_modules_mlp_modules_down_proj_parameters_g_idx_ = l_self_modules_layers_modules_69_modules_mlp_modules_down_proj_parameters_g_idx_sort_indices_ = l_self_modules_layers_modules_69_modules_mlp_modules_down_proj_quant_method_kernel_workspace = l_self_modules_layers_modules_70_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_70_modules_self_attn_modules_qkv_proj_parameters_qweight_ = l_self_modules_layers_modules_70_modules_self_attn_modules_qkv_proj_parameters_scales_ = l_self_modules_layers_modules_70_modules_self_attn_modules_qkv_proj_parameters_qzeros_ = l_self_modules_layers_modules_70_modules_self_attn_modules_qkv_proj_parameters_g_idx_ = l_self_modules_layers_modules_70_modules_self_attn_modules_qkv_proj_parameters_g_idx_sort_indices_ = l_self_modules_layers_modules_70_modules_self_attn_modules_qkv_proj_quant_method_kernel_workspace = l_self_modules_layers_modules_70_modules_self_attn_modules_qkv_proj_parameters_bias_ = None
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/backends.py", line 670, in __call__
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] with torch.cuda.graph(cudagraph, pool=self.graph_pool):
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/torch/cuda/graphs.py", line 186, in __exit__
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] self.cuda_graph.capture_end()
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] File "/usr/local/lib/python3.10/dist-packages/torch/cuda/graphs.py", line 84, in capture_end
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] super().capture_end()
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] RuntimeError: CUDA error: out of memory
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(VllmWorker rank=1 pid=2775) ERROR 03-20 16:54:38 [multiproc_executor.py:375]
(VllmWorker rank=0 pid=2764) ERROR 03-20 16:54:38 [multiproc_executor.py:375] WorkerProc hit an exception: %s
(VllmWorker rank=0 pid=2764) ERROR 03-20 16:54:38 [multiproc_executor.py:375] Traceback (most recent call last):
I set the --max-num-seqs 2 and still got the same error. The visible difference that I see is 0.7.2 uses V0 engine and 0.8.1 uses V1 engine.
However it works in 0.8.1 if I set the --enforce-eager flag. But my project requires to do the Async operations and therefore I use the OpenAIAsync on the client side. Will setting --enforce-eager cause any problem to the async process?
Another thing to also notice is in 0.7.2 I see the INFO 03-20 16:01:36 executor_base.py:115] Maximum concurrency for 16000 tokens per request: 7.26x whereas in 0.8.1 with --enforce-eager I see INFO 03-20 17:00:20 [kv_cache_utils.py:540] Maximum concurrency for 16,000 tokens per request: 5.90x. Why is that happening?
Thanks in advance! š
Before submitting a new issueā¦
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 5
- Comments: 23 (3 by maintainers)
max-num-seqscontrols the maximum number of requests can be running concurrently. In V0 this value is by default 256 but we realized itās a bit conservative given the much more powerful computing capability of GPUs of newer architectures, so in V1 we set this to be 1024 by default.Are you using vLLM V1 for this? If turning on
enforce-eagersolves your problem, then you can try lowing--gpu-memory-utilization(this is whatās reserved for KV cache) to leave more room for cuda graphs.same proble, but āfix itā by:
pip install flashinfer-python==0.2.2Update: I tried something similar with
Mistral-Small-3.1-24B-Instruct-2503When I try to serve using
vllm serve Mistral-Small-3.1-24B-Instruct-2503 --tokenizer_mode mistral --config_format mistral --load_format mistral --limit_mm_per_prompt 'image=10' --dtype auto --api-key token-abc123 --tensor-parallel-size 2 --host 0.0.0.0 --port 8004 --gpu-memory-utilization 0.75 --max-model-len 4000 --enforce-eagerI get the not enough memory error ValueError: No available memory for the cache blocks. Try increasing gpu_memory_utilization when initializing the engine. It cannot even serve when the maximum context length is 4000.I simply use the V0 engine using
VLLM_USE_V1=0 vllm serve ...and I am able to serve the same model with 32000 tokens with 2.28x concurrency!!V1 needs some optimization because it is consuming so much memory for unknown reasons.
This is what I am using for Gemma 3
VLLM_USE_V1=0 vllm serve gemma-3-27b-it --dtype auto --api-key token-abc123 --tensor-parallel-size 4 --host 0.0.0.0 --port 8004 --gpu-memory-utilization 0.75 --max-model-len 16000 --enforce-eagerIf I donāt enable the --enforce-eager (even in V0), the system crashes when concurrent calls are made. But for Qwen 2.5 models, I do not use
--enforce-eagerand still it handles the same load smoothly.I am seeing the same out of memory issues with other quantized models. For example, running https://huggingface.co/neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w4a16 on a single A6000 runs out of memory on V0.8.X but worked on v0.7.X