vllm: [Bug]: serve Llama-3.2-11B-Vision-Instruct with 2 A10 oom
Your current environment
docker image vllm/vllm-openai:v0.6.2 and vllm/vllm-openai:v0.6.3 command:docker run --runtime nvidia --gpus ‘“device=0,1”’ -d -v /data/model/llama:/data/model/llama -p 8001:8000 vllm/vllm-openai:v0.6.2 --model /data/model/llama --max-model-len 1024 --served_model_name Llama-3.2-11B-Vision-Instruct --tensor-parallel-size 2 --gpu_memory_utilization 0.7
I tried v0.6.2 and v0.6.3,both not work,only half of the gpu memory is occupied
nvidia-smi output: ±----------------------------------------------------------------------------------------+ | NVIDIA-SMI 555.42.06 Driver Version: 555.42.06 CUDA Version: 12.5 | |-----------------------------------------±-----------------------±---------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA A10 On | 00000000:00:04.0 Off | 0 | | 0% 47C P0 60W / 150W | 10797MiB / 23028MiB | 0% Default | | | | N/A | ±----------------------------------------±-----------------------±---------------------+ | 1 NVIDIA A10 On | 00000000:00:08.0 Off | 0 | | 0% 47C P0 61W / 150W | 10797MiB / 23028MiB | 0% Default | | | | N/A | ±----------------------------------------±-----------------------±---------------------+ | 2 NVIDIA A10 On | 00000000:00:09.0 Off | 0 | | 0% 55C P0 63W / 150W | 18493MiB / 23028MiB | 0% Default | | | | N/A | ±----------------------------------------±-----------------------±---------------------+
±----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 2750406 C /usr/bin/python3 10788MiB | | 1 N/A N/A 2750455 C /usr/bin/python3 10788MiB | | 2 N/A N/A 2738538 C …nda/miniconda3/envs/vllm/bin/python 18470MiB | ±----------------------------------------------------------------------------------------+
Model Input Dumps
No response
🐛 Describe the bug
INFO 11-05 03:21:18 api_server.py:526] vLLM API server version 0.6.1.dev238+ge2c6e0a82
INFO 11-05 03:21:18 api_server.py:527] args: Namespace(host=None, port=8000, uvicorn_log_level=‘info’, allow_credentials=False, allowed_origins=[‘‘], allowed_methods=[’’], allowed_headers=[‘‘], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role=‘assistant’, ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_auto_tool_choice=False, tool_call_parser=None, model=’/data/model/llama’, tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode=‘auto’, trust_remote_code=False, download_dir=None, load_format=‘auto’, config_format=‘auto’, dtype=‘auto’, kv_cache_dtype=‘auto’, quantization_param_path=None, max_model_len=1024, guided_decoding_backend=‘outlines’, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.7, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type=‘ray’, tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype=‘auto’, long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device=‘auto’, num_scheduler_steps=1, multi_step_stream_outputs=False, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method=‘rejection_sampler’, typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=[‘Llama-3.2-11B-Vision-Instruct’], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False)
INFO 11-05 03:21:18 api_server.py:164] Multiprocessing frontend to use ipc:///tmp/056a940a-9835-44c7-95c2-01e0c84f4ed4 for IPC Path.
INFO 11-05 03:21:18 api_server.py:177] Started engine process with PID 29
INFO 11-05 03:21:18 config.py:899] Defaulting to use mp for distributed inference
junjie_wang@ubuntuT-CCLLM-4804084:~$ sudo docker logs c123d8180a936a7bda26a6b3ce14dbee4964ac79b89f1ae74bb29c3467e0b37f
INFO 11-05 03:21:18 api_server.py:526] vLLM API server version 0.6.1.dev238+ge2c6e0a82
INFO 11-05 03:21:18 api_server.py:527] args: Namespace(host=None, port=8000, uvicorn_log_level=‘info’, allow_credentials=False, allowed_origins=['’], allowed_methods=[‘‘], allowed_headers=[’’], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role=‘assistant’, ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_auto_tool_choice=False, tool_call_parser=None, model=‘/data/model/llama’, tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode=‘auto’, trust_remote_code=False, download_dir=None, load_format=‘auto’, config_format=‘auto’, dtype=‘auto’, kv_cache_dtype=‘auto’, quantization_param_path=None, max_model_len=1024, guided_decoding_backend=‘outlines’, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.7, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type=‘ray’, tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype=‘auto’, long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device=‘auto’, num_scheduler_steps=1, multi_step_stream_outputs=False, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method=‘rejection_sampler’, typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=[‘Llama-3.2-11B-Vision-Instruct’], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False)
INFO 11-05 03:21:18 api_server.py:164] Multiprocessing frontend to use ipc:///tmp/056a940a-9835-44c7-95c2-01e0c84f4ed4 for IPC Path.
INFO 11-05 03:21:18 api_server.py:177] Started engine process with PID 29
INFO 11-05 03:21:18 config.py:899] Defaulting to use mp for distributed inference
INFO 11-05 03:21:22 config.py:899] Defaulting to use mp for distributed inference
INFO 11-05 03:21:22 llm_engine.py:226] Initializing an LLM engine (v0.6.1.dev238+ge2c6e0a82) with config: model=‘/data/model/llama’, speculative_config=None, tokenizer=‘/data/model/llama’, skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=1024, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend=‘outlines’), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Llama-3.2-11B-Vision-Instruct, use_v2_block_manager=False, num_scheduler_steps=1, multi_step_stream_outputs=False, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=True, mm_processor_kwargs=None)
WARNING 11-05 03:21:23 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 16 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 11-05 03:21:23 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 11-05 03:21:23 enc_dec_model_runner.py:140] EncoderDecoderModelRunner requires XFormers backend; overriding backend auto-selection and forcing XFormers.
(VllmWorkerProcess pid=67) INFO 11-05 03:21:23 enc_dec_model_runner.py:140] EncoderDecoderModelRunner requires XFormers backend; overriding backend auto-selection and forcing XFormers.
(VllmWorkerProcess pid=67) INFO 11-05 03:21:23 selector.py:116] Using XFormers backend.
INFO 11-05 03:21:23 selector.py:116] Using XFormers backend.
/usr/local/lib/python3.12/dist-packages/xformers/ops/fmha/flash.py:211: FutureWarning: torch.library.impl_abstract was renamed to torch.library.register_fake. Please use that instead; we will remove torch.library.impl_abstract in a future version of PyTorch.
@torch.library.impl_abstract(“xformers_flash::flash_fwd”)
(VllmWorkerProcess pid=67) /usr/local/lib/python3.12/dist-packages/xformers/ops/fmha/flash.py:211: FutureWarning: torch.library.impl_abstract was renamed to torch.library.register_fake. Please use that instead; we will remove torch.library.impl_abstract in a future version of PyTorch.
(VllmWorkerProcess pid=67) @torch.library.impl_abstract(“xformers_flash::flash_fwd”)
/usr/local/lib/python3.12/dist-packages/xformers/ops/fmha/flash.py:344: FutureWarning: torch.library.impl_abstract was renamed to torch.library.register_fake. Please use that instead; we will remove torch.library.impl_abstract in a future version of PyTorch.
@torch.library.impl_abstract(“xformers_flash::flash_bwd”)
(VllmWorkerProcess pid=67) /usr/local/lib/python3.12/dist-packages/xformers/ops/fmha/flash.py:344: FutureWarning: torch.library.impl_abstract was renamed to torch.library.register_fake. Please use that instead; we will remove torch.library.impl_abstract in a future version of PyTorch.
(VllmWorkerProcess pid=67) @torch.library.impl_abstract(“xformers_flash::flash_bwd”)
(VllmWorkerProcess pid=67) INFO 11-05 03:21:23 multiproc_worker_utils.py:218] Worker ready; awaiting tasks
INFO 11-05 03:21:24 utils.py:992] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=67) INFO 11-05 03:21:24 utils.py:992] Found nccl from library libnccl.so.2
INFO 11-05 03:21:24 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=67) INFO 11-05 03:21:24 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 11-05 03:21:24 custom_all_reduce_utils.py:204] generating GPU P2P access cache in /root/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
INFO 11-05 03:21:35 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
(VllmWorkerProcess pid=67) INFO 11-05 03:21:35 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
INFO 11-05 03:21:35 shm_broadcast.py:241] vLLM message queue communication handle: Handle(connect_ip=‘127.0.0.1’, local_reader_ranks=[1], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7f3afe2f6570>, local_subscribe_port=33245, remote_subscribe_port=None)
INFO 11-05 03:21:35 model_runner.py:1014] Starting to load model /data/model/llama…
(VllmWorkerProcess pid=67) INFO 11-05 03:21:35 model_runner.py:1014] Starting to load model /data/model/llama…
INFO 11-05 03:21:35 selector.py:116] Using XFormers backend.
(VllmWorkerProcess pid=67) INFO 11-05 03:21:35 selector.py:116] Using XFormers backend.
Loading safetensors checkpoint shards: 0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 20% Completed | 1/5 [00:01<00:04, 1.08s/it]
Loading safetensors checkpoint shards: 40% Completed | 2/5 [00:02<00:03, 1.22s/it]
Loading safetensors checkpoint shards: 60% Completed | 3/5 [00:03<00:02, 1.23s/it]
Loading safetensors checkpoint shards: 80% Completed | 4/5 [00:05<00:01, 1.28s/it]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:05<00:00, 1.05it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:05<00:00, 1.07s/it]
INFO 11-05 03:21:41 model_runner.py:1025] Loading model weights took 10.0714 GB (VllmWorkerProcess pid=67) INFO 11-05 03:21:41 model_runner.py:1025] Loading model weights took 10.0714 GB INFO 11-05 03:21:41 enc_dec_model_runner.py:297] Starting profile run for multi-modal models. (VllmWorkerProcess pid=67) INFO 11-05 03:21:41 enc_dec_model_runner.py:297] Starting profile run for multi-modal models. (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] Exception in worker VllmWorkerProcess while processing method determine_num_available_blocks: CUDA out of memory. Tried to allocate 1.95 GiB. GPU 1 has a total capacity of 22.09 GiB of which 758.44 MiB is free. Process 2751070 has 21.35 GiB memory in use. Of the allocated memory 20.90 GiB is allocated by PyTorch, and 85.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables), Traceback (most recent call last): (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/vllm/executor/multiproc_worker_utils.py”, line 226, in _run_worker_process (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] output = executor(*args, **kwargs) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py”, line 116, in decorate_context (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] return func(*args, **kwargs) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py”, line 223, in determine_num_available_blocks (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] self.model_runner.profile_run() (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py”, line 116, in decorate_context (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] return func(*args, **kwargs) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/vllm/worker/enc_dec_model_runner.py”, line 348, in profile_run (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] self.execute_model(model_input, kv_caches, intermediate_tensors) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py”, line 116, in decorate_context (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] return func(*args, **kwargs) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/vllm/worker/enc_dec_model_runner.py”, line 201, in execute_model (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] hidden_or_intermediate_states = model_executable( (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] return self._call_impl(*args, **kwargs) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1562, in _call_impl (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] return forward_call(*args, **kwargs) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/mllama.py”, line 1084, in forward (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] cross_attention_states = self.vision_model(pixel_values, (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] return self._call_impl(*args, **kwargs) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1562, in _call_impl (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] return forward_call(*args, **kwargs) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/mllama.py”, line 508, in forward (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] patch_embeds = self.patch_embedding( (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] return self._call_impl(*args, **kwargs) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1562, in _call_impl (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] return forward_call(*args, **kwargs) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/mllama.py”, line 229, in forward (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] x, _ = self._linear(x) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] return self._call_impl(*args, **kwargs) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1562, in call_impl (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] return forward_call(*args, **kwargs) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py”, line 367, in forward (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] output_parallel = self.quant_method.apply(self, input, bias) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py”, line 135, in apply (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] return F.linear(x, layer.weight, bias) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.95 GiB. GPU 1 has a total capacity of 22.09 GiB of which 758.44 MiB is free. Process 2751070 has 21.35 GiB memory in use. Of the allocated memory 20.90 GiB is allocated by PyTorch, and 85.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) (VllmWorkerProcess pid=67) ERROR 11-05 03:21:52 multiproc_worker_utils.py:233] (VllmWorkerProcess pid=67) INFO 11-05 03:21:53 multiproc_worker_utils.py:244] Worker exiting INFO 11-05 03:21:53 multiproc_worker_utils.py:124] Killing local vLLM worker processes Process SpawnProcess-1: Traceback (most recent call last): File “/usr/lib/python3.12/multiprocessing/process.py”, line 314, in _bootstrap self.run() File “/usr/lib/python3.12/multiprocessing/process.py”, line 108, in run self._target(*self._args, **self._kwargs) File “/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py”, line 388, in run_mp_engine engine = MQLLMEngine.from_engine_args(engine_args=engine_args, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py”, line 138, in from_engine_args return cls( ^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py”, line 78, in init self.engine = LLMEngine(*args, ^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py”, line 339, in init self._initialize_kv_caches() File “/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py”, line 474, in _initialize_kv_caches self.model_executor.determine_num_available_blocks()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/executor/distributed_gpu_executor.py”, line 39, in determine_num_available_blocks num_blocks = self._run_workers(“determine_num_available_blocks”, ) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/executor/multiproc_gpu_executor.py”, line 185, in _run_workers driver_worker_output = driver_worker_method(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py”, line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py”, line 223, in determine_num_available_blocks self.model_runner.profile_run() File “/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py”, line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/worker/enc_dec_model_runner.py”, line 348, in profile_run self.execute_model(model_input, kv_caches, intermediate_tensors) File “/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py”, line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/worker/enc_dec_model_runner.py”, line 201, in execute_model hidden_or_intermediate_states = model_executable( ^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/mllama.py”, line 1084, in forward cross_attention_states = self.vision_model(pixel_values, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/mllama.py”, line 508, in forward patch_embeds = self.patch_embedding( ^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/mllama.py”, line 229, in forward x, _ = self._linear(x) ^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1562, in call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py”, line 367, in forward output_parallel = self.quant_method.apply(self, input, bias) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py”, line 135, in apply return F.linear(x, layer.weight, bias) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.95 GiB. GPU 0 has a total capacity of 22.09 GiB of which 758.44 MiB is free. Process 2751020 has 21.35 GiB memory in use. Of the allocated memory 20.90 GiB is allocated by PyTorch, and 85.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) [rank0]:[W1105 03:21:54.083552440 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (13 by maintainers)
I used following command,it’s worked
command:
docker run --runtime nvidia --gpus '"device=0,1"' -d -v /data/model/llama:/data/model/llama -p 8001:8000 -e CUDA_LAUNCH_BLOCKING=1 --shm-size=24g vllm/vllm-openai:v0.6.3 --model /data/model/llama --served_model_name Llama-3.2-11B-Vision-Instruct --tensor-parallel-size 2 --gpu_memory_utilization 0.7 --max-model-len 4096 --max-num-seqs 1 --enforce-eagerThanks @DarkLight1337 and @youkaichao Regards
this is the error. if you are running docker, it means the shared memory size is not enough. see https://docs.vllm.ai/en/latest/serving/deploying_with_docker.html
the messages are quite long and I don’t get the key points.
You can try reducing
max_model_lenand/ormax_num_seqsto get the model to run (but even if this works, your throughput will be quite poor)