vllm: [Performance]: lmcache cannot work!
Your current environment
My test environment is as follow:
Environment: 2x H100 with NVLink
My testing method:
# P instance
UCX_TLS=cuda_ipc,cuda_copy,tcp \
LMCACHE_CONFIG_FILE=/work/vllm-0.8.5/examples/lmcache/disagg_prefill_lmcache_v1/configs/lmcache-prefiller-config.yaml \
LMCACHE_USE_EXPERIMENTAL=True \
VLLM_ENABLE_V1_MULTIPROCESSING=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
CUDA_VISIBLE_DEVICES=0 \
vllm serve /work/Qwen/Qwen2.5-0.5B-Instruct \
--port 8100 \
--disable-log-requests \
--enforce-eager \
--kv-transfer-config \
'{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_producer","kv_connector_extra_config": {"discard_partial_chunks": false, "lmcache_rpc_port": "producer1"}}'
# D instance
UCX_TLS=cuda_ipc,cuda_copy,tcp \
LMCACHE_CONFIG_FILE=/work/vllm-0.8.5/examples/lmcache/disagg_prefill_lmcache_v1/configs/lmcache-decoder-config.yaml \
LMCACHE_USE_EXPERIMENTAL=True \
VLLM_ENABLE_V1_MULTIPROCESSING=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
CUDA_VISIBLE_DEVICES=1 \
vllm serve /work/Qwen/Qwen2.5-0.5B-Instruct \
--port 8200 \
--disable-log-requests \
--enforce-eager \
--kv-transfer-config \
'{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_consumer","kv_connector_extra_config": {"discard_partial_chunks": false, "lmcache_rpc_port": "consumer1"}}'
# proxy
python3 /work/vllm-0.8.5/examples/lmcache/disagg_prefill_lmcache_v1/disagg_proxy_server.py \
--host localhost \
--port 9000 \
--prefiller-host localhost \
--prefiller-port 8100 \
--decoder-host localhost \
--decoder-port 8200
Errors encountered during operation:
UCX_TLS=cuda_ipc,cuda_copy,tcp \
> LMCACHE_CONFIG_FILE=/work/vllm-0.8.5/examples/lmcache/disagg_prefill_lmcache_v1/configs/lmcache-prefiller-config.yaml \
> LMCACHE_USE_EXPERIMENTAL=True \
> VLLM_ENABLE_V1_MULTIPROCESSING=1 \
> VLLM_WORKER_MULTIPROC_METHOD=spawn \
> CUDA_VISIBLE_DEVICES=0 \
> vllm serve /work/Qwen/Qwen2.5-0.5B-Instruct \
> --port 8100 \
> --disable-log-requests \
> --enforce-eager \
> --kv-transfer-config \
> '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_producer","kv_connector_extra_config": {"discard_partial_chunks": false, "lmcache_rpc_port": "producer1"}}'
INFO 05-14 02:41:16 [__init__.py:239] Automatically detected platform cuda.
INFO 05-14 02:41:20 [api_server.py:1043] vLLM API server version 0.8.5
INFO 05-14 02:41:20 [api_server.py:1044] args: Namespace(subparser='serve', model_tag='/work/Qwen/Qwen2.5-0.5B-Instruct', config='', host=None, port=8100, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/work/Qwen/Qwen2.5-0.5B-Instruct', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', max_model_len=None, guided_decoding_backend='auto', reasoning_parser=None, logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.9, swap_space=4, kv_cache_dtype='auto', num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=True, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', speculative_config=None, ignore_patterns=[], served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=KVTransferConfig(kv_connector='LMCacheConnectorV1', kv_buffer_device='cuda', kv_buffer_size=1000000000.0, kv_role='kv_producer', kv_rank=None, kv_parallel_size=1, kv_ip='127.0.0.1', kv_port=14579, kv_connector_extra_config={'discard_partial_chunks': False, 'lmcache_rpc_port': 'producer1'}), worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x7f5910656e60>)
WARNING 05-14 02:41:20 [utils.py:2267] Found ulimit of 1024 and failed to automatically increase with error current limit exceeds maximum limit. This can cause fd limit errors like `OSError: [Errno 24] Too many open files`. Consider increasing with ulimit -n
INFO 05-14 02:41:28 [config.py:717] This model supports multiple tasks: {'reward', 'generate', 'embed', 'classify', 'score'}. Defaulting to 'generate'.
INFO 05-14 02:41:28 [config.py:2003] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 05-14 02:41:28 [cuda.py:93] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 05-14 02:41:32 [__init__.py:239] Automatically detected platform cuda.
INFO 05-14 02:41:35 [core.py:58] Initializing a V1 LLM engine (v0.8.5) with config: model='/work/Qwen/Qwen2.5-0.5B-Instruct', speculative_config=None, tokenizer='/work/Qwen/Qwen2.5-0.5B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=/work/Qwen/Qwen2.5-0.5B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[],"max_capture_size":0}
WARNING 05-14 02:41:35 [utils.py:2522] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7f79a4660d60>
[rank0]:[W514 02:41:36.836683514 ProcessGroupGloo.cpp:727] Warning: Unable to resolve hostname to a (local) address. Using the loopback address as fallback. Manually set the network interface to bind to with GLOO_SOCKET_IFNAME. (function operator())
INFO 05-14 02:41:36 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 05-14 02:41:36 [factory.py:64] Creating v1 connector with name: LMCacheConnectorV1
WARNING 05-14 02:41:36 [base.py:58] Initializing KVConnectorBase_V1. This API is experimental and subject to change in the future as we iterate the design.
[2025-05-14 02:41:36,425] LMCache INFO: Loading LMCache config file /work/vllm-0.8.5/examples/lmcache/disagg_prefill_lmcache_v1/configs/lmcache-prefiller-config.yaml (utils.py:32:lmcache.integration.vllm.utils)
[2025-05-14 02:41:36,428] LMCache INFO: Creating LMCacheEngine instance vllm-instance (cache_engine.py:444:lmcache.experimental.cache_engine)
[2025-05-14 02:41:36,428] LMCache INFO: Creating LMCacheEngine with config: LMCacheEngineConfig(chunk_size=256, local_cpu=False, max_local_cpu_size=0, local_disk=None, max_local_disk_size=0, remote_url=None, remote_serde=None, save_decode_cache=False, enable_blending=False, blend_recompute_ratio=0.15, blend_min_tokens=256, blend_special_str=' # # ', enable_p2p=False, lookup_url=None, distributed_url=None, error_handling=False, enable_controller=False, lmcache_instance_id='lmcache_default_instance', controller_url=None, lmcache_worker_url=None, enable_nixl=True, nixl_role='sender', nixl_peer_host='localhost', nixl_peer_port=55555, nixl_buffer_size=1073741824, nixl_buffer_device='cuda', nixl_enable_gc=True) (cache_engine.py:73:lmcache.experimental.cache_engine)
[1747215696.528332] [b3ab5fdd703b:1615 :0] ucp_context.c:1268 UCX WARN transports 'cuda_ipc','cuda_copy' are not available, please use one or more of: mm, posix, self, shm, sm, sysv, tcp
Backend UCX was instantiated
Initialized NIXL agent: NixlRole.SENDER
ERROR 05-14 02:41:36 [core.py:396] EngineCore failed to start.
ERROR 05-14 02:41:36 [core.py:396] Traceback (most recent call last):
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 387, in run_engine_core
ERROR 05-14 02:41:36 [core.py:396] engine_core = EngineCoreProc(*args, **kwargs)
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 329, in __init__
ERROR 05-14 02:41:36 [core.py:396] super().__init__(vllm_config, executor_class, log_stats,
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 64, in __init__
ERROR 05-14 02:41:36 [core.py:396] self.model_executor = executor_class(vllm_config)
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 52, in __init__
ERROR 05-14 02:41:36 [core.py:396] self._init_executor()
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 46, in _init_executor
ERROR 05-14 02:41:36 [core.py:396] self.collective_rpc("init_device")
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 05-14 02:41:36 [core.py:396] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 2456, in run_method
ERROR 05-14 02:41:36 [core.py:396] return func(*args, **kwargs)
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 604, in init_device
ERROR 05-14 02:41:36 [core.py:396] self.worker.init_device() # type: ignore
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 135, in init_device
ERROR 05-14 02:41:36 [core.py:396] init_worker_distributed_environment(self.vllm_config, self.rank,
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 329, in init_worker_distributed_environment
ERROR 05-14 02:41:36 [core.py:396] ensure_kv_transfer_initialized(vllm_config)
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 63, in ensure_kv_transfer_initialized
ERROR 05-14 02:41:36 [core.py:396] _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector_v1(
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 73, in create_connector_v1
ERROR 05-14 02:41:36 [core.py:396] return connector_cls(config, role)
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/lmcache_connector.py", line 25, in __init__
ERROR 05-14 02:41:36 [core.py:396] self._lmcache_engine = LMCacheConnectorV1Impl(vllm_config, role, self)
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/lmcache/integration/vllm/vllm_v1_adapter.py", line 314, in __init__
ERROR 05-14 02:41:36 [core.py:396] self.lmcache_engine = init_lmcache_engine(
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/lmcache/integration/vllm/vllm_adapter.py", line 111, in init_lmcache_engine
ERROR 05-14 02:41:36 [core.py:396] engine = LMCacheEngineBuilder.get_or_create(ENGINE_NAME, config, metadata,
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/cache_engine.py", line 449, in get_or_create
ERROR 05-14 02:41:36 [core.py:396] engine = LMCacheEngine(config, metadata, memory_allocator,
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/cache_engine.py", line 98, in __init__
ERROR 05-14 02:41:36 [core.py:396] self.storage_manager = DistributedStorageManager(
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/storage_manager.py", line 535, in __init__
ERROR 05-14 02:41:36 [core.py:396] self.storage_backend = NixlBackend.CreateNixlBackend(config, metadata)
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/nixl_backend.py", line 412, in CreateNixlBackend
ERROR 05-14 02:41:36 [core.py:396] backend = NixlBackend(nixl_config)
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/nixl_backend.py", line 249, in __init__
ERROR 05-14 02:41:36 [core.py:396] self._nixl_channel = NixlChannel(nixl_config)
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/connector/nixl_connector_v2.py", line 454, in __init__
ERROR 05-14 02:41:36 [core.py:396] self._pipe = NixlPipe(nixl_config, self._side_channel)
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/connector/nixl_connector_v2.py", line 190, in __init__
ERROR 05-14 02:41:36 [core.py:396] self._reg_descs = self._agent.register_memory(self._transfer_buffers)
ERROR 05-14 02:41:36 [core.py:396] File "/usr/local/lib/python3.10/dist-packages/nixl/_api.py", line 265, in register_memory
ERROR 05-14 02:41:36 [core.py:396] self.agent.registerMem(reg_descs, handle_list)
ERROR 05-14 02:41:36 [core.py:396] nixl._bindings.nixlBackendError: NIXL_ERR_BACKEND
Process EngineCore_0:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 400, in run_engine_core
raise e
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 387, in run_engine_core
engine_core = EngineCoreProc(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 329, in __init__
super().__init__(vllm_config, executor_class, log_stats,
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 64, in __init__
self.model_executor = executor_class(vllm_config)
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 52, in __init__
self._init_executor()
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 46, in _init_executor
self.collective_rpc("init_device")
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
answer = run_method(self.driver_worker, method, args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 2456, in run_method
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 604, in init_device
self.worker.init_device() # type: ignore
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 135, in init_device
init_worker_distributed_environment(self.vllm_config, self.rank,
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 329, in init_worker_distributed_environment
ensure_kv_transfer_initialized(vllm_config)
File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 63, in ensure_kv_transfer_initialized
_KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector_v1(
File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 73, in create_connector_v1
return connector_cls(config, role)
File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/lmcache_connector.py", line 25, in __init__
self._lmcache_engine = LMCacheConnectorV1Impl(vllm_config, role, self)
File "/usr/local/lib/python3.10/dist-packages/lmcache/integration/vllm/vllm_v1_adapter.py", line 314, in __init__
self.lmcache_engine = init_lmcache_engine(
File "/usr/local/lib/python3.10/dist-packages/lmcache/integration/vllm/vllm_adapter.py", line 111, in init_lmcache_engine
engine = LMCacheEngineBuilder.get_or_create(ENGINE_NAME, config, metadata,
File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/cache_engine.py", line 449, in get_or_create
engine = LMCacheEngine(config, metadata, memory_allocator,
File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/cache_engine.py", line 98, in __init__
self.storage_manager = DistributedStorageManager(
File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/storage_manager.py", line 535, in __init__
self.storage_backend = NixlBackend.CreateNixlBackend(config, metadata)
File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/nixl_backend.py", line 412, in CreateNixlBackend
backend = NixlBackend(nixl_config)
File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/nixl_backend.py", line 249, in __init__
self._nixl_channel = NixlChannel(nixl_config)
File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/connector/nixl_connector_v2.py", line 454, in __init__
self._pipe = NixlPipe(nixl_config, self._side_channel)
File "/usr/local/lib/python3.10/dist-packages/lmcache/experimental/storage_backend/connector/nixl_connector_v2.py", line 190, in __init__
self._reg_descs = self._agent.register_memory(self._transfer_buffers)
File "/usr/local/lib/python3.10/dist-packages/nixl/_api.py", line 265, in register_memory
self.agent.registerMem(reg_descs, handle_list)
nixl._bindings.nixlBackendError: NIXL_ERR_BACKEND
^CException ignored in: <function Context.__del__ at 0x7f7a84a380d0>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/zmq/sugar/context.py", line 142, in __del__
self.destroy()
File "/usr/local/lib/python3.10/dist-packages/zmq/sugar/context.py", line 324, in destroy
self.term()
File "/usr/local/lib/python3.10/dist-packages/zmq/sugar/context.py", line 266, in term
super().term()
File "_zmq.py", line 564, in zmq.backend.cython._zmq.Context.term
File "_zmq.py", line 160, in zmq.backend.cython._zmq._check_rc
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 371, in signal_handler
raise SystemExit()
SystemExit:
[rank0]:[W514 02:57:14.981244545 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
^CTraceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 398, in __init__
self._wait_for_engine_startup()
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 423, in _wait_for_engine_startup
events = poller.poll(STARTUP_POLL_PERIOD_MS)
File "/usr/local/lib/python3.10/dist-packages/zmq/sugar/poll.py", line 106, in poll
return zmq_poll(self.sockets, timeout=timeout)
File "_zmq.py", line 1555, in zmq.backend.cython._zmq.zmq_poll
File "_zmq.py", line 160, in zmq.backend.cython._zmq._check_rc
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/vllm", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/main.py", line 53, in main
args.dispatch_function(args)
File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/serve.py", line 27, in cmd
uvloop.run(run_server(args))
File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 82, in run
return loop.run_until_complete(wrapper())
File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever
File "uvloop/loop.pyx", line 555, in uvloop.loop.Loop._run
File "uvloop/loop.pyx", line 474, in uvloop.loop.Loop._on_idle
File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run
File "uvloop/cbhandles.pyx", line 61, in uvloop.loop.Handle._run
File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 61, in wrapper
return await main
File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 1078, in run_server
async with build_async_engine_client(args) as engine_client:
File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
return await anext(self.gen)
File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
return await anext(self.gen)
File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 178, in build_async_engine_client_from_engine_args
async_llm = AsyncLLM.from_vllm_config(
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 150, in from_vllm_config
return cls(
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 118, in __init__
self.engine_core = core_client_class(
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 642, in __init__
super().__init__(
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 410, in __init__
self._finalizer()
File "/usr/lib/python3.10/weakref.py", line 591, in __call__
return info.func(*info.args, **(info.kwargs or {}))
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 319, in __call__
core_engine.close()
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 295, in close
proc_handle.shutdown()
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/utils.py", line 128, in shutdown
self._finalizer()
File "/usr/lib/python3.10/weakref.py", line 591, in __call__
return info.func(*info.args, **(info.kwargs or {}))
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/utils.py", line 137, in shutdown
proc.join(5)
File "/usr/lib/python3.10/multiprocessing/process.py", line 149, in join
res = self._popen.wait(timeout)
File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 40, in wait
if not wait([self.sentinel], timeout):
File "/usr/lib/python3.10/multiprocessing/connection.py", line 931, in wait
ready = selector.select(timeout)
File "/usr/lib/python3.10/selectors.py", line 416, in select
fd_event_list = self._selector.poll(timeout)
pip list
Package Version
---------------------------------------- --------------------
accelerate 0.34.0
aiofiles 23.2.1
aiohappyeyeballs 2.4.0
aiohttp 3.10.5
aiohttp-cors 0.7.0
aioprometheus 23.12.0
aiosignal 1.3.1
airportsdata 20241001
aliyun-python-sdk-core 2.16.0
aliyun-python-sdk-kms 2.16.5
altair 5.5.0
annotated-types 0.7.0
antlr4-python3-runtime 4.9.3
anyascii 0.3.2
anyio 4.4.0
argcomplete 3.5.3
astor 0.8.1
async-timeout 4.0.3
attrdict 2.0.1
attrs 24.2.0
audioread 3.0.1
auto_gptq 0.7.1
autoawq 0.2.5
autoawq_kernels 0.0.9
av 14.0.1
babel 2.16.0
bcrypt 4.2.1
beautifulsoup4 4.12.3
bitsandbytes 0.45.1
black 24.10.0
blake3 1.0.4
blinker 1.9.0
boto3 1.28.64
botocore 1.31.85
cached_path 1.6.7
cachetools 5.5.1
cdifflib 1.2.9
certifi 2019.11.28
cffi 1.17.1
chardet 3.0.4
charset-normalizer 3.3.2
chattts 0.2.1
click 8.1.7
cloudpickle 3.0.0
cn2an 0.5.23
colorama 0.4.6
coloredlogs 15.0.1
colorful 0.5.6
compressed-tensors 0.9.3
conformer 0.3.2
contourpy 1.3.1
controlnet-aux 0.0.7
crcmod 1.7
cryptography 44.0.0
cuda-python 12.6.2.post1
cupy-cuda12x 13.4.0
cycler 0.12.1
Cython 3.0.11
dataclass-wizard 0.35.0
datamodel-code-generator 0.26.5
datasets 2.21.0
dateparser 1.1.8
dbus-python 1.2.16
decorator 5.1.1
decord 0.6.0
DeepCache 0.1.1
Deprecated 1.2.15
depyf 0.18.0
diffusers 0.32.2
dill 0.3.8
diskcache 5.6.3
Distance 0.1.3
distlib 0.3.9
distro 1.9.0
distro-info 0.23+ubuntu1.1
dnspython 2.7.0
docopt 0.6.2
ecdsa 0.19.0
editdistance 0.8.1
einops 0.8.0
einx 0.3.0
email_validator 2.2.0
encodec 0.1.1
eva-decord 0.6.1
exceptiongroup 1.2.2
fastapi 0.115.11
fastapi-cli 0.0.7
fastrlock 0.8.3
ffmpy 0.5.0
filelock 3.17.0
FlagEmbedding 1.2.11
flashinfer 0.1.6+cu124torch2.4
Flask 3.1.0
flatbuffers 25.1.21
fonttools 4.55.5
frozendict 2.4.6
frozenlist 1.4.1
fsspec 2024.6.1
fugashi 1.4.0
funasr 1.1.16
fvcore 0.1.5.post20221221
g2p-en 2.1.0
gdown 5.2.0
gekko 1.2.1
genson 1.3.0
gguf 0.16.3
google-api-core 2.24.0
google-auth 2.38.0
google-cloud-core 2.4.1
google-cloud-storage 2.19.0
google-crc32c 1.6.0
google-resumable-media 2.7.2
googleapis-common-protos 1.66.0
gradio 4.26.0
gradio_client 0.15.1
grpcio 1.70.0
gruut 2.4.0
gruut-ipa 0.13.0
gruut-lang-de 2.0.1
gruut-lang-en 2.0.1
gruut-lang-es 2.0.1
gruut-lang-fr 2.0.2
h11 0.14.0
h2 4.2.0
hf_transfer 0.1.8
hf-xet 1.0.2
hiredis 3.1.0
hpack 4.1.0
httpcore 1.0.5
httptools 0.6.1
httpx 0.27.2
huggingface-hub 0.30.1
humanfriendly 10.0
hydra-core 1.3.2
Hypercorn 0.17.3
hyperframe 6.1.0
HyperPyYAML 1.2.2
idna 2.8
imageio 2.37.0
imageio-ffmpeg 0.6.0
importlib_metadata 8.0.0
importlib_resources 6.5.2
infinistore 0.2.35
inflect 5.6.2
iniconfig 2.0.0
interegular 0.3.3
iopath 0.1.10
isort 5.13.2
itsdangerous 2.2.0
jaconv 0.4.0
jamo 0.4.1
jieba 0.42.1
Jinja2 3.1.5
jiter 0.5.0
jj-pytorchvideo 0.1.5
jmespath 0.10.0
joblib 1.4.2
jsonlines 1.2.0
jsonschema 4.23.0
jsonschema-specifications 2023.12.1
kaldiio 2.18.0
kiwisolver 1.4.8
lark 1.2.2
lazy_loader 0.4
libnacl 2.1.0
librosa 0.10.2.post1
lightning 2.5.0.post0
lightning-utilities 0.11.9
linkify-it-py 2.0.3
llama_cpp_python 0.3.4
llguidance 0.7.13
llvmlite 0.44.0
lm-format-enforcer 0.10.11
lmcache 0.2.1
loguru 0.7.3
loralib 0.1.2
markdown-it-py 3.0.0
MarkupSafe 2.1.5
matplotlib 3.10.0
mdit-py-plugins 0.4.2
mdurl 0.1.2
mecab-python3 1.0.10
memray 1.15.0
mistral_common 1.5.4
modelscope 1.18.1
mpmath 1.3.0
msgpack 1.0.8
msgspec 0.18.6
multidict 6.0.5
multiprocess 0.70.16
mypy-extensions 1.0.0
nanobind 2.6.1
narwhals 1.23.0
natsort 8.4.0
nemo_text_processing 1.0.2
nest-asyncio 1.6.0
networkx 3.3
ninja 1.11.1.3
nixl 0.2.0
nltk 3.9.1
num2words 0.5.14
numba 0.61.2
numpy 1.26.4
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.2.1.3
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu12 12.3.1.170
nvidia-cusparselt-cu12 0.6.2
nvidia-ml-py 12.560.30
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.4.127
nvtx 0.2.11
omegaconf 2.3.0
onnxruntime 1.20.1
onnxruntime-gpu 1.16.0
openai 1.60.0
opencensus 0.11.4
opencensus-context 0.1.3
opencv-contrib-python-headless 4.11.0.86
opencv-python 4.11.0.86
opencv-python-headless 4.11.0.86
opentelemetry-api 1.26.0
opentelemetry-exporter-otlp 1.26.0
opentelemetry-exporter-otlp-proto-common 1.26.0
opentelemetry-exporter-otlp-proto-grpc 1.26.0
opentelemetry-exporter-otlp-proto-http 1.26.0
opentelemetry-proto 1.26.0
opentelemetry-sdk 1.26.0
opentelemetry-semantic-conventions 0.47b0
opentelemetry-semantic-conventions-ai 0.4.6
optimum 1.23.3
orjson 3.10.15
ormsgpack 1.7.0
oss2 2.19.1
outlines 0.1.11
outlines_core 0.1.26
packaging 24.1
pandas 2.2.2
parameterized 0.9.0
partial-json-parser 0.2.1.1.post4
passlib 1.7.4
pathspec 0.12.1
peft 0.14.0
pillow 10.4.0
pip 24.3.1
platformdirs 4.3.6
pluggy 1.5.0
pooch 1.8.2
portalocker 3.1.1
prettytable 3.14.0
priority 2.0.0
proces 0.1.7
prometheus_client 0.20.0
prometheus-fastapi-instrumentator 7.0.0
proto-plus 1.25.0
protobuf 4.25.7
psutil 6.0.0
py-cpuinfo 9.0.0
py-spy 0.4.0
pyairports 2.1.1
pyarrow 17.0.0
pyasn1 0.6.1
pyasn1_modules 0.4.1
pybase16384 0.3.7
pybind11 2.13.6
pycountry 24.6.1
pycparser 2.22
pycryptodome 3.21.0
pydantic 2.10.6
pydantic_core 2.27.2
pydub 0.25.1
Pygments 2.19.1
PyGObject 3.36.0
pykakasi 2.3.0
pynini 2.1.5
pynndescent 0.5.13
pyparsing 3.2.1
pypinyin 0.53.0
PySocks 1.7.1
pytest 8.3.4
python-apt 2.0.1+ubuntu0.20.4.1
python-crfsuite 0.9.11
python-dateutil 2.9.0.post0
python-dotenv 1.0.1
python-jose 3.3.0
python-json-logger 3.2.1
python-multipart 0.0.20
python-rapidjson 1.20
pytorch-lightning 2.5.0.post0
pytorch-wpe 0.0.1
pytz 2024.1
PyYAML 6.0.2
pyzmq 26.2.0
quantile-python 1.1
Quart 0.20.0
qwen-vl-utils 0.0.8
ray 2.43.0
redis 5.2.1
referencing 0.35.1
regex 2024.7.24
requests 2.32.3
requests-unixsocket 0.2.0
rich 13.9.4
rich-toolkit 0.13.2
rouge 1.0.1
rpds-py 0.20.0
rsa 4.9
ruamel.yaml 0.18.10
ruamel.yaml.clib 0.2.12
ruff 0.9.3
s3transfer 0.7.0
sacremoses 0.1.1
safetensors 0.4.4
scikit-image 0.25.0
scikit-learn 1.6.1
scipy 1.15.1
semantic-version 2.10.0
sentence-transformers 3.4.0
sentencepiece 0.2.0
setproctitle 1.3.4
setuptools 75.8.0
sglang 0.4.1.post7
shellingham 1.5.4
silero-vad 5.1.2
six 1.17.0
smart-open 7.1.0
sniffio 1.3.1
sortedcontainers 2.4.0
soundfile 0.13.0
soupsieve 2.6
soxr 0.5.0.post1
sse-starlette 2.1.3
starlette 0.46.0
sympy 1.13.1
tabulate 0.9.0
taskgroup 0.2.2
tblib 3.0.0
tensorboardX 2.6.2.2
tensorizer 2.9.1
termcolor 2.5.0
text-generation 0.7.0
textual 1.0.0
threadpoolctl 3.5.0
tifffile 2025.1.10
tiktoken 0.7.0
timm 1.0.14
tokenizers 0.21.1
toml 0.10.2
tomli 2.2.1
tomlkit 0.12.0
torch 2.6.0
torch-complex 0.4.4
torchao 0.10.0
torchaudio 2.6.0
torchdiffeq 0.2.5
torchmetrics 1.6.1
torchvision 0.21.0
tqdm 4.66.5
transformers 4.51.3
transformers-stream-generator 0.0.5
triton 3.2.0
tritonclient 2.54.0
typer 0.15.2
typing_extensions 4.12.2
tzdata 2024.1
tzlocal 5.2
uc-micro-py 1.0.3
umap-learn 0.5.7
unattended-upgrades 0.1
unidic-lite 1.0.8
urllib3 2.0.7
uvicorn 0.30.6
uvloop 0.20.0
vector-quantize-pytorch 1.17.3
verovio 4.5.1
virtualenv 20.29.2
vllm 0.8.5
vllm-flash-attn 2.6.1
vocos 0.1.0
watchfiles 0.24.0
wcwidth 0.2.13
websockets 11.0.3
Werkzeug 3.1.3
WeTextProcessing 1.0.3
wget 3.2
wheel 0.34.2
wrapt 1.17.2
wsproto 1.2.0
x-transformers 1.44.6
xformers 0.0.29.post2
xgrammar 0.1.18
xinference 1.2.2
xoscar 0.4.6
xxhash 3.5.0
yacs 0.1.8
yarl 1.9.9
zipp 3.20.1
zstandard 0.23.0
How would you like to use vllm
I want to run inference of a [specific model](put link here). I don’t know how to integrate it with vllm.
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 19 (7 by maintainers)
I have raised an issue in lmcache. I will sync it if there is any progress on the performance issue.
@tensorflowt Hey bro do you run pd disagg with vllm by Lmcache successful? Can I look your script and env configuration? I have some problem blocked here
Also if you r chinese we could be wechat friends and discuss further
I am using vllm version 0.8.5, and at this commit f25e0d1125f873201ae880b50df46a9e3d29f3ba.
It looks like ucx is not installed or configured correctly.
I ran into the same issue. And walkarounded by reinstalling ucx and nixl from source.
You can first test with lmcache nixl test.