llm-compressor: KeyError: 'layers.0.mlp.down_proj.weight_scale'
After executing https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w8a8_int8/llama3_example.py,I load the model with vllm
python -m "vllm.entrypoints.openai.api_server" --model "your/model/path" --tensor-parallel-size 2 --gpu-memory-utilization 0.9 --max-model-len 32768 --max-seq-len-to-capture 32768 --served-model-name default --block-size 16 --port 8849
however I get the error
(VllmWorkerProcess pid=3351580) INFO 12-05 20:24:41 selector.py:135] Using Flash Attention backend.
(VllmWorkerProcess pid=3351580) INFO 12-05 20:24:41 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3351580) INFO 12-05 20:24:42 utils.py:961] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3351580) INFO 12-05 20:24:42 pynccl.py:69] vLLM is using nccl==2.21.5
INFO 12-05 20:24:42 utils.py:961] Found nccl from library libnccl.so.2
INFO 12-05 20:24:42 pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorkerProcess pid=3351580) INFO 12-05 20:24:46 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/hcece/.cache/vllm/gpu_p2p_access_cache_for_4,5.json
INFO 12-05 20:24:46 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/hcece/.cache/vllm/gpu_p2p_access_cache_for_4,5.json
INFO 12-05 20:24:46 shm_broadcast.py:236] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7fb78e16bd30>, local_subscribe_port=48625, remote_subscribe_port=None)
INFO 12-05 20:24:46 model_runner.py:1072] Starting to load model /data/models/qwen2.5-0.5B/quantizedModel...
(VllmWorkerProcess pid=3351580) INFO 12-05 20:24:46 model_runner.py:1072] Starting to load model /data/models/qwen2.5-0.5B/quantizedModel...
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
ERROR 12-05 20:24:46 engine.py:366] 'layers.0.mlp.down_proj.weight_scale'
ERROR 12-05 20:24:46 engine.py:366] Traceback (most recent call last):
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
ERROR 12-05 20:24:46 engine.py:366] engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 119, in from_engine_args
ERROR 12-05 20:24:46 engine.py:366] return cls(ipc_path=ipc_path,
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 71, in __init__
ERROR 12-05 20:24:46 engine.py:366] self.engine = LLMEngine(*args, **kwargs)
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 347, in __init__
ERROR 12-05 20:24:46 engine.py:366] self.model_executor = executor_class(vllm_config=vllm_config, )
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 26, in __init__
ERROR 12-05 20:24:46 engine.py:366] super().__init__(*args, **kwargs)
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 36, in __init__
ERROR 12-05 20:24:46 engine.py:366] self._init_executor()
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 114, in _init_executor
ERROR 12-05 20:24:46 engine.py:366] self._run_workers("load_model",
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 195, in _run_workers
ERROR 12-05 20:24:46 engine.py:366] driver_worker_output = driver_worker_method(*args, **kwargs)
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/worker/worker.py", line 152, in load_model
ERROR 12-05 20:24:46 engine.py:366] self.model_runner.load_model()
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1074, in load_model
ERROR 12-05 20:24:46 engine.py:366] self.model = get_model(vllm_config=self.vllm_config)
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
ERROR 12-05 20:24:46 engine.py:366] return loader.load_model(vllm_config=vllm_config)
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 334, in load_model
ERROR 12-05 20:24:46 engine.py:366] model.load_weights(self._get_all_weights(model_config, model))
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 503, in load_weights
ERROR 12-05 20:24:46 engine.py:366] loader.load_weights(weights)
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 229, in load_weights
ERROR 12-05 20:24:46 engine.py:366] autoloaded_weights = list(self._load_module("", self.module, weights))
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 190, in _load_module
ERROR 12-05 20:24:46 engine.py:366] yield from self._load_module(prefix,
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 175, in _load_module
ERROR 12-05 20:24:46 engine.py:366] module_load_weights(weights)
ERROR 12-05 20:24:46 engine.py:366] File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 371, in load_weights
ERROR 12-05 20:24:46 engine.py:366] param = params_dict[name]
ERROR 12-05 20:24:46 engine.py:366] KeyError: 'layers.0.mlp.down_proj.weight_scale'
INFO 12-05 20:24:46 multiproc_worker_utils.py:120] Killing local vLLM worker processes
Process SpawnProcess-1:
Traceback (most recent call last):
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 368, in run_mp_engine
raise e
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 119, in from_engine_args
return cls(ipc_path=ipc_path,
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 71, in __init__
self.engine = LLMEngine(*args, **kwargs)
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 347, in __init__
self.model_executor = executor_class(vllm_config=vllm_config, )
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 26, in __init__
super().__init__(*args, **kwargs)
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 36, in __init__
self._init_executor()
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 114, in _init_executor
self._run_workers("load_model",
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 195, in _run_workers
driver_worker_output = driver_worker_method(*args, **kwargs)
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/worker/worker.py", line 152, in load_model
self.model_runner.load_model()
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1074, in load_model
self.model = get_model(vllm_config=self.vllm_config)
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
return loader.load_model(vllm_config=vllm_config)
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 334, in load_model
model.load_weights(self._get_all_weights(model_config, model))
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 503, in load_weights
loader.load_weights(weights)
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 229, in load_weights
autoloaded_weights = list(self._load_module("", self.module, weights))
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 190, in _load_module
yield from self._load_module(prefix,
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 175, in _load_module
module_load_weights(weights)
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 371, in load_weights
param = params_dict[name]
KeyError: 'layers.0.mlp.down_proj.weight_scale'
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
[rank0]:[W1205 20:24:46.112955842 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
Traceback (most recent call last):
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 643, in <module>
uvloop.run(run_server(args))
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run
return loop.run_until_complete(wrapper())
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper
return await main
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 609, in run_server
async with build_async_engine_client(args) as engine_client:
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/contextlib.py", line 199, in __aenter__
return await anext(self.gen)
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 113, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/contextlib.py", line 199, in __aenter__
return await anext(self.gen)
File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 210, in build_async_engine_client_from_engine_args
raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.
/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
How to solve this? Many thanks!
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17
@dsikka hello! I update vllm to 0.6.6 and the problem has been solved! Many thanks for your help!
Here are my package list:
@Kha-Zix-1 You shouldn’t need to make any changes. Based on the config you’ve provided, it doesn’t seem like quantization was applied. Can you confirm the script you used to quantize the model, as well as the version of llmcompressor you’re using?
Thanks