llm-compressor: KeyError: 'layers.0.mlp.down_proj.weight_scale'

After executing https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w8a8_int8/llama3_example.py,I load the model with vllm

python -m "vllm.entrypoints.openai.api_server" --model "your/model/path" --tensor-parallel-size 2 --gpu-memory-utilization 0.9 --max-model-len 32768 --max-seq-len-to-capture 32768 --served-model-name default --block-size 16 --port 8849

however I get the error

(VllmWorkerProcess pid=3351580) INFO 12-05 20:24:41 selector.py:135] Using Flash Attention backend.
(VllmWorkerProcess pid=3351580) INFO 12-05 20:24:41 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3351580) INFO 12-05 20:24:42 utils.py:961] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3351580) INFO 12-05 20:24:42 pynccl.py:69] vLLM is using nccl==2.21.5
INFO 12-05 20:24:42 utils.py:961] Found nccl from library libnccl.so.2
INFO 12-05 20:24:42 pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorkerProcess pid=3351580) INFO 12-05 20:24:46 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/hcece/.cache/vllm/gpu_p2p_access_cache_for_4,5.json
INFO 12-05 20:24:46 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/hcece/.cache/vllm/gpu_p2p_access_cache_for_4,5.json
INFO 12-05 20:24:46 shm_broadcast.py:236] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7fb78e16bd30>, local_subscribe_port=48625, remote_subscribe_port=None)
INFO 12-05 20:24:46 model_runner.py:1072] Starting to load model /data/models/qwen2.5-0.5B/quantizedModel...
(VllmWorkerProcess pid=3351580) INFO 12-05 20:24:46 model_runner.py:1072] Starting to load model /data/models/qwen2.5-0.5B/quantizedModel...
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
ERROR 12-05 20:24:46 engine.py:366] 'layers.0.mlp.down_proj.weight_scale'
ERROR 12-05 20:24:46 engine.py:366] Traceback (most recent call last):
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
ERROR 12-05 20:24:46 engine.py:366]     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 119, in from_engine_args
ERROR 12-05 20:24:46 engine.py:366]     return cls(ipc_path=ipc_path,
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 71, in __init__
ERROR 12-05 20:24:46 engine.py:366]     self.engine = LLMEngine(*args, **kwargs)
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 347, in __init__
ERROR 12-05 20:24:46 engine.py:366]     self.model_executor = executor_class(vllm_config=vllm_config, )
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 26, in __init__
ERROR 12-05 20:24:46 engine.py:366]     super().__init__(*args, **kwargs)
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 36, in __init__
ERROR 12-05 20:24:46 engine.py:366]     self._init_executor()
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 114, in _init_executor
ERROR 12-05 20:24:46 engine.py:366]     self._run_workers("load_model",
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 195, in _run_workers
ERROR 12-05 20:24:46 engine.py:366]     driver_worker_output = driver_worker_method(*args, **kwargs)
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/worker/worker.py", line 152, in load_model
ERROR 12-05 20:24:46 engine.py:366]     self.model_runner.load_model()
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1074, in load_model
ERROR 12-05 20:24:46 engine.py:366]     self.model = get_model(vllm_config=self.vllm_config)
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
ERROR 12-05 20:24:46 engine.py:366]     return loader.load_model(vllm_config=vllm_config)
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 334, in load_model
ERROR 12-05 20:24:46 engine.py:366]     model.load_weights(self._get_all_weights(model_config, model))
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 503, in load_weights
ERROR 12-05 20:24:46 engine.py:366]     loader.load_weights(weights)
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 229, in load_weights
ERROR 12-05 20:24:46 engine.py:366]     autoloaded_weights = list(self._load_module("", self.module, weights))
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 190, in _load_module
ERROR 12-05 20:24:46 engine.py:366]     yield from self._load_module(prefix,
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 175, in _load_module
ERROR 12-05 20:24:46 engine.py:366]     module_load_weights(weights)
ERROR 12-05 20:24:46 engine.py:366]   File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 371, in load_weights
ERROR 12-05 20:24:46 engine.py:366]     param = params_dict[name]
ERROR 12-05 20:24:46 engine.py:366] KeyError: 'layers.0.mlp.down_proj.weight_scale'
INFO 12-05 20:24:46 multiproc_worker_utils.py:120] Killing local vLLM worker processes
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 368, in run_mp_engine
    raise e
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 119, in from_engine_args
    return cls(ipc_path=ipc_path,
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 71, in __init__
    self.engine = LLMEngine(*args, **kwargs)
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 347, in __init__
    self.model_executor = executor_class(vllm_config=vllm_config, )
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 26, in __init__
    super().__init__(*args, **kwargs)
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 36, in __init__
    self._init_executor()
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 114, in _init_executor
    self._run_workers("load_model",
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 195, in _run_workers
    driver_worker_output = driver_worker_method(*args, **kwargs)
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/worker/worker.py", line 152, in load_model
    self.model_runner.load_model()
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1074, in load_model
    self.model = get_model(vllm_config=self.vllm_config)
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
    return loader.load_model(vllm_config=vllm_config)
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 334, in load_model
    model.load_weights(self._get_all_weights(model_config, model))
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 503, in load_weights
    loader.load_weights(weights)
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 229, in load_weights
    autoloaded_weights = list(self._load_module("", self.module, weights))
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 190, in _load_module
    yield from self._load_module(prefix,
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 175, in _load_module
    module_load_weights(weights)
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 371, in load_weights
    param = params_dict[name]
KeyError: 'layers.0.mlp.down_proj.weight_scale'
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]

[rank0]:[W1205 20:24:46.112955842 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
Traceback (most recent call last):
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 643, in <module>
    uvloop.run(run_server(args))
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run
    return loop.run_until_complete(wrapper())
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 609, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 113, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 210, in build_async_engine_client_from_engine_args
    raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.
/data/projects/hcece/miniconda3/envs/vllm_wjy_python3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

How to solve this? Many thanks!

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17

Most upvoted comments

@dsikka hello! I update vllm to 0.6.6 and the problem has been solved! Many thanks for your help!

Here are my package list:

Package                           Version       Editable project location
--------------------------------- ------------- -------------------------------------------------------------
absl-py                           2.1.0
accelerate                        1.1.1
aiohappyeyeballs                  2.4.3
aiohttp                           3.11.7
aiohttp-cors                      0.7.0
aiosignal                         1.3.1
airportsdata                      20241001
annotated-types                   0.7.0
anyio                             4.6.2.post1
astor                             0.8.1
attrs                             24.2.0
blake3                            1.0.0
cachetools                        5.5.0
certifi                           2024.8.30
chardet                           5.2.0
charset-normalizer                3.4.0
click                             8.1.7
cloudpickle                       3.1.0
colorama                          0.4.6
colorful                          0.5.6
compressed-tensors                0.8.1
DataProperty                      1.0.1
datasets                          2.16.1
depyf                             0.18.0
dill                              0.3.7
diskcache                         5.6.3
distlib                           0.3.9
distro                            1.9.0
easydict                          1.13
einops                            0.8.0
evaluate                          0.4.3
fast_hadamard_transform           1.0.4.post1   /data/models/testw8a8/QQQ/third-party/fast-hadamard-transform
fastapi                           0.115.5
filelock                          3.16.1
frozenlist                        1.5.0
fsspec                            2023.10.0
gguf                              0.10.0
google-api-core                   2.24.0
google-auth                       2.37.0
googleapis-common-protos          1.66.0
grpcio                            1.68.1
h11                               0.14.0
httpcore                          1.0.7
httptools                         0.6.4
httpx                             0.27.2
huggingface-hub                   0.26.2
idna                              3.10
importlib_metadata                8.5.0
iniconfig                         2.0.0
interegular                       0.3.3
Jinja2                            3.1.3
jiter                             0.7.1
joblib                            1.4.2
jsonlines                         4.0.0
jsonschema                        4.23.0
jsonschema-specifications         2024.10.1
lark                              1.2.2
linkify-it-py                     2.0.3
llmcompressor                     0.3.1
llvmlite                          0.43.0
lm_eval                           0.4.2
lm-format-enforcer                0.10.9
loguru                            0.7.2
lxml                              5.3.0
markdown-it-py                    3.0.0
MarkupSafe                        2.1.5
mbstrdecoder                      1.1.3
mdit-py-plugins                   0.4.2
mdurl                             0.1.2
memray                            1.15.0
mistral_common                    1.5.1
more-itertools                    10.5.0
mpmath                            1.3.0
msgpack                           1.1.0
msgspec                           0.18.6
multidict                         6.1.0
multiprocess                      0.70.15
nest-asyncio                      1.6.0
networkx                          3.2.1
ninja                             1.11.1.2
nltk                              3.9.1
numba                             0.60.0
numexpr                           2.10.2
numpy                             1.26.3
nvidia-cublas-cu12                12.1.3.1
nvidia-cuda-cupti-cu12            12.1.105
nvidia-cuda-nvrtc-cu12            12.1.105
nvidia-cuda-runtime-cu12          12.1.105
nvidia-cudnn-cu12                 9.1.0.70
nvidia-cufft-cu12                 11.0.2.54
nvidia-curand-cu12                10.3.2.106
nvidia-cusolver-cu12              11.4.5.107
nvidia-cusparse-cu12              12.1.0.106
nvidia-ml-py                      12.560.30
nvidia-nccl-cu12                  2.21.5
nvidia-nvjitlink-cu12             12.1.105
nvidia-nvtx-cu12                  12.1.105
openai                            1.55.0
opencensus                        0.11.4
opencensus-context                0.1.3
opencv-python-headless            4.10.0.84
outlines                          0.1.11
outlines_core                     0.1.26
packaging                         24.2
pandas                            2.2.3
partial-json-parser               0.2.1.1.post4
pathvalidate                      3.2.1
peft                              0.14.0
pillow                            10.4.0
pip                               24.3.1
platformdirs                      4.3.6
pluggy                            1.5.0
portalocker                       3.0.0
prometheus_client                 0.21.0
prometheus-fastapi-instrumentator 7.0.0
propcache                         0.2.0
proto-plus                        1.25.0
protobuf                          5.28.3
psutil                            6.1.0
py-cpuinfo                        9.0.0
py-spy                            0.4.0
pyairports                        2.1.1
pyarrow                           18.0.0
pyarrow-hotfix                    0.6
pyasn1                            0.6.1
pyasn1_modules                    0.4.1
pybind11                          2.13.6
pycountry                         24.6.1
pydantic                          2.10.1
pydantic_core                     2.27.1
Pygments                          2.18.0
pynvml                            11.5.3
pytablewriter                     1.2.0
pytest                            8.3.4
python-dateutil                   2.9.0.post0
python-dotenv                     1.0.1
pytz                              2024.2
PyYAML                            6.0.2
pyzmq                             26.2.0
QQQ                               0.0.0         /data/models/testw8a8/QQQ
ray                               2.39.0
referencing                       0.35.1
regex                             2024.11.6
requests                          2.32.3
rich                              13.9.4
rouge-chinese                     1.0.3
rouge_score                       0.1.2
rpds-py                           0.21.0
rsa                               4.9
sacrebleu                         2.4.3
safetensors                       0.4.5
scikit-learn                      1.5.2
scipy                             1.14.1
sentence-transformers             3.1.1
sentencepiece                     0.2.0
setuptools                        75.6.0
six                               1.16.0
smart-open                        7.1.0
sniffio                           1.3.1
sqlitedict                        2.1.0
starlette                         0.41.3
sympy                             1.13.1
tabledata                         1.3.3
tabulate                          0.9.0
tcolorpy                          0.1.6
textual                           1.0.0
threadpoolctl                     3.5.0
tiktoken                          0.7.0
tokenizers                        0.21.0
torch                             2.5.1+cu121
torchaudio                        2.5.1+cu121
torchvision                       0.20.1+cu121
tqdm                              4.67.1
tqdm-multiprocess                 0.0.11
transformers                      4.47.1
triton                            3.1.0
typepy                            1.3.2
typing_extensions                 4.12.2
tzdata                            2024.2
uc-micro-py                       1.0.3
urllib3                           2.2.3
uvicorn                           0.32.1
uvloop                            0.21.0
virtualenv                        20.28.0
vllm                              0.6.6.post1
watchfiles                        1.0.0
websockets                        14.1
wheel                             0.45.1
word2number                       1.1
wrapt                             1.17.0
xformers                          0.0.28.post3
xgrammar                          0.1.8
xxhash                            3.5.0
yarl                              1.18.0
zipp                              3.21.0
zstandard                         0.23.0

@Kha-Zix-1 You shouldn’t need to make any changes. Based on the config you’ve provided, it doesn’t seem like quantization was applied. Can you confirm the script you used to quantize the model, as well as the version of llmcompressor you’re using?

Thanks