vllm: [Usage]: deepseek v3 can not set tensor_parallel_size=32

Your current environment

The output of `python collect_env.py`

How would you like to use vllm

[rank0]: model = _initialize_model(vllm_config=vllm_config) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py”, line 116, in _initialize_model [rank0]: return model_class(vllm_config=vllm_config, prefix=prefix) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v3.py”, line 505, in init [rank0]: self.model = DeepseekV3Model(vllm_config=vllm_config, [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v3.py”, line 440, in init [rank0]: self.start_layer, self.end_layer, self.layers = make_layers( [rank0]: ^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py”, line 551, in make_layers [rank0]: maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}")) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v3.py”, line 442, in <lambda> [rank0]: lambda prefix: DeepseekV3DecoderLayer( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v3.py”, line 375, in init [rank0]: self.mlp = DeepseekV3MLP( [rank0]: ^^^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v3.py”, line 69, in init [rank0]: self.gate_up_proj = MergedColumnParallelLinear( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py”, line 429, in init [rank0]: super().init(input_size=input_size, [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py”, line 309, in init [rank0]: self.quant_method.create_weights( [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/fp8.py”, line 186, in create_weights [rank0]: raise ValueError( [rank0]: ValueError: Weight output_partition_size = 576 is not divisible by weight quantization block_n = 128.

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18

Most upvoted comments

if rewrite the config.json weight_block_size to 64

"quantization_config": {
    "activation_scheme": "dynamic",
    "fmt": "e4m3",
    "quant_method": "fp8",
    "weight_block_size": [
      64,
      64
    ]

but error as

[rank0]:     self.model_runner.load_model()
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1096, in load_model
[rank0]:     self.model = get_model(vllm_config=self.vllm_config)
[rank0]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
[rank0]:     return loader.load_model(vllm_config=vllm_config)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 366, in load_model
[rank0]:     loaded_weights = model.load_weights(
[rank0]:                      ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v3.py", line 628, in load_weights
[rank0]:     weight_loader(param,
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/layer.py", line 431, in weight_loader
[rank0]:     self._load_model_weight_or_group_weight_scale(
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/layer.py", line 276, in _load_model_weight_or_group_weight_scale
[rank0]:     self._load_w2(shard_id=shard_id,
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/layer.py", line 330, in _load_w2
[rank0]:     expert_data.copy_(loaded_weight)
[rank0]: RuntimeError: The size of tensor a (112) must match the size of tensor b (56) at non-singleton dimension 0