vllm: [Usage]: deepseek v3 can not set tensor_parallel_size=32
Your current environment
The output of `python collect_env.py`
How would you like to use vllm
[rank0]: model = _initialize_model(vllm_config=vllm_config) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py”, line 116, in _initialize_model [rank0]: return model_class(vllm_config=vllm_config, prefix=prefix) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v3.py”, line 505, in init [rank0]: self.model = DeepseekV3Model(vllm_config=vllm_config, [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v3.py”, line 440, in init [rank0]: self.start_layer, self.end_layer, self.layers = make_layers( [rank0]: ^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py”, line 551, in make_layers [rank0]: maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}")) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v3.py”, line 442, in <lambda> [rank0]: lambda prefix: DeepseekV3DecoderLayer( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v3.py”, line 375, in init [rank0]: self.mlp = DeepseekV3MLP( [rank0]: ^^^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v3.py”, line 69, in init [rank0]: self.gate_up_proj = MergedColumnParallelLinear( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py”, line 429, in init [rank0]: super().init(input_size=input_size, [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py”, line 309, in init [rank0]: self.quant_method.create_weights( [rank0]: File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/fp8.py”, line 186, in create_weights [rank0]: raise ValueError( [rank0]: ValueError: Weight output_partition_size = 576 is not divisible by weight quantization block_n = 128.
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18
if rewrite the
config.jsonweight_block_size to 64but error as