transformers: NotImplementedError: Cannot copy out of meta tensor; no data!

System Info

transformers==4.34.0.dev0 accelerate==0.23.0 torch==2.0.1 cuda==11.7

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, …)
• My own task or dataset (give details below)

Reproduction

import transformers model = transformers.MistralForCausalLM.from_pretrained(model_path)

Error: Traceback (most recent call last): File “./trainer.py”, line 198, in <module> train() File “./trainer.py”, line 152, in train model = transformers.MistralForCausalLM.from_pretrained( File “/opt/conda/envs/ptca/lib/python3.8/site-packages/transformers/modeling_utils.py”, line 3301, in from_pretrained ) = cls._load_pretrained_model( File “/opt/conda/envs/ptca/lib/python3.8/site-packages/transformers/modeling_utils.py”, line 3689, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File “/opt/conda/envs/ptca/lib/python3.8/site-packages/transformers/modeling_utils.py”, line 741, in _load_state_dict_into_meta_model set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs) File “/opt/conda/envs/ptca/lib/python3.8/site-packages/accelerate/utils/modeling.py”, line 317, in set_module_tensor_to_device new_value = value.to(device) NotImplementedError: Cannot copy out of meta tensor; no data!

Expected behavior

model loads sucessfully