pyreft: [P1] [Error] can not use bfloat16 and TypeError: Object of type type is not JSON serializable

Thanks for your wonderful model, but I have got some problems.

  1. can not use bfloat16.
File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1211, in forward
    outputs = self.model(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 992, in forward
    causal_mask = self._update_causal_mask(attention_mask, inputs_embeds, cache_position, past_seen_tokens)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1095, in _update_causal_mask
    causal_mask = torch.triu(causal_mask, diagonal=1)
RuntimeError: "triu_tril_cuda_template" not implemented for 'BFloat16'
  1. I run the main_demo.ipynb, but got the error:
Traceback (most recent call last):                                                                                                                                                
  File "/mnt/geogpt-gpfs/pyreft/inference.py", line 61, in <module>                                                                  
    _ = trainer.train()                                                                                                                                                           
  File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1859, in train                                                                                     
    return inner_training_loop(                                                                                                                                                   
  File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 2116, in _inner_training_loop                                                                      
    self.control = self.callback_handler.on_train_begin(args, self.state, self.control)                                                                                           
  File "/opt/conda/lib/python3.10/site-packages/transformers/trainer_callback.py", line 371, in on_train_begin                                                                    
    return self.call_event("on_train_begin", args, state, control)                                                                                                                
  File "/opt/conda/lib/python3.10/site-packages/transformers/trainer_callback.py", line 415, in call_event                                                                        
    result = getattr(callback, event)(                                                                                                                                            
  File "/opt/conda/lib/python3.10/site-packages/transformers/integrations/integration_utils.py", line 636, in on_train_begin                                                      
    model_config_json = model.config.to_json_string()                                                                                                                             
  File "/opt/conda/lib/python3.10/site-packages/transformers/configuration_utils.py", line 938, in to_json_string                                                                 
    return json.dumps(config_dict, indent=2, sort_keys=True) + "\n"
  File "/opt/conda/lib/python3.10/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/opt/conda/lib/python3.10/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/opt/conda/lib/python3.10/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/opt/conda/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/opt/conda/lib/python3.10/json/encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "/opt/conda/lib/python3.10/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/opt/conda/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type type is not JSON serializable

I find the issues 69, but I use the main_demo.ipynb, so it does not work for me.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22

Most upvoted comments

Gradient_cccumulation_steps

Thank you for your reply. Now I understand how to set Gradient_cccumulation_steps and batch size.

@mrsempress this is expected i think, i am using -gradient_accumulation_steps 8 -batch_size 4, so if you are running with a batch size of 8, it can be doubled?

So, should I use the command -gradient_accumulation_steps 16 -batch_size 8 or -gradient_accumulation_steps 4 -batch_size 8?

I also want to know how to set hyperparameters like gradient_accumulation_steps, as we cannot directly follow the command in examples/loreft/README.md.

Sorry about the confusion - but I think there is nothing being changed here, since for hyperparameter, what matters is the effective batch size, which is batch size (bounded by the GPU MEM) times the gradient accumulation step (you can set whatever you want to match the effective batch size). I think for the script you sent earlier and my settings, they all have an effective batch size of 32.

Note in the paper, we only report effective batch size, not per device batch size.

Hope these help. Thanks.