F5-TTS: [Finetune Exploration] KeyError: 'model_state_dict' when using test_train.py

Hey guys, appreciate the project, the model has been great so far!

So I’ve set up a dataset to be compatible with dataset.py, set the parameters inside of the test_train.py, and on trying to load F5TTS_Base, it seems like it’s missing some keys:

Exception has occurred: KeyError
'model_state_dict'
  File "C:\Users\jarod\OneDrive\Desktop\code\F5-TTS\model\trainer.py", line 154, in load_checkpoint
    self.accelerator.unwrap_model(self.model).load_state_dict(checkpoint['model_state_dict'])
                                                              ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jarod\OneDrive\Desktop\code\F5-TTS\model\trainer.py", line 240, in train
    start_step = self.load_checkpoint()
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jarod\OneDrive\Desktop\code\F5-TTS\test_train.py", line 85, in main
    trainer.train(train_dataset,
  File "C:\Users\jarod\OneDrive\Desktop\code\F5-TTS\test_train.py", line 92, in <module>
    main()
KeyError: 'model_state_dict'

Were the pretrained models distilled down to remove these keys? If so, got any tips for loading it up into further training?

I played around with just trying to load ema_model_state_dict, but it seems like I ended up just starting from scratch with this.

Related to https://github.com/SWivid/F5-TTS/issues/16

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 15 (7 by maintainers)

Most upvoted comments

@JarodMica error with keys mismatch solved. Previously we replaced the prefix of keys, ema_model. with model., though it should just be removed. We fix this in 46d391a8766d8eabb967b326ccb473bc2d8b9a6c, and we did an actual run to make sure it is correct now.

I can infer that EMA doesn’t update/get trained during training.

Thought it was updated, but still dominated by previous updates, say pretraining, as the finetune process just goes few updates.

@JarodMica we currently deal it with 0d7b47bc3b7064ce1bdb4688392c9b907e4fbf3f Give it a try~

We just missed that. Thanks for mentioning this ~

I will test on it, but place the loading in init as we’re using accelerator, cuz ema is only recorded at main_process of ddp

Thanks for the response!

So I made the modifications:

def load_checkpoint(self):
        if not exists(self.checkpoint_path) or not os.path.exists(self.checkpoint_path) or not os.listdir(self.checkpoint_path):
            return 0
        
        self.accelerator.wait_for_everyone()
        if "model_last.pt" in os.listdir(self.checkpoint_path):
            latest_checkpoint = "model_last.pt"
        else:
            latest_checkpoint = sorted(os.listdir(self.checkpoint_path), key=lambda x: int(''.join(filter(str.isdigit, x))))[-1]
        # checkpoint = torch.load(f"{self.checkpoint_path}/{latest_checkpoint}", map_location=self.accelerator.device)  # rather use accelerator.load_state ಥ_ಥ
        checkpoint = torch.load(f"{self.checkpoint_path}/{latest_checkpoint}", map_location="cpu")
        print("Checkpoint keys:", checkpoint.keys())

        # self.accelerator.unwrap_model(self.model).load_state_dict(checkpoint['ema_model_state_dict'])
        # self.accelerator.unwrap_model(self.optimizer).load_state_dict(checkpoint['optimizer_state_dict'])


        if self.is_main:
            self.ema_model.load_state_dict(checkpoint['ema_model_state_dict'])
            
        self.ema_model.copy_params_from_ema_to_model()

        # if self.scheduler:
        #     self.scheduler.load_state_dict(checkpoint['scheduler_state_dict'])
        
        # step = checkpoint['step']
        step = 0
        
        del checkpoint; gc.collect()
        return step

And training seems to be running.

I did set the model to save just to try it out and make sure it’s still working, so after 100 steps, the model still outputs comprehensible speech so I think we’re good here