transformers: ValueError: Found `optimizer` configured in the DeepSpeed config, but no `scheduler`. Please configure a scheduler in the DeepSpeed config.

Hello, the supported combinations now are:

1. Trainer optimizer + Trainer scheduler - Don’t specify these in the DS config and use trainer args
2. DeepSpeed optimizer + DeeepSpeed Scheduler - Specify both in DeepSpeed config and no need to use/specify them via Trainer args (@jackapbutler, please note this as you happen to be doing both)
3. Trainer optimizer + DeepSpeed Scheduler - Don’t specify optimizer in DS config; only set the scheduler there. Don’t specify the scheduler via Trainer args.

@luohao123, the case you want is DeepSpeed Optimizer + Trainer Scheduler which isn’t supported now. The suggested approach in your case would be to use Trainer optimizer + Trainer scheduler (Settting 1. above).

Hope this helps.

TLDR; if you’re in a rush, downgrading to version <4.30 (4.29.2) worked for me

I’ve had the same issue 👇 I believe the previous behaviour allowed you to not include any DeepSpeed configuration scheduler key and the one specified in your TrainerArguments would be used. Now it seems you have to include the corresponding scheduler between DeepSpeed and Hugging Face Trainer.

i.e.

DeepSpeed scheduler	Trainer scheduler	Resulting scheduler
WarmupLR	constant_with_warmup	constant_with_warmup
WarmupDecayLR	linear	linear

whereas before you could just ignore the first column and leave it blank to get the same result

DeepSpeed scheduler	Trainer scheduler	Resulting scheduler
	constant_with_warmup	constant_with_warmup
	linear	linear

personally, I found it handier before where I only had to specify the scheduler in one place rather than tracking this over a DeepSpeed config and a Trainer config which are generally separate objects.

Hello @awasthiabhijeet, it should be part of the latest release, could you recheck it?

Seems to work well so far @pacman100. Thanks!

I’m trying to use DeepSpeed optimizer + Trainer scheduler because DeepSpeed has the most best optimizer (fused Adam) and Trainer has the best scheduler for my use case (cosine). DeepSpeed does not support cosine. Why was DeepSpeed optimizer + Trainer scheduler deprecate without any warning? I think this is a mistake and that you should reconsider @pacman100.