pytorch-lightning: PyTorch Lightning DDP crashes with unused parameters

Got solved for me after specifying strategy='ddp_find_unused_parameters_true',

I was manually turning off gradient computation for some parts of the model which seems to be the reason for the error.

trainer = pl.Trainer(accelerator="gpu", devices=-1, max_epochs=args.epochs,   
                        strategy='ddp_find_unused_parameters_true',       
                        callbacks=[ checkpoint_callback,lr_monitor], 
                            )

BUT yes this does slow down training drastically!

This error can appear when there are unused parameters that are not part of the computational graph, because of an ‘if’ statement for example. For me it was a layer that didn’t get used during a forward pass. (also it may look obvious, but if you have multiple models involved in your architecture, the probleme might not come from where you think, particularly if the aformentioned answers show nothing)

There is a way to get the names of the unused parameters. You have to set some environment variable as follows: os.environ[“TORCH_CPP_LOG_LEVEL”]=“INFO” os.environ[“TORCH_DISTRIBUTED_DEBUG”] = “DETAIL”

This way the debug logs are much more detailed and the names of the parameters that did not get a gradient during the backwards pass are shown.

I have/had the same issue. Removing all the if-else branches in the model init didn’t help. Using strategy='ddp_find_unused_parameters_true in the Trainer bypassed the error, but resulted in a very slow performance (training on 4 GPUs were barely faster than training on just one) - as @rushi-the-neural-arch also observed. Then I tried @LightingMc’s init tip which more than doubled the speed! There is obviously something there. At this point training with 4 GPUs is around 2.87x faster than training on just one, which I imagine could be better, but for now is acceptable. However I also wonder how I could properly solve the issue. I can’t see any parameters in the model that would only be initialized but then never used. I also don’t have parameters that have requires_grad=False. I have to try @MehdiDeb’s debug tip and see what I find.

@awaelchli Since an exception gets raised anyway, do you think we could hook into it and print the list of unused parameters? This seems to be a recurring issue with new users and printing the list of troublemakers might help new users find possible solutions for their model.

Any updates on this issue? I guess the best question to ask is “how to have untrainable parameters in the pl.LightningModule” or “example showing multiple resnets inside a pl.LightningModule”?

I am trying to have a non-trainable ensemble backbone whose features get passed into a trainable resnet.

Been getting the same error. Tried lighting<2.0 as well and am getting the same issue. The only workaround is to declare the “DDPStrategy(find_unused_parameters=True)”, which is very slow.

I dont have any unused parameters. Just declaring ensemble backbones inside the pl.LightningModule gives me the same error.

@MehdiDeb suggestion didn’t give any more information. @rob-hen suggestion also doesn’t work for me. @meghana-kshirsagar suggestion on number workers also didn’t work.

Is there any update on this issue? I get this error if I use either find_unused_parameters=True or find_unused_parameters=False. Setting static_graph=True also doesn’t help.

Hi all,

I’m on pytorch_lightning==2.2.0, I dealt with the same error which I resolved by the following workaround that is a combination of setting the log levels as described by MehdiDeb. This showed me which paramters were unused (in my case contact_head and position_embeddings i’m training an ESM model on a masked language modeling task). Subsequently, I added the following line to my train/validation step. I first tried to add a callback but that didn’t do the trick.

  def training_step(self, batch, batch_idx):
        outputs = self.forward(batch)
        loss = outputs.loss
        loss += sum(
            0.0 * param.sum()
            for name, param in self.named_parameters()
            if "contact_head" in name or "position_embeddings" in name
        )
        return loss

Hopefully this helps someone!

Hey, same issue here. I solved this problem by adding on_after_backward() in my LightningModule.

def on_after_backward(self):
    for name, param in self.named_parameters():
        if param.grad is None:
            print(name)

It will print the params that were not used. After freezing these params, DDP works in my case. Hope this can help.

I solved this by removing the part of the model architecture that was initialized but never used in the training workflow, considered as unused parameters with no gradients.

Solution: I was able to resolve this error for my particular use case. I was modifying the backbone of a resnet 18. I had to delete the components of the backbone which were being modified.

class Model(nn.module): def init(self): backbone = resnet18() if small==True else resnet50() backbone.requires_grad=False del backbone.fc # This is the new line that is being added which fixes my error. backbone.fc = nn.Identity()

@ArtemSivtsov @carmocca lmk what you think.

@ArtemSivtsov interesting. I am not facing that issue, but my lighting training does incredibly slow down after about 35 epochs sometimes.

I train pipeline with Unet+EfficientNet backbone and meet the same issue on lightning==2.0.8. Usual “ddp” fails with topic starters error, but ddp_find_unused_parameters_true works fine with previous speed. No huge increase.

Dear authors, how do I need to change my LightningModule to solve this problem and get all power of DDP strategy? Maybe I need to change something in training step as error says?

Same problem when i train ControlNet。If i understand correctly, PL ddp is not suitable for those finetuneing task like SD, Lora or controlnet？

@rob-hen Yes this makes sense to me. The model gets wrapped with DDP before optimizers get configured, so that means DDP sees all parameters before you set p.requires_grad=False in configure_optimizers. Doing this right away in the constructor means when DDP wraps the model, it can immediately mark these parameters as “unused”. So this observation makes sense to me.

I encountered the same issue. I was manually setting p.requires_grad=False for some parameters. When I was doing this within the configure_optimizers method, I got the same error message as @athn-nik. However, performing the same settings within the constructor __init__, I could run without the error message. @awaelchli Is this intended by design ?

I meet the same problem. It outputs the names of the parameters that I need to freeze.

I think I am having the same issue. Although setting ddp_find_unused_parameters_true is not that slow in my case.

I double-checked with the following code and printed nothing in between enter and exit.

def on_before_optimizer_step(self, optimizer) -> None:
    print("on_before_opt enter")
    for p in self.trainable_params:
        if p.grad is None:
            print(p)
    print("on_before_opt exit")