ignite: Using ignite.distributed with 3 or more processes hangs indefinitely
❓ Questions/Help/Support
Trying to use the ignite.distributed to train a model with DDP. The issue I encounter is that when spawning 3 or processes to run my code, it seems to hang indefinitely. Works fine with 2 processes. I even tried a very basic script and still hangs (similar to the tutorial).
# run.py
import torch
import ignite.distributed as idist
def run(rank, config):
print(f"Running basic DDP example on rank {rank}.")
def main():
world_size = 4 # if this is 3 or more it hangs
# some dummy config
config = {}
# run task
idist.spawn("nccl", run, args=(config,), nproc_per_node=world_size)
# the same happens even in this case
# with idist.Parallel(backend="nccl", nproc_per_node=world_size) as parallel:
# parallel.run(run, config)
if __name__ == "__main__":
main()
Executing this with:
python -m module.run
I’d be very grateful if anyone can weigh in on this.
Environment
- PyTorch Version: 1.9.0
- Ignite Version: 0.4.6
- OS: Ubuntu 20.04.2 LTS
- How you installed Ignite (
conda,pip, source): conda - Python version: 3.9.6
- Any other relevant information: Running on 4 A100-PCIE-40GB GPUs
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 16 (3 by maintainers)
Thanks for the quick feedback again. We are working on a package (aitlas) that has multiple modules in it. I am rnning the script in the root of package with
python -m package.test. Thepackage/test.pyscript has the code I shared earlier.The server has 4 A100-PCIE-40GB GPUs. It has 2TB of RAM memory and 256 AMD EPYC 7742 64-Core Processors.
Pytorch (1.9.0) and Pytorch-ignite (0.4.6.) are installed with conda (4.9.2).
This is appears in the logs when running:
And, it indefinetly stays in this state.
Thanks for the feedback, @ivankitanovski ! I’d expect gloo a bit slower than nccl. For completeness sake, could you still please run above commands to understand what happened with nccl.
This did it.
Don’t why I didn’t try this in the first place. But, any how, it works now. Not sure, if there are major drawbacks to using this over
nccl, but I see the GPUs are getting used and things are processed.Thanks for the help @vfdev-5 @sdesrozis
@ivankitanovski thanks for the info. I’m really interested to understand what happens in your case on your infrastructure. There are few things which seem a bit strange to me.
nvidiachannel.Could you please try the following things:
NCCL_DEBUG=INFOand report the full outputNCCL_DEBUG=INFO)I think this should work
and run it
I would expect this also to hang.
Thanks a lot for helping to debug that.
@ivankitanovski thanks for the info. I was suspecting that if your server has more than 4 GPUs than when ignite calls barrier it could somehow freeze everything… (even if I checked that with 8 GPUs server) Could you please also run this script to get more env info:
Do you have any chance to check the same code on other multi-gpu machine ? I would like to understand if it is related to A100 (hardware) or software, for example if you run the same test on a clean conda env with pytorch and ignite only ?
EDIT: I also yesterday checked on 4 RTX GPUs having the same env as for previous repro tests (pytorch 1.9.0_cuda11.1, python 3.9 and nvidia drivers 460, cuda 11.2) and could not neither reproduce the issue, code snippet worked for 3 and 4 processes.
@ivankitanovski can you please share more info on your infrastucture, total number of gpus available on the node. How exactly you run the code and if any logs are printed before hanging…
Tested all above code snippets on 3 and 4 V100 GPUs (aws) with pytorch 1.9.0 and python 3.9 and everything works fine. Here are logs for the first code with spawn:
Spawn on 3 GPUs
Spawn on 4 GPUs
nvidia-smi
EDIT: same on a server with 8 V100 using 3 or 4 GPUs:
Spawn on 3 GPUs (p3.16xlarge = 8 V100)
Spawn on 4 GPUs (p3.16xlarge = 8 V100)
nvidia-smi
I will try asap on another machine without using the slurm configuration.
Could you please check if pytorch distributed only works with A100 on 4 gpus. Please try to run this code:
@ivankitanovski Could you try using the gloo backend instead of nccl ? Thanks in advance.
@ivankitanovski thanks for reporting, we’ll investigate what happens exactly.
Meanwhile, can you try to run the training with torch dist launcher :