verl: Actor Model Hangs at loss.backward() During Multi-GPU Training
Environment
- OS: Ubuntu 24.04
- CUDA Version: 12.4
- GPU: NVIDIA H100 80GB HBM3
- Framework version: Verl 0.2.0.dev; EasyR1 cf1d7ef
- Pytorch: 2.6.0
- vLLM: 0.8.3
Problem Description
When training the qwen2.5-vl-3b model with multiple GPUs (single node, trainer.n_gpus_per_node > 1), the actor model hangs indefinitely. The issue does not occur when training with a single GPU. The utilization ratio of gpu and cpu is 100%.
I used py-spy dump --pid to investigate the hanging process and the code stuck at loss.backward() in function update_policy(), class DataParallelPPOActor.
As illustrated in the above figure, one gpu cannot achieve Backward 9.
This issue occurs both in Verl and EasyR1.
What I’ve Tried
- Training works correctly with
trainer.n_gpus_per_node = 1 - Increasing or decreasing actor’s parameters, like
micro_batch_size_per_device_for_update,offloaddoes not work, and even leads to stuck at computing logits, per #1188
Please let me know if you need more information.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 26
@ra-MANUJ-an yes, my code is function calling based, so I need to change the code
@qilong-zhang Thanks for sharing this resource! @ra-MANUJ-an qilong-zhang is working on the tool calling agent training, and the tool calling causes the stuck. But I am working on the vanilla verl framework, and the solution may not work for me. According to https://github.com/volcengine/verl/issues/176#issuecomment-2674541444, the code gets stuck during Rollout, but my code gets stuck during the bp in the actor model.
@qilong-zhang Yeah, me neither. I guess we have to try it out
@ra-MANUJ-an I don’t know and I am studying how to solve it
I see, I will try other library versions as well and check if it works.
@ra-MANUJ-an Not yet. I spent a lot of time changing my code and configurations, but they did not work. It seems the issue comes from the verl/ray/pytorch, but I have not figured it out yet.
Maybe I will try to change the torch/verl/vllm version first to see if it works.