verl: Actor Model Hangs at loss.backward() During Multi-GPU Training

Environment

  • OS: Ubuntu 24.04
  • CUDA Version: 12.4
  • GPU: NVIDIA H100 80GB HBM3
  • Framework version: Verl 0.2.0.dev; EasyR1 cf1d7ef
  • Pytorch: 2.6.0
  • vLLM: 0.8.3

Problem Description

When training the qwen2.5-vl-3b model with multiple GPUs (single node, trainer.n_gpus_per_node > 1), the actor model hangs indefinitely. The issue does not occur when training with a single GPU. The utilization ratio of gpu and cpu is 100%.

Image

I used py-spy dump --pid to investigate the hanging process and the code stuck at loss.backward() in function update_policy(), class DataParallelPPOActor.

Image

Image

As illustrated in the above figure, one gpu cannot achieve Backward 9.

This issue occurs both in Verl and EasyR1.

What I’ve Tried

  1. Training works correctly with trainer.n_gpus_per_node = 1
  2. Increasing or decreasing actor’s parameters, like micro_batch_size_per_device_for_update, offload does not work, and even leads to stuck at computing logits, per #1188

Please let me know if you need more information.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 1
  • Comments: 26

Most upvoted comments

@qilong-zhang Thanks for sharing this resource! @ra-MANUJ-an qilong-zhang is working on the tool calling agent training, and the tool calling causes the stuck. But I am working on the vanilla verl framework, and the solution may not work for me. According to #176 (comment), the code gets stuck during Rollout, but my code gets stuck during the bp in the actor model.

@ra-MANUJ-an yes, my code is function calling based, so I need to change the code

@qilong-zhang Thanks for sharing this resource! @ra-MANUJ-an qilong-zhang is working on the tool calling agent training, and the tool calling causes the stuck. But I am working on the vanilla verl framework, and the solution may not work for me. According to https://github.com/volcengine/verl/issues/176#issuecomment-2674541444, the code gets stuck during Rollout, but my code gets stuck during the bp in the actor model.

@qilong-zhang Yeah, me neither. I guess we have to try it out

@ra-MANUJ-an I don’t know and I am studying how to solve it

I see, I will try other library versions as well and check if it works.

@ra-MANUJ-an Not yet. I spent a lot of time changing my code and configurations, but they did not work. It seems the issue comes from the verl/ray/pytorch, but I have not figured it out yet.

Maybe I will try to change the torch/verl/vllm version first to see if it works.