verl: Ray OOM causes the process to be killed
I found that as the training progressed, the System Memory Utilization (%) skyrocketed, and after a fixed point ray would report an out of memory error that would crash the training process.
- Error
- System Memory Utilization (%)
- Script
set -x
MODEL_PATH=<local_path>
export VLLM_ATTENTION_BACKEND=XFORMERS
python3 -m verl.trainer.main_ppo \
algorithm.adv_estimator=grpo \
data.train_files=<local_path> \
data.val_files=<local_path> \
data.train_batch_size=64 \
data.val_batch_size=64 \
data.max_prompt_length=768 \
data.max_response_length=3328 \
actor_rollout_ref.model.path=$MODEL_PATH \
actor_rollout_ref.actor.optim.lr=1e-6 \
actor_rollout_ref.actor.ppo_mini_batch_size=64 \
actor_rollout_ref.actor.ppo_max_token_len_per_gpu=12288 \
actor_rollout_ref.actor.use_kl_loss=False \
actor_rollout_ref.actor.kl_loss_coef=0. \
actor_rollout_ref.actor.kl_loss_type=low_var_kl \
actor_rollout_ref.actor.use_dynamic_bsz=True \
actor_rollout_ref.actor.ulysses_sequence_parallel_size=1 \
actor_rollout_ref.model.use_remove_padding=True \
actor_rollout_ref.model.enable_gradient_checkpointing=True \
actor_rollout_ref.actor.fsdp_config.param_offload=False \
actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
actor_rollout_ref.rollout.name=vllm \
actor_rollout_ref.rollout.temperature=1.0 \
actor_rollout_ref.rollout.gpu_memory_utilization=0.72 \
actor_rollout_ref.rollout.n=32 \
actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=12288 \
actor_rollout_ref.rollout.enforce_eager=False \
actor_rollout_ref.rollout.free_cache_engine=False \
actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=12288 \
actor_rollout_ref.ref.fsdp_config.param_offload=True \
algorithm.kl_ctrl.kl_coef=0. \
trainer.critic_warmup=0 \
trainer.logger=['wandb'] \
trainer.project_name=<wandb> \
trainer.experiment_name=<wandb> \
trainer.n_gpus_per_node=8 \
trainer.nnodes=1 \
trainer.default_local_dir=<local_path> \
trainer.default_hdfs_dir=null \
+trainer.val_before_train=False \
trainer.save_freq=200 \
trainer.test_freq=200 \
trainer.total_epochs=3
Or is there some parameter I haven’t configured correctly that is causing the memory usage to keep increasing?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 45 (3 by maintainers)
@hiyouga It works! So it’s a vLLM problem, thanks a lot!
An example script to fix this problem:
same issue
@PKU-Fgx We have found this issue is caused by https://github.com/vllm-project/vllm/pull/14326 and this PR could solve the memory leak problem
I encountered the same problem, which I feel is a vllm issue. Leakage at least happens in 0.7+ and 0.6.6.
In my case, the memory jump occurs at two places: generate_sequences and compute log_prob.
I printed the cpu_memory, and I found that the vllm memory usage increases when _add_to_request is called, and llm_engine.step() does not cause memory gain. But after llm_engine.step(), the increased memory is not released.
I suspect the seq_group in vLLM is not corrected released.
Another finding is that a larger dataset causes more leakage (all other hyperparams are the same).
Will keep updated when I have more progress.
yes I met the same issue . my vllm is 0.6.3 too
Tried to comment this line “#export VLLM_ATTENTION_BACKEND=XFORMERS”, but still gets memory leakage.
But I only saved the checkpoint once or twice during the training process, so if it’s a checkpoint saving memory leak, then there shouldn’t be a linear increase in memory usage before saving.
@aaa559 Did you solve the problem? I also meet it (vllm=0.6.3,memory=200GB use the qwen2.5-7B-instruct GPU = 2A10080GB)
@eric-haibin-lin here is mine. I use the following commands to build the env.
the following is the training script:
I comment out this line to resolve the issue #export VLLM_ATTENTION_BACKEND=XFORMERS
@hiyouga Thanks, but I actually met the same issue when using verl with vllm == 0.6.3, i doubt if it’s the same problem.
Thanks for the fix! Does anyone know how to fix the problem in vllm == 0.6.3? The PR seems not compatible with vllm == 0.6.3. Thanks!
it does not work well for me, my version is 0.6.3
it does not work well for me, my version is 0.7.2
same issue on 32B with 16 nodes. use vllm 0.6.3
@PKU-Fgx, this could be either a veRL issue, a Ray issue, or both. You can use jemalloc to profile it (see https://github.com/ray-project/ray/pull/51031). If it turns out to be a Ray issue after you profile it, I’ll take a look.
@PeterSH6 would you mind adding the label
rayto the issue? Thanks!