verl: Ray OOM causes the process to be killed

I found that as the training progressed, the System Memory Utilization (%) skyrocketed, and after a fixed point ray would report an out of memory error that would crash the training process.

  • Error

Image

  • System Memory Utilization (%)

Image

  • Script
set -x
MODEL_PATH=<local_path>
export VLLM_ATTENTION_BACKEND=XFORMERS
python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files=<local_path> \
    data.val_files=<local_path> \
    data.train_batch_size=64 \
    data.val_batch_size=64 \
    data.max_prompt_length=768 \
    data.max_response_length=3328 \
    actor_rollout_ref.model.path=$MODEL_PATH \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.actor.ppo_mini_batch_size=64 \
    actor_rollout_ref.actor.ppo_max_token_len_per_gpu=12288 \
    actor_rollout_ref.actor.use_kl_loss=False \
    actor_rollout_ref.actor.kl_loss_coef=0. \
    actor_rollout_ref.actor.kl_loss_type=low_var_kl \
    actor_rollout_ref.actor.use_dynamic_bsz=True \
    actor_rollout_ref.actor.ulysses_sequence_parallel_size=1 \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.model.enable_gradient_checkpointing=True \
    actor_rollout_ref.actor.fsdp_config.param_offload=False \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
    actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.temperature=1.0 \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.72 \
    actor_rollout_ref.rollout.n=32 \
    actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=12288 \
    actor_rollout_ref.rollout.enforce_eager=False \
    actor_rollout_ref.rollout.free_cache_engine=False \
    actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=12288 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    algorithm.kl_ctrl.kl_coef=0. \
    trainer.critic_warmup=0 \
    trainer.logger=['wandb'] \
    trainer.project_name=<wandb> \
    trainer.experiment_name=<wandb> \
    trainer.n_gpus_per_node=8 \
    trainer.nnodes=1 \
    trainer.default_local_dir=<local_path> \
    trainer.default_hdfs_dir=null \
    +trainer.val_before_train=False \
    trainer.save_freq=200 \
    trainer.test_freq=200 \
    trainer.total_epochs=3

Or is there some parameter I haven’t configured correctly that is causing the memory usage to keep increasing?

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 45 (3 by maintainers)

Most upvoted comments

@hiyouga It works! So it’s a vLLM problem, thanks a lot!

Image

An example script to fix this problem:

export VLLM_COMMIT=227578480d71fc94ef46ca77fb69496412158d68
sudo pip install vllm --pre --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}
git clone -b verl_v1 https://github.com/hiyouga/vllm.git
sudo cp -r vllm/vllm/ /usr/local/lib/python3.10/dist-packages/

same issue

@PKU-Fgx We have found this issue is caused by https://github.com/vllm-project/vllm/pull/14326 and this PR could solve the memory leak problem

I encountered the same problem, which I feel is a vllm issue. Leakage at least happens in 0.7+ and 0.6.6.

In my case, the memory jump occurs at two places: generate_sequences and compute log_prob.

Image

I printed the cpu_memory, and I found that the vllm memory usage increases when _add_to_request is called, and llm_engine.step() does not cause memory gain. But after llm_engine.step(), the increased memory is not released.

I suspect the seq_group in vLLM is not corrected released.

Another finding is that a larger dataset causes more leakage (all other hyperparams are the same).

Image

Will keep updated when I have more progress.

@hiyouga Thanks, but I actually met the same issue when using verl with vllm == 0.6.3, i doubt if it’s the same problem.

Image

yes I met the same issue . my vllm is 0.6.3 too

Same issue, maybe bacasue of xformers ?

Tried to comment this line “#export VLLM_ATTENTION_BACKEND=XFORMERS”, but still gets memory leakage.

I suspect that this is memory leak when saving checkpoint.

But I only saved the checkpoint once or twice during the training process, so if it’s a checkpoint saving memory leak, then there shouldn’t be a linear increase in memory usage before saving.

@hiyouga Thanks, but I actually met the same issue when using verl with vllm == 0.6.3, i doubt if it’s the same problem. Image

yes I met the same issue . my vllm is 0.6.3 too

@aaa559 Did you solve the problem? I also meet it (vllm=0.6.3,memory=200GB use the qwen2.5-7B-instruct GPU = 2A10080GB)

@eric-haibin-lin here is mine. I use the following commands to build the env.

docker run -it --rm --entrypoint bash -v /data/models:/root/code/models -v /data/datasets/gsm8k:/root/code/gsm8k --shm-size 64G harbor.dev.jsfund.cn/aigos/verl:vemlp-th2.4.0-cu124-vllm0.6.3-ray2.10-te2.0-megatron0.11.0-v0.0.6
pip install verl -i http://10.0.27.61:8081/repository/pypi/simple --trusted-host  10.0.27.61

the following is the training script:

data_name=gsm8k

export CUDA_VISIBLE_DEVICES=0,1
# export CUDA_VISIBLE_DEVICES=5
export DATA_DIR=/root/code/${data_name} 
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

TRAIN_PROJECT="Test-R1"

export BASE_MODEL='/root/code/models/Qwen2.5-0.5B'
export EXPERIMENT_NAME=${data_name}-test-r1-grpo-qwen2.5-0.5b-em

# set -x
export VLLM_ATTENTION_BACKEND=XFORMERS # vllm + qwen2-7b with flash_attn has some issues

# max_prompt_length = (config['training']['max_start_length'] + config['training']['max_response_length'] * (config['training']['max_turns'] - 1) + config['training']['max_obs_length'] * config['training']['max_turns'])

PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
    data.train_files=$DATA_DIR/train.parquet \
    data.val_files=$DATA_DIR/test.parquet \
    data.train_batch_size=64 \
    data.val_batch_size=64 \
    data.max_prompt_length=1024 \
    data.max_response_length=2048 \
    data.shuffle=True \
    actor_rollout_ref.model.path=$BASE_MODEL \
    actor_rollout_ref.model.enable_gradient_checkpointing=true \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.actor.optim.lr_warmup_steps_ratio=0.95 \
    actor_rollout_ref.actor.ppo_mini_batch_size=64 \
    actor_rollout_ref.actor.ppo_micro_batch_size=32 \
    actor_rollout_ref.actor.use_kl_loss=true \
    actor_rollout_ref.actor.kl_loss_coef=0.001 \
    actor_rollout_ref.actor.kl_loss_type=low_var_kl \
    actor_rollout_ref.actor.fsdp_config.param_offload=True \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=True \
    actor_rollout_ref.ref.log_prob_micro_batch_size=64 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.5 \
    actor_rollout_ref.rollout.temperature=1 \
    actor_rollout_ref.rollout.log_prob_micro_batch_size=64 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
    algorithm.adv_estimator=grpo \
    trainer.logger='[console, tensorboard]' \
    trainer.total_epochs=15 \
    trainer.total_training_steps=305 \
    trainer.project_name=$TRAIN_PROJECT \
    trainer.experiment_name=$EXPERIMENT_NAME \
    trainer.n_gpus_per_node=2 \
    trainer.nnodes=1 \
    trainer.save_freq=100 \
    trainer.test_freq=50 \
    +trainer.val_only=false \
    +trainer.val_before_train=true \
    trainer.default_hdfs_dir=null \
    trainer.default_local_dir=verl_checkpoints/$EXPERIMENT_NAME \
    2>&1 | tee -a $EXPERIMENT_NAME.log```

I comment out this line to resolve the issue #export VLLM_ATTENTION_BACKEND=XFORMERS

@hiyouga Thanks, but I actually met the same issue when using verl with vllm == 0.6.3, i doubt if it’s the same problem.

Image

Thanks for the fix! Does anyone know how to fix the problem in vllm == 0.6.3? The PR seems not compatible with vllm == 0.6.3. Thanks!

In my case, I set actor_rollout_ref.rollout.free_cache_engine=False to solve this problem. My vllm version is 0.6.3.

it does not work well for me, my version is 0.6.3

In my case, I set actor_rollout_ref.rollout.free_cache_engine=False to solve this problem. My vllm version is 0.6.3.

it does not work well for me, my version is 0.7.2

Did you use vLLM 0.7+? It seems that the recent versions of vLLM are the cause of memory leak. You can try vllm 0.6.3

same issue on 32B with 16 nodes. use vllm 0.6.3

@PKU-Fgx, this could be either a veRL issue, a Ray issue, or both. You can use jemalloc to profile it (see https://github.com/ray-project/ray/pull/51031). If it turns out to be a Ray issue after you profile it, I’ll take a look.

@PeterSH6 would you mind adding the label ray to the issue? Thanks!