verl: RuntimeError: batch size must be positive in flash_attn_varlen_forward when use_remove_padding=True
Environment:
• Framework: verl
• Components: PPO Trainer, FSDP Worker, Flash Attention
Problem Description:
When running GRPO training with use_remove_padding=True, the code fails with:
RuntimeError: batch size must be positive
The error occurs in flash_attn_varlen_forward when processing unpadded inputs.
**Error Stack Trace **:
Traceback (most recent call last):
File "[PROJECT_ROOT]/verl/trainer/main_ppo.py", line 175, in run
trainer.fit()
File "[PROJECT_ROOT]/verl/trainer/ppo/ray_trainer.py", line 910, in fit
actor_output = self.actor_rollout_wg.update_actor(batch)
File "[PROJECT_ROOT]/verl/single_controller/ray/base.py", line 42, in func
output = ray.get(output)
File "[PROJECT_ROOT]/verl/workers/fsdp_workers.py", line 459, in update_actor
metrics = self.actor.update_policy(data=data)
File "[PROJECT_ROOT]/verl/workers/actor/dp_actor.py", line 300, in update_policy
entropy, log_prob = self._forward_micro_batch(micro_batch=data, temperature=temperature)
File "[PROJECT_ROOT]/verl/workers/actor/dp_actor.py", line 110, in _forward_micro_batch
output = self.actor_module(input_ids=input_ids_rmpad, ...)
File "[CONDA_ENV_PATH]/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 930, in forward
out_padded, softmax_lse, S_dmask, rng_state = _wrapped_flash_attn_varlen_forward(...)
RuntimeError: batch size must be positive
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 16 (2 by maintainers)
I do in-batch shuffle. While the randomized index for input_ids and position_ids are not same.