verl: RuntimeError: batch size must be positive in flash_attn_varlen_forward when use_remove_padding=True

Environment:
• Framework: verl
• Components: PPO Trainer, FSDP Worker, Flash Attention

Problem Description:
When running GRPO training with use_remove_padding=True, the code fails with:

RuntimeError: batch size must be positive

The error occurs in flash_attn_varlen_forward when processing unpadded inputs.

**Error Stack Trace **:

Traceback (most recent call last):
  File "[PROJECT_ROOT]/verl/trainer/main_ppo.py", line 175, in run
    trainer.fit()
  File "[PROJECT_ROOT]/verl/trainer/ppo/ray_trainer.py", line 910, in fit
    actor_output = self.actor_rollout_wg.update_actor(batch)
  File "[PROJECT_ROOT]/verl/single_controller/ray/base.py", line 42, in func
    output = ray.get(output)
  File "[PROJECT_ROOT]/verl/workers/fsdp_workers.py", line 459, in update_actor
    metrics = self.actor.update_policy(data=data)
  File "[PROJECT_ROOT]/verl/workers/actor/dp_actor.py", line 300, in update_policy
    entropy, log_prob = self._forward_micro_batch(micro_batch=data, temperature=temperature)
  File "[PROJECT_ROOT]/verl/workers/actor/dp_actor.py", line 110, in _forward_micro_batch
    output = self.actor_module(input_ids=input_ids_rmpad, ...)
  File "[CONDA_ENV_PATH]/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 930, in forward
    out_padded, softmax_lse, S_dmask, rng_state = _wrapped_flash_attn_varlen_forward(...)
RuntimeError: batch size must be positive

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 16 (2 by maintainers)

Most upvoted comments

In my case, the problem stem from: std::vectorat::Tensor mha_varlen_fwd(at::Tensor &q, // total_q x num_heads x head_size, total_q := \sum_{i=0}^{b} s_i const at::Tensor &k, // total_k x num_heads_k x head_size, total_k := \sum_{i=0}^{b} s_i or num_blocks x page_block_size x num_heads_k x head_size if there’s a block_table. const at::Tensor &v, // total_k x num_heads_k x head_size, total_k := \sum_{i=0}^{b} s_i or num_blocks x page_block_size x num_heads_k x head_size if there’s a block_table. std::optionalat::Tensor &out_, // total_q x num_heads x head_size, total_k := \sum_{i=0}^{b} s_i const at::Tensor &cu_seqlens_q, // b+1 const at::Tensor &cu_seqlens_k, // b+1 std::optionalat::Tensor &seqused_k, // b. If given, only this many elements of each batch element’s keys are used. std::optional &leftpad_k_, // batch_size std::optionalat::Tensor &block_table_, // batch_size x max_num_blocks_per_seq std::optionalat::Tensor &alibi_slopes_, // num_heads or b x num_heads int max_seqlen_q, const int max_seqlen_k, const float p_dropout, const float softmax_scale, const bool zero_tensors, bool is_causal, int window_size_left, int window_size_right, const float softcap, const bool return_softmax, std::optionalat::Generator gen_) { … where check const int batch_size = cu_seqlens_q.numel() - 1; // … TORCH_CHECK(batch_size > 0, “batch size must be positive”); And I find my input cu_seqlens_q.numel() == 1. And in verl/models/transformers/monkey_patch.py, _ulysses_flash_attention_forward calls attn_output = _flash_attention_forward( query_states, key_states, value_states, *args, position_ids=position_ids, **kwargs ) while kwargs={'query_length': 3233, 'is_causal': True, 'dropout': 0.0, 'softmax_scale': 0.08838834764831845, 'sliding_window': None, 'softcap': None, 'use_top_left_mask': False, 'target_dtype': None, 'attn_implementation': 'flash_attention_2', 'output_attentions': False, 'use_cache': False} So inside the _flash_attention_forward it goes to

elif is_fa2_with_varlen_kwargs or is_fa2_with_position_ids:
    batch_size = query_states.size(0)

    if cu_seq_lens_q is None or cu_seq_lens_k is None:
        query_states, key_states, value_states, indices_q, cu_seq_lens, max_seq_lens = (
            _prepare_flash_attention_from_position_ids(query_states, key_states, value_states, position_ids)
        )

        cu_seq_lens_q, cu_seq_lens_k = cu_seq_lens
        max_length_q, max_length_k = max_seq_lens

    else:
        query_states = query_states.reshape(-1, query_states.size(-2), query_states.size(-1))
        key_states = key_states.reshape(-1, key_states.size(-2), key_states.size(-1))
        value_states = value_states.reshape(-1, value_states.size(-2), value_states.size(-1))

    attn_output = _flash_attn_varlen_func(
        query_states,
        key_states,
        value_states,
        cu_seqlens_q=cu_seq_lens_q,
        cu_seqlens_k=cu_seq_lens_k,
        max_seqlen_q=max_length_q,
        max_seqlen_k=max_length_k,
        softmax_scale=softmax_scale,
        causal=causal,
        **flash_kwargs,
    )

where query_states, key_states, value_states, indices_q, cu_seq_lens, max_seq_lens = ( _prepare_flash_attention_from_position_ids(query_states, key_states, value_states, position_ids) ) build the wrong cu_seq_lens when there is only one sequence in this batch. So the problem potential comes from the transformers implementation of _prepare_flash_attention_from_position_ids. And Claude suggest the following implementation which solve the problem in my case. def _prepare_flash_attention_from_position_ids(query, key, value, position_ids): “”" This function returns necessary arguments to call flash_attn_varlen_func. All three query, key, value states will be flattened. Cumulative lengths of each examples in the batch will be extracted from position_ids.

NOTE: ideally cumulative lengths should be prepared at the data collator stage

Arguments:
    query (`torch.Tensor`):
        Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim).
    key (`torch.Tensor`):
        Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim).
    value (`torch.Tensor`):
        Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim).
    position_ids (`torch.Tensor`):
        Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid.

Return:
    query (`torch.Tensor`):
        Query state without padding. Shape: (total_target_length, num_heads, head_dim).
    key (`torch.Tensor`):
        Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim).
    value (`torch.Tensor`):
        Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim).
    indices_q (`torch.Tensor`):
        The indices of non-masked tokens from the flattened input target sequence.
    (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`):
        The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,).
    (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`):
        Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value).
"""
query = query.view(-1, query.size(-2), query.size(-1))
key = key.contiguous().view(-1, key.size(-2), key.size(-1))
value = value.contiguous().view(-1, value.size(-2), value.size(-1))
position_ids = position_ids.flatten()
indices_q = torch.arange(position_ids.size(0), device=position_ids.device, dtype=torch.int32)

# 修复:正确处理cumulative sequence lengths
# 找到每个序列的开始位置(position_id == 0的位置)
seq_starts = indices_q[position_ids == 0]

# 如果没有找到任何0,说明只有一个序列,从0开始
if len(seq_starts) == 0:
    cu_seq_lens = torch.tensor([0, position_ids.size(0)], device=position_ids.device, dtype=torch.int32)
else:
    # 添加序列结束位置
    cu_seq_lens = torch.cat([
        seq_starts,
        torch.tensor([position_ids.size(0)], device=position_ids.device, dtype=torch.int32)
    ])

max_length = position_ids.max() + 1

return (query, key, value, indices_q, (cu_seq_lens, cu_seq_lens), (max_length, max_length))

Never mind, the essential problem in my case is that input_ids and position_ids don’t correspond. 🤦‍♂️

How is your input_ids and position_ids don't correspond ?

I do in-batch shuffle. While the randomized index for input_ids and position_ids are not same.