verl: [performance] verl 0.4.0 slower than 0.3.0 in actor fwd/bwd part

Hi there,

We test the new version verl 0.4.0, using SGLang as the rollout backend. However, we found that the performance of verl 0.4.0 is slower than verl 0.3.0 when running the same script. In verl 0.3.0, the running time of 10 steps in average is as follows:

{
    'timing_s/gen': 81.85025,
    'timing_s/old_log_prob': 56.87424999999999,
    'timing_s/ref': 54.18241666666666,
    'timing_s/update_actor': 151.79700000000003,
    'timing_s/step': 348.087
}

In verl 0.4.0, the running time of 10 steps in average is as follows:

{
    'timing_s/gen': 70.84241666666667,
    'timing_s/old_log_prob': 74.38166666666667,
    'timing_s/ref': 71.17658333333333,
    'timing_s/update_actor': 167.71325000000002,
    'timing_s/step': 387.78775
}

The running time of old_log_prob, ref and update_actor in verl 0.4.0 increases obvisously compared to 0.3.0. And the GPU untilization rate decreased from 95%+ to around 85% when executing old_log_prob, ref and update_actor part.

The key differences of the two running environment is the version of verl and the version of verl dependency. The differences of the verl dependency are listed below:

verl 0.3.0 dependency verl 0.4.0 dependency
sglang 0.4.5 sglang 0.4.6.post5
sgl-kernel 0.0.5.post3 sglkernel 0.1.4
megatron 0.11.0 megatron 0.12.0
- transformer-engine 2.2.0
- cuda-bindings 12.8.0
flashinfer 0.2.4 flashinfer 0.2.5

I think the performance of actor is most related to megatron, which is upgraded from 0.11.0 to 0.12.0, and megatron 0.12.0 depends on transformer-engine package. But I can not understand why the higher version of megatron get worse performance. Is there any suggestions to solve this problem? Thanks in advance.

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 18 (3 by maintainers)

Most upvoted comments

Great job! Mark here, and we will collect some performance tuning options of GPTModel and TransformerConfig.

The moe model also needs optimization, can try to test when in convenient~

Hi @ETOgaosion @ISEEKYAN . We located the performance problem yesterday and it was related to rope embedding, but the conclusion we listed above is not totally correct. We find that megatron has provide a rope fusion kernel implementation, which can be enabled by setting apply_rope_fusion=True of TransformerConfig before constructing GPTModel in verl/models/mcore/model_initializer.py:

        self.tfconfig.apply_rope_fusion=True

        model = GPTModel(
            config=self.tfconfig,
            transformer_layer_spec=transformer_layer_spec,
            vocab_size=self.hf_config.vocab_size,
            max_sequence_length=self.hf_config.max_position_embeddings,
            pre_process=pre_process,
            post_process=post_process,
            share_embeddings_and_output_weights=share_embeddings_and_output_weights,
            position_embedding_type="rope",
            rotary_base=self.hf_config.rope_theta,
            **rope_scaling_args,
            mtp_block_spec=mtp_block_spec,
        )

In verl 0.4.0, the default value of apply_rope_fusion is False in our settings running DeepSeek-R1-Distill-Qwen-7B, thus megatron choose the running path of a PyTorch native implementation(see megatron/core/models/common/embeddings/rope_utils.py#L229 ), causing this performance issue. Setting apply_rope_fusion=True can solve our problem, and the running time of a step decrease from about 390 seconds to 330 seconds, which is as expected.

We compare the difference between verl 0.3.0 and 0.4.0, and find that in verl 0.3.0, the Qwen2ForCausalLM model we used is initialized in verl/models/registry.py:

_MODELS = {
    "LlamaForCausalLM":
        ("llama", ("ParallelLlamaForCausalLMRmPadPP", "ParallelLlamaForValueRmPadPP", "ParallelLlamaForCausalLMRmPad")),
    "Qwen2ForCausalLM":
        ("qwen2", ("ParallelQwen2ForCausalLMRmPadPP", "ParallelQwen2ForValueRmPadPP", "ParallelQwen2ForCausalLMRmPad")),
    "MistralForCausalLM": ("mistral", ("ParallelMistralForCausalLMRmPadPP", "ParallelMistralForValueRmPadPP",
                                       "ParallelMistralForCausalLMRmPad"))
}

While in verl 0.4.0, the model is initialized using API in megatron in verl/models/mcore/model_initializer.py:

from megatron.core.models.gpt.gpt_model import GPTModel

class BaseModelInitializer(ABC):
    """Base class for model initializers."""

    def initialize(
        self,
        pre_process: bool = True,
        post_process: bool = True,
        share_embeddings_and_output_weights: bool = False,
        value: bool = False,
        **extra_kwargs,
    ) -> GPTModel:
        """Initialize a GPT model with the given configuration.
        https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/models/gpt/gpt_model.py

        Args:
            pre_process (bool): include embedding layer.
            post_process (bool): including an output layer.
            share_embeddings_and_output_weights (bool): input embeddings and output logit weights are shared.
            value (bool): add an extra linear layer for classification or regression.

        Returns:
            GPTModel: An initialized GPT model instance
        """
        transformer_layer_spec = self.get_transformer_layer_spec()
        rope_scaling_args = self.get_rope_scaling_args()
        mtp_block_spec = extra_kwargs.get("mtp_block_spec", None)
        model = GPTModel(
            config=self.tfconfig,
            transformer_layer_spec=transformer_layer_spec,
            vocab_size=self.hf_config.vocab_size,
            max_sequence_length=self.hf_config.max_position_embeddings,
            pre_process=pre_process,
            post_process=post_process,
            share_embeddings_and_output_weights=share_embeddings_and_output_weights,
            position_embedding_type="rope",
            rotary_base=self.hf_config.rope_theta,
            **rope_scaling_args,
            mtp_block_spec=mtp_block_spec,
        )

Is this the reason for performance gap between 0.3.0 and 0.4.0? According to our experiments, the performance of original implementation in verl 0.3.0 is better than the megatron’s GPTModel.

Hi @ETOgaosion @ISEEKYAN . We located the performance problem yesterday and it was related to rope embedding, but the conclusion we listed above is not totally correct. We find that megatron has provide a rope fusion kernel implementation, which can be enabled by setting apply_rope_fusion=True of TransformerConfig before constructing GPTModel in verl/models/mcore/model_initializer.py:

    self.tfconfig.apply_rope_fusion=True

    model = GPTModel(
        config=self.tfconfig,
        transformer_layer_spec=transformer_layer_spec,
        vocab_size=self.hf_config.vocab_size,
        max_sequence_length=self.hf_config.max_position_embeddings,
        pre_process=pre_process,
        post_process=post_process,
        share_embeddings_and_output_weights=share_embeddings_and_output_weights,
        position_embedding_type="rope",
        rotary_base=self.hf_config.rope_theta,
        **rope_scaling_args,
        mtp_block_spec=mtp_block_spec,
    )

In verl 0.4.0, the default value of apply_rope_fusion is False in our settings running DeepSeek-R1-Distill-Qwen-7B, thus megatron choose the running path of a PyTorch native implementation(see megatron/core/models/common/embeddings/rope_utils.py#L229 ), causing this performance issue. Setting apply_rope_fusion=True can solve our problem, and the running time of a step decrease from about 390 seconds to 330 seconds, which is as expected.

does that mean more megatron kernel fusions can be manually turned on

Yes, if you have tried some args with a significant performance gain, please let us know. We will set it up for the default arguments.

It’s totally true, perhaps a performance tuning guide related to Megatron could be provided to help users tune by themselves, similar to the Performance Tuning Guide. Anyway, it’s fun to locate the performance issue’s root cause, and we deeply appreciate the contribution of the verl team.

We dive into the source code of verl 0.3.0 and megatron GPTModel and add many logs to compare the running time of different layers, and we find that the performance gap of GPTModel is caused by rotary embedding in megatron. The performance issue can be solved by implementing cached rotary embedding module and hack to the megatron source code, which is implemented in verl/models/qwen2/megatron/layers/parallel_attention.py#L35. But we not sure why megatron does not implement rotary embedding cache and propose an issue. The issue detail can be viewed at megatron#1646 . Hope this helpful to someone who encounter the same issue, and looking forward to your insights.

Hi @UnpureRationalist , thanks for reporting this performance issue~

Actually dynamic batch size is a key feature in verl v0.4.0, could you try to enable this and do some performance tuning?

use_dynamic_bsz: False
ppo_max_token_len_per_gpu: 16384

OK, I will try to use dynamic_bsz feature and report performance later.