verl: [performance] verl 0.4.0 slower than 0.3.0 in actor fwd/bwd part
Hi there,
We test the new version verl 0.4.0, using SGLang as the rollout backend. However, we found that the performance of verl 0.4.0 is slower than verl 0.3.0 when running the same script. In verl 0.3.0, the running time of 10 steps in average is as follows:
{
'timing_s/gen': 81.85025,
'timing_s/old_log_prob': 56.87424999999999,
'timing_s/ref': 54.18241666666666,
'timing_s/update_actor': 151.79700000000003,
'timing_s/step': 348.087
}
In verl 0.4.0, the running time of 10 steps in average is as follows:
{
'timing_s/gen': 70.84241666666667,
'timing_s/old_log_prob': 74.38166666666667,
'timing_s/ref': 71.17658333333333,
'timing_s/update_actor': 167.71325000000002,
'timing_s/step': 387.78775
}
The running time of old_log_prob, ref and update_actor in verl 0.4.0 increases obvisously compared to 0.3.0. And the GPU untilization rate decreased from 95%+ to around 85% when executing old_log_prob, ref and update_actor part.
The key differences of the two running environment is the version of verl and the version of verl dependency. The differences of the verl dependency are listed below:
| verl 0.3.0 dependency | verl 0.4.0 dependency |
|---|---|
| sglang 0.4.5 | sglang 0.4.6.post5 |
| sgl-kernel 0.0.5.post3 | sglkernel 0.1.4 |
| megatron 0.11.0 | megatron 0.12.0 |
| - | transformer-engine 2.2.0 |
| - | cuda-bindings 12.8.0 |
| flashinfer 0.2.4 | flashinfer 0.2.5 |
I think the performance of actor is most related to megatron, which is upgraded from 0.11.0 to 0.12.0, and megatron 0.12.0 depends on transformer-engine package. But I can not understand why the higher version of megatron get worse performance. Is there any suggestions to solve this problem? Thanks in advance.
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 18 (3 by maintainers)
Great job! Mark here, and we will collect some performance tuning options of
GPTModelandTransformerConfig.The moe model also needs optimization, can try to test when in convenient~
Hi @ETOgaosion @ISEEKYAN . We located the performance problem yesterday and it was related to rope embedding, but the conclusion we listed above is not totally correct. We find that megatron has provide a rope fusion kernel implementation, which can be enabled by setting
apply_rope_fusion=TrueofTransformerConfigbefore constructingGPTModelinverl/models/mcore/model_initializer.py:In verl 0.4.0, the default value of
apply_rope_fusionisFalsein our settings running DeepSeek-R1-Distill-Qwen-7B, thus megatron choose the running path of a PyTorch native implementation(see megatron/core/models/common/embeddings/rope_utils.py#L229 ), causing this performance issue. Settingapply_rope_fusion=Truecan solve our problem, and the running time of a step decrease from about 390 seconds to 330 seconds, which is as expected.We compare the difference between verl 0.3.0 and 0.4.0, and find that in verl 0.3.0, the
Qwen2ForCausalLMmodel we used is initialized inverl/models/registry.py:While in verl 0.4.0, the model is initialized using API in
megatroninverl/models/mcore/model_initializer.py:Is this the reason for performance gap between 0.3.0 and 0.4.0? According to our experiments, the performance of original implementation in verl 0.3.0 is better than the megatron’s
GPTModel.Yes, if you have tried some args with a significant performance gain, please let us know. We will set it up for the default arguments.
It’s totally true, perhaps a performance tuning guide related to Megatron could be provided to help users tune by themselves, similar to the Performance Tuning Guide. Anyway, it’s fun to locate the performance issue’s root cause, and we deeply appreciate the contribution of the verl team.
We dive into the source code of verl 0.3.0 and megatron
GPTModeland add many logs to compare the running time of different layers, and we find that the performance gap ofGPTModelis caused by rotary embedding in megatron. The performance issue can be solved by implementing cached rotary embedding module and hack to the megatron source code, which is implemented in verl/models/qwen2/megatron/layers/parallel_attention.py#L35. But we not sure why megatron does not implement rotary embedding cache and propose an issue. The issue detail can be viewed at megatron#1646 . Hope this helpful to someone who encounter the same issue, and looking forward to your insights.OK, I will try to use dynamic_bsz feature and report performance later.