Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, Shengyi Huang, Siran Yang, Jiamang Wang, Wenbo Su, Bo Zheng
强化学习(RL)在LLM推理领域的应用已迅速成为重要研究方向，相关研究在算法创新和实际应用方面均呈现显著增长。尽管取得进展，该领域仍存在若干关键挑战：缺乏使用RL技术的标准化指南，对其底层机制的理解也较为碎片化。此外，实验设置不一致、训练数据差异以及模型初始化不同等问题导致研究结论相互矛盾，既模糊了这些技术的核心特征，也使从业者在选择合适技术时产生困惑。本文通过严格复现和统一开源框架下的隔离评估，系统回顾了广泛采用的RL技术。我们通过细粒度实验（包括不同难度数据集、模型规模和架构）分析了各项技术的内部机制、适用场景和核心原理。基于这些发现，我们提出了针对特定配置选择RL技术的清晰指南，为LLM领域的强化学习实践者提供了可靠路线图。最后，我们发现两种技术的最小化组合可以通过原始PPO损失实现无评判器策略的学习能力。实验结果表明，我们的简单组合能持续提升性能，超越GRPO和DAPO等策略。
Reinforcement learning for LLM reasoning has rapidly emerged as a prominent research area, marked by a significant surge in related studies on both algorithmic innovations and practical applications. Despite this progress, several critical challenges remain, including the absence of standardized guidelines for employing RL techniques and a fragmented understanding of their underlying mechanisms. Additionally, inconsistent experimental settings, variations in training data, and differences in mod...