BAPO: 通过自适应裁剪平衡策略优化稳定LLM的离策略强化学习 BAPO: Stabilizing Off-Policy RL for LLMs via Balanced Policy Optimization

实现快速、稳定、数据高效的强化学习训练 Achieving Fast, Stable, and Data-Efficient RL Training
Zhiheng Xi¹*, Xin Guo¹*, Yang Nan¹, Enyu Zhou¹, Junrui Shen¹, Wenxiang Chen¹, Jiaqi Liu¹, Jixuan Huang¹, Zhihao Zhang¹, Honglin Guo¹, Xun Deng², Zhikai Lei², Miao Zheng², Guoteng Wang², Shuo Zhang², Peng Sun², Rui Zheng², Hang Yan², Tao Gui¹³†, Qi Zhang¹†, Xuanjing Huang¹
¹复旦大学 ²上海奇绩智峰有限公司 ³上海创智学院 ¹Fudan University ²Shanghai Qiji Zhifeng Co., Ltd. ³Shanghai Innovation Institute

研究背景 Background

离策略RL在LLM中对齐中具有高样本效率和容忍数据陈旧的优势 Off-policy RL offers high sample efficiency and tolerance to data staleness for LLM alignment
离策略训练面临优化不稳定、策略熵急剧下降和训练崩溃的挑战 Off-policy training suffers from unstable optimization, sharp entropy decline, and training collapse
现有PPO-like方法使用固定对称裁剪，无法适应不同数据分布 Existing PPO-like methods use fixed symmetric clipping, unable to adapt to different data distributions
Data Staleness Impact
数据陈旧度对训练稳定性的影响：随着陈旧度增加，训练奖励波动、熵下降、梯度爆炸 Impact of data staleness on training stability: increased staleness leads to reward fluctuation, entropy decline, and gradient explosion

核心洞察 Key Insights

优化失衡：负优势样本主导策略梯度，抑制有用行为并导致梯度爆炸 Imbalanced optimization: negative-advantage samples dominate policy gradient, suppressing useful behaviors and causing gradient explosions
熵-裁剪规则：固定裁剪机制系统性地阻止熵增更新，导致过度利用 Entropy-Clip Rule: fixed clipping systematically blocks entropy-increasing updates, leading to over-exploitation

主要贡献 Main Contributions

理论分析：识别并分析离策略RL中优化失衡和熵-裁剪规则两大关键问题 Theoretical analysis: Identify and analyze imbalanced optimization and Entropy-Clip Rule
新算法BAPO：通过动态调整裁剪边界平衡正负信号，保持探索能力 Novel BAPO algorithm: Dynamically adjust clipping bounds to balance signals and preserve exploration
全面验证：在多种模型架构、规模和离策略设置下验证方法有效性 Comprehensive validation: Validate across multiple backbones, scales, and off-policy settings
BAPO Method Overview
BAPO方法示意图：动态调整裁剪边界以平衡正负样本贡献并保持熵稳定 BAPO method illustration: dynamically adjusting clipping bounds to balance positive/negative contributions and maintain entropy stability

方法设计 Method Design

自适应裁剪：动态调整上下裁剪边界 chigh 和 clow Adaptive clipping: dynamically adjust upper and lower bounds chigh and clow
平衡目标：确保正样本贡献达到目标比例 ρ0 Balancing objective: ensure positive samples reach target contribution ratio ρ0
熵保持：引入低概率正样本，过滤过度负样本 Entropy preservation: incorporate low-probability positives while filtering excessive negatives

算法优势 Algorithm Benefits

稳定优化：防止负样本主导和梯度爆炸 Stable optimization: prevent negative sample dominance and gradient explosion
探索平衡：保持策略熵，避免过度利用 Exploration balance: maintain policy entropy, avoid over-exploitation
无需复杂调参：自动适应不同数据分布 No complex tuning: automatically adapt to different data distributions

实验配置 Experimental Setup

数据集：SkyWork-OR1-RL-Data，AIME 2024/2025评估基准 Dataset: SkyWork-OR1-RL-Data, AIME 2024/2025 benchmarks
模型：DeepSeek-R1-Distill、Qwen2.5-Math、OctoThinker-Llama等 Models: DeepSeek-R1-Distill, Qwen2.5-Math, OctoThinker-Llama
参数设置：ρ₀=0.4，裁剪范围[0.6,0.9]-[1.2,3.0]，学习率2e-6 Hyperparameters: ρ₀=0.4, clip range [0.6,0.9]-[1.2,3.0], lr=2e-6
Performance Results
BAPO在AIME 2024/2025基准测试中的性能表现：超越开源和商业模型 BAPO performance on AIME 2024/2025 benchmarks: surpassing open-source and commercial models

核心结果 Key Results

AIME 2024 (32B) AIME 2024 (32B)
87.1%
AIME 2025 (32B) AIME 2025 (32B)
80.0%
AIME 2024 (7B) AIME 2024 (7B)
70.8%
AIME 2025 (7B) AIME 2025 (7B)
62.5%

性能对比 Performance Comparison

模型 Model AIME 2024 AIME 2024 AIME 2025 AIME 2025
BAPO-32B 87.1 80.0
Qwen3-32B 81.4 72.9
SkyWork-OR1-32B 82.2 73.3
o3-mini 79.6 76.7