Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization
Policy optimization (PO) for Large Language Models faces a stability--exploration trade-off, currently mediated by an action-side Policy-KL regularizer. This puts practitioners in a double bind: keeping Policy-KL constrains response behavior and consumes the action-side exploration budget, while dropping it leaves the optimization without an explicit drift control. We argue for an alternative that breaks the dilemma by moving regularization to the input side. As training progresses, the distribution over training queries induced by the current policy drifts unchecked from its pre-RL reference
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- FuzzyOverlapping authors or contributors · 62%modular/modular →
“Shared author/contributor keys: liu”
- FuzzyOverlapping authors or contributors · 62%sgl-project/sglang →
“Shared author/contributor keys: zhou”
- FuzzyOverlapping authors or contributors · 62%Zeyi-Lin/HivisionIDPhotos →
“Shared author/contributor keys: lin”
- FuzzyOverlapping authors or contributors · 62%hiyouga/LlamaFactory →
“Shared author/contributor keys: lin”
- LinkedLinked via arxiv author · 85%Xianlei Zhou →
“Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization”
- LinkedLinked via arxiv author · 85%Xiangdi Meng →
“Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization”
- LinkedLinked via arxiv author · 85%Yu He →
“Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization”
- LinkedLinked via arxiv author · 85%Tianyu Qi →
“Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization”