LLM作为教练：面向不可验证任务的体验式学习

摘要

基於LLM的評估在開放式任務上進行強化學習時，會將評分量表轉化為單一標量獎勵，導致豐富的文字反饋被丟棄，且不同品質特徵的回應被混為一談。為此，我們提出「體驗式學習」（Experiential Learning, EL），將原本用作「裁判」的LLM反饋模型重新定位為「教練」。該教練將其對每個在線策略（on-policy）回應的評估提煉成可遷移的體驗知識，用以條件化教師模型，並透過在線策略的上下文蒸餾（on-policy context distillation）使策略內化此知識。相比標量獎勵，此高頻寬反饋通道能提供密集監督，並保留高品質回應間的細粒度偏好。在兩種策略家族中，無論反饋來自策略自身還是專有模型，EL在保留與未見過的開放式任務上均持續優於基於評分表的強化學習。值得注意的是，EL在訓練分布之外具有更佳的泛化能力，並能緩解獎勵破解（reward hacking）問題。這些發現證實，體驗知識可作為非可驗證任務後期訓練中更豐富且更具泛化性的學習訊號。

English

Reinforcement learning (RL) on open-ended tasks compresses an LLM's rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles. We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach. The coach distills its assessment of each on-policy response into transferable experiential knowledge, which conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared with scalar rewards, this higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences among high-quality responses. Across two policy families, with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks. Notably, EL generalizes better beyond the training distribution, and mitigates reward hacking. These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.