LLM作为导师：面向不可验证强化学习的策略感知提示适配

摘要

针对不可验证指令遵循的强化学习（RL）越来越依赖配备提示特定评分标准的LLM裁判作为奖励信号。尽管近期方法在训练过程中使这些评分标准随策略演化而自适应调整，但训练提示本身仍保持静态，源自固定语料库。这种静态方法常导致提示难度与策略能力之间的严重错配，当提示无法在策略生成样本中引发质量差异时，裁判便无法恢复可区分的奖励信号。为解决这一错配问题，我们提出"LLM作为导师"（LLM-as-a-Tutor）框架，将LLM的角色从裁判扩展为导师：单一模型既充当考官，通过成对比较策略生成样本来检测非挑战性提示；又充当生成器，为这些提示追加原子约束。这种仅追加的设计使提示难度随策略能力同步单调递增，无需外部难度调度即可产生自校准训练信号。在三个复杂指令遵循基准上，我们的方法始终优于不考虑策略的基线方法以及之前通过调整评分标准或改写提示来适应策略的方法，这表明提示适应是面向策略感知的不可验证强化学习中一个缺失的维度。

English

Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design monotonically raises difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method consistently outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, suggesting prompt adaptation as a missing axis of policy-awareness in non-verifiable RL.