LLM作為導師：針對不可驗證強化學習的策略感知提示調整

摘要

強化學習（RL）在非可驗證指令遵循任務中，越來越依賴搭配特定提示評分標準的 LLM 裁判作為獎勵訊號。近期方法雖能在訓練過程中根據策略演進調整評分標準，但訓練提示本身仍維持靜態，來自固定語料庫。此種靜態作法常導致提示難度與策略能力之間出現嚴重錯位：當提示無法讓不同生成結果之間產生有效的品質差異時，裁判便無法提取出具區分力的獎勵訊號。為解決此錯位問題，我們提出「LLM-as-a-Tutor」框架，將 LLM 的角色從裁判擴展為導師：單一模型同時扮演考官與生成器——考官對策略的輸出進行成對比較，藉此偵測不具挑戰性的提示；生成器則為這些提示附加原子約束。此種僅追加新約束的設計，使難度隨策略能力同步單調提升，從而產生無需外部難度調度機制的自我校準訓練訊號。在三項複雜的指令遵循基準測試中，我們的方法持續優於不考慮策略的基線方法，以及先前調整評分標準或改寫提示的策略自適應方法，顯示提示調適是非可驗證強化學習中一個被忽視的策略感知維度。

English

Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design monotonically raises difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method consistently outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, suggesting prompt adaptation as a missing axis of policy-awareness in non-verifiable RL.