法官編碼化：透過程式蒸餾實現可擴展評估

摘要

以LLM為評判已成為自動評估的標準做法，但其成本高昂、延遲顯著且決策過程不透明，這些限制削弱了其可擴展性與可靠性。為解決此問題，我們提出一個簡單且高效的替代方案：程式蒸餾。此方法在評估階段不直接提示LLM，而是將其決策邏輯蒸餾至一個由多個程式組成的委員會中，由這些程式直接對候選項目進行評分。這些基於程式的評判者具備可解釋性，易於檢視或修改，並消除了每次樣本的API成本。基於此概念，我們推出PAJAMA系統，該系統能綜合生成多個評判程式，將其決策匯總為聯合裁定結果，並內建備援機制，選擇性地將低可信度案件升級至LLM處理。在五個資料集與四個模型系列上的實驗顯示，基於程式的評判者可達到13B參數LLM評判者的效能水準。當使用程式輸出作為路由訊號時，PAJAMA能同時提升準確率與吞吐量，並推進帕雷托前沿。在評估範疇之外，基於程式的評判者還能提供低成本且有效的獎勵訊號：在RewardBench上，從程式裁定結果蒸餾出的獎勵模型，其效能優於訓練自專有LLM標籤的模型，而API成本卻降低了兩個數量級。

English

LLM-as-a-judge has become the standard for automated evaluation, but it suffers from high cost, significant latency, and opaque decisions -- limitations that undermine its scalability and reliability. We address these with a simple, efficient alternative: program distillation. Instead of prompting an LLM at the evaluation time, we distill its decision logic into a committee of programs that score candidates directly. These programmatic judges offer transparency, are easily inspected or edited, and eliminate per-sample API costs. Building on this notion, we introduce PAJAMA, a system that synthesizes programs as judges, aggregates their decisions into a joint verdict, and incorporates a fallback mechanism to selectively escalate low-confidence cases to an LLM. Across five datasets and four model families, we show that programmatic judges can match the performance of a 13B-size LLM judge. When using program outputs as routing signals, PAJAMA improves both accuracy and throughput and advances the Pareto frontier. Beyond evaluation, programmatic judges produce cheap and effective reward signals: on RewardBench, a reward model distilled from programs' verdicts outperforms one trained on a proprietary LLM's labels at two orders of magnitude lower API cost.