法官编码化：通过程序蒸馏实现可扩展评估

摘要

LLM作为评判者已成为自动化评估的标准方法，但存在成本高、延迟显著和决策不透明等局限性——这些缺陷削弱了其可扩展性和可靠性。我们提出一种简单高效的替代方案：程序蒸馏。不同于在评估时提示LLM，我们将其决策逻辑蒸馏为一个程序委员会，由该委员会直接对候选结果进行评分。这些程序化评判者具有透明性，易于审查或编辑，并且消除了每次样本的API成本。基于这一概念，我们引入了PAJAMA系统，该系统能够合成程序作为评判者，将它们的决策聚合为联合裁决，并集成了一种降级机制，可选择性地将低置信度案例升级至LLM处理。在五个数据集和四个模型家族上的实验表明，程序化评判者能够达到与13B规模LLM评判者相当的性能。当将程序输出用作路由信号时，PAJAMA在提升准确率和吞吐量的同时，进一步扩展了帕累托前沿。在评估之外，程序化评判者还能产生廉价且有效的奖励信号：在RewardBench上，从程序裁决中蒸馏得到的奖励模型，其性能优于基于专有LLM标签训练的模型，而API成本降低了两个数量级。

English

LLM-as-a-judge has become the standard for automated evaluation, but it suffers from high cost, significant latency, and opaque decisions -- limitations that undermine its scalability and reliability. We address these with a simple, efficient alternative: program distillation. Instead of prompting an LLM at the evaluation time, we distill its decision logic into a committee of programs that score candidates directly. These programmatic judges offer transparency, are easily inspected or edited, and eliminate per-sample API costs. Building on this notion, we introduce PAJAMA, a system that synthesizes programs as judges, aggregates their decisions into a joint verdict, and incorporates a fallback mechanism to selectively escalate low-confidence cases to an LLM. Across five datasets and four model families, we show that programmatic judges can match the performance of a 13B-size LLM judge. When using program outputs as routing signals, PAJAMA improves both accuracy and throughput and advances the Pareto frontier. Beyond evaluation, programmatic judges produce cheap and effective reward signals: on RewardBench, a reward model distilled from programs' verdicts outperforms one trained on a proprietary LLM's labels at two orders of magnitude lower API cost.