透過預測驅動推斷進行統計上可靠的基於LLM之排序評估

摘要

採用PRECISE方法，我們擴展了預測驅動推論（Prediction-Powered Inference），透過結合少量人工標註數據與大量大型語言模型（LLM）評判結果，生成排序評估指標的無偏修正估計。PPI方法具有可證明的無偏性，無論LLM評判的誤差分佈為何。針對Precision@K這類分層指標（標註以單篇文檔為單位，但指標以單一查詢為單位），我們將輸出空間的計算複雜度從O(2^|C|)降至O(2^K)。在ESCI基準測試中，將30個人工標註與Claude 3 Sonnet的評判結果結合後，Precision@4估計值的標準誤差從4.45降至3.50（相對降低21%）。在生產系統中，我們的框架僅憑100個人工標籤與2小時的領域專家標註時間，即可正確識別三個系統變體中的最佳方案；A/B測試驗證了此排序結果，對應的日銷售額提升達407個基點。

English

With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set. PPI is provably unbiased regardless of the LLM judge's error profile. We make it applicable to hierarchical metrics like Precision@K, where annotations are per-document but the metric is per-query, by reducing the output-space computation from O(2^|C|) to O(2^K). On the ESCI benchmark, augmenting 30 human annotations with Claude 3 Sonnet judgments reduces the standard error of Precision@4 estimates from 4.45 to 3.50 (a 21% relative reduction). In a production system, our framework correctly identified the best of three system variants from 100 human labels and 2 hours of domain-expert annotation; A/B testing confirmed this ranking with +407 bps in daily sales.