零差距並非恢復：基準測試污染的分層逐問題機率評估與逐步緩解策略

摘要

公開基準測試的測試資料無可避免地滲入預訓練語料庫，一旦被記憶便會虛增評估分數。污染緩解評估透過介入解碼過程來壓制記憶，並恢復受污染模型的真實能力，但其主流指標 G-AP（聚合效能差距，Gap of Aggregate Performance）存在缺陷。離散的正確／錯誤讀數無法刻畫每題表現；先平均再相減會讓過度抑制與不足抑制相互抵消；而每題均勻加權則容易誘使策略將解題機率推向乾淨模型的高頻值。我們提出 SA-PPG（分層聚合的每題機率差距，Stratified Aggregate of Per-question Probability Gaps）：透過抽樣估計每題的解題機率，逐題與乾淨模型相減，並以乾淨模型解題機率定義的群組內進行聚合。現有緩解策略先估計污染所在，再基於該估計運作，因此其正確性取決於估計的正確性。RailCap 則在生成過程中直接判斷污染：每當樣本落回貪婪軌跡時，下一個軌跡 token 便被限制為次高選項，累積抑制直到回應分佈變得足夠分散。在多個受污染模型與基準測試中，SA-PPG 揭示先前策略的恢復效果被大幅高估，而 RailCap 達到最低的 SA-PPG。

English

Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized. Contamination mitigation evaluation intervenes in the decoding process to suppress memorization and restore a contaminated model's genuine capability, but its prevailing metric, the G-AP (Gap of Aggregate Performance), is flawed. Discrete correct/incorrect readouts cannot characterize per-question performance, averaging before differencing lets over- and under-suppression cancel out, and uniform per-question weighting invites strategies to push solve probabilities onto the clean model's high-frequency values. We propose SA-PPG (Stratified Aggregate of Per-question Probability Gaps): estimate each question's solve probability by sampling, difference it against the clean model per question, and aggregate within groups defined by the clean model's solve probability. Existing mitigation strategies first estimate where contamination lies and then operate on the estimate, so they are only as correct as the estimate. RailCap instead judges contamination during generation: whenever a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner-up, accumulating suppression until the response distribution becomes sufficiently dispersed. Across multiple contaminated models and benchmarks, SA-PPG reveals that prior strategies' restoration is substantially overestimated, while RailCap attains the lowest SA-PPG.