零差距并非恢复：基准污染的分层逐题概率评估与逐步缓解

摘要

公开基准中的测试数据不可避免地泄漏到预训练语料中，一旦被记忆就会导致评估分数虚高。污染缓解评估通过干预解码过程来抑制记忆并恢复受污染模型的真实能力，但其主流指标 G-AP（Gap of Aggregate Performance）存在缺陷：离散的正确/错误读数无法刻画逐题表现；先平均再求差会让过度抑制和抑制不足相互抵消；均匀的逐题加权会诱导策略将解答概率推向干净模型的高频值。我们提出 SA-PPG（Stratified Aggregate of Per-question Probability Gaps）：通过采样估计每道题的解答概率，与干净模型逐题求差，并在由干净模型解答概率定义的组内进行聚合。现有缓解策略首先估计污染所在，再基于该估计进行操作，因此其正确性取决于估计本身。相反，RailCap 在生成过程中判断污染：每当样本回落到贪心轨迹上时，将下一个轨迹词元限制为次优词元，持续累积抑制，直到响应分布足够分散。在多个受污染模型和基准上，SA-PPG 显示先前策略的恢复效果被显著高估，而 RailCap 取得了最低的 SA-PPG。

English

Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized. Contamination mitigation evaluation intervenes in the decoding process to suppress memorization and restore a contaminated model's genuine capability, but its prevailing metric, the G-AP (Gap of Aggregate Performance), is flawed. Discrete correct/incorrect readouts cannot characterize per-question performance, averaging before differencing lets over- and under-suppression cancel out, and uniform per-question weighting invites strategies to push solve probabilities onto the clean model's high-frequency values. We propose SA-PPG (Stratified Aggregate of Per-question Probability Gaps): estimate each question's solve probability by sampling, difference it against the clean model per question, and aggregate within groups defined by the clean model's solve probability. Existing mitigation strategies first estimate where contamination lies and then operate on the estimate, so they are only as correct as the estimate. RailCap instead judges contamination during generation: whenever a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner-up, accumulating suppression until the response distribution becomes sufficiently dispersed. Across multiple contaminated models and benchmarks, SA-PPG reveals that prior strategies' restoration is substantially overestimated, while RailCap attains the lowest SA-PPG.