DyePack：利用後門技術可證明地標記大型語言模型中的測試集污染

DyePack: Provably Flagging Test Set Contamination in LLMs Using Backdoors

May 29, 2025

作者: Yize Cheng, Wenxiao Wang, Mazda Moayeri, Soheil Feizi

cs.AI

摘要

開放式基準測試對於評估和推進大型語言模型至關重要，它們提供了可重現性和透明度。然而，其易於獲取的特點也使其容易成為測試集污染的目標。在本研究中，我們引入了DyePack框架，該框架利用後門攻擊來識別在訓練過程中使用了基準測試集的模型，而無需訪問模型的損失、logits或任何內部細節。就像銀行將染料包與鈔票混合以標記搶劫者一樣，DyePack將後門樣本與測試數據混合，以標記那些在訓練中使用過這些數據的模型。我們提出了一種結合多個具有隨機目標的後門的原則性設計，使得在標記每個模型時能夠精確計算假陽性率（FPR）。這在理論上防止了錯誤指控，同時為每個檢測到的污染案例提供了強有力的證據。我們在三個數據集上的五個模型中評估了DyePack，涵蓋了多項選擇題和開放式生成任務。對於多項選擇題，它成功檢測出了所有受污染的模型，在MMLU-Pro和Big-Bench-Hard上使用八個後門時，保證的FPR分別低至0.000073%和0.000017%。對於開放式生成任務，它表現出良好的泛化能力，在Alpaca上使用六個後門時，以僅0.127%的保證假陽性率識別出了所有受污染的模型。

English

Open benchmarks are essential for evaluating and advancing large language models, offering reproducibility and transparency. However, their accessibility makes them likely targets of test set contamination. In this work, we introduce DyePack, a framework that leverages backdoor attacks to identify models that used benchmark test sets during training, without requiring access to the loss, logits, or any internal details of the model. Like how banks mix dye packs with their money to mark robbers, DyePack mixes backdoor samples with the test data to flag models that trained on it. We propose a principled design incorporating multiple backdoors with stochastic targets, enabling exact false positive rate (FPR) computation when flagging every model. This provably prevents false accusations while providing strong evidence for every detected case of contamination. We evaluate DyePack on five models across three datasets, covering both multiple-choice and open-ended generation tasks. For multiple-choice questions, it successfully detects all contaminated models with guaranteed FPRs as low as 0.000073% on MMLU-Pro and 0.000017% on Big-Bench-Hard using eight backdoors. For open-ended generation tasks, it generalizes well and identifies all contaminated models on Alpaca with a guaranteed false positive rate of just 0.127% using six backdoors.