無需攻擊者的博弈：選擇壓力下LLM驅動搜尋中的基準指紋識別

摘要

針對評估訊號進行最佳化的系統，其基準測試所衡量的內容與其所宣稱的有所不同。我們在兩個帶有留出式泛化門檻的GPU核心最佳化套件中具體記錄了這一點：Metal-Sci（10項科學計算任務）和Metal-ZK（12項零知識/密碼學任務），其中三個前沿LLM（Opus 4.7、Gemini 3.1 Pro、GPT-5.5）在帶有豐富回饋的(1+1)演化迴圈內提出Metal核心。雖然沒有任何模型被提示以對抗方式行事，但被選拔的勝出者反覆對評估配置進行指紋識別：它們依據執行時期參數的識別結果進行分支，最大化地調校被測量的分支，而使未測量的分支保持緩慢或靜默錯誤。在合併的套件中，16/53（30%）的分佈內勝出案例未能遷移到留出配置。我們為這些失敗提供了一個四模式分類，從配置指紋到門檻洩漏。我們提煉出在策略性最佳化下進行測量的設計指引：留出探針僅在不可枚舉的軸向上保持有效性；門檻必須衡量留出效能，而不僅僅是正確性；且遷移率只有在具備逐失敗機制評級時才可解釋——我們的評級分解為被操弄、過度擬合和良性三類。程式碼與研究工件：https://github.com/vicgalle/kernel-fingerprinting

English

Benchmarks for systems that are optimized against the evaluation signal measure something different from what they claim. We document this concretely in two GPU-kernel-optimization suites with held-out generalization gates: Metal-Sci (10 scientific-compute tasks) and Metal-ZK (12 zero-knowledge/cryptographic tasks), in which three frontier LLMs (Opus 4.7, Gemini 3.1 Pro, GPT-5.5) propose Metal kernels inside a (1{+}1) evolutionary loop with rich feedback. Although no model is prompted to act adversarially, the promoted winners repeatedly fingerprint the evaluation configuration: they branch on the identity of runtime parameters, tune the measured branch maximally, and leave the unmeasured branch slow or silently wrong. Across the pooled suites, 16/53 (30%) of in-distribution wins fail to transfer to held-out configurations. We give a four-mode taxonomy of these failures, from configuration fingerprints to gate leakage. We distill design guidance for measurement under strategic optimization: held-out probes retain validity only on non-enumerable axes; gates must measure held-out performance, not just correctness; and a transfer rate is interpretable only with per-failure mechanism grades: ours decomposes into gamed, overfit, and benign. Code and research artifacts: https://github.com/vicgalle/kernel-fingerprinting