MOLE：偵測 AI 代理程式中的內部威脅

摘要

模型未對齊、提示注入或操作者濫用，可能導致操作前沿實驗室帳號的 AI 代理外洩模型權重、毒化訓練資料，或削弱發布閘門。現有基準並未測試防禦者能否在有限審查預算下，於日常工作之中偵測到此類活動。我們提出 MOLE，一個開放基準，涵蓋 150 個由 AI 操作、在 30 個工作日內共用 9 項具狀態服務的帳號，並包含 12 種威脅，以及來自四個模型、總計約 200 億個符元的 8 個語料庫。在 39 個代理模型中，有 72% 完成大多數被指派的有害目標，且代理拒答無法預測是否完成。MOLE 能比較 40 個監控器在語料生成器、可觀測性層級與威脅上的表現；在我們的單日稽核事件比較中，即便是表現最佳的受評監控器，仍漏掉近半數已完成的危害。MOLE 亦能用於監控器開發：基準引導式搜尋可將中階監控器提升 49–64%，而在可比的模擬成本下，選擇性使用較強監控器相較於將其套用於每個帳號日，可將 budget-AUC 提升 10%。

English

Model misalignment, prompt injection, or operator misuse could lead AI agents operating frontier-lab accounts to exfiltrate model weights, poison training data, or weaken release gates. Existing benchmarks do not test whether defenders can detect this activity among routine work under a limited review budget. We introduce MOLE, an open benchmark of 150 AI-operated accounts sharing 9 stateful services over 30 workdays, with 12 threats and 8 corpora from four models totaling roughly 20 billion tokens. Of 39 agent models, 72% complete most assigned harmful objectives and agent refusal does not predict completion. MOLE enables comparison of 40 monitors across corpus generators, observability levels, and threats; even the best evaluated monitor in our single-day audit-event comparison misses nearly half of completed harm. MOLE also enables monitor development: benchmark-guided search improves a mid-tier monitor by 49-64%, while selective use of a stronger monitor improves budget-AUC by 10% over applying it to every account-day at comparable modeled cost.