「同一智能體，不同答案：檢索增強問答中語料庫引發之答案變動的重複感知稽核」

摘要

檢索增強問答系統在索引擴充後，即使其請求的模型識別碼、提示、檢索策略、證據深度、渲染方式及暴露的生成控制設定均保持固定，仍可能回傳不同答案。當增益與損失相互抵銷時，總體準確度可能掩蓋這些變化；而一般的生成變異又會使一次性比較高估更新效果。我們將此隱藏現象稱為準確度盲區答案更迭（accuracy-blind answer churn），並引入快照相容性稽核（Snapshot Compatibility Audit），其透過從跨快照分歧中減去同一快照重複執行的分歧，來估計超額答案更迭。我們透過將一個凍結的 FineWeb 前綴從一個分片擴充至七個分片來實例化此方法。在一項預先註冊的 400 題 Natural Questions 研究中，標準化精確與盲式語義的超額更迭分別為 6.44 與 10.25 個百分點，而精確匹配準確度僅變化 -1.50 個百分點。事後分析發現 40/400 題存在重複穩定的語義翻轉。另一項另行預先註冊的 200 題 TriviaQA 研究產生較小且方向一致的超額更迭，而精確匹配準確度則朝相反方向變動。一項結果盲式的事後 100 題子集複現，使用第二個 DeepSeek 生成器與部署配置，發現語義超額更迭達 8.75 個百分點，即使精確匹配上升了 3.00 個百分點。因此，答案層級的相容性可能在沒有明顯或方向一致的效用變動下失效。檢索增強的發布應在評估效用之際一併稽核相容性。

English

A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval policy, evidence depth, rendering, and exposed generation controls are held fixed. Aggregate accuracy may hide these changes when gains and losses cancel, while ordinary generation variability makes one-shot comparisons overstate update effects. We call the hidden phenomenon accuracy-blind answer churn and introduce the Snapshot Compatibility Audit, which estimates excess answer churn by subtracting same-snapshot repeat disagreement from cross-snapshot disagreement. We instantiate it by expanding one frozen FineWeb prefix from one to seven shards. In a preregistered 400-question Natural Questions study, normalized-exact and blinded-semantic excess churn are 6.44 and 10.25 percentage points while exact-match accuracy changes by only -1.50 points. A post-hoc analysis finds repeat-stable semantic flips on 40/400 questions. A separately preregistered 200-question TriviaQA study yields smaller, directionally consistent excess churn while exact-match accuracy moves in the opposite direction. An outcome-blind post-hoc 100-question subset replication with a second DeepSeek generator and serving configuration finds 8.75 pp of semantic excess churn even as exact match rises by 3.00 percentage points. Answer-level compatibility can therefore fail without a conspicuous or consistently directed utility shift. Retrieval-augmented releases should audit compatibility alongside utility.