同一智能体，不同答案：检索增强问答中语料库引发答案更替的可重复性审计

摘要

检索增强问答系统在索引扩展后可能返回不同的答案，即使其请求的模型标识符、提示词、检索策略、证据深度、渲染方式及暴露的生成控制均保持不变。当增益与损失相互抵消时，总体准确率可能掩盖这些变化，而普通的生成变异性会使单次比较夸大更新效果。我们将这一隐藏现象称为"准确率盲区答案波动"（accuracy-blind answer churn），并引入快照兼容性审计（Snapshot Compatibility Audit），该方法通过从跨快照不一致中减去同快照重复不一致来估计超额答案波动。我们通过将一个冻结的FineWeb前缀从一个分片扩展到七个分片来实例化该方法。在一项预注册的400题Natural Questions研究中，归一化精确匹配和盲评语义的超额波动分别为6.44和10.25个百分点，而精确匹配准确率仅变化了-1.50个百分点。事后分析发现，40/400的问题存在重复稳定的语义翻转。另一项单独预注册的200题TriviaQA研究产生了较小且方向一致的超额波动，而精确匹配准确率则朝相反方向变化。一项使用第二个DeepSeek生成器及服务配置的结果盲评事后100题子集复现发现，即使在精确匹配率上升3.00个百分点的情况下，语义超额波动仍达8.75个百分点。因此，在没有显著或方向一致的效用变化时，答案级兼容性仍可能失效。检索增强发布应在评估效用的同时审计兼容性。

English

A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval policy, evidence depth, rendering, and exposed generation controls are held fixed. Aggregate accuracy may hide these changes when gains and losses cancel, while ordinary generation variability makes one-shot comparisons overstate update effects. We call the hidden phenomenon accuracy-blind answer churn and introduce the Snapshot Compatibility Audit, which estimates excess answer churn by subtracting same-snapshot repeat disagreement from cross-snapshot disagreement. We instantiate it by expanding one frozen FineWeb prefix from one to seven shards. In a preregistered 400-question Natural Questions study, normalized-exact and blinded-semantic excess churn are 6.44 and 10.25 percentage points while exact-match accuracy changes by only -1.50 points. A post-hoc analysis finds repeat-stable semantic flips on 40/400 questions. A separately preregistered 200-question TriviaQA study yields smaller, directionally consistent excess churn while exact-match accuracy moves in the opposite direction. An outcome-blind post-hoc 100-question subset replication with a second DeepSeek generator and serving configuration finds 8.75 pp of semantic excess churn even as exact match rises by 3.00 percentage points. Answer-level compatibility can therefore fail without a conspicuous or consistently directed utility shift. Retrieval-augmented releases should audit compatibility alongside utility.