在無標記策略下，準確度與順序敏感度呈現分歧

摘要

選擇題基準被廣泛用於評估大型語言模型，但選擇題分數將知識與對選項順序的敏感性混為一談，使其成為模型知識不可靠的衡量指標。在本文中，我們測試在模型作出選擇時防止其看到選項標籤，是否能去除位置影響，進而提升表現。我們評估了兩種減輕偏差的不同策略。第一種採用先生成後匹配的方法，第二種則單獨對選項評分，這種方法在設計上即無位置偏差。兩者都無法可靠地提升準確度。完整的分解分析顯示，瓶頸在於隱藏選項，而非匹配步驟。唯一能持續與基線相符的配置，是向模型展示所有選項並搭配 LLM 匹配器的配置。然而，完全消除位置影響仍然無法可靠地帶來準確度提升，而循環排列則經常能提升準確度。對於兩階段提示，召回率不平衡的總體指標以及每個問題順序敏感度的直接指標，都無法顯示可靠的去偏效果。

English

Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, which makes them unreliable measures of model knowledge. In this paper, we test whether preventing a model from seeing option labels while committing to an answer removes positional influence and, in turn, improves performance. We evaluate two different strategies for mitigating bias. The first uses a generation-then-matching approach, and the second scores options in isolation, which is positionally unbiased by construction. Neither reliably improves accuracy. A complete decomposition shows that the bottleneck is withholding options, not the matching step. The only configuration that consistently matches the baseline is the one that shows the model all options paired with an LLM matcher. However, eliminating positional influence entirely still does not reliably yield accuracy gains, while cyclic permutation often improves them. For two-stage prompting, an aggregate measure of recall imbalance and a direct per-question measure of order sensitivity both fail to show reliable debiasing.