无标签策略下准确性与顺序敏感性出现分歧

摘要

多项选择基准广泛用于评估大型语言模型，但多项选择（MCQ）分数将知识与对选项顺序的敏感性混为一谈，使其成为模型知识不可靠的衡量标准。在本文中，我们测试了在模型确定答案时阻止其看到选项标签是否能消除位置影响，并进而提高性能。我们评估了两种不同的去偏策略。第一种采用“先生成后匹配”的方法，第二种则对选项进行独立评分，该方法在构造上对位置无偏。两种方法均未可靠地提高准确率。完整的分解分析表明，瓶颈在于隐藏选项，而非匹配步骤。唯一能够稳定匹配基线的配置是向模型展示所有选项并配以LLM匹配器的配置。然而，完全消除位置影响仍然无法可靠地带来准确率提升，而循环置换反而常常提高准确率。对于两阶段提示，召回不平衡的聚合度量以及直接的逐问题顺序敏感性度量均未能显示出可靠的去偏效果。

English

Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, which makes them unreliable measures of model knowledge. In this paper, we test whether preventing a model from seeing option labels while committing to an answer removes positional influence and, in turn, improves performance. We evaluate two different strategies for mitigating bias. The first uses a generation-then-matching approach, and the second scores options in isolation, which is positionally unbiased by construction. Neither reliably improves accuracy. A complete decomposition shows that the bottleneck is withholding options, not the matching step. The only configuration that consistently matches the baseline is the one that shows the model all options paired with an LLM matcher. However, eliminating positional influence entirely still does not reliably yield accuracy gains, while cyclic permutation often improves them. For two-stage prompting, an aggregate measure of recall imbalance and a direct per-question measure of order sensitivity both fail to show reliable debiasing.