優化分解以實現最佳聲明驗證

Optimizing Decomposition for Optimal Claim Verification

March 19, 2025

作者: Yining Lu, Noah Ziems, Hy Dang, Meng Jiang

cs.AI

摘要

当前关于“分解-验证”范式用于评估长文本事实性的研究，通常将分解与验证孤立对待，忽视了它们之间的相互作用及潜在的不一致性。我们发现，现有的分解策略（通常为手工设计的示例）在原子性（一种量化信息密度的新指标）方面与下游验证器并不匹配，导致验证结果欠佳。我们将寻找最优分解策略以实现最优验证的问题形式化为一个双层优化问题。为了近似求解这一强NP难问题，我们提出了动态分解方法，这是一个强化学习框架，利用验证器的反馈来学习一种策略，动态地将声明分解为验证器偏好的原子性。实验结果表明，动态分解方法优于现有的分解策略，在不同验证器、数据集及输入声明原子性的情况下，平均将验证置信度提高了0.07，准确率提升了0.12（基于0-1评分标准）。

English

Current research on the Decompose-Then-Verify paradigm for evaluating the factuality of long-form text typically treats decomposition and verification in isolation, overlooking their interactions and potential misalignment. We find that existing decomposition policies, typically hand-crafted demonstrations, do not align well with downstream verifiers in terms of atomicity -- a novel metric quantifying information density -- leading to suboptimal verification results. We formulate finding the optimal decomposition policy for optimal verification as a bilevel optimization problem. To approximate a solution for this strongly NP-hard problem, we propose dynamic decomposition, a reinforcement learning framework that leverages verifier feedback to learn a policy for dynamically decomposing claims to verifier-preferred atomicity. Experimental results show that dynamic decomposition outperforms existing decomposition policies, improving verification confidence by 0.07 and accuracy by 0.12 (on a 0-1 scale) on average across varying verifiers, datasets, and atomcities of input claims.