优化分解以实现最优声明验证

Optimizing Decomposition for Optimal Claim Verification

March 19, 2025

作者: Yining Lu, Noah Ziems, Hy Dang, Meng Jiang

cs.AI

摘要

当前关于评估长篇文本事实性的“分解-验证”范式研究，通常将分解与验证过程孤立对待，忽视了二者间的相互作用及潜在的不匹配问题。我们发现，现有的分解策略，多为手工设计的示例，在原子性（一种量化信息密度的新指标）方面与下游验证器未能良好对齐，导致验证效果欠佳。为此，我们将寻找最优分解策略以实现最佳验证的问题，建模为一个双层优化问题。针对这一强NP难问题，我们提出了动态分解方法，这是一个强化学习框架，它利用验证器的反馈来学习一种策略，动态地将声明分解为验证器偏好的原子性水平。实验结果表明，动态分解策略优于现有分解方法，在不同验证器、数据集及输入声明原子性的条件下，平均提升了0.07的验证置信度和0.12的准确率（基于0-1评分标准）。

English

Current research on the Decompose-Then-Verify paradigm for evaluating the factuality of long-form text typically treats decomposition and verification in isolation, overlooking their interactions and potential misalignment. We find that existing decomposition policies, typically hand-crafted demonstrations, do not align well with downstream verifiers in terms of atomicity -- a novel metric quantifying information density -- leading to suboptimal verification results. We formulate finding the optimal decomposition policy for optimal verification as a bilevel optimization problem. To approximate a solution for this strongly NP-hard problem, we propose dynamic decomposition, a reinforcement learning framework that leverages verifier feedback to learn a policy for dynamically decomposing claims to verifier-preferred atomicity. Experimental results show that dynamic decomposition outperforms existing decomposition policies, improving verification confidence by 0.07 and accuracy by 0.12 (on a 0-1 scale) on average across varying verifiers, datasets, and atomcities of input claims.