图迭代对齐

Iterative Graph Alignment

August 29, 2024

作者: Fangyuan Yu, Hardeep Singh Arora, Matt Johnson

cs.AI

摘要

通过压缩多样化叙事，LLM超越了仅仅记忆的范畴，通过捕捉可泛化的因果关系实现了智能化。然而，由于训练数据多样性不足，它们存在局部的“表示间隙”，限制了它们在现实世界中的实用性，特别是在需要严格遵循规则的任务中。依赖于繁重人工标注的传统对齐方法效率低下且不可扩展。最近的自对齐技术也存在不足，因为它们通常依赖于基于自我选择的提示和基于记忆的学习。为了解决这些问题，我们引入了迭代图对齐（IGA），这是一种无需注释的基于规则的对齐算法。一位教师模型（VLM）采用迭代图提示（IGP）来创建逻辑图和参考答案。学生模型（LLM）通过尝试将其响应与这些参考答案对齐来识别局部知识间隙，与辅助模型合作生成多样化的答案。然后，这些对齐的响应被用于迭代监督微调（SFT）。我们在五个基于规则的场景中的评估显示了IGP的有效性，在Claude Sonnet 3.5中实现了73.12\%的对齐改进，Llama3-8B-Instruct实现了86.20%的改进，优于Claude Sonnet 3.5在基于规则的对齐方面。

English

By compressing diverse narratives, LLMs go beyond memorization, achieving intelligence by capturing generalizable causal relationships. However, they suffer from local 'representation gaps' due to insufficient training data diversity, limiting their real-world utility, especially in tasks requiring strict alignment to rules. Traditional alignment methods relying on heavy human annotations are inefficient and unscalable. Recent self-alignment techniques also fall short, as they often depend on self-selection based prompting and memorization-based learning. To address these issues, we introduce Iterative Graph Alignment (IGA), an annotation-free rule-based alignment algorithm. A teacher model (VLM) employs Iterative Graph Prompting (IGP) to create logical graphs and reference answers. The student model (LLM) identifies local knowledge gaps by attempting to align its responses with these references, collaborating with helper models to generate diverse answers. These aligned responses are then used for iterative supervised fine-tuning (SFT). Our evaluations across five rule-based scenarios demonstrate IGP's effectiveness, with a 73.12\% alignment improvement in Claude Sonnet 3.5, and Llama3-8B-Instruct achieving an 86.20\% improvement, outperforming Claude Sonnet 3.5 in rule-based alignment.