重新格式化對齊

摘要

Fine-tuning 資料的品質對於調整大型語言模型（LLMs）與人類價值觀之間的一致性至關重要。目前改善資料品質的方法要麼耗時耗力，要麼容易出現因LLM幻覺而導致的事實錯誤。本文探討如何提升現有指示資料的品質，以更好地與人類價值觀保持一致，並介紹了一種名為ReAlign的簡單有效方法，該方法將指示資料的回應重新格式化為更符合預先確定標準和匯總證據的格式。這種方法最小化了人類標註、幻覺和擴展困難，與現有的對齊技術保持正交。在實驗中，ReAlign 顯著提升了LLMs的一般對齊能力、數學推理、事實性和可讀性。令人鼓舞的是，在不引入任何額外資料或高級訓練技術的情況下，僅通過重新格式化回應，LLaMA-2-13B在GSM8K上的數學推理能力從46.77%提高到56.63%的準確度。此外，僅使用 5% 的 ReAlign 資料就使 Alpaca 資料集測量的一般對齊能力提升了 67%。這項工作凸顯了對LLMs的科學和機械解釋能力進行進一步研究的必要性。我們已經將相關的代碼和資料公開，以支持未來研究，網址為 https://github.com/GAIR-NLP/ReAlign。

English

The quality of finetuning data is crucial for aligning large language models (LLMs) with human values. Current methods to improve data quality are either labor-intensive or prone to factual errors caused by LLM hallucinations. This paper explores elevating the quality of existing instruction data to better align with human values, introducing a simple and effective approach named ReAlign, which reformats the responses of instruction data into a format that better aligns with pre-established criteria and the collated evidence. This approach minimizes human annotation, hallucination, and the difficulty in scaling, remaining orthogonal to existing alignment techniques. Experimentally, ReAlign significantly boosts the general alignment ability, math reasoning, factuality, and readability of the LLMs. Encouragingly, without introducing any additional data or advanced training techniques, and merely by reformatting the response, LLaMA-2-13B's mathematical reasoning ability on GSM8K can be improved from 46.77% to 56.63% in accuracy. Additionally, a mere 5% of ReAlign data yields a 67% boost in general alignment ability measured by the Alpaca dataset. This work highlights the need for further research into the science and mechanistic interpretability of LLMs. We have made the associated code and data publicly accessible to support future studies at https://github.com/GAIR-NLP/ReAlign.

重新格式化對齊

Reformatted Alignment

摘要

Summary

Support

Support