Coffee-Gym：一个用于评估和改进自然语言反馈错误代码的环境

Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code

September 29, 2024

作者: Hyungjoo Chae, Taeyoon Kwon, Seungjun Moon, Yongho Song, Dongjin Kang, Kai Tzu-iunn Ong, Beong-woo Kwak, Seonghyeon Bae, Seung-won Hwang, Jinyoung Yeo

cs.AI

摘要

本文介绍了Coffee-Gym，一个用于训练能够提供代码编辑反馈的模型的全面RL环境。Coffee-Gym包括两个主要组件：(1) Coffee，一个包含人类代码编辑痕迹和针对错误代码编辑的机器生成反馈的数据集；(2) CoffeeEval，一个奖励函数，通过评估修订后代码在单元测试中的性能，忠实地反映反馈的帮助性。借助这两个组件，Coffee-Gym解决了用于训练RL反馈模型的高质量数据集不可用的问题，并提供比当前最先进的奖励模型（即GPT-4）更准确的奖励。通过应用Coffee-Gym，我们获得了优于基线的反馈模型，能够增强开源代码LLM的代码编辑，使其与闭源LLM相媲美。我们已公开提供数据集和模型检查点。

English

This paper presents Coffee-Gym, a comprehensive RL environment for training models that provide feedback on code editing. Coffee-Gym includes two major components: (1) Coffee, a dataset containing humans' code edit traces for coding questions and machine-written feedback for editing erroneous code; (2) CoffeeEval, a reward function that faithfully reflects the helpfulness of feedback by assessing the performance of the revised code in unit tests. With them, Coffee-Gym addresses the unavailability of high-quality datasets for training feedback models with RL, and provides more accurate rewards than the SOTA reward model (i.e., GPT-4). By applying Coffee-Gym, we elicit feedback models that outperform baselines in enhancing open-source code LLMs' code editing, making them comparable with closed-source LLMs. We make the dataset and the model checkpoint publicly available.