SUPER：評估代理人在設置和執行任務的研究存儲庫上的表現

SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories

September 11, 2024

作者: Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, Tushar Khot

cs.AI

摘要

鑑於大型語言模型（LLMs）在編寫程式碼方面取得了顯著進展，它們現在能否被用於自主復制研究存儲庫中的結果？這種能力將對研究社區帶來幫助，幫助研究人員驗證、理解和擴展先前的工作。為了朝著這個目標邁進，我們引入了SUPER，這是第一個旨在評估LLMs在設置和執行來自研究存儲庫任務能力的基準。SUPER旨在捕捉與機器學習（ML）和自然語言處理（NLP）研究存儲庫中工作的研究人員所面臨的現實挑戰。我們的基準包括三個不同的問題集：45個具有註釋專家解決方案的端對端問題，從專家集合中衍生出的152個專注於特定挑戰（例如配置訓練器）的子問題，以及自動生成的602個用於更大規模開發的問題。我們引入了各種評估措施來評估任務成功和進展，利用金標準解決方案（如果可用）或其他近似值。我們展示了最先進的方法在解決這些問題時遇到困難，最佳模型（GPT-4o）僅解決了端對端集合的16.3％，以及46.1％的情境。這說明了這個任務的挑戰，並表明SUPER可以作為社區製定和衡量進展的寶貴資源。

English

Given that Large Language Models (LLMs) have made significant progress in writing code, can they now be used to autonomously reproduce results from research repositories? Such a capability would be a boon to the research community, helping researchers validate, understand, and extend prior work. To advance towards this goal, we introduce SUPER, the first benchmark designed to evaluate the capability of LLMs in setting up and executing tasks from research repositories. SUPERaims to capture the realistic challenges faced by researchers working with Machine Learning (ML) and Natural Language Processing (NLP) research repositories. Our benchmark comprises three distinct problem sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems derived from the expert set that focus on specific challenges (e.g., configuring a trainer), and 602 automatically generated problems for larger-scale development. We introduce various evaluation measures to assess both task success and progress, utilizing gold solutions when available or approximations otherwise. We show that state-of-the-art approaches struggle to solve these problems with the best model (GPT-4o) solving only 16.3% of the end-to-end set, and 46.1% of the scenarios. This illustrates the challenge of this task, and suggests that SUPER can serve as a valuable resource for the community to make and measure progress.