CoffeeBench：在異質多智能體經濟中評估長時程LLM智能體

摘要

隨著大型語言模型代理能夠執行越來越長期的任務，評估它們在經濟系統中的表現變得越發重要。有別於現有主要評估單一代理與被動環境互動的基準，經濟系統本質上涉及多個代理，要求自主代理在長期追求自身目標的過程中，進行溝通、協商與交易。我們提出 CoffeeBench，這是一個用於評估大型語言模型代理在由異質企業組成的長期多代理經濟體中的基準。在 CoffeeBench 中，兩名農民、兩名烘焙師與兩名零售商在為期 90 天的模擬中自主經營其業務，各自透過溝通與交易管理現金、庫存與定價，力求最大化累積淨收入。被評估的模型控制一名咖啡烘焙師，其餘企業則由固定的參考代理所控制。在近期多個開源權重與專有的大型語言模型中，所有模型皆優於不採取任何行動的被動基準，大多數模型都能獲得正的淨收入。代理行為分析顯示，長期經濟互動存在顯著差異：表現較佳的模型與其他企業溝通更為積極，而 Claude Haiku 4.5 則表現出「閒置漂移」的失敗模式，即便能產出連貫的評估與計畫，仍反覆選擇不行動。我們釋出程式碼與代理軌跡，以支持未來的研究。

English

As LLM agents become capable of increasingly long-horizon tasks, evaluating their performance in economic systems is becoming increasingly important. Unlike existing benchmarks that primarily evaluate a single agent interacting with a passive environment, economic systems are inherently multi-agent, requiring autonomous agents to communicate, negotiate, and transact while pursuing their own objectives over extended periods. We introduce CoffeeBench, a benchmark for evaluating LLM agents in a long-horizon multi-agent economy composed of heterogeneous firms. In CoffeeBench, two farmers, two roasters, and two retailers autonomously operate their businesses over a 90-day simulation, each seeking to maximize cumulative net income through communication and transactions while managing cash, inventory, and pricing. The evaluated model controls one coffee roaster, while the remaining firms are controlled by fixed reference agents. Across several recent open-weight and proprietary LLMs, all models outperform a passive baseline that takes no actions, with most achieving positive net income. Analysis of agent behavior reveals substantial differences in long-horizon economic interaction: higher-performing models communicate more actively with other firms, whereas Claude~Haiku~4.5 exhibits an idle-drift failure mode, repeatedly choosing inaction despite producing coherent assessments and plans. We release our code and agent trajectories to support future research.