FM-Bench：競爭性智能體長期管理基準測試

摘要

語言模型代理目前能可靠地執行有限任務。然而，它們能否在長時間跨度中維持有效決策——在這種情境下，行動具有累積性後果，且環境會對其選擇做出回應——在很大程度上仍未被測量。FM-Bench（足球管理基準）正是用於衡量此能力。一個LLM代理透過26種工具及約340至400個決策節點，經營一家足球俱樂部達20個遊戲內賽季。它以與所有對手相同的預算組建陣容、交易球員、協商合約、投資設施與青訓、設定先發陣容，並向可將其解僱的董事會負責；與此同時，一個確定性引擎逐年累積所有結果，最終形成單一最終分數，全程無需LLM裁判或人類評分員。單人賽道讓15個前沿模型各自對抗一個固定腳本化世界，而競技場則將相同模型加上一個腳本化錨點置於同一個共享的20年世界之中；據我們所知，這是此規模下首次的正面對決評估。我們衡量了分數背後六項行為能力。在三個隨機種子下，所有15個模型均完成每個時間跨度，而盲目腳本化基線則在它們的大多數時間跨度中消亡；claude-fable-5在平均分數上位居單人賽道榜首，亦在競技場奪冠，儘管冠軍頭銜在競技場中仍輪流落在十個模型之間。規模、價格或供應商均無法預測排名；排名要到時間跨度後期才趨於穩定，且最佳的人類首秀玩家僅落在模型排行榜末位。區分各模型的關鍵在於管理行為而非計算能力。得分較高的模型會在接近尾聲時減少延遲回報投資、保持資金持續投入而非閒置，並在截止日期前及早開啟續約；而Token消耗則無法預測任何結果。沒有任何模型能從數百次被拒絕的出價中學會市場的隱藏價格，而自我管理記憶則以兩種相反的模式失效：檔案庫只增不減，或計劃每賽季被重寫。程式碼可於 https://github.com/Analogy-AI/fm-bench 取得。

English

Language model agents now execute bounded tasks reliably. Whether they can sustain effective decision-making over long horizons, where actions have cumulative consequences and the environment responds to their choices, remains largely unmeasured. FM-Bench (Football Management Benchmark) measures this. An LLM agent runs a football club for 20 in-game years through 26 tools and roughly 340 to 400 decision stops. It drafts a squad on the same budget as every rival, trades players, negotiates contracts, invests in facilities and youth, sets lineups, and answers to a board that can fire it, while a deterministic engine accumulates every year into one final score with no LLM judge or human rater. The solo track plays each of 15 frontier models against a frozen scripted world, and the Arena places the same models plus a scripted anchor in one shared 20-year world; to our knowledge, the first head-to-head evaluation at this scale. We measure six behavioral capabilities behind the score. Across three seeds, all 15 models complete every horizon while the blind scripted baselines die out in most of theirs, and claude-fable-5 tops the solo board on mean score and the Arena, where the title nonetheless rotates among ten models. Neither scale, price, nor vendor predicts the order; the order settles only late in the horizon, and the best first-play human lands only at the bottom of the model board. What separates the models is managerial behavior rather than computation. Higher-scoring models reduce slow-payoff investment near the end, keep cash invested rather than idle, and open renewals well before the deadline, while token spend predicts nothing. No model learns the market's hidden prices from hundreds of rejected bids, and self-managed memory fails in two opposite modes: an archive that only grows or a plan rewritten every season. Code is available at https://github.com/Analogy-AI/fm-bench.