FM-Bench：面向竞争性智能体的长期管理基准

摘要

语言模型智能体如今已能可靠地执行有界任务。然而，它们能否在长时间跨度内维持有效决策——其间行动会带来累积性后果，环境也会对它们的选择作出响应——在很大程度上仍未得到衡量。FM-Bench（足球管理基准）正是为此而设：一个LLM智能体通过26个工具以及约340至400个决策节点，在游戏中经营一家足球俱乐部长达20年。它以与所有竞争对手相同的预算组建阵容、交易球员、谈判合同、投资设施与青训、排定出场阵容，并向一个可以解雇它的董事会负责；与此同时，一个确定性引擎将每一年的表现累积为唯一的最终得分，无需LLM裁判或人工评分员参与。单模型赛道让15个前沿模型各自面对一个冻结的脚本化世界进行对局；Arena则将这些相同模型外加一个脚本化锚点模型置于同一个共享的20年世界中——据我们所知，这是该规模下首次进行的同场正面交锋式评估。我们衡量了得分背后的六种行为能力。在三个随机种子下，全部15个模型均完成了每一段完整历程，而盲脚本基线在大多数对应历程中出局；claude-fable-5在平均得分上位居单模型赛道榜首，并同样在Arena中胜出——不过Arena的冠军头衔会在十个模型之间轮换。模型规模、价格与供应商均无法预测最终排序；排序要到历程后期才尘埃落定，而首次游玩的人类最佳选手也仅能排在模型排行榜的末尾。将各个模型区分开来的并非计算能力，而是管理行为：得分较高的模型会在接近结束时减少迟效投资，让资金保持运转而非闲置，并远在截止期限之前就开启续约；而token消耗量则无法预测任何结果。没有模型能从数百次被拒的报价中学到市场的隐藏价格；自管理记忆则呈现出两种相反的失败模式——要么是一个只增不减的档案，要么是一份每个赛季都被重写的计划。代码可在 https://github.com/Analogy-AI/fm-bench 获取。

English

Language model agents now execute bounded tasks reliably. Whether they can sustain effective decision-making over long horizons, where actions have cumulative consequences and the environment responds to their choices, remains largely unmeasured. FM-Bench (Football Management Benchmark) measures this. An LLM agent runs a football club for 20 in-game years through 26 tools and roughly 340 to 400 decision stops. It drafts a squad on the same budget as every rival, trades players, negotiates contracts, invests in facilities and youth, sets lineups, and answers to a board that can fire it, while a deterministic engine accumulates every year into one final score with no LLM judge or human rater. The solo track plays each of 15 frontier models against a frozen scripted world, and the Arena places the same models plus a scripted anchor in one shared 20-year world; to our knowledge, the first head-to-head evaluation at this scale. We measure six behavioral capabilities behind the score. Across three seeds, all 15 models complete every horizon while the blind scripted baselines die out in most of theirs, and claude-fable-5 tops the solo board on mean score and the Arena, where the title nonetheless rotates among ten models. Neither scale, price, nor vendor predicts the order; the order settles only late in the horizon, and the best first-play human lands only at the bottom of the model board. What separates the models is managerial behavior rather than computation. Higher-scoring models reduce slow-payoff investment near the end, keep cash invested rather than idle, and open renewals well before the deadline, while token spend predicts nothing. No model learns the market's hidden prices from hundreds of rejected bids, and self-managed memory fails in two opposite modes: an archive that only grows or a plan rewritten every season. Code is available at https://github.com/Analogy-AI/fm-bench.