探索语言智能体与非语言智能体之间的协作

摘要

大语言模型（LLM）正越来越多地被部署为编排器，通过自然语言协调专门的子代理来解决复杂任务。然而，在游戏对局和机器人等许多重要领域，目前可用的最强智能体并非语言模型。将非语言智能体与 LLM 集成需要语言化：即在每个交互步骤，将其丰富的连续表示压缩为稀疏的文本摘要。为了研究语言化是否会构成瓶颈，我们提出了 LLAMIA-Bench——一个包含六项多样化协作国际象棋任务的基准套件，涵盖行为模仿、状态评估和自然语言解释三个方面。每项任务都实例化了一个公认的国际象棋难题，而该难题单独靠 LLM 或单独靠国际象棋引擎都无法解决。为解决 LLM 与非语言智能体的协作问题，我们引入潜在状态内化：将子代理的连续表示作为学习得到的状态 token 直接投射到 LLM 的 token 流中，并在动作推进环境状态时进行动态重新编码。通过将内化与语言化集成进行比较，实验一致揭示出一种语言化债务：性能差距在训练过程中不断扩大，并在 LLM 从 4B 扩展到 14B 参数时持续存在。采用潜在状态内化训练的单一 14B 模型 LLAMIA，在所有基准任务上均能达到甚至超越任务专家和前沿模型，包括具备工具访问能力的 GPT-5.1；同时，它还能在任务特定微调模型失效的分布外场景中实现泛化。

English

LLMs are increasingly deployed as orchestrators that coordinate specialized subagents to solve complex tasks through natural language. However, in many important domains like game playing and robotics, the strongest available agents are not language models. Integrating non-language agents with LLMs would require verbalization: compressing their rich continuous representations into sparse textual summaries at each interaction step. To study whether verbalization constitutes a bottleneck, we introduce LLAMIA-Bench, a suite of six diverse collaborative chess tasks spanning three facets: behavioral imitation, state assessment, and natural-language explanation. Each task instantiates a well-established chess problem that neither the LLM nor the chess engine can solve alone. To solve LLM collaboration with non-language agents, we introduce latent state internalization, which projects the subagent's continuous representations directly into the LLM's token stream as learned state tokens, with dynamic re-encoding as actions advance the environment state. Comparing internalization to verbalized integration, our experiments reveal a consistent verbalization debt: the performance gap widens throughout training and persists as the LLM scales from 4B to 14B parameters. A single 14B model, LLAMIA, trained with latent state internalization, matches or exceeds task specialists and frontier models including GPT-5.1 with tool access across all benchmark tasks, and generalizes out-of-distribution where task-specific finetunes collapse