探索語言智能體與非語言智能體之間的協作

摘要

大型語言模型（LLM）日益被部署為協調器，透過自然語言協調專門的子代理來解決複雜任務。然而，在遊戲對弈與機器人學等許多重要領域中，現有最強的代理並非語言模型。若要將非語言代理與LLM整合，就必須進行語言化：在每個互動步驟，將它們豐富的連續表徵壓縮成稀疏的文字摘要。為了探討語言化是否構成瓶頸，我們提出 LLAMIA-Bench：一套涵蓋三個面向、共六項多樣化西洋棋協作任務的基準，三個面向分別是行為模仿、狀態評估與自然語言解釋。每個任務都具體呈現一個成熟的西洋棋問題，而該問題無法由LLM或西洋棋引擎單獨解決。為了解決LLM與非語言代理的協作問題，我們引入潛在狀態內化法，將子代理的連續表徵作為學習到的狀態token直接投影到LLM的token串流中，並在行動推進環境狀態時動態重新編碼。透過將內化與語言化整合進行比較，我們的實驗揭示了持續存在的語言化債：效能差距在訓練過程中不斷擴大，且在LLM從4B擴展至14B參數時依然存在。單一14B模型LLAMIA以潛在狀態內化法訓練，在所有基準任務上均能匹配或超越任務專用模型及具備工具使用能力的前沿模型（包括GPT-5.1），並能泛化至任務特定微調模型會崩潰的分布外情境。

English

LLMs are increasingly deployed as orchestrators that coordinate specialized subagents to solve complex tasks through natural language. However, in many important domains like game playing and robotics, the strongest available agents are not language models. Integrating non-language agents with LLMs would require verbalization: compressing their rich continuous representations into sparse textual summaries at each interaction step. To study whether verbalization constitutes a bottleneck, we introduce LLAMIA-Bench, a suite of six diverse collaborative chess tasks spanning three facets: behavioral imitation, state assessment, and natural-language explanation. Each task instantiates a well-established chess problem that neither the LLM nor the chess engine can solve alone. To solve LLM collaboration with non-language agents, we introduce latent state internalization, which projects the subagent's continuous representations directly into the LLM's token stream as learned state tokens, with dynamic re-encoding as actions advance the environment state. Comparing internalization to verbalized integration, our experiments reveal a consistent verbalization debt: the performance gap widens throughout training and persists as the LLM scales from 4B to 14B parameters. A single 14B model, LLAMIA, trained with latent state internalization, matches or exceeds task specialists and frontier models including GPT-5.1 with tool access across all benchmark tasks, and generalizes out-of-distribution where task-specific finetunes collapse