多頭潛在控制：LLM 智能體決策制定的統一介面

摘要

大型語言模型日益被部署為代理，但可靠的代理行為所需的遠不止於下一個token的預測。在推理階段，代理最好能自行決定是否繼續當前的推理過程、是否交由更強大的模型處理、是否請求額外資訊、是否調用外部工具，或是在當前設定下選擇放棄。現有方法透過提示層級路由、外部編排或任務專屬微調來處理這些決策，這些方法主要依賴輸入端的訊號，且隨著模型主幹的演進，往往成本高昂且難以維護。我們探討這類控制決策能否直接從模型的潛在生成過程中推斷出來。我們提出多頭潛在控制（Multi-Head Latent Control），這是一個輕量層，能從已凍結的大型語言模型（LLM）或視覺語言模型（VLM）中讀取隱藏狀態軌跡，以產生部署時的控制信號。能力頭（Capability Head）預測當前模型是否能解決該實例，或應交由更強大的協作者處理；而決議頭（Resolution Head）則預測適當的決議決策：釐清、工具使用、放棄，或直接回答。這兩個頭僅在相同凍結LLM主幹的潛在軌跡上進行訓練，從而實現無需修改模型的事後適應。在語言與視覺語言設定中，多頭潛在控制一貫改善了多模型系統的品質-成本權衡，實現了基於部分生成內容的早期交接與更準確的干預決策。在路由執行（小模型+大模型）中，它在AndroidWorld上減少了高達90.7%的大模型使用量，在各基準測試中平均減少27-53%，同時保留了大部分大模型的性能。此外，學習到的控制信號提升了工具使用決策的品質，帶來高達+158%的相對分數提升，並減少了65.5%的必要工具調用遺漏。

English

Large language models are increasingly deployed as agents, but reliable agentic behavior requires more than next-token prediction. At inference time, it is preferred that an agent can decide whether to proceed with its current reasoning, defer to a stronger model, request additional information, invoke external tools, or abstain under the given setup. Existing approaches address these decisions through prompt-level routing, external orchestration, or task-specific fine-tuning, which primarily rely on input-side signals, and are often costly and difficult to maintain as model backbones evolve. We ask whether such control decisions can be inferred directly from a model's latent generation process. We introduce Multi-Head Latent Control, a lightweight layer that reads hidden-state trajectories from a frozen LLM or VLM to produce deployment-time control signals. A Capability Head predicts whether the current model can solve the instance or should defer to a stronger collaborator, while a Resolution Head predicts appropriate resolution decision Clarification, Tool Use, Abstention, or Direct Answering. Both heads are trained only on latent traces from the same frozen LLM backbone, enabling post hoc adaptation without modifying the model. Across language and vision-language settings, Multi-Head Latent Control consistently improves the quality-cost tradeoff of multi-model systems, enabling early handoff from partial generations and more accurate intervention decisions. In routed execution (small + large model), it reduces large-model usage by up to 90.7 percent on AndroidWorld and 27-53 percent on average across benchmarks, while retaining most of large-model performance. Additionally, the learned control signals improve tool-use decision quality, yielding up to +158 percent relative score gain and 65.5 percent fewer missed-required tool calls.