多头潜在控制：LLM智能体决策的统一接口

摘要

大型语言模型日益被部署为智能代理，但可靠的代理行为需要超越简单的下一个词预测。在推理阶段，理想情况下，代理应能自主决定是继续当前推理、将任务移交给更强模型、请求额外信息、调用外部工具，还是在给定设置下放弃回答。现有方法通过提示级路由、外部编排或任务特定微调来处理这些决策，这些方法主要依赖输入侧信号，且随着模型主干迭代升级，往往成本高昂且难以维护。我们探究能否直接从模型的潜在生成过程中推断此类控制决策。为此，我们提出多头部潜在控制层，这是一种轻量级架构，通过读取冻结大语言模型或视觉语言模型的隐藏状态轨迹，生成部署时的控制信号。能力头部预测当前模型能否解决该实例，或应移交至更强协作模型；决策头部则预测适当的解决决策类型——澄清、工具调用、放弃或直接回答。两个头部仅基于相同冻结大语言模型主干的潜在轨迹进行训练，从而在无需修改模型的前提下实现事后适配。在语言和视觉-语言场景中，多头部潜在控制持续改善多模型系统的质量-成本权衡，支持基于部分生成结果的早期移交和更精准的干预决策。在路由执行（小模型+大模型）场景中，该方法在AndroidWorld上减少大模型使用量高达90.7%，跨基准测试平均减少27%-53%，同时保留大模型大部分性能。此外，学习到的控制信号提升了工具调用决策质量，相对得分提升最高达158%，且遗漏必要工具调用的次数减少65.5%。

English

Large language models are increasingly deployed as agents, but reliable agentic behavior requires more than next-token prediction. At inference time, it is preferred that an agent can decide whether to proceed with its current reasoning, defer to a stronger model, request additional information, invoke external tools, or abstain under the given setup. Existing approaches address these decisions through prompt-level routing, external orchestration, or task-specific fine-tuning, which primarily rely on input-side signals, and are often costly and difficult to maintain as model backbones evolve. We ask whether such control decisions can be inferred directly from a model's latent generation process. We introduce Multi-Head Latent Control, a lightweight layer that reads hidden-state trajectories from a frozen LLM or VLM to produce deployment-time control signals. A Capability Head predicts whether the current model can solve the instance or should defer to a stronger collaborator, while a Resolution Head predicts appropriate resolution decision Clarification, Tool Use, Abstention, or Direct Answering. Both heads are trained only on latent traces from the same frozen LLM backbone, enabling post hoc adaptation without modifying the model. Across language and vision-language settings, Multi-Head Latent Control consistently improves the quality-cost tradeoff of multi-model systems, enabling early handoff from partial generations and more accurate intervention decisions. In routed execution (small + large model), it reduces large-model usage by up to 90.7 percent on AndroidWorld and 27-53 percent on average across benchmarks, while retaining most of large-model performance. Additionally, the learned control signals improve tool-use decision quality, yielding up to +158 percent relative score gain and 65.5 percent fewer missed-required tool calls.