边想边说：面向响应式智能对话语音代理的推理时知识迁移

摘要

语音代理面临一个根本性矛盾：使基础模型具备能力的推理、检索和工具调用过程是迭代且缓慢的，而对话交互要求毫秒级的响应速度。较小的实时模型能满足延迟要求，但在复杂任务上无法匹敌基础模型，导致当前语音代理不得不在响应能力与功能完备性之间取舍。我们提出会话填充（conversational infill）方法，即通过一个小型说话模型同时实现：即时生成上下文相关的响应以隐藏外部推理模型的延迟，并在推理过程中流畅地将推理模型的流式知识整合进自身的响应。我们构建了涵盖六个领域的290,571条合成数据集，实验表明，在七个主流小型语言模型（参数量从135M到1.7B不等）中，该任务均可学习。我们的系统实现ConvFill，在将精度差距缩小至对应前沿推理模型性能的6.3%以内时，仍能保持毫秒级的首次响应时间。在基于Apple M2 SoC运行说话模型的18人实时用户研究中，参与者认为ConvFill整体与前沿模型相当，在检索密集型任务中更受青睐，并对其响应速度给予显著更高评价。这些结果表明，会话填充在延迟-能力帕累托前沿上开辟了新平衡点，为打造兼具快速响应与强大能力的语音代理提供了可行路径。代码、模型和数据集已开源：https://github.com/vysri/conversational-infill。

English

Voice agents face a fundamental tension: the reasoning, retrieval, and tool use that make foundation models capable are iterative and slow, while conversational interaction demands responses on a millisecond timescale. Smaller, real-time models meet the latency bar but cannot match foundation models on complex tasks, leaving current voice agents to trade away either responsiveness or capability. We introduce conversational infill, where a small talker model both immediately generates contextually grounded responses to hide the latency of an external reasoner model and fluently integrates streamed reasoner knowledge into its responses during inference. We curate a 290,571-example synthetic dataset spanning six domains and demonstrate that this task is learnable across seven widely used small language models ranging from 135M to 1.7B parameters. Our system implementation, ConvFill, sustains millisecond-level time-to-first-response while closing the accuracy gap to within 6.3% of the corresponding frontier reasoner performance. In a live user study (n=18) with talker deployments running on an Apple M2 SoC, participants rank ConvFill on par with frontier models overall, prefer it for retrieval-heavy tasks, and rate it significantly more responsive. These results show that conversational infill unlocks a new point on the latency-capability Pareto frontier, offering a practical path toward voice agents that are both responsive and highly capable. Code, models, and datasets are available at https://github.com/vysri/conversational-infill.