邊說邊想：推論階段知識轉移以實現反應靈敏且智能的對話式語音代理

摘要

语音代理面临一个根本性矛盾：使基础模型具备推理、检索和工具使用能力的迭代过程缓慢，而对话交互却要求毫秒级的响应时间。小型实时模型能满足低延迟要求，但在复杂任务上无法媲美基础模型，导致现有语音代理不得不在响应速度与能力之间取舍。我们提出对话填充（conversational infill）方法，通过一个小型对话模型在即时生成上下文相关的响应以隐藏外部推理模型的延迟，同时能在推理过程中将流式传输的推理结果知识无缝整合到其输出中。我们构建了一个涵盖六个领域、包含290,571个样本的合成数据集，并在七个广泛使用的小型语言模型（参数量从1.35亿到17亿）上验证了该任务的可学习性。我们的系统实现ConvFill，在保持毫秒级首响时间的同时，将准确率差距缩小至对应前沿推理模型性能的6.3%以内。在基于Apple M2芯片运行对话模型的实际用户研究（n=18）中，参与者认为ConvFill整体表现与前沿模型相当，在检索密集型任务中更倾向选择它，并认为其响应速度显著更快。这些结果表明，对话填充在延迟-能力帕累托前沿上开辟了新节点，为构建兼具快速响应与高能力的语音代理提供了可行路径。代码、模型和数据集已开源：https://github.com/vysri/conversational-infill。

English

Voice agents face a fundamental tension: the reasoning, retrieval, and tool use that make foundation models capable are iterative and slow, while conversational interaction demands responses on a millisecond timescale. Smaller, real-time models meet the latency bar but cannot match foundation models on complex tasks, leaving current voice agents to trade away either responsiveness or capability. We introduce conversational infill, where a small talker model both immediately generates contextually grounded responses to hide the latency of an external reasoner model and fluently integrates streamed reasoner knowledge into its responses during inference. We curate a 290,571-example synthetic dataset spanning six domains and demonstrate that this task is learnable across seven widely used small language models ranging from 135M to 1.7B parameters. Our system implementation, ConvFill, sustains millisecond-level time-to-first-response while closing the accuracy gap to within 6.3% of the corresponding frontier reasoner performance. In a live user study (n=18) with talker deployments running on an Apple M2 SoC, participants rank ConvFill on par with frontier models overall, prefer it for retrieval-heavy tasks, and rate it significantly more responsive. These results show that conversational infill unlocks a new point on the latency-capability Pareto frontier, offering a practical path toward voice agents that are both responsive and highly capable. Code, models, and datasets are available at https://github.com/vysri/conversational-infill.