TRACER：基于追踪的大型语言模型分类自适应成本优化路由

摘要

每次调用大语言模型分类终端时，都会生成已保存在生产日志中的带标签输入-输出对。这些数据对构成了一个免费且持续扩增的训练集：基于此训练的轻量级替代模型能以近乎零边际推理成本吸收未来大量流量。核心问题在于何时部署具备足够可靠性的替代模型、其处理与转交任务的边界如何划分，以及该边界如何随数据积累动态演化。我们推出TRACER（基于追踪的自适应成本效益路由系统），这一开源系统利用大语言模型自身生产轨迹训练机器学习替代模型，并通过一致性校验门控部署机制：仅当替代模型与LLM的共识率超过用户设定阈值α时才激活运行。为实现路由边界的透明化，TRACER生成可解释性分析报告，清晰阐述替代模型处理的输入范围、性能瓶颈区域及转交决策依据。在77类意图识别基准测试中（以Sonnet 4.6作为教师模型），TRACER实现的替代模型覆盖率可达83-100%（具体取决于质量目标α）；在150类基准测试中，替代模型可完全取代教师模型。对于自然语言推理任务，系统通过一致性校验门控正确拒绝部署，因为嵌入表示无法支持可靠的分类决策。本系统已作为开源软件发布。

English

Every call to an LLM classification endpoint produces a labeled input-output pair already retained in production logs. These pairs constitute a free, growing training set: a lightweight surrogate trained on them can absorb a significant portion of future traffic at near-zero marginal inference cost. The open questions are when the surrogate is reliable enough to deploy, what it handles versus defers, and how that boundary evolves as data accumulates. We introduce TRACER (Trace-based Adaptive Cost-Efficient Routing), an open-source system that trains ML surrogates on an LLM's own production traces and governs deployment through a parity gate: the surrogate is activated only when its agreement with the LLM exceeds a user-specified threshold α. To make the routing boundary transparent, TRACER generates interpretability artifacts describing which input regions the surrogate handles, where it plateaus, and why it defers. On a 77-class intent benchmark with a Sonnet 4.6 teacher, TRACER achieves 83-100% surrogate coverage depending on the quality target α; on a 150-class benchmark, the surrogate fully replaces the teacher. On a natural language inference task, the parity gate correctly refuses deployment because the embedding representation cannot support reliable separation. The system is available as open-source software.

TRACER：基于追踪的大型语言模型分类自适应成本优化路由

TRACER: Trace-Based Adaptive Cost-Efficient Routing for LLM Classification

摘要

Support