CoIn：商用不透明大语言模型API中不可见推理令牌的计数

CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM APIs

May 19, 2025

作者: Guoheng Sun, Ziyao Wang, Bowei Tian, Meng Liu, Zheyu Shen, Shwai He, Yexiao He, Wanghao Ye, Yiting Wang, Ang Li

cs.AI

摘要

随着后训练技术的不断演进，大型语言模型（LLMs）正日益增强其结构化多步推理能力，这一过程常通过强化学习进行优化。这些具备增强推理能力的模型在复杂任务上超越了标准LLMs，并已成为众多商用LLM API的核心支撑。然而，为了保护专有行为并减少冗余输出，服务提供商通常隐藏推理过程，仅返回最终答案。这种不透明性引入了一个关键的透明度缺口：用户为不可见的推理令牌付费，这些令牌往往占据成本的大头，却无法验证其真实性。这为令牌计数膨胀打开了方便之门，即服务提供商可能虚报令牌使用量或注入低质量合成令牌以抬高费用。针对这一问题，我们提出了CoIn，一个验证框架，用于审计隐藏令牌的数量及语义有效性。CoIn通过构建基于令牌嵌入指纹的可验证哈希树来检查令牌计数，并利用基于嵌入的相关性匹配来检测伪造的推理内容。实验表明，CoIn作为可信第三方审计工具部署时，能有效检测令牌计数膨胀，成功率高达94.7%，展现了其在恢复不透明LLM服务计费透明度方面的强大能力。数据集与代码已公开于https://github.com/CASE-Lab-UMD/LLM-Auditing-CoIn。

English

As post-training techniques evolve, large language models (LLMs) are increasingly augmented with structured multi-step reasoning abilities, often optimized through reinforcement learning. These reasoning-enhanced models outperform standard LLMs on complex tasks and now underpin many commercial LLM APIs. However, to protect proprietary behavior and reduce verbosity, providers typically conceal the reasoning traces while returning only the final answer. This opacity introduces a critical transparency gap: users are billed for invisible reasoning tokens, which often account for the majority of the cost, yet have no means to verify their authenticity. This opens the door to token count inflation, where providers may overreport token usage or inject synthetic, low-effort tokens to inflate charges. To address this issue, we propose CoIn, a verification framework that audits both the quantity and semantic validity of hidden tokens. CoIn constructs a verifiable hash tree from token embedding fingerprints to check token counts, and uses embedding-based relevance matching to detect fabricated reasoning content. Experiments demonstrate that CoIn, when deployed as a trusted third-party auditor, can effectively detect token count inflation with a success rate reaching up to 94.7%, showing the strong ability to restore billing transparency in opaque LLM services. The dataset and code are available at https://github.com/CASE-Lab-UMD/LLM-Auditing-CoIn.