CoIn：商用不透明大語言模型API中隱性推理代幣的計數

CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM APIs

May 19, 2025

作者: Guoheng Sun, Ziyao Wang, Bowei Tian, Meng Liu, Zheyu Shen, Shwai He, Yexiao He, Wanghao Ye, Yiting Wang, Ang Li

cs.AI

摘要

隨著後訓練技術的發展，大型語言模型（LLMs）正日益增強其結構化的多步推理能力，這些能力通常通過強化學習進行優化。這些增強了推理能力的模型在複雜任務上超越了標準的LLMs，並已成為許多商業LLM API的基礎。然而，為了保護專有行為並減少冗餘，提供商通常隱藏推理過程，僅返回最終答案。這種不透明性引入了一個關鍵的透明度缺口：用戶為不可見的推理令牌付費，這些令牌往往佔據成本的大部分，卻無法驗證其真實性。這為令牌計數膨脹打開了大門，提供商可能過度報告令牌使用量或注入合成的、低成本的令牌以增加收費。為解決這一問題，我們提出了CoIn，一個驗證框架，用於審計隱藏令牌的數量和語義有效性。CoIn通過從令牌嵌入指紋構建可驗證的哈希樹來檢查令牌計數，並使用基於嵌入的相關性匹配來檢測偽造的推理內容。實驗表明，CoIn作為受信任的第三方審計者部署時，能夠有效檢測令牌計數膨脹，成功率最高可達94.7%，顯示出在恢復不透明LLM服務計費透明度方面的強大能力。數據集和代碼可在https://github.com/CASE-Lab-UMD/LLM-Auditing-CoIn獲取。

English

As post-training techniques evolve, large language models (LLMs) are increasingly augmented with structured multi-step reasoning abilities, often optimized through reinforcement learning. These reasoning-enhanced models outperform standard LLMs on complex tasks and now underpin many commercial LLM APIs. However, to protect proprietary behavior and reduce verbosity, providers typically conceal the reasoning traces while returning only the final answer. This opacity introduces a critical transparency gap: users are billed for invisible reasoning tokens, which often account for the majority of the cost, yet have no means to verify their authenticity. This opens the door to token count inflation, where providers may overreport token usage or inject synthetic, low-effort tokens to inflate charges. To address this issue, we propose CoIn, a verification framework that audits both the quantity and semantic validity of hidden tokens. CoIn constructs a verifiable hash tree from token embedding fingerprints to check token counts, and uses embedding-based relevance matching to detect fabricated reasoning content. Experiments demonstrate that CoIn, when deployed as a trusted third-party auditor, can effectively detect token count inflation with a success rate reaching up to 94.7%, showing the strong ability to restore billing transparency in opaque LLM services. The dataset and code are available at https://github.com/CASE-Lab-UMD/LLM-Auditing-CoIn.