Ventor-QTest：威脅模型驅動的供應商託管LLM API驗證方法

摘要

隨著大型語言模型日益普及，部署開放權重模型的第三方供應商已成為生態系統的重要組成部分。因此，稽核其推論API的品質是一項開放性問題。我們將託管模型路由形式化為隨機過程，並提出\textbf{Ventor-QTest}，一種無需目標API提供任何機率資訊的複合式黑箱稽核方法。其重複請求組件會將每個凍結的受約束語境多次發送至目標，根據返回的文字計數重建類別輸出分布，並以平均忠實度損失（AFL）作為虛無假設偏差校正的窗內平均粗化KL統計量。其長序列組件使用獨立運行，透過運行層級以參考分布為中心的驚訝度統計量的經驗上尾來報告極端忠實度損失（EFL）。在三種具備對數機率能力的路由條件下，AFL與由對數機率推導的粗化KL比較器呈現強烈的線性描述性一致性。在七個路由快照中，20次運行的序列探測揭示了路由特定的EFL變異。AFL與EFL與GPQA-Diamond準確率之間幾乎沒有可偵測的路由層級關聯。相反地，明顯的EFL伴隨著Terminal-Bench通過率在任務暴露度增加時的下降。此模式可能源於長時程任務的正確性對極端忠實度損失更為敏感。這些結果支持在稽核長時程代理型任務時聯合報告AFL與EFL。開源實作可於https://github.com/Tencent/AI-Infra-Guard/tree/main/services/api_checker/ventor_qtest取得。

English

As large language models become increasingly widespread, third-party providers that deploy open-weight models have become an important part of the ecosystem. Auditing the quality of their inference APIs is therefore an open problem. We formalize hosted model routing as a stochastic process and propose \textbf{Ventor-QTest}, a composite black-box audit that requires no probability information from the target API. Its repeated-request component sends each frozen constrained context to the target multiple times, reconstructs a categorical output distribution from the returned text counts, and reports average fidelity loss (AFL) as a null-bias-corrected, within-window mean coarsened-KL statistic. Its long-sequence component uses independent runs to report extreme fidelity loss (EFL) through the empirical upper tail of a run-level reference-centered-surprisal statistic. Across three logprob-capable route conditions, AFL shows strong linear descriptive agreement with a logprob-derived coarsened-KL comparator. Across seven route snapshots, 20-run sequence probes reveal route-specific EFL variation. AFL and EFL have little detectable route-level association with GPQA-Diamond accuracy. In contrast, pronounced EFL coincides with a decline in Terminal-Bench pass rate as task exposure increases. This pattern may arise because correctness in long-horizon tasks is more sensitive to extreme fidelity loss. These results motivate reporting AFL and EFL jointly, particularly when auditing long-horizon agentic tasks. The open-source implementation is available at https://github.com/Tencent/AI-Infra-Guard/tree/main/services/api_checker/ventor_qtest.