UniProbe：一種適用於大型視覺語言模型、基於多結構內部表徵之可學習詞元層級幻覺偵測器

摘要

大型視覺語言模型（LVLMs）展現出卓越的視覺推理與對話能力，然而經常產生缺乏視覺輸入支持的幻覺內容。有效的緩解需要詞元級定位，才能在不丟棄整個回應的情況下進行針對性干預。現有的偵測器需要昂貴的全模型微調、依賴忽略模型生成過程的外部驗證器，或將內部訊號簡化為孤立特徵與手工統計量，因而丟棄了空間、序列與關係結構。我們提出 UniProbe，一種輕量級、統一且可學習的偵測器，能從單次前向傳播中建模凍結 LVLM 的異質計算軌跡。UniProbe 在影像區塊、查詢詞元與生成詞元上建構有向圖，以注意力權重編碼其關係。它使用交替的結構感知模組處理此軌跡：GNN 處理關係證據、ViT 處理二維視覺幾何、GRU 處理回應順序。交織這些模組使空間、關係與序列證據能在整個偵測器中互動。我們進一步開發串流變體以進行幻覺感知解碼，在生成期間偵測並重新取樣幻覺詞元，並提出自我適應策略，使偵測器與 LVLM 自身生成內容對齊。在各種 LVLM 骨幹上，UniProbe 達到詞元級與物件幻覺偵測的最新技術水準。在解碼期間，它以標準生成 1.06 倍的延遲，將物件幻覺減少高達 55%。

English

Large Vision-Language Models (LVLMs) achieve impressive visual reasoning and dialogue capabilities, yet frequently hallucinate content unsupported by the visual input. Effective mitigation requires token-level localization, enabling targeted intervention without discarding the entire response. Existing detectors require expensive full-model fine-tuning, rely on external verifiers that ignore the model's generation process, or reduce internal signals to isolated features and hand-crafted statistics, discarding spatial, sequential, and relational structure. We introduce UniProbe, a lightweight, unified, learnable detector that models a frozen LVLM's heterogeneous computational trace from a single forward pass. UniProbe constructs a directed graph over image patches, query tokens, and generated tokens, with attention weights encoding their relations. It processes this trace with alternating structure-aware modules: a GNN for relational evidence, a ViT for 2-D visual geometry, and a GRU for response order. Interleaving them allows spatial, relational, and sequential evidence to interact throughout the detector. We further develop a streaming variant for hallucination-aware decoding, which detects and resamples hallucinated tokens during generation, and a self-adaptation strategy aligning the detector with the LVLM's own generations. Across diverse LVLM backbones, UniProbe achieves state-of-the-art token-level and object-hallucination detection. During decoding, it reduces object hallucinations by up to 55\% at 1.06times the latency of standard generation.