UniProbe：一种利用多结构内部表示的大型视觉语言模型可学习词元级幻觉检测器

摘要

大型视觉-语言模型（LVLMs）在视觉推理和对话能力上表现出色，但常常产生缺乏视觉输入支持的幻觉内容。要有效缓解这一问题，需要进行词元级定位，以便在不丢弃整个响应的情况下进行针对性干预。现有检测器要么需要昂贵的全模型微调，要么依赖忽略模型生成过程的外部验证器，要么将内部信号简化为孤立的特征和手工设计的统计量，从而丢弃了空间、序列和关系结构。我们提出UniProbe，一种轻量级、统一的可学习检测器，它从单次前向传播中对冻结LVLM的异构计算轨迹进行建模。UniProbe在图像块、查询词元和生成词元之间构建有向图，并用注意力权重编码它们之间的关系。它通过交替的结构感知模块处理该轨迹：图神经网络用于关系证据，视觉Transformer用于二维视觉几何，门控循环单元用于响应顺序。交错这些模块使得空间、关系和序列证据能够在整个检测器中相互作用。我们进一步开发了一种面向幻觉感知解码的流式变体，该变体在生成过程中检测并重新采样幻觉词元；同时，我们还提出了一种自适应策略，使检测器与LVLM自身的生成结果对齐。在多种LVLM骨干网络上，UniProbe在词元级检测和物体幻觉检测方面达到了最先进的性能。在解码过程中，它将物体幻觉减少高达55%，而延迟仅为标准生成的1.06倍。

English

Large Vision-Language Models (LVLMs) achieve impressive visual reasoning and dialogue capabilities, yet frequently hallucinate content unsupported by the visual input. Effective mitigation requires token-level localization, enabling targeted intervention without discarding the entire response. Existing detectors require expensive full-model fine-tuning, rely on external verifiers that ignore the model's generation process, or reduce internal signals to isolated features and hand-crafted statistics, discarding spatial, sequential, and relational structure. We introduce UniProbe, a lightweight, unified, learnable detector that models a frozen LVLM's heterogeneous computational trace from a single forward pass. UniProbe constructs a directed graph over image patches, query tokens, and generated tokens, with attention weights encoding their relations. It processes this trace with alternating structure-aware modules: a GNN for relational evidence, a ViT for 2-D visual geometry, and a GRU for response order. Interleaving them allows spatial, relational, and sequential evidence to interact throughout the detector. We further develop a streaming variant for hallucination-aware decoding, which detects and resamples hallucinated tokens during generation, and a self-adaptation strategy aligning the detector with the LVLM's own generations. Across diverse LVLM backbones, UniProbe achieves state-of-the-art token-level and object-hallucination detection. During decoding, it reduces object hallucinations by up to 55\% at 1.06times the latency of standard generation.