NoLan：通过动态抑制语言先验缓解大型视觉语言模型中的物体幻觉

摘要

物体幻觉是大规模视觉语言模型（LVLM）中的一个关键问题，表现为输出内容包含输入图像中未出现的物体。这一现象引发出一个核心问题：LVLM流程中的哪个组件是物体幻觉的主要成因？是负责感知视觉信息的视觉编码器，还是生成文本响应的语言解码器？本研究通过设计系统性实验分析视觉编码器与语言解码器在幻觉生成中的作用，试图解答这一问题。我们的观察表明，物体幻觉主要与语言解码器的强先验知识相关。基于此发现，我们提出了一种简单无需训练的框架——无语言幻觉解码（NoLan），该框架通过动态抑制语言先验来优化输出分布，其调节机制基于多模态输入与纯文本输入之间的输出分布差异。实验结果表明，NoLan在不同任务的各种LVLM上均能有效减少物体幻觉。例如在POPE基准测试中，NoLan显著提升了LLaVA-1.5 7B和Qwen-VL 7B模型的准确率，分别实现6.45和7.21的增益。代码已开源：https://github.com/lingfengren/NoLan。

English

Object hallucination is a critical issue in Large Vision-Language Models (LVLMs), where outputs include objects that do not appear in the input image. A natural question arises from this phenomenon: Which component of the LVLM pipeline primarily contributes to object hallucinations? The vision encoder to perceive visual information, or the language decoder to generate text responses? In this work, we strive to answer this question through designing a systematic experiment to analyze the roles of the vision encoder and the language decoder in hallucination generation. Our observations reveal that object hallucinations are predominantly associated with the strong priors from the language decoder. Based on this finding, we propose a simple and training-free framework, No-Language-Hallucination Decoding, NoLan, which refines the output distribution by dynamically suppressing language priors, modulated based on the output distribution difference between multimodal and text-only inputs. Experimental results demonstrate that NoLan effectively reduces object hallucinations across various LVLMs on different tasks. For instance, NoLan achieves substantial improvements on POPE, enhancing the accuracy of LLaVA-1.5 7B and Qwen-VL 7B by up to 6.45 and 7.21, respectively. The code is publicly available at: https://github.com/lingfengren/NoLan.

NoLan：通过动态抑制语言先验缓解大型视觉语言模型中的物体幻觉

NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors

摘要

Support