一瞥足矣：基于SimLoss的单遍细粒度图像描述

摘要

一幅图可能胜过千言万语，但大多数图像描述模型仅用寥寥数语来描述它。现代视觉语言模型能够生成流畅的高层次描述，却经常遗漏那些使图像在视觉上具备具体性的属性、数量、纹理、材质和空间关系。近期多阶段系统通过生成、分解、验证和重写等步骤恢复部分此类细节，但代价是推理延迟大幅增加。我们提出SimLoss，一种用于单遍细粒度图像描述的无参考嵌入空间目标函数。SimLoss通过InfoNCE对比损失训练视觉语言模型，使其投影后的隐状态表示与冻结的图像嵌入对齐，从而在解码任何文本之前提供密集的视觉监督信号，且既不需要人工撰写的细粒度描述，也不需来自多阶段流程的伪描述。我们将其实例化为SimLoss FFT（通过本地可用的嵌入模型进行反向传播）和SimLoss GRPO（将该模型视为黑盒奖励）。与单遍、多阶段验证、奖励优化及感知感知基线相比，全可微微调变体SimLoss FFT取得了最高的精确率，同时F1分数几乎与多阶段方法持平，并且保持单遍推理，运行速度约为多阶段流程的20倍。基于奖励的变体SimLoss GRPO则获得了最强的召回率。综合来看，这些结果表明，嵌入空间监督能够在单遍描述器的延迟水平下，恢复多阶段验证的质量。

English

An image may be worth a thousand words, but most captioning models describe it in only a few. Modern vision-language models produce fluent high-level captions, yet routinely miss the attributes, counts, textures, materials, and spatial relations that make an image visually specific. Recent multi-stage systems recover some of these details through generation, decomposition, verification, and rewriting, but they do so at the expense of substantially higher inference latency. We propose SimLoss, a reference-free embedding-space objective for single-pass fine-grained image captioning. SimLoss trains a vision-language model to align its projected hidden-state representation with a frozen image embedding through an InfoNCE contrastive loss, supplying a dense visual supervision signal before any text is decoded, and requiring neither human-written fine-grained captions nor pseudo-captions from a multi-stage pipeline. We instantiate it as SimLoss FFT, which backpropagates through a locally available embedding model, and SimLoss GRPO, which treats that model as a black-box reward. Compared with single-pass, multi-stage verification, reward-optimized, and perception-aware baselines, the fully differentiable fine-tuning variant, SimLoss FFT, achieves the highest precision while nearly matching the F1 score of the multi-stage method, all while retaining single-pass inference and running roughly 20 times faster than the multi-stage pipeline. The reward-based variant SimLoss GRPO attains the strongest recall. Together, these results show that embedding-space supervision can recover the quality of multi-stage verification at the latency of a single-pass captioner.