一眼足矣：基於SimLoss的單遍細粒度圖像描述

摘要

一張影像或許勝過千言萬語，但大多數影像描述生成模型卻只用寥寥數語來描述它。現代的視覺-語言模型能生成流暢的高層次描述，然而經常遺漏那些構成影像視覺具體性的屬性、數量、紋理、材質與空間關係。近期的多階段系統透過生成、分解、驗證與改寫來恢復部分這類細節，但代價是推理延遲大幅提高。我們提出 SimLoss——一種無需參考的嵌入空間目標函數，專用於單次前向的細粒度影像描述生成。SimLoss 透過 InfoNCE 對比損失，訓練視覺-語言模型將其投影後的隱藏狀態表徵與凍結的影像嵌入對齊；如此一來，在解碼任何文字之前便能提供密集的視覺監督信號，既不需要人工撰寫的細粒度描述，也不需要來自多階段管線的偽描述。我們將其實例化為兩種變體：SimLoss FFT 會對本地可用的嵌入模型進行反向傳播，而 SimLoss GRPO 則將該嵌入模型視為黑箱獎勵。與單次前向、多階段驗證、獎勵最佳化及感知導向的基線方法相比，完全可微的微調變體 SimLoss FFT 達到了最高的精確率，同時其 F1 分數幾乎與多階段方法相當；此外，它仍維持單次前向推理，運行速度約比多階段管線快 20 倍。基於獎勵的變體 SimLoss GRPO 則取得了最強的召回率。綜合而言，這些結果表明，嵌入空間監督能以單次前向描述生成器的延遲，達到與多階段驗證相當的品質。

English

An image may be worth a thousand words, but most captioning models describe it in only a few. Modern vision-language models produce fluent high-level captions, yet routinely miss the attributes, counts, textures, materials, and spatial relations that make an image visually specific. Recent multi-stage systems recover some of these details through generation, decomposition, verification, and rewriting, but they do so at the expense of substantially higher inference latency. We propose SimLoss, a reference-free embedding-space objective for single-pass fine-grained image captioning. SimLoss trains a vision-language model to align its projected hidden-state representation with a frozen image embedding through an InfoNCE contrastive loss, supplying a dense visual supervision signal before any text is decoded, and requiring neither human-written fine-grained captions nor pseudo-captions from a multi-stage pipeline. We instantiate it as SimLoss FFT, which backpropagates through a locally available embedding model, and SimLoss GRPO, which treats that model as a black-box reward. Compared with single-pass, multi-stage verification, reward-optimized, and perception-aware baselines, the fully differentiable fine-tuning variant, SimLoss FFT, achieves the highest precision while nearly matching the F1 score of the multi-stage method, all while retaining single-pass inference and running roughly 20 times faster than the multi-stage pipeline. The reward-based variant SimLoss GRPO attains the strongest recall. Together, these results show that embedding-space supervision can recover the quality of multi-stage verification at the latency of a single-pass captioner.