Marigold V2：重新審視擴散 Transformer 於單目深度估計

摘要

單目深度估計是一項無所不在卻高度不適定的電腦視覺任務，其下游應用包括場景重建、計算攝影與機器人學等。儘管該領域已日趨成熟，近期模型仍難以泛化到分佈外輸入，也難以產生銳利且細節豐富的深度圖。在本論文中，我們重新審視 Marigold，一套將由擴散 Transformer（DiT）架構驅動的現代影像生成與編輯模型，改造為最先進單目深度估計器的技術。我們的方法旨在對預訓練的多步流匹配模型進行單步推論，並視需要量化，在保持模型容量之餘仍維持低廉的執行成本。我們分析樸素訓練的偽影，並找出兩項有效補救措施：將模型的內部表徵與從真實標註中擷取的語意特徵對齊，以及採用圍繞新穎 Sinkhorn 損失所建構的兩階段微調流程。結果是更銳利、更乾淨的深度圖，能良好泛化到分佈外，並在 KITTI 與 ETH3D 上，AbsRel 相較先前最佳成果提升 16–26%。在質性上，我們的模型能解析先前模型難以捕捉的毛皮、枝葉與髮絲般纖細的邊緣。此外，Marigold V2 應用於其他密集回歸任務時，例如表面法線估計與本質影像分解，達到最先進的結果。專案網站：https://hf.co/spaces/huawei-bayerlab/marigold-v2-web

English

Monocular depth estimation is a ubiquitous yet highly ill-posed computer vision task, with downstream applications in scene reconstruction, computational photography, and robotics, among others. Despite the field's maturity, recent models still struggle to generalize to out-of-distribution inputs and to produce sharp and detailed depth maps. In this paper, we revisit Marigold, a set of techniques for repurposing modern image generation and editing models, powered by the diffusion transformer (DiT) architecture, into state-of-the-art monocular depth estimators. Our recipes target single-step inference from pretrained multi-step flow-matching models, with quantization where needed, preserving model capacity while remaining cheap to run. We analyze the artifacts of naive training and identify two effective remedies: aligning the model's internal representations with semantic features extracted from ground-truth, and adopting a 2-stage fine-tuning protocol built around a novel Sinkhorn-based loss. The results are crisper, cleaner depth maps that generalize well out-of-distribution, with 16-26% improvement in AbsRel over the previous best on KITTI and ETH3D. Qualitatively, our model resolves fur, foliage, and hair-thin edges that have eluded prior models. Furthermore, Marigold V2 achieves state-of-the-art results when applied to other dense regression tasks, such as surface normals estimation and intrinsic image decomposition. Project website: https://hf.co/spaces/huawei-bayerlab/marigold-v2-web