關於低層次視覺任務中語言引導的魯棒性：深度估計的研究結果

On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation

April 12, 2024

作者: Agneet Chatterjee, Tejas Gokhale, Chitta Baral, Yezhou Yang

cs.AI

摘要

最近在單眼深度估計方面取得了進展，透過將自然語言作為額外指導來提高效果。儘管取得了令人印象深刻的結果，但語言先驗的影響，特別是在泛化和魯棒性方面，仍未被探討。本文通過量化這種先驗的影響，並引入方法來評估其在各種情況下的效果，來填補這一空白。我們生成傳達以物為中心、三維空間關係的“低級”句子，將它們作為額外的語言先驗，並評估它們對深度估計的影響。我們的主要發現是，當前的語言引導深度估計器只有在場景級描述時才能表現最佳，而在低級描述時表現出乎意料地更差。儘管利用了額外數據，這些方法對定向對抗攻擊不具魯棒性，並且隨著分布轉變的增加而性能下降。最後，為了為未來研究奠定基礎，我們確定了失敗點並提供見解，以更好地理解這些缺陷。隨著越來越多的方法在深度估計中使用語言，我們的研究發現突顯了機遇和陷阱，需要在實際應用中仔細考慮。

English

Recent advances in monocular depth estimation have been made by incorporating natural language as additional guidance. Although yielding impressive results, the impact of the language prior, particularly in terms of generalization and robustness, remains unexplored. In this paper, we address this gap by quantifying the impact of this prior and introduce methods to benchmark its effectiveness across various settings. We generate "low-level" sentences that convey object-centric, three-dimensional spatial relationships, incorporate them as additional language priors and evaluate their downstream impact on depth estimation. Our key finding is that current language-guided depth estimators perform optimally only with scene-level descriptions and counter-intuitively fare worse with low level descriptions. Despite leveraging additional data, these methods are not robust to directed adversarial attacks and decline in performance with an increase in distribution shift. Finally, to provide a foundation for future research, we identify points of failures and offer insights to better understand these shortcomings. With an increasing number of methods using language for depth estimation, our findings highlight the opportunities and pitfalls that require careful consideration for effective deployment in real-world settings