关于语言引导在低层视觉任务中的鲁棒性： 深度估计的研究结果

On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation

April 12, 2024

作者: Agneet Chatterjee, Tejas Gokhale, Chitta Baral, Yezhou Yang

cs.AI

摘要

最近在单目深度估计领域取得了进展，通过将自然语言作为额外指导进行。尽管取得了令人印象深刻的结果，但语言先验的影响，特别是在泛化和鲁棒性方面，仍未被探索。本文通过量化这一先验的影响，并引入方法来评估其在各种情境下的有效性，填补了这一空白。我们生成了传达以物体为中心的三维空间关系的“低级”句子，将其作为额外的语言先验，并评估其对深度估计的下游影响。我们的关键发现是，当前的语言引导深度估计器只有在场景级描述的情况下才能表现最佳，而在低级描述下表现出乎意料的更差。尽管利用了额外数据，这些方法对有针对性的对抗攻击不具有鲁棒性，并且随着分布偏移的增加而性能下降。最后，为了为未来研究奠定基础，我们确定了失败点，并提供了更好理解这些缺陷的见解。随着越来越多的方法在深度估计中使用语言，我们的研究结果突显了需要在实际环境中有效部署时认真考虑的机遇和风险。

English

Recent advances in monocular depth estimation have been made by incorporating natural language as additional guidance. Although yielding impressive results, the impact of the language prior, particularly in terms of generalization and robustness, remains unexplored. In this paper, we address this gap by quantifying the impact of this prior and introduce methods to benchmark its effectiveness across various settings. We generate "low-level" sentences that convey object-centric, three-dimensional spatial relationships, incorporate them as additional language priors and evaluate their downstream impact on depth estimation. Our key finding is that current language-guided depth estimators perform optimally only with scene-level descriptions and counter-intuitively fare worse with low level descriptions. Despite leveraging additional data, these methods are not robust to directed adversarial attacks and decline in performance with an increase in distribution shift. Finally, to provide a foundation for future research, we identify points of failures and offer insights to better understand these shortcomings. With an increasing number of methods using language for depth estimation, our findings highlight the opportunities and pitfalls that require careful consideration for effective deployment in real-world settings