逃離柏拉圖洞穴：邁向3D與文本潛在空間的對齊

摘要

近期研究表明，當進行大規模訓練時，單模態的二維視覺與文本編碼器所學習到的特徵，儘管源自不同的表示方式，卻展現出顯著的結構相似性。然而，三維編碼器相對於其他模態的角色仍未被探索。此外，現有的利用大數據集的三維基礎模型，通常通過與來自其他表示的凍結編碼器進行顯式對齊目標來訓練。在本研究中，我們探討了單模態三維編碼器與基於文本的特徵空間之間進行後驗對齊的可能性。我們發現，對單模態文本和三維編碼器進行簡單的訓練後特徵對齊，其性能有限。隨後，我們專注於提取相應特徵空間的子空間，並發現通過將學習到的表示投影到精心選擇的低維子空間上，對齊質量顯著提高，從而提升了匹配和檢索任務的準確性。我們的分析進一步揭示了這些共享子空間的本質，它們大致區分了語義與幾何數據表示。總的來說，我們的工作首次為三維單模態與文本特徵空間的訓練後對齊建立了基準，並有助於凸顯三維數據相較於其他表示的共享與獨特屬性。

English

Recent works have shown that, when trained at scale, uni-modal 2D vision and text encoders converge to learned features that share remarkable structural properties, despite arising from different representations. However, the role of 3D encoders with respect to other modalities remains unexplored. Furthermore, existing 3D foundation models that leverage large datasets are typically trained with explicit alignment objectives with respect to frozen encoders from other representations. In this work, we investigate the possibility of a posteriori alignment of representations obtained from uni-modal 3D encoders compared to text-based feature spaces. We show that naive post-training feature alignment of uni-modal text and 3D encoders results in limited performance. We then focus on extracting subspaces of the corresponding feature spaces and discover that by projecting learned representations onto well-chosen lower-dimensional subspaces the quality of alignment becomes significantly higher, leading to improved accuracy on matching and retrieval tasks. Our analysis further sheds light on the nature of these shared subspaces, which roughly separate between semantic and geometric data representations. Overall, ours is the first work that helps to establish a baseline for post-training alignment of 3D uni-modal and text feature spaces, and helps to highlight both the shared and unique properties of 3D data compared to other representations.

逃離柏拉圖洞穴：邁向3D與文本潛在空間的對齊

Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent Spaces

摘要

Summary

Support

Support