逐一对齐文本、图像与三维结构表征

摘要

构建能够理解三维世界的机器，对于辅助设计师创建和编辑三维环境，以及帮助机器人在三维空间中导航和交互至关重要。受语言和图像建模进展的启发，我们探索了自回归模型在一种新模态——结构化三维场景中的潜力。为此，我们提出了一个统一的LLM框架，该框架对齐了语言、图像和三维场景，并提供了一个详细的“操作手册”，阐述了实现最佳训练和性能的关键设计选择，涵盖了数据表示、模态特定目标等核心问题。我们在四个核心三维任务——渲染、识别、指令跟随和问答——以及四个合成与现实世界三维数据集上评估了性能。通过引入量化形状编码来丰富我们的三维模态，我们进一步扩展了方法以重建复杂的三维物体形状，并在现实世界的三维物体识别任务中展示了模型的有效性。项目网页：https://glab-caltech.github.io/kyvo/

English

Creating machines capable of understanding the world in 3D is essential in assisting designers that build and edit 3D environments and robots navigating and interacting within a three-dimensional space. Inspired by advances in language and image modeling, we investigate the potential of autoregressive models for a new modality: structured 3D scenes. To this end, we propose a unified LLM framework that aligns language, images, and 3D scenes and provide a detailed ''cookbook'' outlining critical design choices for achieving optimal training and performance addressing key questions related to data representation, modality-specific objectives, and more. We evaluate performance across four core 3D tasks -- rendering, recognition, instruction-following, and question-answering -- and four 3D datasets, synthetic and real-world. We extend our approach to reconstruct complex 3D object shapes by enriching our 3D modality with quantized shape encodings, and show our model's effectiveness on real-world 3D object recognition tasks. Project webpage: https://glab-caltech.github.io/kyvo/

逐一对齐文本、图像与三维结构表征

Aligning Text, Images, and 3D Structure Token-by-Token

摘要

Support