ConceptGraphs：用於感知和規劃的開放詞彙3D場景圖

摘要

為了讓機器人能夠執行各種任務，它們需要一個在語義上豐富、同時又緊湊高效，以供任務驅動的感知和規劃之用的世界3D表示。最近的方法試圖利用來自大型視覺語言模型的特徵來編碼3D表示中的語義。然而，這些方法往往會產生具有每點特徵向量的地圖，在較大環境中無法良好擴展，也不包含環境中實體之間的語義空間關係，這對於下游規劃是有用的。在這項工作中，我們提出了ConceptGraphs，這是一種用於3D場景的開放詞彙圖結構表示。ConceptGraphs是通過利用2D基礎模型並通過多視圖關聯將它們的輸出融合到3D中而構建的。結果表示能夠泛化到新的語義類別，而無需收集大量3D數據集或微調模型。我們通過一些通過抽象（語言）提示指定並需要對空間和語義概念進行複雜推理的下游規劃任務來展示此表示的效用。（項目頁面：https://concept-graphs.github.io/ 解說視頻：https://youtu.be/mRhNkQwRYnc）

English

For robots to perform a wide variety of tasks, they require a 3D representation of the world that is semantically rich, yet compact and efficient for task-driven perception and planning. Recent approaches have attempted to leverage features from large vision-language models to encode semantics in 3D representations. However, these approaches tend to produce maps with per-point feature vectors, which do not scale well in larger environments, nor do they contain semantic spatial relationships between entities in the environment, which are useful for downstream planning. In this work, we propose ConceptGraphs, an open-vocabulary graph-structured representation for 3D scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. The resulting representations generalize to novel semantic classes, without the need to collect large 3D datasets or finetune models. We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer video: https://youtu.be/mRhNkQwRYnc )

ConceptGraphs：用於感知和規劃的開放詞彙3D場景圖

ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning

摘要

Support