FirePlace：基於幾何精煉的LLM常識推理於3D物件擺放

FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement

March 6, 2025

作者: Ian Huang, Yanan Bao, Karen Truong, Howard Zhou, Cordelia Schmid, Leonidas Guibas, Alireza Fathi

cs.AI

摘要

利用3D資產進行場景生成面臨著複雜的挑戰，這既需要高層次的語義理解，也需具備低層次的幾何推理能力。雖然多模態大型語言模型（MLLMs）在語義任務上表現卓越，但其在3D場景生成中的應用卻因對3D幾何基礎的有限掌握而受限。本文探討了如何在物件放置任務中最佳地運用MLLMs。為此，我們提出了一個新穎的框架——FirePlace，該框架將現有的MLLMs應用於：(1) 3D幾何推理及從3D場景中提取相關幾何細節，(2) 基於提取的低層次幾何構建並解決幾何約束，(3) 篩選出符合常識的最終放置方案。通過將幾何推理與MLLMs對現實世界的理解相結合，我們的方法能夠提出既滿足幾何約束又符合高層次語義常識考量的物件放置方案。實驗結果表明，這些能力使我們的方法在具有複雜幾何結構的場景中更有效地放置物件，超越了先前工作的質量。

English

Scene generation with 3D assets presents a complex challenge, requiring both high-level semantic understanding and low-level geometric reasoning. While Multimodal Large Language Models (MLLMs) excel at semantic tasks, their application to 3D scene generation is hindered by their limited grounding on 3D geometry. In this paper, we investigate how to best work with MLLMs in an object placement task. Towards this goal, we introduce a novel framework, FirePlace, that applies existing MLLMs in (1) 3D geometric reasoning and the extraction of relevant geometric details from the 3D scene, (2) constructing and solving geometric constraints on the extracted low-level geometry, and (3) pruning for final placements that conform to common sense. By combining geometric reasoning with real-world understanding of MLLMs, our method can propose object placements that satisfy both geometric constraints as well as high-level semantic common-sense considerations. Our experiments show that these capabilities allow our method to place objects more effectively in complex scenes with intricate geometry, surpassing the quality of prior work.