FirePlace：基于几何优化的LLM常识推理在3D物体摆放中的应用

FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement

March 6, 2025

作者: Ian Huang, Yanan Bao, Karen Truong, Howard Zhou, Cordelia Schmid, Leonidas Guibas, Alireza Fathi

cs.AI

摘要

利用3D资产进行场景生成是一项复杂的挑战，既需要高层次的语义理解，又需低层次的几何推理。尽管多模态大语言模型（MLLMs）在语义任务上表现出色，但其在3D场景生成中的应用却受限于对3D几何的有限理解。本文探讨了如何在物体布局任务中最佳地运用MLLMs。为此，我们提出了一个新颖的框架——FirePlace，该框架将现有MLLMs应用于：(1) 3D几何推理及从3D场景中提取相关几何细节，(2) 构建并解决基于提取的低层次几何的约束条件，以及(3) 筛选出符合常识的最终布局方案。通过将几何推理与MLLMs对现实世界的理解相结合，我们的方法能够提出既满足几何约束又兼顾高层次语义常识考量的物体布局方案。实验结果表明，这些能力使我们的方法在具有复杂几何结构的场景中更有效地布置物体，超越了先前工作的质量。

English

Scene generation with 3D assets presents a complex challenge, requiring both high-level semantic understanding and low-level geometric reasoning. While Multimodal Large Language Models (MLLMs) excel at semantic tasks, their application to 3D scene generation is hindered by their limited grounding on 3D geometry. In this paper, we investigate how to best work with MLLMs in an object placement task. Towards this goal, we introduce a novel framework, FirePlace, that applies existing MLLMs in (1) 3D geometric reasoning and the extraction of relevant geometric details from the 3D scene, (2) constructing and solving geometric constraints on the extracted low-level geometry, and (3) pruning for final placements that conform to common sense. By combining geometric reasoning with real-world understanding of MLLMs, our method can propose object placements that satisfy both geometric constraints as well as high-level semantic common-sense considerations. Our experiments show that these capabilities allow our method to place objects more effectively in complex scenes with intricate geometry, surpassing the quality of prior work.