3D-GPT：利用大型语言模型进行程序化3D建模

3D-GPT: Procedural 3D Modeling with Large Language Models

October 19, 2023

作者: Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zishan Qin, Stephen Gould

cs.AI

摘要

为了追求高效的自动化内容创作，程序生成成为一种有前途的方法，利用可修改参数和基于规则的系统。然而，考虑到其复杂性，这可能是一项具有挑战性的工作，需要深刻理解规则、算法和参数。为了减轻工作量，我们引入了3D-GPT，这是一个利用大型语言模型（LLMs）进行指令驱动的3D建模的框架。3D-GPT将LLMs定位为熟练的问题解决者，将程序化3D建模任务分解为易访问的部分，并为每个任务指定适当的代理。3D-GPT集成了三个核心代理：任务分派代理、概念化代理和建模代理。它们共同实现了两个目标。首先，它增强了简洁的初始场景描述，将其演变为详细形式，同时根据后续指令动态调整文本。其次，它集成了程序生成，从丰富文本中提取参数值，以便轻松地与3D软件进行资产创建的接口。我们的实证调查证实，3D-GPT不仅解释和执行指令，提供可靠的结果，而且还与人类设计师有效合作。此外，它与Blender无缝集成，拓展了操控可能性。我们的工作突显了LLMs在3D建模中的潜力，为未来在场景生成和动画方面的进展提供了基本框架。

English

In the pursuit of efficient automated content creation, procedural generation, leveraging modifiable parameters and rule-based systems, emerges as a promising approach. Nonetheless, it could be a demanding endeavor, given its intricate nature necessitating a deep understanding of rules, algorithms, and parameters. To reduce workload, we introduce 3D-GPT, a framework utilizing large language models~(LLMs) for instruction-driven 3D modeling. 3D-GPT positions LLMs as proficient problem solvers, dissecting the procedural 3D modeling tasks into accessible segments and appointing the apt agent for each task. 3D-GPT integrates three core agents: the task dispatch agent, the conceptualization agent, and the modeling agent. They collaboratively achieve two objectives. First, it enhances concise initial scene descriptions, evolving them into detailed forms while dynamically adapting the text based on subsequent instructions. Second, it integrates procedural generation, extracting parameter values from enriched text to effortlessly interface with 3D software for asset creation. Our empirical investigations confirm that 3D-GPT not only interprets and executes instructions, delivering reliable results but also collaborates effectively with human designers. Furthermore, it seamlessly integrates with Blender, unlocking expanded manipulation possibilities. Our work highlights the potential of LLMs in 3D modeling, offering a basic framework for future advancements in scene generation and animation.