L3GO：具有三维思维链的语言代理，用于生成非常规对象

L3GO: Language Agents with Chain-of-3D-Thoughts for Generating Unconventional Objects

February 14, 2024

作者: Yutaro Yamada, Khyathi Chandu, Yuchen Lin, Jack Hessel, Ilker Yildirim, Yejin Choi

cs.AI

摘要

基于扩散的图像生成模型，如DALL-E 3和Stable Diffusion-XL，展现了在生成具有逼真和独特构图的图像方面的显著能力。然而，这些模型在精确推理物体的物理和空间配置方面并不稳健，特别是在接收到非常规、即属于分布之外的描述时，比如“一把有五条腿的椅子”。本文提出了一种具有三维思维链（L3GO）的语言代理，在推理时可以处理当前基于数据驱动的扩散模型难以应对的非常规对象的基于部件的三维网格生成。更具体地说，我们利用大型语言模型作为代理，在三维模拟环境中通过试错来组合所需的对象。为了促进我们的研究，我们开发了一个新的基准测试，名为非常规可行对象（UFO），以及SimpleBlenv，这是建立在Blender之上的一个包装环境，语言代理可以通过API调用构建和组合原子建筑块。人工和自动GPT-4V评估表明，我们的方法在ShapeNet上的三维网格生成方面超越了标准GPT-4和其他语言代理（例如ReAct和Reflexion）。此外，当在我们的UFO基准测试上进行测试时，我们的方法在人类评估中胜过了其他基于文本到二维图像和文本到三维模型的最新技术。

English

Diffusion-based image generation models such as DALL-E 3 and Stable Diffusion-XL demonstrate remarkable capabilities in generating images with realistic and unique compositions. Yet, these models are not robust in precisely reasoning about physical and spatial configurations of objects, especially when instructed with unconventional, thereby out-of-distribution descriptions, such as "a chair with five legs". In this paper, we propose a language agent with chain-of-3D-thoughts (L3GO), an inference-time approach that can reason about part-based 3D mesh generation of unconventional objects that current data-driven diffusion models struggle with. More concretely, we use large language models as agents to compose a desired object via trial-and-error within the 3D simulation environment. To facilitate our investigation, we develop a new benchmark, Unconventionally Feasible Objects (UFO), as well as SimpleBlenv, a wrapper environment built on top of Blender where language agents can build and compose atomic building blocks via API calls. Human and automatic GPT-4V evaluations show that our approach surpasses the standard GPT-4 and other language agents (e.g., ReAct and Reflexion) for 3D mesh generation on ShapeNet. Moreover, when tested on our UFO benchmark, our approach outperforms other state-of-the-art text-to-2D image and text-to-3D models based on human evaluation.