L3GO：具有三維思維鏈的語言代理，用於生成非傳統物體

L3GO: Language Agents with Chain-of-3D-Thoughts for Generating Unconventional Objects

February 14, 2024

作者: Yutaro Yamada, Khyathi Chandu, Yuchen Lin, Jack Hessel, Ilker Yildirim, Yejin Choi

cs.AI

摘要

基於擴散的圖像生成模型，如DALL-E 3和Stable Diffusion-XL，在生成具有逼真且獨特構圖的圖像方面展現出卓越能力。然而，這些模型在精確推理物體的物理和空間配置方面並不堅固，特別是在接收到非傳統、即屬於分布之外的描述時，例如“一張有五條腿的椅子”。本文提出了一種具有三維思維鏈（L3GO）的語言代理，在推理時能夠處理當前基於數據驅動的擴散模型難以應對的非傳統物體的基於部件的三維網格生成。更具體地說，我們使用大型語言模型作為代理，在3D模擬環境中通過試錯來構建所需物體。為了促進我們的研究，我們開發了一個新的基準測試，名為Unconventionally Feasible Objects（UFO），以及SimpleBlenv，這是建立在Blender之上的包裝環境，語言代理可以通過API調用構建和組合原子建築塊。人類和自動GPT-4V評估表明，我們的方法在ShapeNet的三維網格生成方面勝過標準GPT-4和其他語言代理（例如ReAct和Reflexion）。此外，當在我們的UFO基準測試上進行測試時，我們的方法在人類評估方面優於其他最先進的文本到二維圖像和文本到三維模型。

English

Diffusion-based image generation models such as DALL-E 3 and Stable Diffusion-XL demonstrate remarkable capabilities in generating images with realistic and unique compositions. Yet, these models are not robust in precisely reasoning about physical and spatial configurations of objects, especially when instructed with unconventional, thereby out-of-distribution descriptions, such as "a chair with five legs". In this paper, we propose a language agent with chain-of-3D-thoughts (L3GO), an inference-time approach that can reason about part-based 3D mesh generation of unconventional objects that current data-driven diffusion models struggle with. More concretely, we use large language models as agents to compose a desired object via trial-and-error within the 3D simulation environment. To facilitate our investigation, we develop a new benchmark, Unconventionally Feasible Objects (UFO), as well as SimpleBlenv, a wrapper environment built on top of Blender where language agents can build and compose atomic building blocks via API calls. Human and automatic GPT-4V evaluations show that our approach surpasses the standard GPT-4 and other language agents (e.g., ReAct and Reflexion) for 3D mesh generation on ShapeNet. Moreover, when tested on our UFO benchmark, our approach outperforms other state-of-the-art text-to-2D image and text-to-3D models based on human evaluation.