YouDream：生成具有解剖学可控一致性的文本到三维动物

YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals

June 24, 2024

作者: Sandeep Mishra, Oindrila Saha, Alan C. Bovik

cs.AI

摘要

由文本到图像扩散模型引导的3D生成使得创作出视觉上引人注目的资产成为可能。然而，先前的方法探索基于图像或文本的生成。创造力的边界受限于通过文字表达或可获取的图像。我们提出了YouDream，一种生成高质量解剖可控动物的方法。YouDream受2D视图控制的3D姿势先验引导文本到图像扩散模型。我们的方法生成了以往文本到3D生成方法无法创造的3D动物。此外，我们的方法能够在生成的动物中保持解剖一致性，这是先前文本到3D方法经常面临困难的领域。此外，我们设计了一个用于生成常见动物的完全自动化流程。为了避免需要人工干预来创建3D姿势，我们提出了一个多智能体LLM，从有限的动物3D姿势库中调整姿势以代表所需的动物。对YouDream结果的用户研究表明，我们方法生成的动物模型优于其他方法。旋转展示结果和代码发布在https://youdream3d.github.io/。

English

3D generation guided by text-to-image diffusion models enables the creation of visually compelling assets. However previous methods explore generation based on image or text. The boundaries of creativity are limited by what can be expressed through words or the images that can be sourced. We present YouDream, a method to generate high-quality anatomically controllable animals. YouDream is guided using a text-to-image diffusion model controlled by 2D views of a 3D pose prior. Our method generates 3D animals that are not possible to create using previous text-to-3D generative methods. Additionally, our method is capable of preserving anatomic consistency in the generated animals, an area where prior text-to-3D approaches often struggle. Moreover, we design a fully automated pipeline for generating commonly found animals. To circumvent the need for human intervention to create a 3D pose, we propose a multi-agent LLM that adapts poses from a limited library of animal 3D poses to represent the desired animal. A user study conducted on the outcomes of YouDream demonstrates the preference of the animal models generated by our method over others. Turntable results and code are released at https://youdream3d.github.io/