YouDream：生成具有解剖可控一致性的文本至3D動物

YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals

June 24, 2024

作者: Sandeep Mishra, Oindrila Saha, Alan C. Bovik

cs.AI

摘要

由文本到圖像擴散模型引導的3D生成技術能夠創建引人入勝的視覺資產。然而，先前的方法探索基於圖像或文本的生成。創造力的界限受限於能夠通過文字表達或可以獲取的圖像。我們提出了YouDream，一種生成高質量解剖可控動物的方法。YouDream使用由3D姿勢先前的2D視圖控制的文本到圖像擴散模型進行引導。我們的方法生成了以往的文本到3D生成方法無法創建的3D動物。此外，我們的方法能夠在生成的動物中保持解剖一致性，這是先前的文本到3D方法常常難以應對的領域。此外，我們設計了一個用於生成常見動物的完全自動化流程。為了避免需要人工干預來創建3D姿勢，我們提出了一種多智能體LLM，從有限的動物3D姿勢庫中調整姿勢以表示所需的動物。對YouDream的結果進行的用戶研究表明，我們方法生成的動物模型優於其他方法。轉盤結果和代碼已在https://youdream3d.github.io/ 上發布。

English

3D generation guided by text-to-image diffusion models enables the creation of visually compelling assets. However previous methods explore generation based on image or text. The boundaries of creativity are limited by what can be expressed through words or the images that can be sourced. We present YouDream, a method to generate high-quality anatomically controllable animals. YouDream is guided using a text-to-image diffusion model controlled by 2D views of a 3D pose prior. Our method generates 3D animals that are not possible to create using previous text-to-3D generative methods. Additionally, our method is capable of preserving anatomic consistency in the generated animals, an area where prior text-to-3D approaches often struggle. Moreover, we design a fully automated pipeline for generating commonly found animals. To circumvent the need for human intervention to create a 3D pose, we propose a multi-agent LLM that adapts poses from a limited library of animal 3D poses to represent the desired animal. A user study conducted on the outcomes of YouDream demonstrates the preference of the animal models generated by our method over others. Turntable results and code are released at https://youdream3d.github.io/