DeepMesh：基於強化學習的自回歸藝術家網格生成

DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learning

March 19, 2025

作者: Ruowen Zhao, Junliang Ye, Zhengyi Wang, Guangce Liu, Yiwen Chen, Yikai Wang, Jun Zhu

cs.AI

摘要

三角網格在3D應用中扮演著至關重要的角色，用於高效的操作與渲染。雖然自回歸方法通過預測離散的頂點標記來生成結構化網格，但它們往往受限於有限的面數和網格的不完整性。為應對這些挑戰，我們提出了DeepMesh框架，該框架通過兩項關鍵創新來優化網格生成：(1) 一種高效的預訓練策略，結合了新型的標記化算法，並在數據整理與處理方面進行了改進；(2) 將強化學習（RL）引入3D網格生成，通過直接偏好優化（DPO）實現與人類偏好的對齊。我們設計了一個結合人類評估與3D指標的評分標準，以收集用於DPO的偏好對，確保視覺吸引力和幾何精確性。基於點雲和圖像的條件下，DeepMesh生成的網格具有精細的細節和精確的拓撲結構，在精度和質量上均超越了現有的最先進方法。項目頁面：https://zhaorw02.github.io/DeepMesh/

English

Triangle meshes play a crucial role in 3D applications for efficient manipulation and rendering. While auto-regressive methods generate structured meshes by predicting discrete vertex tokens, they are often constrained by limited face counts and mesh incompleteness. To address these challenges, we propose DeepMesh, a framework that optimizes mesh generation through two key innovations: (1) an efficient pre-training strategy incorporating a novel tokenization algorithm, along with improvements in data curation and processing, and (2) the introduction of Reinforcement Learning (RL) into 3D mesh generation to achieve human preference alignment via Direct Preference Optimization (DPO). We design a scoring standard that combines human evaluation with 3D metrics to collect preference pairs for DPO, ensuring both visual appeal and geometric accuracy. Conditioned on point clouds and images, DeepMesh generates meshes with intricate details and precise topology, outperforming state-of-the-art methods in both precision and quality. Project page: https://zhaorw02.github.io/DeepMesh/