DeepMesh：基于强化学习的自回归艺术家网格生成

DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learning

March 19, 2025

作者: Ruowen Zhao, Junliang Ye, Zhengyi Wang, Guangce Liu, Yiwen Chen, Yikai Wang, Jun Zhu

cs.AI

摘要

三角网格在三维应用中扮演着关键角色，以实现高效的操控与渲染。尽管自回归方法通过预测离散顶点标记来生成结构化网格，但它们常受限于面数不足及网格不完整的问题。为应对这些挑战，我们提出了DeepMesh框架，该框架通过两大创新优化网格生成：(1) 采用一种结合新型标记化算法的高效预训练策略，并改进数据整理与处理流程；(2) 将强化学习（RL）引入三维网格生成，通过直接偏好优化（DPO）实现与人类偏好的对齐。我们设计了一套评分标准，融合人类评估与三维度量指标，以收集用于DPO的偏好对，确保视觉吸引力与几何精度兼备。基于点云和图像条件，DeepMesh能够生成细节丰富、拓扑精确的网格，在精度与质量上均超越了现有最先进方法。项目页面：https://zhaorw02.github.io/DeepMesh/

English

Triangle meshes play a crucial role in 3D applications for efficient manipulation and rendering. While auto-regressive methods generate structured meshes by predicting discrete vertex tokens, they are often constrained by limited face counts and mesh incompleteness. To address these challenges, we propose DeepMesh, a framework that optimizes mesh generation through two key innovations: (1) an efficient pre-training strategy incorporating a novel tokenization algorithm, along with improvements in data curation and processing, and (2) the introduction of Reinforcement Learning (RL) into 3D mesh generation to achieve human preference alignment via Direct Preference Optimization (DPO). We design a scoring standard that combines human evaluation with 3D metrics to collect preference pairs for DPO, ensuring both visual appeal and geometric accuracy. Conditioned on point clouds and images, DeepMesh generates meshes with intricate details and precise topology, outperforming state-of-the-art methods in both precision and quality. Project page: https://zhaorw02.github.io/DeepMesh/