One4D：基于解耦LoRA控制的统一四维生成与重建框架

摘要

我们提出One4D——一个统一的4D生成与重建框架，能生成同步的RGB帧与点云图的动态4D内容。通过统一掩码条件机制（UMC）对输入帧的不同稀疏度进行一致性处理，该框架可实现从单张图像生成4D内容、完整视频重建4D内容，以及基于稀疏帧的混合生成与重建之间的无缝切换。我们通过精心设计的网络架构，将强大视频生成模型适配于RGB与点云图的联合生成。针对深度图或点云图重建的常用扩散模型微调策略在联合生成任务中常导致基础视频模型快速退化，为此我们提出解耦LoRA控制（DLC），采用两个模态专用LoRA适配器构建RGB帧与点云图的解耦计算分支，并通过轻量级零初始化控制链接逐步学习像素级互一致性。在有限算力下使用合成与真实4D数据集混合训练后，One4D在生成与重建任务中均能产出高质量RGB帧与精准点云图。本工作标志着基于视频扩散模型实现通用高质量几何化4D世界建模的重要进展。项目页面：https://mizhenxing.github.io/One4D

English

We present One4D, a unified framework for 4D generation and reconstruction that produces dynamic 4D content as synchronized RGB frames and pointmaps. By consistently handling varying sparsities of conditioning frames through a Unified Masked Conditioning (UMC) mechanism, One4D can seamlessly transition between 4D generation from a single image, 4D reconstruction from a full video, and mixed generation and reconstruction from sparse frames. Our framework adapts a powerful video generation model for joint RGB and pointmap generation, with carefully designed network architectures. The commonly used diffusion finetuning strategies for depthmap or pointmap reconstruction often fail on joint RGB and pointmap generation, quickly degrading the base video model. To address this challenge, we introduce Decoupled LoRA Control (DLC), which employs two modality-specific LoRA adapters to form decoupled computation branches for RGB frames and pointmaps, connected by lightweight, zero-initialized control links that gradually learn mutual pixel-level consistency. Trained on a mixture of synthetic and real 4D datasets under modest computational budgets, One4D produces high-quality RGB frames and accurate pointmaps across both generation and reconstruction tasks. This work represents a step toward general, high-quality geometry-based 4D world modeling using video diffusion models. Project page: https://mizhenxing.github.io/One4D

One4D：基于解耦LoRA控制的统一四维生成与重建框架

One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Control

摘要

Support