One4D：基于解耦LoRA控制的统一4维生成与重建

摘要

我们提出One4D——一个统一的四维生成与重建框架，能够生成同步的RGB帧与点云图的动态四维内容。通过统一掩码条件机制（UMC）对输入帧的不同稀疏度进行一致性处理，该框架可实现从单张图像的四维生成、完整视频的四维重建到稀疏帧混合生成与重建的无缝切换。我们基于强大的视频生成模型，通过精心设计的网络架构实现了RGB与点云的联合生成。传统基于扩散模型的深度图或点云重建微调策略在联合生成任务中常导致基础视频模型性能退化，为此我们提出解耦LoRA控制技术（DLC），采用两个模态特定的LoRA适配器构建RGB帧与点云的解耦计算分支，并通过轻量级零初始化控制链接逐步学习像素级一致性。在有限算力下使用合成与真实四维数据集进行训练后，One4D在生成与重建任务中均能产出高质量RGB帧与精确点云。这项研究标志着基于视频扩散模型实现通用高质量几何四维世界建模的重要进展。项目页面：https://mizhenxing.github.io/One4D

English

We present One4D, a unified framework for 4D generation and reconstruction that produces dynamic 4D content as synchronized RGB frames and pointmaps. By consistently handling varying sparsities of conditioning frames through a Unified Masked Conditioning (UMC) mechanism, One4D can seamlessly transition between 4D generation from a single image, 4D reconstruction from a full video, and mixed generation and reconstruction from sparse frames. Our framework adapts a powerful video generation model for joint RGB and pointmap generation, with carefully designed network architectures. The commonly used diffusion finetuning strategies for depthmap or pointmap reconstruction often fail on joint RGB and pointmap generation, quickly degrading the base video model. To address this challenge, we introduce Decoupled LoRA Control (DLC), which employs two modality-specific LoRA adapters to form decoupled computation branches for RGB frames and pointmaps, connected by lightweight, zero-initialized control links that gradually learn mutual pixel-level consistency. Trained on a mixture of synthetic and real 4D datasets under modest computational budgets, One4D produces high-quality RGB frames and accurate pointmaps across both generation and reconstruction tasks. This work represents a step toward general, high-quality geometry-based 4D world modeling using video diffusion models. Project page: https://mizhenxing.github.io/One4D

One4D：基于解耦LoRA控制的统一4维生成与重建

One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Control

摘要

Support