Lift4D: 协调单视图3D估计，用于野外4D重建

摘要

从单目视频重建动态非刚体对象需要将直接观测的视觉线索与几何和外观的数据驱动先验相结合。现有方法要么直接从视觉输入学习预测4D表示，要么初始化一个3D表示，随后基于视频证据对其进行变形和优化。然而，前者受限于4D训练数据的稀缺性，后者仅在初始重建阶段利用先验，之后完全依赖视频监督；两者均难以处理具有大形变和遮挡的复杂野外场景。我们提出Lift4D，一种测试时优化框架，同时解决了这两个局限性。首先，我们调整现有单视图3D重建模型，通过因果潜在条件生成时间一致性的逐帧预测，为可变形3D高斯飞溅表示提供连贯的初始化。然后，我们通过遮挡感知优化对此表示进行“雕刻”，使其匹配输入视频：在忠实恢复可见表面细节的同时，利用视图条件扩散先验完成未观测区域。我们证明，Lift4D在4D重建方面显著优于先前方法，尤其在具有严重遮挡和非刚性运动的挑战性野外序列上表现突出。

English

Reconstructing dynamic non-rigid objects from monocular video requires integrating visual cues from direct observations with data-driven priors over geometry and appearance. Prior approaches either learn to directly predict 4D representations from visual input or initialize a 3D representation that is subsequently deformed and refined based on video evidence. However, the former are constrained by the scarcity of 4D training data, while the latter leverage priors only for the initial reconstruction and rely solely on video supervision thereafter; neither handles complex in-the-wild scenarios with large deformations and occlusions well. We present Lift4D, a test-time optimization framework that addresses both limitations. First, we adapt an existing single-view 3D reconstruction model to yield temporally consistent per-frame predictions via causal latent conditioning, providing a coherent initialization for a deformable 3D Gaussian Splatting representation. We then ``sculpt'' this representation to match the input video through an occlusion-aware optimization that faithfully recovers visible surface details while completing unobserved regions using a view-conditioned diffusion prior. We demonstrate that Lift4D clearly improves over prior 4D reconstruction methods, particularly on challenging in-the-wild sequences with severe occlusions and non-rigid motion.