4DSloMo：基于异步捕捉的高速场景四维重建

摘要

从多视角视频中重建快速动态场景对于高速运动分析和逼真的4D重建至关重要。然而，大多数4D捕捉系统的帧率限制在30 FPS（每秒帧数）以下，直接从低帧率输入进行高速运动的4D重建可能会导致不理想的结果。在本研究中，我们提出了一种仅使用低帧率相机的高速4D捕捉系统，通过新颖的捕捉和处理模块实现。在捕捉方面，我们提出了一种异步捕捉方案，通过错开相机的启动时间来提高有效帧率。通过将相机分组并利用25 FPS的基础帧率，我们的方法实现了100-200 FPS的等效帧率，而无需使用专门的高速相机。在处理方面，我们还提出了一种新的生成模型，用于修复由4D稀疏视图重建引起的伪影，因为异步性会减少每个时间戳的视角数量。具体而言，我们提出训练一种基于视频扩散的伪影修复模型，用于稀疏4D重建，该模型能够细化缺失细节、保持时间一致性并提高整体重建质量。实验结果表明，与同步捕捉相比，我们的方法显著提升了高速4D重建的效果。

English

Reconstructing fast-dynamic scenes from multi-view videos is crucial for high-speed motion analysis and realistic 4D reconstruction. However, the majority of 4D capture systems are limited to frame rates below 30 FPS (frames per second), and a direct 4D reconstruction of high-speed motion from low FPS input may lead to undesirable results. In this work, we propose a high-speed 4D capturing system only using low FPS cameras, through novel capturing and processing modules. On the capturing side, we propose an asynchronous capture scheme that increases the effective frame rate by staggering the start times of cameras. By grouping cameras and leveraging a base frame rate of 25 FPS, our method achieves an equivalent frame rate of 100-200 FPS without requiring specialized high-speed cameras. On processing side, we also propose a novel generative model to fix artifacts caused by 4D sparse-view reconstruction, as asynchrony reduces the number of viewpoints at each timestamp. Specifically, we propose to train a video-diffusion-based artifact-fix model for sparse 4D reconstruction, which refines missing details, maintains temporal consistency, and improves overall reconstruction quality. Experimental results demonstrate that our method significantly enhances high-speed 4D reconstruction compared to synchronous capture.

4DSloMo：基于异步捕捉的高速场景四维重建

4DSloMo: 4D Reconstruction for High Speed Scene with Asynchronous Capture

摘要

Support