使用预训练的文本到图像扩散模型进行点云补全。

摘要

在现实世界的应用中收集的点云数据通常是不完整的。数据通常缺失是因为对象是从部分视角观察到的，这些视角只捕获特定的透视或角度。此外，数据可能由于遮挡和低分辨率采样而不完整。现有的完成方法依赖于预定义对象的数据集，以指导嘈杂和不完整的点云的完成。然而，这些方法在测试时表现不佳，当测试对象是训练数据集中较少代表的Out-Of-Distribution (OOD)对象时。在这里，我们利用了最近在文本引导图像生成方面取得的进展，这些进展导致了文本引导形状生成方面的重大突破。我们描述了一种名为SDS-Complete的方法，它使用预训练的文本到图像扩散模型，并利用给定对象的不完整点云的文本语义，以获得完整的表面表示。SDS-Complete可以使用测试时间优化完成各种对象，而无需昂贵地收集3D信息。我们在由现实世界深度传感器和激光雷达扫描仪捕获的不完整扫描对象上评估了SDS Complete。我们发现，与当前方法相比，它有效地重建了常见数据集中缺失的对象，平均减少了50%的Chamfer损失。项目页面：https://sds-complete.github.io/

English

Point-cloud data collected in real-world applications are often incomplete. Data is typically missing due to objects being observed from partial viewpoints, which only capture a specific perspective or angle. Additionally, data can be incomplete due to occlusion and low-resolution sampling. Existing completion approaches rely on datasets of predefined objects to guide the completion of noisy and incomplete, point clouds. However, these approaches perform poorly when tested on Out-Of-Distribution (OOD) objects, that are poorly represented in the training dataset. Here we leverage recent advances in text-guided image generation, which lead to major breakthroughs in text-guided shape generation. We describe an approach called SDS-Complete that uses a pre-trained text-to-image diffusion model and leverages the text semantics of a given incomplete point cloud of an object, to obtain a complete surface representation. SDS-Complete can complete a variety of objects using test-time optimization without expensive collection of 3D information. We evaluate SDS Complete on incomplete scanned objects, captured by real-world depth sensors and LiDAR scanners. We find that it effectively reconstructs objects that are absent from common datasets, reducing Chamfer loss by 50% on average compared with current methods. Project page: https://sds-complete.github.io/

使用预训练的文本到图像扩散模型进行点云补全。

Point-Cloud Completion with Pretrained Text-to-image Diffusion Models

摘要

Support