多视图图像编辑的注意力特征整合

摘要

大规模文本到图像模型实现了广泛的图像编辑技术，使用文本提示甚至空间控制。然而，将这些编辑方法应用于描绘单个场景的多视图图像会导致3D不一致的结果。在这项工作中，我们专注于基于空间控制的几何操作，并介绍一种方法来统一各种视图上的编辑过程。我们基于两个观点：（1）在生成过程中始终保持一致的特征有助于实现多视图编辑的一致性，（2）自注意力层中的查询显著影响图像结构。因此，我们提出通过强化查询的一致性来改善编辑图像的几何一致性。为此，我们引入了QNeRF，这是一个基于编辑图像的内部查询特征训练的神经辐射场。一旦训练完成，QNeRF可以渲染出3D一致的查询，然后在生成过程中软性注入回自注意力层，极大地提高了多视图的一致性。我们通过渐进迭代方法对这一过程进行了改进，更好地统一了扩散时间步中的查询。我们将我们的方法与一系列现有技术进行了比较，并证明它能够实现更好的多视图一致性，并更忠实于输入场景。这些优势使我们能够训练出更少视觉伪影、更好地与目标几何形状对齐的神经辐射场。

English

Large-scale text-to-image models enable a wide range of image editing techniques, using text prompts or even spatial controls. However, applying these editing methods to multi-view images depicting a single scene leads to 3D-inconsistent results. In this work, we focus on spatial control-based geometric manipulations and introduce a method to consolidate the editing process across various views. We build on two insights: (1) maintaining consistent features throughout the generative process helps attain consistency in multi-view editing, and (2) the queries in self-attention layers significantly influence the image structure. Hence, we propose to improve the geometric consistency of the edited images by enforcing the consistency of the queries. To do so, we introduce QNeRF, a neural radiance field trained on the internal query features of the edited images. Once trained, QNeRF can render 3D-consistent queries, which are then softly injected back into the self-attention layers during generation, greatly improving multi-view consistency. We refine the process through a progressive, iterative method that better consolidates queries across the diffusion timesteps. We compare our method to a range of existing techniques and demonstrate that it can achieve better multi-view consistency and higher fidelity to the input scene. These advantages allow us to train NeRFs with fewer visual artifacts, that are better aligned with the target geometry.

多视图图像编辑的注意力特征整合

Consolidating Attention Features for Multi-view Image Editing

摘要

Support