大理石：CLIP空间中的材料重组与混合

摘要

基於示例圖像對圖像中物體材質進行編輯，是計算機視覺與圖形學領域的一個活躍研究方向。本文提出MARBLE方法，該方法通過在CLIP空間中尋找材質嵌入並利用其控制預訓練的文本到圖像模型，實現了材質混合與細粒度材質屬性的重構。我們通過定位去噪UNet中負責材質歸因的模塊，改進了基於示例的材質編輯。給定兩張材質示例圖像，我們在CLIP空間中找到了混合材質的方向。此外，利用淺層網絡預測期望材質屬性變化的方向，我們能夠實現對粗糙度、金屬感、透明度及發光等細粒度材質屬性的參數化控制。通過定性與定量分析，我們驗證了所提方法的有效性。同時，我們展示了該方法在單次前向傳播中執行多重編輯的能力及其在繪畫中的應用潛力。項目頁面：https://marblecontrol.github.io/

English

Editing materials of objects in images based on exemplar images is an active area of research in computer vision and graphics. We propose MARBLE, a method for performing material blending and recomposing fine-grained material properties by finding material embeddings in CLIP-space and using that to control pre-trained text-to-image models. We improve exemplar-based material editing by finding a block in the denoising UNet responsible for material attribution. Given two material exemplar-images, we find directions in the CLIP-space for blending the materials. Further, we can achieve parametric control over fine-grained material attributes such as roughness, metallic, transparency, and glow using a shallow network to predict the direction for the desired material attribute change. We perform qualitative and quantitative analysis to demonstrate the efficacy of our proposed method. We also present the ability of our method to perform multiple edits in a single forward pass and applicability to painting. Project Page: https://marblecontrol.github.io/

大理石：CLIP空间中的材料重组与混合

MARBLE: Material Recomposition and Blending in CLIP-Space

摘要

Support