基於生成先驗蒸餾的無配對監督3D編輯學習

摘要

指令引導的3D編輯對於互動式內容創作至關重要，然而它面臨一個重大瓶頸：高品質成對訓練資料的嚴重匱乏。現有方法試圖透過依賴緩慢的測試時優化，或是在透過複雜流程建構的偽成對資料上進行訓練來繞過此問題，但這些方法往往會引入結構漂移與幾何偽影。在本文中，我們提出一個新穎框架，透過生成先驗蒸餾，學習無需成對3D監督的前饋式3D編輯。我們的核心概念並非依賴真實標註的3D成對資料，而是將來自強大基礎模型的視覺、語義與幾何知識直接蒸餾到3D編輯模型中。具體而言，我們透過可微分渲染管線，使用兩個互補訊號來監督3D表徵：一個是在主編輯視角來自影像編輯模型的2D視覺先驗，另一個是在新視角來自視覺-語言模型的語義先驗，以確保嚴格遵循指令與來源身份保留。至關重要的是，為了解決2D投影監督固有的幾何崩潰與多視角不一致性，我們引入3D感知的分布匹配正則化。此項作為幾何先驗，作用於3D潛在空間，約束編輯輸出保持在由預訓練的影像到3D教師模型所定義的真實3D資產流形內。大量實驗證明，我們的方法達到優異的指令保真度與跨視角一致性，顯著優於最先進的基線方法。我們的專案位於：https://github.com/thiamine128/PriorEdit3D。

English

Instruction-guided 3D editing is essential for interactive content creation, yet it faces a significant bottleneck: the severe scarcity of high-quality paired training data. Existing approaches attempt to bypass this by either relying on slow test-time optimization or training on pseudo-pairs constructed via complex pipelines, which often introduce structural drift and geometric artifacts. In this paper, we propose a novel framework that learns feed-forward 3D editing without paired 3D supervision via Generative Prior Distillation. Instead of relying on ground-truth 3D pairs, our core idea is to distill visual, semantic, and geometric knowledge from powerful foundation models directly into a 3D editing model. Specifically, through a differentiable rendering pipeline, we supervise the 3D representation using two complementary signals: a 2D visual prior from an image editing model at the main editing view, and a semantic prior from a Vision-Language Model at novel views to ensure strict instruction following and source identity preservation. Crucially, to address the geometric collapse and multi-view inconsistencies inherent in 2D projection supervision, we introduce a 3D-aware Distribution Matching regularization. Acting as a geometric prior, this term operates in the 3D latent space, constraining the edited output to remain within the manifold of realistic 3D assets defined by a pretrained image to 3D teacher model. Extensive experiments demonstrate that our method achieves superior instruction fidelity and cross-view consistency, significantly outperforming state-of-the-art baselines. Our project is available at: https://github.com/thiamine128/PriorEdit3D.