Chat-Edit-3D++：基於大型語言模型的互動式3D與4D場景編輯

摘要

近期，基於視覺-語言預訓練模型的影像內容操作研究已被有效延伸至文字驅動的三維場景編輯。然而，現有的三維場景編輯方案仍存在若干不足，阻礙其作為互動式設計工具的進一步發展。此類方案通常遵循固定的輸入模式，限制了文字輸入的靈活性。此外，其編輯能力受限於單一或少數二維視覺模型，且需要複雜的流程設計，方能將這些模型整合至三維重建過程中。為了解決上述問題，我們提出 Hash-Atlas 網路，將三維場景編輯重新表述為對二維圖集影像的操作，從而實現二維編輯與三維重建流程的工作流解耦。在此基礎上，我們引入一種基於對話的三維場景編輯方法，稱為 CE3D++，其以大型語言模型（LLM）為核心，允許使用者任意輸入文字並解讀其意圖，進而促成相應視覺模型的自動調用。此外，我們透過對移動物體施加運動約束，將 CE3D++ 延伸至單目四維場景，並透過建立與編輯任務相關的軌跡資料集進一步微調 LLM，從而使較小的 LLM 能夠準確調度高達 30 種不同的視覺工具。實驗結果表明，CE3D++ 能有效整合多個視覺模型，實現多樣化的視覺編輯效果，並具備強大的場景理解與多輪對話能力。原始碼與訓練模型可在 https://github.com/Fangkang515/CE3D 取得。

English

Recent work on image content manipulation based on vision-language pre-training models has been effectively extended to text-driven 3D scene editing. However, existing schemes for 3D scene editing still have certain shortcomings, hindering their further development as interactive design tools. Such schemes typically adhere to fixed input patterns, limiting flexibility in text input. Furthermore, their editing capabilities are constrained by a single or a few 2D visual models and require intricate pipeline design to integrate these models into 3D reconstruction processes. To address the aforementioned issues, we propose the Hash-Atlas network, which reformulates 3D scene editing as operations on 2D atlas images, thereby achieving a workflow decoupling of the 2D editing and 3D reconstruction processes. Building on this foundation, we introduce a dialogue-based 3D scene editing approach, termed CE3D++, which is centered on a large language model (LLM) that allows arbitrary textual input from users and interprets their intentions, subsequently facilitating the autonomous invocation of the corresponding visual models. Additionally, we extend CE3D++ to monocular 4D scenes by imposing motion constraints on moving objects and further fine-tuning the LLM by creating a trajectory dataset related to editing tasks, which enables the smaller LLM to schedule up to 30 different visual tools accurately. Experimental results demonstrate that CE3D++ effectively integrates multiple visual models to achieve diverse visual editing effects, possessing strong scene comprehension and multi-round dialog capabilities. The source codes and trained models are available at https://github.com/Fangkang515/CE3D.