Chat-Edit-3D++：基于大语言模型的交互式3D和4D场景编辑

摘要

近期，基于视觉-语言预训练模型的图像内容编辑工作已被有效扩展到文本驱动的3D场景编辑中。然而，现有的3D场景编辑方案仍存在一些不足，阻碍了其作为交互式设计工具的进一步发展。这些方案通常遵循固定的输入模式，限制了文本输入的灵活性。此外，其编辑能力受限于单个或少数几个2D视觉模型，并且需要复杂的流水线设计才能将这些模型集成到3D重建过程中。为了解决上述问题，我们提出了Hash-Atlas网络，将3D场景编辑重新表述为对2D图集图像的操作，从而实现了2D编辑与3D重建过程的工作流解耦。在此基础上，我们提出了一种基于对话的3D场景编辑方法CE3D++，该方法以大语言模型（LLM）为核心，允许用户输入任意文本并理解其意图，进而促进相应视觉模型的自主调用。此外，我们通过对移动物体施加运动约束将CE3D++扩展到单目4D场景，并通过创建与编辑任务相关的轨迹数据集进一步微调LLM，使较小的LLM能够准确调度多达30种不同的视觉工具。实验结果表明，CE3D++有效集成了多种视觉模型以实现多样化的视觉编辑效果，具备强大的场景理解能力和多轮对话能力。源代码和训练好的模型可在 https://github.com/Fangkang515/CE3D 获取。

English

Recent work on image content manipulation based on vision-language pre-training models has been effectively extended to text-driven 3D scene editing. However, existing schemes for 3D scene editing still have certain shortcomings, hindering their further development as interactive design tools. Such schemes typically adhere to fixed input patterns, limiting flexibility in text input. Furthermore, their editing capabilities are constrained by a single or a few 2D visual models and require intricate pipeline design to integrate these models into 3D reconstruction processes. To address the aforementioned issues, we propose the Hash-Atlas network, which reformulates 3D scene editing as operations on 2D atlas images, thereby achieving a workflow decoupling of the 2D editing and 3D reconstruction processes. Building on this foundation, we introduce a dialogue-based 3D scene editing approach, termed CE3D++, which is centered on a large language model (LLM) that allows arbitrary textual input from users and interprets their intentions, subsequently facilitating the autonomous invocation of the corresponding visual models. Additionally, we extend CE3D++ to monocular 4D scenes by imposing motion constraints on moving objects and further fine-tuning the LLM by creating a trajectory dataset related to editing tasks, which enables the smaller LLM to schedule up to 30 different visual tools accurately. Experimental results demonstrate that CE3D++ effectively integrates multiple visual models to achieve diverse visual editing effects, possessing strong scene comprehension and multi-round dialog capabilities. The source codes and trained models are available at https://github.com/Fangkang515/CE3D.