Hallo4D: 缓解多模态幻觉以实现一致时空生成

摘要

尽管近期3D生成技术的进步实现了令人印象深刻的视觉合成效果，但现有方法往往依赖2D扩散监督，缺乏显式的几何一致性机制，导致产生重复结构、几何错位等空间幻觉。在4D生成中，这些问题更为严峻——需同时维持多视角与时序演变的一致性的挑战，会引发抖动、身份闪烁和结构漂移等现象。我们提出Hallo4D，一个统一且与模型无关的框架，用于缓解3D和4D内容生成中的时空幻觉。Hallo4D引入"生成-检测-修正"范式，利用大型多模态语言模型（LMM）从多视角和多帧渲染结果中识别并总结空间与时间不一致性。这些洞察引导基于共识的图像空间一致性优化，通过多模型投票机制，由LMM驱动的选择器评估候选修正方案，无需重新训练或修改模型架构。为进一步提升时间一致性与优化效率，Hallo4D融合了运动感知关键帧采样、LMM引导初始化与外观对齐技术。我们还引入曝光感知优化与可见性剪枝策略，增强对挑战性视角的鲁棒性。大量实验表明，Hallo4D在多种3D和4D生成场景中持续超越强基线方法，为一致性感知的内容生成提供了可扩展且泛化能力强的解决方案。

English

While recent advances in 3D generation have enabled impressive visual synthesis, existing methods often rely on 2D diffusion supervision without explicit mechanisms for geometric consistency, leading to spatial hallucinations such as duplicated structures and misaligned geometry. These issues become more severe in 4D generation, where maintaining consistency across viewpoints and temporal evolution introduces additional challenges, including jitter, identity flicker, and structural drift. We present Hallo4D, a unified and model-agnostic framework for mitigating spatiotemporal hallucinations in 3D and 4D content generation. Hallo4D introduces a generation-detection-correction paradigm that leverages large multimodal language models (LMMs) to identify and summarize spatial and temporal inconsistencies from multi-view and multi-frame renderings. These insights guide a consensus-driven image-space consistency optimization, where an LMM-based selector evaluates candidate corrections through multi-model voting, without requiring retraining or architectural modifications. To further improve temporal consistency and optimization efficiency, Hallo4D incorporates motion-aware keyframe sampling, LMM-guided initialization, and appearance alignment. We additionally introduce exposure-aware optimization and visibility pruning to enhance robustness under challenging viewpoints. Extensive experiments demonstrate that Hallo4D consistently outperforms strong baselines across diverse 3D and 4D generation settings, providing a scalable and generalizable solution for consistency-aware content generation.