超越NL2Code：多模态代码智能的结构化综述

摘要

尽管大型语言模型（LLMs）在文本到代码合成方面取得了显著进展，但许多实际编程任务通过截图、图表、矢量绘图、视频和交互状态等视觉元素来指定意图。这些任务要求模型将视觉感知与可执行程序相连接，因为正确性不仅取决于语法，还取决于执行后适用的布局、数据语义、交互行为及领域特定约束。本综述探讨了多模态代码智能，涵盖在视觉引导的输入和输出下生成、编辑、优化或推理代码的系统。我们首先通过代码在每项任务中扮演的角色来界定该领域，区分代码作为渲染制品、可编辑的符号结构、科学表示、中间推理痕迹，以及可执行的策略或工具接口。随后，我们将基准测试和方法组织为四个领域：图形用户界面、科学可视化、结构化图形，以及前沿任务与框架。这一分类将成熟的制品生成问题与新兴的智能体和统一设置联系起来，使我们能够比较不同任务如何处理正确性证据。展望未来，我们认为未来研究可能受益于四个以验证为核心的方向：多信号验证可结合互补的正确性证据，多状态验证可测试跨执行轨迹的行为，跨任务迁移测试可探索可复用的视觉-代码技能，而可验证的智能体痕迹可揭示智能体行动是否基于视觉证据。这些方向共同可能推动该领域从单一输出的模仿走向基于证据的可执行系统。相关持续项目与资源可访问 https://github.com/xjywhu/Awesome-Multimodal-LLM-for-Code{GitHub}。

English

While Large Language Models (LLMs) have substantially advanced text-to-code synthesis, many real programming tasks specify intent through visual artifacts such as screenshots, charts, vector drawings, videos, and interactive states. These tasks require models to connect visual perception to executable programs, because correctness depends not only on syntax but also on layout, data semantics, interaction behavior, and domain-specific constraints that apply after execution. This survey examines Multimodal Code Intelligence, covering systems that generate, edit, refine, or reason with code under visually grounded inputs and outputs. We first formulate the field by the role that code plays in each task, distinguishing code as a rendered artifact, an editable symbolic structure, a scientific representation, an intermediate reasoning trace, or an executable policy or tool interface. We then organize benchmarks and methods into four domains: Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks. This taxonomy connects mature artifact-generation problems to emerging agentic and unified settings and allows us to compare how different tasks treat evidence of correctness. Looking ahead, we argue that future research may benefit from four verification-centered directions. Multi-signal validation can combine complementary evidence of correctness, multi-state verification can test behavior across execution trajectories, cross-task transfer testing can probe reusable visual-code skills, and verifiable agent traces can reveal whether agent actions are grounded in visual evidence. Together, these directions may move this field from single-output imitation toward evidence-grounded executable systems. An ongoing project and resources are available on https://github.com/xjywhu/Awesome-Multimodal-LLM-for-Code{GitHub}.