超越NL2Code：多模态代碼智能的結構化綜述

摘要

雖然大型語言模型（LLMs）已大幅推進文字轉程式碼的合成技術，但許多真實的程式設計任務仍透過螢幕截圖、圖表、向量圖形、影片及互動狀態等視覺化成品來指定意圖。這類任務要求模型將視覺感知與可執行程式碼連結，因為正確與否不僅取決於語法，更取決於佈局、資料語義、互動行為，以及在執行後適用的領域特定約束。本調查探討多模態程式碼智慧（Multimodal Code Intelligence），涵蓋在以視覺為基礎的輸入與輸出下，生成、編輯、完善或推理程式碼的系統。我們首先根據程式碼在各項任務中所扮演的角色來界定此領域，區分程式碼作為渲染成品、可編輯符號結構、科學表徵、中間推理軌跡，或可執行策略與工具介面。接著，我們將基準測試與方法組織為四個領域：圖形使用者介面、科學視覺化、結構化圖形，以及前沿任務與框架。此分類法將成熟的成品生成問題，與新興的代理式（agentic）及統一化情境連結起來，並讓我們得以比較不同任務如何處理正確性的證據。展望未來，我們認為未來研究可能受益於四個以驗證為核心的方向：多訊號驗證可結合互補的正確性證據；多狀態驗證可測試跨執行軌跡的行為；跨任務遷移測試可探索可重複使用的視覺-程式碼技能；而可驗證的代理軌跡則可揭示代理行為是否奠基於視覺證據。綜合來看，這些方向可能推動本領域從單一輸出模仿，邁向以證據為基礎的可執行系統。相關持續進行的專案與資源可見於 https://github.com/xjywhu/Awesome-Multimodal-LLM-for-Code{GitHub}。

English

While Large Language Models (LLMs) have substantially advanced text-to-code synthesis, many real programming tasks specify intent through visual artifacts such as screenshots, charts, vector drawings, videos, and interactive states. These tasks require models to connect visual perception to executable programs, because correctness depends not only on syntax but also on layout, data semantics, interaction behavior, and domain-specific constraints that apply after execution. This survey examines Multimodal Code Intelligence, covering systems that generate, edit, refine, or reason with code under visually grounded inputs and outputs. We first formulate the field by the role that code plays in each task, distinguishing code as a rendered artifact, an editable symbolic structure, a scientific representation, an intermediate reasoning trace, or an executable policy or tool interface. We then organize benchmarks and methods into four domains: Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks. This taxonomy connects mature artifact-generation problems to emerging agentic and unified settings and allows us to compare how different tasks treat evidence of correctness. Looking ahead, we argue that future research may benefit from four verification-centered directions. Multi-signal validation can combine complementary evidence of correctness, multi-state verification can test behavior across execution trajectories, cross-task transfer testing can probe reusable visual-code skills, and verifiable agent traces can reveal whether agent actions are grounded in visual evidence. Together, these directions may move this field from single-output imitation toward evidence-grounded executable systems. An ongoing project and resources are available on https://github.com/xjywhu/Awesome-Multimodal-LLM-for-Code{GitHub}.