UI2App：可执行Web应用生成中的视觉交互推理基准测试

摘要

大型语言模型（LLMs）在网页生成方面展现出日益增强的能力。然而，现有的文本驱动方法依赖于复杂提示，对用户提出较高要求，且在页面布局和跨页面视觉一致性方面表达力有限。以图像为驱动的范式将UI截图作为输入，更贴近实际的开发工作流。然而，当前基准测试主要关注视觉保真度，缺乏对生成产物交互能力的系统性评估。为填补这一空白，我们提出了首个针对交互推断（即仅凭截图恢复应用程序行为的能力，无需任何文本或行为指导）的基准测试——UI2App。UI2App包含327张截图，分为45个状态一致的截图集，对应可运行的多路径Web应用程序。我们设计了一个端到端评估流水线，沿四个维度对每个产物进行评测：可执行性、导航可达性、视觉保真度和交互推断。其中交互指标（IIS）根据功能正确性和状态管理复杂度评估推断出的交互，认可任何有效实现而非匹配单一参考答案。在六个前沿视觉语言模型上的实验揭示了视觉重建与交互实现之间的显著能力差距：视觉保真度领先模型在IIS上仅得7.5分，排名第四，落后IIS领先者5.2倍。跨页面状态等高复杂度交互仍然是普遍存在的瓶颈，半数受评模型在此维度上得分为零。整体结果表明，从静态截图中推断完整的交互行为仍然是模型面临的关键挑战。

English

Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex prompts that impose substantial demands on users and offer limited expressivity for page layout and cross-page visual coherence. Image-driven paradigms, which take UI screenshots as input, align more closely with real development workflows. However, current benchmarks focus primarily on visual fidelity and lack a systematic evaluation of the interaction capabilities in generated artifacts. To address this gap, we introduce UI2App, the first benchmark targeting interaction inference, the ability to recover application behavior from screenshots alone, without any textual or behavioral guidance. UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web applications. We design an end-to-end pipeline that evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference. The interaction metric (IIS) assesses inferred interactions by functional correctness and state-management complexity, crediting any valid implementation rather than matching a single reference. Experiments on six frontier vision-language models reveal a marked capability mismatch between visual reconstruction and interaction realization: the visual-fidelity leader scores only 7.5 on IIS, ranking fourth and trailing the IIS leader by 5.2x. High-complexity interactions such as cross-page state remain a pervasive bottleneck, with half of the evaluated models scoring exactly zero on this dimension. Overall, the results indicate that inferring complete interaction behavior from static screenshots remains a key challenge for models.