UI2App：可執行網頁應用程式生成中的視覺互動推論基準測試

摘要

大型語言模型（LLMs）在網頁生成方面展現出日益增強的能力。然而，現有的文字驅動方法依賴於複雜的提示詞，這對使用者提出較高要求，且在頁面佈局與跨頁視覺一致性方面的表達力有限。以圖像為驅動的範式（將UI截圖作為輸入）更貼近實際開發流程。然而，現有基準主要聚焦於視覺保真度，缺乏對生成成品互動能力的系統性評估。為填補此缺口，我們提出UI2App——首個針對互動推論（即僅從截圖還原應用行為，無需任何文字或行為指引）的評測基準。UI2App包含327張截圖，組成45組狀態一致的截圖集，對應可執行的多路徑網頁應用。我們設計了一套端到端評估流程，從可執行性、導航可達性、視覺保真度及互動推論四個維度對每個成品進行評估。互動指標（IIS）根據功能正確性與狀態管理複雜度來評估推論出的互動行為，只要任何有效實現皆可獲得評分，而非匹配單一參考答案。在六個前沿視覺語言模型上的實驗顯示，視覺重建與互動實現之間存在顯著的能力落差：視覺保真度最高的模型在IIS上僅得7.5分，排名第四，落後IIS冠軍5.2倍。高複雜度的互動（如跨頁狀態管理）仍是普遍瓶頸，半數受評模型在此維度得分恰好為零。整體而言，結果表明從靜態截圖推論完整互動行為仍是模型面臨的主要挑戰。

English

Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex prompts that impose substantial demands on users and offer limited expressivity for page layout and cross-page visual coherence. Image-driven paradigms, which take UI screenshots as input, align more closely with real development workflows. However, current benchmarks focus primarily on visual fidelity and lack a systematic evaluation of the interaction capabilities in generated artifacts. To address this gap, we introduce UI2App, the first benchmark targeting interaction inference, the ability to recover application behavior from screenshots alone, without any textual or behavioral guidance. UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web applications. We design an end-to-end pipeline that evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference. The interaction metric (IIS) assesses inferred interactions by functional correctness and state-management complexity, crediting any valid implementation rather than matching a single reference. Experiments on six frontier vision-language models reveal a marked capability mismatch between visual reconstruction and interaction realization: the visual-fidelity leader scores only 7.5 on IIS, ranking fourth and trailing the IIS leader by 5.2x. High-complexity interactions such as cross-page state remain a pervasive bottleneck, with half of the evaluated models scoring exactly zero on this dimension. Overall, the results indicate that inferring complete interaction behavior from static screenshots remains a key challenge for models.