Code2World：基於可渲染程式碼生成的圖形使用者介面世界模型

Code2World: A GUI World Model via Renderable Code Generation

February 10, 2026

作者: Yuhao Zheng, Li'an Zhong, Yi Wang, Rui Dai, Kaikui Liu, Xiangxiang Chu, Linyuan Lv, Philip Torr, Kevin Qinghong Lin

cs.AI

摘要

自主GUI代理通過感知介面並執行操作來與環境互動。GUI世界模型作為虛擬沙盒，透過動作條件預測使代理具備類人預見能力。然而現有基於文本和像素的方法難以同時實現高視覺保真度與細粒度結構可控性。為此，我們提出Code2World——一種透過可渲染代碼生成來模擬下一視覺狀態的視覺語言編碼器。具體而言，為解決數據稀缺問題，我們構建AndroidCode數據集：將GUI軌跡轉譯為高保真HTML，並透過視覺反饋修正機制精煉合成代碼，最終獲得包含8萬餘高質量螢幕-動作對的語料庫。為使現有VLM適應代碼預測任務，我們首先進行SFT作為格式佈局跟蹤的冷啟動，隨後應用渲染感知強化學習——以渲染結果作為獎勵信號，強化視覺語義保真度與動作一致性。大量實驗表明，Code2World-8B在下一代UI預測任務中表現頂尖，可與競爭對手GPT-5和Gemini-3-Pro-Image相媲美。值得注意的是，Code2World以靈活方式顯著提升下游導航成功率，在AndroidWorld導航任務中將Gemini-2.5-Flash的性能提升9.5%。代碼已開源於：https://github.com/AMAP-ML/Code2World。

English

Autonomous GUI agents interact with environments by perceiving interfaces and executing actions. As a virtual sandbox, the GUI World model empowers agents with human-like foresight by enabling action-conditioned prediction. However, existing text- and pixel-based approaches struggle to simultaneously achieve high visual fidelity and fine-grained structural controllability. To this end, we propose Code2World, a vision-language coder that simulates the next visual state via renderable code generation. Specifically, to address the data scarcity problem, we construct AndroidCode by translating GUI trajectories into high-fidelity HTML and refining synthesized code through a visual-feedback revision mechanism, yielding a corpus of over 80K high-quality screen-action pairs. To adapt existing VLMs into code prediction, we first perform SFT as a cold start for format layout following, then further apply Render-Aware Reinforcement Learning which uses rendered outcome as the reward signal by enforcing visual semantic fidelity and action consistency. Extensive experiments demonstrate that Code2World-8B achieves the top-performing next UI prediction, rivaling the competitive GPT-5 and Gemini-3-Pro-Image. Notably, Code2World significantly enhances downstream navigation success rates in a flexible manner, boosting Gemini-2.5-Flash by +9.5% on AndroidWorld navigation. The code is available at https://github.com/AMAP-ML/Code2World.