評分量規作為自我進化UI轉程式碼生成之視覺修復情境

摘要

大型視覺-語言模型在介面轉程式碼生成方面展現出顯著進展，然而其在測試時的自我演化仍不穩定。我們首先發現一個根本性障礙，稱之為視覺修復耦合：局部程式碼修改可能透過佈局、樣式和元件依賴關係傳播，在修正某一視覺不一致的同時，卻導致先前符合要求的區域退化。為了解決此問題，我們提出 RubSE，一個基於評分量表引導的自我演化框架，利用評分量表將視覺回饋表示為結構化的視覺修復語境。在每一輪求精過程中，RubSE 生成具類型的候選評分量表，選擇一個優先修復目標，並將先前選取的評分量表儲存為歷史紀錄，從而引導每次修訂朝向範圍明確的視覺修復，同時抑制重複或過度廣泛的變更。在六個視覺-語言模型和三個介面轉程式碼基準上的評估顯示，RubSE 在最終輪次和最佳輪次設定下均大幅優於樸素的自我演化方法，實現更穩定的求精軌跡和更高的軌跡層級效能上限。進一步分析表明，RubSE 透過改善對嚴重視覺退化的恢復能力來緩解軌跡崩潰，且更強的評分量表生成器能將有效的視覺修復指引遷移至較弱的程式碼改進器。

English

Large vision-language models have shown strong progress in UI-to-code generation, yet their test-time self-evolution remains unstable. We first identify a fundamental obstacle, termed visual repair coupling: a local code edit may propagate through layout, style, and component dependencies, correcting one visual mismatch while degrading regions that were previously faithful. To address this issue, we present RubSE, a Rubric-guided Self-Evolution framework that uses rubrics to represent visual feedback as a structured visual-repair context. At each refinement round, RubSE generates typed candidate rubrics, selects one prioritized repair target, and stores previously selected rubrics as history, thereby steering each revision toward a well-scoped visual repair while discouraging repeated or over-broad changes. Evaluations across six VLMs and three UI-to-code benchmarks demonstrate that RubSE substantially outperforms naïve self-evolution in final-round and best-round settings, achieving more stable refinement trajectories and a higher trajectory-level performance ceiling. Further analysis shows that RubSE mitigates trajectory collapse by improving recovery from severe visual regressions, and that stronger rubric generators can transfer effective visual-repair guidance to weaker code improvers.