GUI-Primitives：診斷視覺-語言GUI定位中的空間推理缺陷

摘要

電腦使用代理透過將自然語言指令對應到截圖來定位介面元素，然而現有基準測試並未單獨檢驗模型是否能將關係語言綁定到正確元素。我們提出 GUI-Primitives，一個包含 994 個項目的基準測試，涵蓋圖形使用者介面中七種空間關係的對比指令對（左/右、上/下、包含、對齊、鄰近、列表序數、遮蔽）。每個指令對在保持截圖和錨點固定的情況下改變關係表達式，使得正確目標在兩個指定候選之間移動。五位標註者驗證了 196 個項目的子集（格式良好性 κ= 0.94；目標選擇 κ= 0.79）。十九個視覺語言模型的點在框內嚴格準確率最高僅達 32%。由於模型輸出非約束座標，我們根據預測落入的候選區域對每個預測進行分類。在 60–92% 的項目中，預測落在兩個候選之外。在落入候選區域的條件下，水平位置、垂直位置、鄰近和列表序數的目標選擇準確率達到 0.82–0.90，但包含和遮蔽的準確率與 0.50 無顯著差異：大多數失敗反映的是候選定位問題，而非關係理解問題。在十個模型中，基準測試準確率與 ScreenSpot-Pro 準確率相關（斯皮爾曼 ρ= +0.74），在此樣本量下為探索性關聯。標記兩個指定候選可將選擇準確率提高 35–57 個百分點，這是一種提供候選集合的神諭式診斷，而非可部署的方法。我們公開基準測試、預測和程式碼。

English

Computer-use agents ground natural-language instructions in screenshots to locate interface elements, yet existing benchmarks do not isolate whether models bind relational language to the correct element. We introduce GUI-Primitives, a 994-item benchmark of contrastive instruction pairs over seven spatial relations in graphical user interfaces (left/right, above/below, containment, alignment, proximity, list ordinal, occlusion). Each pair holds the screenshot and anchor fixed while changing the relation expression, so the correct target moves between two designated candidates. Five annotators validate a 196-item subset (κ= 0.94 well-formedness; κ= 0.79 target selection). Nineteen vision-language models reach at most 32% strict point-in-box accuracy. Because models emit unconstrained coordinates, we classify each prediction by the candidate region it falls within. Predictions fall outside both candidates on 60-92% of items. Conditional on falling within a candidate region, target selection reaches 0.82-0.90 for horizontal position, vertical position, proximity, and list ordinal, but does not differ significantly from 0.50 for containment and occlusion: most failures reflect candidate localization rather than relation understanding. Across ten models, benchmark accuracy correlates with ScreenSpot-Pro accuracy (Spearman ρ= +0.74), an exploratory association at this sample size. Marking the two designated candidates raises selection accuracy by 35--57 percentage points, an oracle diagnostic that supplies the candidate set rather than a deployable method. We release the benchmark, predictions, and code.