GUI-Primitives：诊断视觉-语言GUI定位中的空间推理失败

摘要

计算机使用智能体需要将自然语言指令与截图中的界面元素进行对应，然而现有基准测试并未检验模型能否将关系语言绑定到正确的元素上。我们提出了GUI-Primitives，一个包含994个对比指令对的基准测试，覆盖图形用户界面中的七种空间关系（左/右、上/下、包含、对齐、邻近、列表序数、遮挡）。每个指令对在保持截图和锚点不变的情况下仅改变关系表达，从而使正确目标在两个指定候选元素之间移动。五位标注者对196个条目的子集进行了验证（规范性κ=0.94；目标选择κ=0.79）。十九个视觉语言模型的严格框内点准确率最高仅为32%。由于模型输出不受约束的坐标，我们依据每个预测落入的候选区域对其分类。在60-92%的条目中，预测落在两个候选区域之外。在落入候选区域的前提下，水平位置、垂直位置、邻近和列表序数的目标选择准确率达到0.82-0.90，但包含和遮挡的准确率与0.50无显著差异：大多数失败反映的是候选定位问题而非关系理解问题。在十个模型上，基准准确率与ScreenSpot-Pro准确率相关（斯皮尔曼ρ=+0.74），在此样本量下属于探索性关联。标记两个指定候选元素可使选择准确率提高35-57个百分点，这是一种提供候选集合的预言机诊断方法，而非可部署的方法。我们发布了该基准测试、预测结果和代码。

English

Computer-use agents ground natural-language instructions in screenshots to locate interface elements, yet existing benchmarks do not isolate whether models bind relational language to the correct element. We introduce GUI-Primitives, a 994-item benchmark of contrastive instruction pairs over seven spatial relations in graphical user interfaces (left/right, above/below, containment, alignment, proximity, list ordinal, occlusion). Each pair holds the screenshot and anchor fixed while changing the relation expression, so the correct target moves between two designated candidates. Five annotators validate a 196-item subset (κ= 0.94 well-formedness; κ= 0.79 target selection). Nineteen vision-language models reach at most 32% strict point-in-box accuracy. Because models emit unconstrained coordinates, we classify each prediction by the candidate region it falls within. Predictions fall outside both candidates on 60-92% of items. Conditional on falling within a candidate region, target selection reaches 0.82-0.90 for horizontal position, vertical position, proximity, and list ordinal, but does not differ significantly from 0.50 for containment and occlusion: most failures reflect candidate localization rather than relation understanding. Across ten models, benchmark accuracy correlates with ScreenSpot-Pro accuracy (Spearman ρ= +0.74), an exploratory association at this sample size. Marking the two designated candidates raises selection accuracy by 35--57 percentage points, an oracle diagnostic that supplies the candidate set rather than a deployable method. We release the benchmark, predictions, and code.