Android野外环境下的研究：用于Android设备控制的大规模数据集

Android in the Wild: A Large-Scale Dataset for Android Device Control

July 19, 2023

作者: Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, Timothy Lillicrap

cs.AI

摘要

对能够解释人类自然语言指令并通过直接控制数字设备的用户界面在数字设备上执行这些指令的设备控制系统越来越感兴趣。我们提出了一个用于设备控制研究的数据集Android in the Wild（AITW），其规模比当前数据集大几个数量级。该数据集包含人类演示的设备交互，包括屏幕和操作，以及相应的自然语言指令。它包含了715k个情节，涵盖了30k个独特指令，四个Android版本（v10-13），以及八种设备类型（从Pixel 2 XL到Pixel 6），具有不同的屏幕分辨率。数据集包含需要语义理解和视觉背景的多步任务。该数据集提出了一个新挑战：必须从视觉外观推断用户界面中的可用操作。而且，动作空间不是简单的基于UI元素的动作，而是包括精确手势（例如，水平滚动以操作走马灯小部件）。我们组织了数据集以促进对设备控制系统的鲁棒性分析，即系统在面对新任务描述、新应用程序或新平台版本时的表现如何。我们开发了两个代理程序，并报告了整个数据集上的性能。数据集可在https://github.com/google-research/google-research/tree/master/android_in_the_wild 上获得。

English

There is a growing interest in device-control systems that can interpret human natural language instructions and execute them on a digital device by directly controlling its user interface. We present a dataset for device-control research, Android in the Wild (AITW), which is orders of magnitude larger than current datasets. The dataset contains human demonstrations of device interactions, including the screens and actions, and corresponding natural language instructions. It consists of 715k episodes spanning 30k unique instructions, four versions of Android (v10-13),and eight device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It contains multi-step tasks that require semantic understanding of language and visual context. This dataset poses a new challenge: actions available through the user interface must be inferred from their visual appearance. And, instead of simple UI element-based actions, the action space consists of precise gestures (e.g., horizontal scrolls to operate carousel widgets). We organize our dataset to encourage robustness analysis of device-control systems, i.e., how well a system performs in the presence of new task descriptions, new applications, or new platform versions. We develop two agents and report performance across the dataset. The dataset is available at https://github.com/google-research/google-research/tree/master/android_in_the_wild.