LUMOS：一种面向基于无障碍性的AI智能体的语义操作系统层

摘要

当前操作系统所暴露的接口是为人类用户优化设计的，而非面向AI智能体。人类从中受益于像素、图标、窗口、视觉分组、鼠标移动和键盘快捷键；而AI智能体需要的则是紧凑的语义状态、可落地的动作以及可靠的反馈。因此，许多计算机操作智能体被迫通过截图、OCR输出和视觉裁剪来理解界面，这带来了高昂的token成本、视觉模糊性、延迟以及坐标不确定性。本文引入LUMOS（语言模型统一机器可读操作系统语义），这是一个介于AI智能体与操作系统之间的语义交互层。LUMOS将原生无障碍元数据和浏览器UI结构转换为机器可读的语义蓝图，其中包含稳定的标识符、角色、名称、值、边界和动作能力。它还支持通过操作系统自动化API查询光标下方或附近的UI元素，实现实时的语义指针定位。随后，大语言模型通过基于无障碍性构建的“观察-行动”循环，使用受约束的可见UI原语（而非特定于应用的脚本）来执行操作。LUMOS并非试图取代视觉智能体，而是旨在当操作系统本身已提供语义结构时，减少对截图的依赖。这些研究结果表明了一条通往AI原生操作系统和机器可读交互层的可行路径。

English

Current operating systems expose interfaces optimized for human users but not for AI agents. Humans benefit from pixels, icons, windows, visual grouping, mouse movement, and keyboard shortcuts; AI agents instead need compact semantic state, grounded actions, and reliable feedback. As a result, many computer-use agents are forced to interpret screenshots, OCR output, and visual crops, introducing high token costs, visual ambiguity, latency, and coordinate uncertainty. This paper introduces LUMOS (Language Model Unified Machine-Readable Operating-System Semantics), a semantic interaction layer between AI agents and operating systems. LUMOS converts native accessibility metadata and browser UI structures into machine readable semantic blueprints with stable identifiers, roles, names, values, bounds, and action affordances. It also supports live semantic pointer grounding by querying the UI element under or near the cursor through operating-system automation APIs. An LLM then acts through an accessibility grounded observe act loop using constrained visible-UI primitives rather than application-specific scripts. LUMOS does not claim to replace visual agents; instead, it reduces dependence on screenshots when operating systems already provide semantic structure. These results suggest a path toward AI-native operating systems and machine-readable interaction layers.