LUMOS：為以無障礙為基礎的AI代理設計的語義作業系統層

摘要

當前作業系統所暴露的介面專為人類使用者最佳化，而非為AI代理。人類受惠於像素、圖示、視窗、視覺分群、滑鼠移動及鍵盤快捷鍵；而AI代理需要的則是緊湊的語意狀態、扎根動作與可靠的回饋。因此，許多電腦使用代理被迫解讀螢幕截圖、OCR輸出及視覺裁切，導致高額的token成本、視覺模糊性、延遲及座標不確定性。本文提出LUMOS（Language Model Unified Machine-Readable Operating-System Semantics，語言模型統一機器可讀作業系統語意），作為AI代理與作業系統之間的語意互動層。LUMOS將原生的輔助功能元數據與瀏覽器UI結構轉換為機器可讀的語意藍圖，其中包含穩定的識別碼、角色、名稱、值、邊界及動作賦能。此外，LUMOS也支援即時語意指標扎根，透過作業系統自動化API查詢游標下方或附近的UI元素。大型語言模型（LLM）隨後透過以輔助功能為基礎的觀察-動作迴圈來執行操作，使用受限的可視UI基本元素，而非應用程式專用腳本。LUMOS並不宣稱取代視覺代理；而是主張在作業系統已提供語意結構的場景下，降低對螢幕截圖的依賴。這些結果指向一條邁向AI原生作業系統與機器可讀互動層的途徑。

English

Current operating systems expose interfaces optimized for human users but not for AI agents. Humans benefit from pixels, icons, windows, visual grouping, mouse movement, and keyboard shortcuts; AI agents instead need compact semantic state, grounded actions, and reliable feedback. As a result, many computer-use agents are forced to interpret screenshots, OCR output, and visual crops, introducing high token costs, visual ambiguity, latency, and coordinate uncertainty. This paper introduces LUMOS (Language Model Unified Machine-Readable Operating-System Semantics), a semantic interaction layer between AI agents and operating systems. LUMOS converts native accessibility metadata and browser UI structures into machine readable semantic blueprints with stable identifiers, roles, names, values, bounds, and action affordances. It also supports live semantic pointer grounding by querying the UI element under or near the cursor through operating-system automation APIs. An LLM then acts through an accessibility grounded observe act loop using constrained visible-UI primitives rather than application-specific scripts. LUMOS does not claim to replace visual agents; instead, it reduces dependence on screenshots when operating systems already provide semantic structure. These results suggest a path toward AI-native operating systems and machine-readable interaction layers.