神經操作系統：基於神經生成模型的操作系統模擬探索

摘要

我们引入了NeuralOS，这是一个通过直接预测屏幕帧来模拟操作系统图形用户界面（GUI）的神经框架，其响应包括鼠标移动、点击和键盘事件在内的用户输入。NeuralOS结合了追踪计算机状态的循环神经网络（RNN）与基于扩散的神经渲染器，后者负责生成屏幕图像。该模型在Ubuntu XFCE的大规模记录数据集上进行训练，这些数据集涵盖了随机生成的交互以及由AI代理产生的真实交互。实验表明，NeuralOS成功渲染了逼真的GUI序列，准确捕捉了鼠标交互，并可靠地预测了如应用程序启动等状态转换。尽管精确建模细粒度的键盘交互仍具挑战，但NeuralOS为未来人机交互系统创建完全自适应、生成式的神经界面迈出了重要一步。

English

We introduce NeuralOS, a neural framework that simulates graphical user interfaces (GUIs) of operating systems by directly predicting screen frames in response to user inputs such as mouse movements, clicks, and keyboard events. NeuralOS combines a recurrent neural network (RNN), which tracks computer state, with a diffusion-based neural renderer that generates screen images. The model is trained on a large-scale dataset of Ubuntu XFCE recordings, which include both randomly generated interactions and realistic interactions produced by AI agents. Experiments show that NeuralOS successfully renders realistic GUI sequences, accurately captures mouse interactions, and reliably predicts state transitions like application launches. Although modeling fine-grained keyboard interactions precisely remains challenging, NeuralOS offers a step toward creating fully adaptive, generative neural interfaces for future human-computer interaction systems.

神經操作系統：基於神經生成模型的操作系統模擬探索

NeuralOS: Towards Simulating Operating Systems via Neural Generative Models

摘要

Support