NeuralOS：迈向基于神经生成模型的操作系统仿真

摘要

我们推出NeuralOS，这是一个神经框架，通过直接预测屏幕帧来模拟操作系统的图形用户界面（GUI），响应用户输入如鼠标移动、点击和键盘事件。NeuralOS结合了循环神经网络（RNN）——用于追踪计算机状态——与基于扩散的神经渲染器，后者负责生成屏幕图像。该模型在Ubuntu XFCE的大规模录制数据集上进行训练，这些数据包括随机生成的交互和由AI代理产生的真实交互。实验表明，NeuralOS成功渲染出逼真的GUI序列，准确捕捉鼠标交互，并可靠预测如应用程序启动等状态转换。尽管精确建模细粒度的键盘交互仍具挑战，但NeuralOS为未来人机交互系统创建完全自适应、生成式的神经界面迈出了重要一步。

English

We introduce NeuralOS, a neural framework that simulates graphical user interfaces (GUIs) of operating systems by directly predicting screen frames in response to user inputs such as mouse movements, clicks, and keyboard events. NeuralOS combines a recurrent neural network (RNN), which tracks computer state, with a diffusion-based neural renderer that generates screen images. The model is trained on a large-scale dataset of Ubuntu XFCE recordings, which include both randomly generated interactions and realistic interactions produced by AI agents. Experiments show that NeuralOS successfully renders realistic GUI sequences, accurately captures mouse interactions, and reliably predicts state transitions like application launches. Although modeling fine-grained keyboard interactions precisely remains challenging, NeuralOS offers a step toward creating fully adaptive, generative neural interfaces for future human-computer interaction systems.

NeuralOS：迈向基于神经生成模型的操作系统仿真

NeuralOS: Towards Simulating Operating Systems via Neural Generative Models

摘要

Support