超越语言模型：字节模型是数字世界的模拟器。

摘要

传统的深度学习经常忽视字节，这是数字世界的基本单位，所有形式的信息和操作都是以二进制格式编码和操作的。受自然语言处理中下一个标记预测成功的启发，我们引入了bGPT，这是一个具有下一个字节预测的模型，用于模拟数字世界。bGPT在各种模态中，包括文本、音频和图像方面的性能与专用模型相匹敌，并为预测、模拟和诊断算法或硬件行为提供了新的可能性。它几乎无缺地复制了将符号音乐数据转换的过程，在将ABC记谱转换为MIDI格式时，实现了每字节0.0011比特的低错误率。此外，bGPT在模拟CPU行为方面表现出色，执行各种操作的准确率超过99.99%。利用下一个字节预测，像bGPT这样的模型可以直接从大量的二进制数据中学习，有效地模拟数字世界复杂的模式。

English

Traditional deep learning often overlooks bytes, the basic units of the digital world, where all forms of information and operations are encoded and manipulated in binary format. Inspired by the success of next token prediction in natural language processing, we introduce bGPT, a model with next byte prediction to simulate the digital world. bGPT matches specialized models in performance across various modalities, including text, audio, and images, and offers new possibilities for predicting, simulating, and diagnosing algorithm or hardware behaviour. It has almost flawlessly replicated the process of converting symbolic music data, achieving a low error rate of 0.0011 bits per byte in converting ABC notation to MIDI format. In addition, bGPT demonstrates exceptional capabilities in simulating CPU behaviour, with an accuracy exceeding 99.99% in executing various operations. Leveraging next byte prediction, models like bGPT can directly learn from vast binary data, effectively simulating the intricate patterns of the digital world.

超越语言模型：字节模型是数字世界的模拟器。

Beyond Language Models: Byte Models are Digital World Simulators

摘要

Support