ByteFlow：基于自适应字节压缩的无分词器语言建模

摘要

当代语言模型仍依赖固定的预定义子词分词机制。一旦分词器训练完成，模型只能在此固定粒度层面运行，这常导致即使具备强大推理能力的模型也会出现脆弱且反直觉的行为。我们提出ByteFlow Net——一种新型分层架构，完全摒弃分词器，使模型能够自主将原始字节流分割为语义单元。该架构基于潜在表示编码率进行压缩驱动的分割，通过Top-K选择在保持静态计算图的同时生成自适应边界。与依赖人工设计归纳偏置的脆弱启发式自分词方法不同，ByteFlow Net能根据输入内容动态调整内部表示粒度。实验表明，这种基于压缩的分块策略带来显著性能提升，ByteFlow Net在表现上优于基于BPE的Transformer模型及先前字节级架构。这些结果证明端到端的无分词器建模不仅可行且更高效，为开发更具自适应性和信息基础的语言模型开辟了新路径。

English

Modern language models still rely on fixed, pre-defined subword tokenizations. Once a tokenizer is trained, the LM can only operate at this fixed level of granularity, which often leads to brittle and counterintuitive behaviors even in otherwise strong reasoning models. We introduce ByteFlow Net, a new hierarchical architecture that removes tokenizers entirely and instead enables models to learn their own segmentation of raw byte streams into semantically meaningful units. ByteFlow Net performs compression-driven segmentation based on the coding rate of latent representations, yielding adaptive boundaries while preserving a static computation graph via Top-K selection. Unlike prior self-tokenizing methods that depend on brittle heuristics with human-designed inductive biases, ByteFlow Net adapts its internal representation granularity to the input itself. Experiments demonstrate that this compression-based chunking strategy yields substantial performance gains, with ByteFlow Net outperforming both BPE-based Transformers and previous byte-level architectures. These results suggest that end-to-end, tokenizer-free modeling is not only feasible but also more effective, opening a path toward more adaptive and information-grounded language models.

ByteFlow：基于自适应字节压缩的无分词器语言建模

ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer

摘要

Support