动态多字节预测与层次化语言模型

摘要

字节级分层语言模型（LMs）近年来已成为使用子词分词的主流模型的一种稳健替代方案。然而，逐字节生成仍是推理速度的瓶颈。为解决这一问题，我们引入了多字节预测（MBP），它能够并行生成多个字节，在几乎不影响性能且不增加额外参数的情况下加速推理。MBP建立在流行的多词元预测（MTP）范式之上，并包含两项关键创新。首先，我们引入了一个可变长度预测窗口，使其与分层LM的潜在词元（即分段）对齐。其次，我们实现了一种新颖的注意力掩码方案，使得在不违反因果性的前提下实现并行字节预测成为可能。我们证明，多字节预测在多项生成任务中实现了帕累托最优的权衡，包括指令跟随、问答、摘要生成和机器翻译，在性能与推理吞吐量之间取得了最佳平衡。

English

Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains a bottleneck for inference speed. To address this, we introduce multi-byte prediction (MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters. MBP builds on the popular multi-token prediction (MTP) paradigm with two crucial innovations. First, we introduce a variable-length prediction window that aligns with the latent tokens, or segments, of a hierarchical LM. Second, we implement a novel attention-masking scheme that enables parallel byte prediction without violating causality. We show that multi-byte prediction strikes a Pareto-optimal trade-off across multiple generative tasks, instruction following, question answering, summarization, and machine translation, achieving the best trade-off between performance and inference throughput.