階層式語言模型的動態多位元組預測

摘要

位元組層級的分層語言模型（LMs）近年來已成為使用子詞分詞之主流對應模型的一種穩健替代方案。然而，一次生成一個位元組仍是推論速度的瓶頸。為了解決此問題，我們引入了多位元組預測（MBP），透過並行生成多個位元組來加速推論，同時將性能影響降至最低且無需增加額外參數。MBP建立在流行的多詞元預測（MTP）範式之上，並具備兩項關鍵創新。首先，我們引入了可變長度的預測視窗，使其與分層語言模型的潛在詞元或分段對齊。其次，我們實作了一種新穎的注意力遮罩機制，使並行位元組預測得以在不違反因果性的前提下實現。我們證明，多位元組預測在多種生成任務中達到了帕累托最優權衡，包括指令遵循、問答、摘要與機器翻譯，在性能與推論吞吐量之間實現了最佳取捨。

English

Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains a bottleneck for inference speed. To address this, we introduce multi-byte prediction (MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters. MBP builds on the popular multi-token prediction (MTP) paradigm with two crucial innovations. First, we introduce a variable-length prediction window that aligns with the latent tokens, or segments, of a hierarchical LM. Second, we implement a novel attention-masking scheme that enables parallel byte prediction without violating causality. We show that multi-byte prediction strikes a Pareto-optimal trade-off across multiple generative tasks, instruction following, question answering, summarization, and machine translation, achieving the best trade-off between performance and inference throughput.