基因组下一标记预测器具备上下文学习能力

摘要

情境学习（ICL）——即模型从输入中的示例推断并应用抽象模式的能力——已在基于人类文本进行下一词预测训练的大语言模型中得到广泛研究。事实上，先前研究常将这种涌现能力归因于人类语言独特的统计特性。这引出一个根本性问题：情境学习能否通过纯大规模预测训练，在其他序列领域自然涌现？为探究此问题，我们转向基因组序列这一富含统计结构的替代符号领域。具体而言，我们研究了以下一核苷酸（A/T/C/G）预测为主要训练目标、规模与中型LLM相当的Evo2基因组模型。我们构建了受控实验框架，包含语言和基因组形式下的符号推理任务，从而实现对基因组与语言模型情境学习的直接比较。研究结果表明，随着情境示例数量的增加，基因组模型与语言模型类似，在模式归纳上表现出对数线性增益。据我们所知，这是基因组序列中自然涌现情境学习的首个证据，支持了“情境学习是大规模预测建模在丰富数据上的必然产物”这一假说。这些发现将涌现元学习拓展至语言之外，为构建跨模态的情境学习统一理论指明了方向。

English

In-context learning (ICL) -- the capacity of a model to infer and apply abstract patterns from examples provided within its input -- has been extensively studied in large language models trained for next-token prediction on human text. In fact, prior work often attributes this emergent behavior to distinctive statistical properties in human language. This raises a fundamental question: can ICL arise organically in other sequence domains purely through large-scale predictive training? To explore this, we turn to genomic sequences, an alternative symbolic domain rich in statistical structure. Specifically, we study the Evo2 genomic model, trained predominantly on next-nucleotide (A/T/C/G) prediction, at a scale comparable to mid-sized LLMs. We develop a controlled experimental framework comprising symbolic reasoning tasks instantiated in both linguistic and genomic forms, enabling direct comparison of ICL across genomic and linguistic models. Our results show that genomic models, like their linguistic counterparts, exhibit log-linear gains in pattern induction as the number of in-context demonstrations increases. To the best of our knowledge, this is the first evidence of organically emergent ICL in genomic sequences, supporting the hypothesis that ICL arises as a consequence of large-scale predictive modeling over rich data. These findings extend emergent meta-learning beyond language, pointing toward a unified, modality-agnostic view of in-context learning.

基因组下一标记预测器具备上下文学习能力

Genomic Next-Token Predictors are In-Context Learners

摘要

Support