理解數據時間性對大型語言模型預訓練的影響

摘要

大语言模型（LLMs）通常基于打乱顺序的语料库进行训练，这导致模型的知识在训练时固定不变，且其时间关联性难以被充分理解。本研究重点关注预训练动态对模型获取时间敏感事实性知识的影响，尤其聚焦于数据排序问题。我们的主要贡献有两方面。首先，我们构建了一个包含超过7,000个时间锚定问题的综合基准测试，并提出了一套评估协议，能够分析模型是否将事实与其对应的时间段正确关联。其次，我们利用按时间顺序排列的Common Crawl快照对60亿参数模型进行预训练，并将其与标准的乱序预训练模型进行对比。结果表明，按时间顺序训练的模型在通用语言理解与常识知识方面与乱序基线模型表现相当，同时始终展现出更即时、更精确的时间化知识。按时间顺序预训练可提升事实的新鲜度，而乱序预训练则更倾向于老旧数据，这可能是由于事实重复率更高所致。这些发现，连同我们在https://github.com/kyutai-labs/kairos 发布的代码、检查点及数据集（https://huggingface.co/collections/kyutai/kairos ），为LLMs持续学习的未来研究提供了基础。

English

Large language models (LLMs) are typically trained on shuffled corpora, yielding models whose knowledge is frozen at train time and whose temporal grounding remains poorly understood. In this work, we study the impact of pre-training dynamics on the acquisition of time-sensitive factual knowledge, focusing specifically on data ordering. Our main contributions are twofold. First, we introduce a comprehensive benchmark of over 7,000 temporally grounded questions and an evaluation protocol that enables analysis of whether models correctly associate facts with their corresponding time periods. Second, we pretrain 6B-parameter models on temporally ordered Common Crawl snapshots and compare them against standard shuffled pre-training. Our results show that sequentially trained models match shuffled baselines on general language understanding and common knowledge while consistently exhibiting more up-to-date and temporally precise knowledge. Temporally ordered pre-training yields improved factual freshness, while shuffled pre-training peaks on older data, possibly due to increased factual repetition. These findings, along with the release of our code at https://github.com/kyutai-labs/kairos , checkpoints, and datasets at https://huggingface.co/collections/kyutai/kairos provide a foundation for future research on continual learning for LLMs.