表徵網路規模大型語言模型預訓練資料中的敘事內容

摘要

網絡規模的大型語言模型預訓練語料庫的敘事構成在很大程度上仍未被探索，儘管敘事是人類溝通的基本模式。我們首次對Dolma（一個包含3兆詞元的開放預訓練語料庫）中的敘事特徵進行了細粒度研究。借鑒敘事理論，我們設計了一個框架，涵蓋三個核心敘事元素（能動性、場景與事件），並將其操作化為11個可解釋維度。在抽樣並標註400篇多樣化段落後，我們微調並驗證了NarraBERT——一個基於RoBERTa的模型，用於細粒度敘事預測。我們將NarraBERT應用於300萬篇段落，產生了新數據集NarraDolma。我們發現：(i) 敘事結構可在極度異質的數據中大規模測量；(ii) 我們揭示了網路文本背後存在連續的多維敘事結構；(iii) 敘事品質在預訓練來源與主題間分佈不均，而現行的數據篩選實務既未測量亦未考慮此現象。我們的框架、數據集與分析為理解敘事品質如何在大型語言模型預訓練資料中分佈，以及研究資料構成如何影響敘事推理任務奠定了基礎。我們公開發佈NarraDolma與NarraBERT。

English

The narrative composition of web-scale LLM pretraining corpora remains largely unexplored even though narrative is a fundamental mode of human communication. We present the first fine-grained study of narrative features in Dolma, a 3-trillion-token open pretraining corpus. Drawing on narrative theory, we design a framework spanning three core narrative elements (agency, setting, and events) operationalized as 11 interpretable dimensions. After sampling and annotating a diverse set of 400 passages, we finetune and validate NarraBERT, a RoBERTa-based model for fine-grained narrative prediction. We apply NarraBERT to 3M passages, resulting in a new dataset, NarraDolma. We find (i) narrative structure is measurable at scale across extremely heterogeneous data, (ii) we uncover a continuous, multidimensional narrative structure underlying web text, and (iii) narrative qualities are unequally distributed across pretraining sources and topics in ways that current curation practices neither measure nor account for. Our framework, dataset, and analyses provide a foundation for understanding how narrative qualities are distributed in LLM pretraining data and for studying how data composition affects narrative reasoning tasks. We publicly release NarraDolma and NarraBERT.