機構報紙管線：從歷史報紙中提取數十億高品質詞元

摘要

歷史報紙是公共生活的豐富紀錄，但其版面編排密集、不規則，且時有雜訊干擾，使得以計算方式存取這些材料既具挑戰性，也受限甚多。我們提出一套與波士頓公共圖書館（Boston Public Library）共同設計的模組化系統——Institutional Newspapers Pipeline（機構報紙管線），旨在從歷史報紙掃描中萃取高品質的結構化資料集。此管線在架構上確保每個步驟都可解釋且可自訂，同時使整體管線的運算需求保持節約，足以運行於工作站等級的硬體之上。管線會對每份掃描執行多步驟處理：先將掃描分割為與類型無關的個別圖像區塊（crops），並對每個產生的區塊執行光學字元辨識（OCR）；隨後再對每個區塊進行文本分析、類型分類、閱讀順序偵測、命名實體辨識、主題分類、語言偵測，並預先計算及生成嵌入向量。我們已將此管線應用於波士頓公共圖書館的部分館藏，並以開放資料集形式發布結果。該 OCR 輸出包含163億個 o200k_base 詞元，涵蓋8,310萬個獨立圖像區塊，內容擷取自1,473,635份於1795年至1930年間出版的公有領域報紙掃描。本報告說明每個處理步驟所使用的方法、我們訓練的小型模型，以及在此過程中所收集的評估結果與資料集規模量測數據。本報告伴隨管線、模型與資料集的發布，並將此項工作定位為朝向從數千萬份報紙掃描中取得高品質資料所邁出的重要一步。

English

Historical newspapers are an abundant record of public life, but their dense, irregular and sometimes noisy layouts make computational access to these materials both challenging and limited. We present the Institutional Newspapers Pipeline, a modular system we jointly designed with Boston Public Library to extract high-quality, structured datasets from historical newspaper scans. It was architected so that each step remains interpretable and customizable, and so that the pipeline as a whole remains computationally frugal enough to run on workstation-level hardware. The pipeline runs each scan through a multi-step process: it segments scans into individual type-agnostic crops and performs OCR on each resulting segment before then performing text analysis, type classification, reading order detection, named entities recognition, subject classification, language detection, and pre-computed embeddings generation on every crop. We ran this pipeline against a portion of Boston Public Library's holdings and released the results as an open dataset. The optical character recognition (OCR) output represents 16.3 billion o200k_base tokens across 83.1 million individual crops, extracted from 1,473,635 public domain newspaper scans published between 1795 and 1930. This report describes our methods for each processing step, the small models we trained, as well as the evaluation results and dataset-scale measurements we collected in the process. It accompanies the release of the pipeline, models, and dataset. We position this work as a substantial step towards unlocking high-quality data from tens of millions of newspaper scans.