机构报纸流水线：从历史报纸中提取数十亿高质量词元

摘要

历史报纸是公共生活的丰富记录，但其密集、不规则且有时杂乱的版面使得对这些材料的计算访问既具挑战性又十分有限。我们提出了机构报纸流水线（Institutional Newspapers Pipeline），这是一个我们与波士顿公共图书馆联合设计的模块化系统，用于从历史报纸扫描件中提取高质量、结构化的数据集。该系统的架构设计确保每个步骤保持可解释性和可定制性，同时整个流水线在计算上足够精简，能够在工作站级硬件上运行。该流水线对每份扫描件执行多步骤处理：将扫描件分割为独立的、不区分字体的图像块（crops），对每个分割结果执行OCR，随后对每个图像块进行文本分析、字体分类、阅读顺序检测、命名实体识别、主题分类、语言检测以及预计算嵌入的生成。我们对波士顿公共图书馆馆藏的一部分运行了该流水线，并将结果作为开放数据集发布。光学字符识别（OCR）输出涵盖163亿个o200k_base词元，来自8310万个独立图像块，这些内容提取自1795年至1930年间出版的1,473,635份公有领域报纸扫描件。本报告描述了我们在每个处理步骤中使用的方法、训练的小型模型，以及在此过程中收集的评估结果和数据集规模测量数据。本报告随流水线、模型和数据集的发布一同提供。我们将这项工作定位为向解锁数千万份报纸扫描件中的高质量数据迈出的重要一步。

English

Historical newspapers are an abundant record of public life, but their dense, irregular and sometimes noisy layouts make computational access to these materials both challenging and limited. We present the Institutional Newspapers Pipeline, a modular system we jointly designed with Boston Public Library to extract high-quality, structured datasets from historical newspaper scans. It was architected so that each step remains interpretable and customizable, and so that the pipeline as a whole remains computationally frugal enough to run on workstation-level hardware. The pipeline runs each scan through a multi-step process: it segments scans into individual type-agnostic crops and performs OCR on each resulting segment before then performing text analysis, type classification, reading order detection, named entities recognition, subject classification, language detection, and pre-computed embeddings generation on every crop. We ran this pipeline against a portion of Boston Public Library's holdings and released the results as an open dataset. The optical character recognition (OCR) output represents 16.3 billion o200k_base tokens across 83.1 million individual crops, extracted from 1,473,635 public domain newspaper scans published between 1795 and 1930. This report describes our methods for each processing step, the small models we trained, as well as the evaluation results and dataset-scale measurements we collected in the process. It accompanies the release of the pipeline, models, and dataset. We position this work as a substantial step towards unlocking high-quality data from tens of millions of newspaper scans.