RIBOSPAN：用於多功能RNA建模的長上下文RNA基礎模型

摘要

全長RNA，尤其是信使RNA（mRNA），通常超出既有RNA基礎模型預訓練時所使用的上下文長度，限制了在單核苷酸解析度下對完整轉錄本進行建模的能力。我們提出RIBOSPAN，一個具有16.1億參數的雙向RNA基礎模型，其原生預訓練的上下文長度可達10,240個核苷酸（nt）。RIBOSPAN結合了密集雙向自注意力、單核苷酸標記化，以及注意力隔離的序列打包，使完整的長鏈RNA得以進行高解析度建模。我們透過核苷酸重建、受控的長上下文表徵基準測試，以及凍結RNA類型表徵分析來評估該模型。原生10K預訓練在10,240個token下保留了強勁的重建能力，而使用40%遮罩的持續預訓練則在重度破壞下提升恢復能力，同時保持表徵品質。長上下文基準測試進一步顯示，原生10K模型維持了強烈的上下文響應性及上下文特異性的表徵分離，同時使擾動引起的表徵變化保持高度局部化。推論階段的YaRN擴展恢復了短上下文模型直接外推所損失的大部分上下文組織，但卻導致遠端表徵擴散顯著增加。凍結表徵評估進一步展示了最先進的RNA表徵品質，RIBOSPAN在多種RNA類型中取得最強的整體表現，並在長鏈RNA上保持明顯優勢。基於相同的骨幹架構，我們開發了一個多維條件離散擴散框架，用於全長mRNA的生成與重新設計，其中包括用於保留蛋白質的CDS最佳化的同義密碼子擴散。整體而言，RIBOSPAN為可轉移的RNA表徵學習與全轉錄本mRNA設計奠定了強大的長上下文基礎。

English

Full-length RNAs, particularly messenger RNAs, often exceed the context lengths used to pretrain existing RNA foundation models, limiting complete-transcript modeling at single-nucleotide resolution. We present RIBOSPAN, a 1.61-billion-parameter bidirectional RNA foundation model natively pretrained with context lengths up to 10,240 nt. RIBOSPAN combines dense bidirectional self-attention, single-nucleotide tokenization, and attention-isolated sequence packing to enable high-resolution modeling of complete long RNAs. We evaluate the model through nucleotide reconstruction, a controlled long-context representation benchmark, and frozen RNA-type representation analysis. Native 10K pretraining preserves strong reconstruction at 10,240 tokens, while continued pretraining with 40% masking improves recovery under heavy corruption while preserving representation quality. The long-context benchmark further shows that native 10K models maintain strong contextual responsiveness and context-specific representation separation while keeping perturbation-induced representation changes highly localized. Inference-time YaRN scaling recovers much of the contextual organization lost by direct extrapolation of short-context models, but induces substantially greater distal representation diffusion. Frozen-representation evaluations further demonstrate state-of-the-art RNA representation quality, with RIBOSPAN achieving the strongest overall performance across diverse RNA types and retaining a clear advantage on long RNAs. Building on the same backbone, we develop a multidimensionally conditioned discrete-diffusion framework for full-length mRNA generation and redesign, including synonymous-codon diffusion for protein-preserving CDS optimization. Together, RIBOSPAN establishes a powerful long-context foundation for transferable RNA representation learning and full-transcript mRNA design.