RIBOSPAN: 面向多功能RNA建模的长上下文RNA基础模型

摘要

全长RNA，特别是信使RNA，其长度往往超过现有RNA基础模型预训练时使用的上下文长度，这限制了在单核苷酸分辨率下对完整转录本的建模。我们提出RIBOSPAN，一个包含16.1亿参数的双向RNA基础模型，其原生预训练上下文长度可达10,240个核苷酸。RIBOSPAN结合了密集双向自注意力、单核苷酸分词以及注意力隔离的序列打包技术，实现了对完整长RNA的高分辨率建模。我们通过核苷酸重构、受控长上下文表征基准测试以及冻结RNA类型表征分析来评估该模型。原生10K预训练在10,240个token下保持了强大的重构能力，而采用40%掩码率的持续预训练则在重度损坏条件下改善了序列恢复能力，同时保持了表征质量。长上下文基准测试进一步表明，原生10K模型保持了强上下文响应性和上下文特异性表征分离，同时将扰动引发的表征变化限制在高度局部范围内。推理时的YaRN缩放恢复了大部?#124;因短上下文模型直接外推而丢失的上下文组织结构，但同时也导致了显著更大的远端表征扩散。冻结表征评估进一步证明了RIBOSPAN达到了最先进的RNA表征质量，在多种RNA类型中取得了整体最优性能，并在长RNA上保持了明显优势。基于同一骨干架构，我们开发了一个多维条件约束的离散扩散框架，用于全长mRNA生成和重新设计，其中包括同义密码子扩散，以实现蛋白质序列保持的CDS优化。综上，RIBOSPAN为可迁移的RNA表征学习和全转录本mRNA设计建立了强大的长上下文基础。

English

Full-length RNAs, particularly messenger RNAs, often exceed the context lengths used to pretrain existing RNA foundation models, limiting complete-transcript modeling at single-nucleotide resolution. We present RIBOSPAN, a 1.61-billion-parameter bidirectional RNA foundation model natively pretrained with context lengths up to 10,240 nt. RIBOSPAN combines dense bidirectional self-attention, single-nucleotide tokenization, and attention-isolated sequence packing to enable high-resolution modeling of complete long RNAs. We evaluate the model through nucleotide reconstruction, a controlled long-context representation benchmark, and frozen RNA-type representation analysis. Native 10K pretraining preserves strong reconstruction at 10,240 tokens, while continued pretraining with 40% masking improves recovery under heavy corruption while preserving representation quality. The long-context benchmark further shows that native 10K models maintain strong contextual responsiveness and context-specific representation separation while keeping perturbation-induced representation changes highly localized. Inference-time YaRN scaling recovers much of the contextual organization lost by direct extrapolation of short-context models, but induces substantially greater distal representation diffusion. Frozen-representation evaluations further demonstrate state-of-the-art RNA representation quality, with RIBOSPAN achieving the strongest overall performance across diverse RNA types and retaining a clear advantage on long RNAs. Building on the same backbone, we develop a multidimensionally conditioned discrete-diffusion framework for full-length mRNA generation and redesign, including synonymous-codon diffusion for protein-preserving CDS optimization. Together, RIBOSPAN establishes a powerful long-context foundation for transferable RNA representation learning and full-transcript mRNA design.