後訓練如何塑造生物推理模型

摘要

生物學科學推理模型結合了語言模型與基於多模態生物數據（包括DNA、RNA與蛋白質）訓練的基礎模型。這類模型通過後訓練階段構建，然而每個階段如何塑造推理能力與泛化性能，目前仍缺乏深入理解。本研究探討後訓練何時能提升性能，以及何時會導致過度專業化。在基因組學、轉錄組學與蛋白質組學領域，我們在可控變因下（包括主幹模型、持續預訓練、監督微調與強化學習），訓練並評估超過100個生物推理模型，同時衡量其域內與域外表現。研究發現，每個後訓練階段並非帶來均勻的效能提升，而是以不同方式重塑泛化能力：持續預訓練通過使模型與生物語言對齊來改善下游性能；監督微調能穩定提升域內表現，但域外表現會先達到峰值，隨後隨模型過度擬合訓練分佈而下降；當在強監督微調檢查點上結合對齊獎勵進行強化學習時，可改善域外表現並部分恢復泛化能力。這些結果表明，生物推理能力並非隨額外監督或計算資源的增加而單調提升，其表現取決於訓練階段的組合方式。在固定後訓練預算下，最優的域內-域外權衡來自於簡短的監督微調、較多的強化學習配置，以及各階段間不對稱的適應能力。

English

Scientific reasoning models for biology combine language models with foundation models trained on multimodal biological data, including DNA, RNA, and proteins. These models are built through post-training, yet how each stage shapes reasoning and generalization remains poorly understood. We study when post-training improves performance and when it induces over-specialization. Across genomics, transcriptomics, and proteins, we train and evaluate more than 100 biological reasoning models under controlled variation in backbone, continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL), measuring both in-domain (ID) and out-of-domain (OOD) performance. We find that each post-training stage reshapes generalization in a distinct way rather than contributing uniform gains. CPT improves downstream performance by aligning models with biological language. SFT consistently increases ID performance but causes OOD performance to peak early and decline as models fit the training distribution. RL, when applied to strong SFT checkpoints with aligned rewards, improves OOD performance and partially recovers generalization. These results show that biological reasoning does not improve monotonically with additional supervision or compute. Instead, performance depends on how training stages are composed. Under fixed post-training budgets, the strongest ID-OOD trade-off comes from brief SFT, larger RL allocations, and asymmetric adaptation capacity across stages.