DecoupleMix: 解耦比例搜索与凸分配实现可扩展的VLM数据配方

摘要

尽管面向视觉语言模型（VLM）的数据整理工作日益活跃，但构建预训练混合数据的公开实践仍高度依赖经验法则：从业者堆叠通过质量过滤的数据集，凭直觉设定跨领域比例，缺乏原则性且可归因的标准来接纳新数据，而前沿方案则始终秘而不宣。我们将数据构建形式化为一个系统的混合优化问题，通过将混合数据解耦为两个正交子问题——跨能力类间比例与同类能力内类内比例——将其转化为可复现的工程准则。针对类间分配，我们采用单变量迭代搜索；针对类内组合，我们运用多维数据集级评分（评估质量与难度），并将选择过程建模为带多样性目标的带约束凸优化问题。DecoupleMix框架提供两项关键能力：指导下一步数据采集方向，以及将数据集验证转化为可控可归因的实验。实验表明，我们的方法持续超越经验基线方法。此外，在小规模代理上发现的最优比例可直接迁移至更大规模场景而无需重新调参。仅使用800亿额外多模态持续预训练token，我们的VLM即可与使用显著更大多模态预算训练的强开源模型相匹敌。

English

While data curation for Vision Language Models (VLMs) is increasingly active, public practice for constructing pretraining mixtures remains largely heuristic: practitioners stack datasets that pass quality filters, set cross-domain ratios by intuition, and lack a principled, attributable criterion for admitting new data, while frontier recipes remain undisclosed. We formulate data construction as a systematic mixture-optimization problem and turn it into a reproducible engineering discipline by decoupling the mixture into two orthogonal sub-problems: inter-class ratios across capabilities and intra-class ratios within a category. For inter-class allocation, we use a single-variable iterative search; for intra-class composition, we apply a multidimensional, dataset-level assessment scoring Quality and Difficulty, and formulate selection as a constrained convex optimization with a diversity objective. The DecoupleMix framework delivers two critical capabilities: guiding what data to collect next and rendering dataset validation a controlled, attributable experiment. Experiments show our approach consistently surpasses heuristic baselines. Moreover, optimal ratios discovered on small-scale proxies transfer seamlessly to larger scales without retuning. Using 80B additional multimodal continue-pretraining tokens, our VLM is competitive with strong open-source models trained with substantially larger multimodal budgets.