DecoupleMix: 解耦比率搜索与凸分配实现可扩展的VLM数据配方

摘要

隨著視覺語言模型（VLM）的資料策展日益活躍，建構預訓練混合集的公開實務仍大致停留於啟發式層面：實務者疊加通過品質過濾的資料集、憑直覺設定跨領域比例、缺乏原則性且可歸因的標準來接納新資料，而前沿配方則保持不公開。我們將資料建構系統性地表述為一個混合優化問題，並通過將混合集解耦為兩個正交的子問題——跨能力類別的間比例與單一類別內的內部比例——將其轉化為可再現的工程學科。對於類別間分配，我們採用單變量迭代搜索；對於類別內組成，我們應用基於資料集層級的多維度評估，評分品質與難度，並將選擇表述為帶有多樣性目標的約束凸優化。DecoupleMix框架提供了兩項關鍵能力：引導下一步該收集哪些資料，以及使資料集驗證成為可控且可歸因的實驗。實驗結果顯示，我們的方法始終優於啟發式基線。此外，在小規模代理上發現的最優比例可直接遷移至更大規模，無需重新調整。利用額外800億個多模態持續預訓練token，我們的VLM在競爭性上可與使用更大多模態預算訓練的強開源模型相媲美。

English

While data curation for Vision Language Models (VLMs) is increasingly active, public practice for constructing pretraining mixtures remains largely heuristic: practitioners stack datasets that pass quality filters, set cross-domain ratios by intuition, and lack a principled, attributable criterion for admitting new data, while frontier recipes remain undisclosed. We formulate data construction as a systematic mixture-optimization problem and turn it into a reproducible engineering discipline by decoupling the mixture into two orthogonal sub-problems: inter-class ratios across capabilities and intra-class ratios within a category. For inter-class allocation, we use a single-variable iterative search; for intra-class composition, we apply a multidimensional, dataset-level assessment scoring Quality and Difficulty, and formulate selection as a constrained convex optimization with a diversity objective. The DecoupleMix framework delivers two critical capabilities: guiding what data to collect next and rendering dataset validation a controlled, attributable experiment. Experiments show our approach consistently surpasses heuristic baselines. Moreover, optimal ratios discovered on small-scale proxies transfer seamlessly to larger scales without retuning. Using 80B additional multimodal continue-pretraining tokens, our VLM is competitive with strong open-source models trained with substantially larger multimodal budgets.