通过基座模型协同优化生成内容的多样性与质量

摘要

對齊技術在提升大型語言模型（LLMs）輸出質量的同時，卻以多樣性喪失為代價，導致多次生成結果高度相似。我們提出基礎對齊模型協作框架（BACo），這是一種推理階段的詞元級模型協作框架，通過動態結合基礎LLM與其對齊版本來優化多樣性與質量。受前人研究（Fei et al., 2025）啟發，BACo採用路由策略，根據下一詞元預測的不確定性及預測內容的語義角色，逐詞元決策解碼來源模型。現有的多樣性提升方法（如重訓練、提示工程、多重採樣）雖能改善多樣性，但往往犧牲質量或需耗費大量解碼/訓練成本。相比之下，BACo僅需單次推理即可事後兼顧高多樣性與高質量，並具備強可控性。我們在三大開放式生成任務中驗證了多種路由策略，基於涵蓋多樣性與質量的13項指標，BACo持續超越現有推理階段基準方法。採用最佳路由策略時，BACo實現了多樣性與質量21.3%的綜合提升，人類評估亦印證此改進。結果表明基礎模型與對齊模型的協作能有效優化並控制多樣性與質量。

English

Alignment has greatly improved large language models (LLMs)' output quality at the cost of diversity, yielding highly similar outputs across generations. We propose Base-Aligned Model Collaboration (BACo), an inference-time token-level model collaboration framework that dynamically combines a base LLM with its aligned counterpart to optimize diversity and quality. Inspired by prior work (Fei et al., 2025), BACo employs routing strategies that determine, at each token, from which model to decode based on next-token prediction uncertainty and predicted contents' semantic role. Prior diversity-promoting methods, such as retraining, prompt engineering, and multi-sampling methods, improve diversity but often degrade quality or require costly decoding or post-training. In contrast, BACo achieves both high diversity and quality post hoc within a single pass, while offering strong controllability. We explore a family of routing strategies, across three open-ended generation tasks and 13 metrics covering diversity and quality, BACo consistently surpasses state-of-the-art inference-time baselines. With our best router, BACo achieves a 21.3% joint improvement in diversity and quality. Human evaluations also mirror these improvements. The results suggest that collaboration between base and aligned models can optimize and control diversity and quality.

通过基座模型协同优化生成内容的多样性与质量

Optimizing Diversity and Quality through Base-Aligned Model Collaboration

摘要

Support