通过基座模型协同优化实现多样性提升与质量保障

摘要

对齐技术虽显著提升了大语言模型（LLM）的输出质量，却以牺牲多样性为代价，导致多次生成的内容高度趋同。我们提出基座-对齐模型协同框架（BACo），该推理阶段令牌级模型协作框架通过动态结合基座LLM与其对齐版本，以优化多样性与质量。受前人研究（Fei等，2025）启发，BACo采用路由策略，根据下一令牌预测的不确定性及预测内容的语义角色，逐令牌判定解码来源模型。现有提升多样性的方法（如重训练、提示工程、多采样等）虽能改善多样性，但往往导致质量下降或需要高昂的解码/训练后成本。相比之下，BACo在单次推理中即可实现高质量与高多样性的统一，同时具备强可控性。我们在三类开放生成任务中系统评估了多种路由策略，涵盖13项多样性与质量指标。实验表明BACo持续优于最先进的推理阶段基线方法：采用最优路由策略时，其多样性与质量综合提升率达21.3%。人工评估结果亦验证了上述改进。研究表明，基座模型与对齐模型的协同机制能有效优化并控制生成内容的多样性与质量。

English

Alignment has greatly improved large language models (LLMs)' output quality at the cost of diversity, yielding highly similar outputs across generations. We propose Base-Aligned Model Collaboration (BACo), an inference-time token-level model collaboration framework that dynamically combines a base LLM with its aligned counterpart to optimize diversity and quality. Inspired by prior work (Fei et al., 2025), BACo employs routing strategies that determine, at each token, from which model to decode based on next-token prediction uncertainty and predicted contents' semantic role. Prior diversity-promoting methods, such as retraining, prompt engineering, and multi-sampling methods, improve diversity but often degrade quality or require costly decoding or post-training. In contrast, BACo achieves both high diversity and quality post hoc within a single pass, while offering strong controllability. We explore a family of routing strategies, across three open-ended generation tasks and 13 metrics covering diversity and quality, BACo consistently surpasses state-of-the-art inference-time baselines. With our best router, BACo achieves a 21.3% joint improvement in diversity and quality. Human evaluations also mirror these improvements. The results suggest that collaboration between base and aligned models can optimize and control diversity and quality.

通过基座模型协同优化实现多样性提升与质量保障

Optimizing Diversity and Quality through Base-Aligned Model Collaboration

摘要

Support