離散策略最佳化的引導對比標記信用分配

摘要

基於群體優勢的強化學習方法（如GRPO和DAPO）已在多個領域展現出優異性能，包括數學推理與文字到圖像生成。然而，此類方法依賴樣本層級的獎勵，導致無法捕捉細粒度的詞元層級貢獻，因均勻分配信用至所有詞元而引入關鍵限制。為解決此問題，我們提出引導對比策略優化（Guidance Contrastive Policy Optimization, GCPO），這是一種新穎演算法，能透過對比正負提示下的模型預測，實現逐詞元的信用分配。GCPO並非均勻廣播樣本層級的優勢，而是將詞元層級的優勢分配至與這些對比預測差異成比例的程度，從而提供更精確且更具資訊性的學習訊號。實驗結果顯示，GCPO能強調語義相關區域，例如在文字到圖像生成中對齊文字提示的視覺區域，以及在思維鏈任務中推理軌跡內的關鍵關鍵詞。透過廣泛實驗，GCPO在文字到圖像生成與思維鏈推理基準上均持續優於GRPO和DAPO基線，證實其作為離散策略學習中通用且可擴展的最佳化策略之有效性。

English

Group-advantage-based reinforcement learning methods, such as GRPO and DAPO, have demonstrated strong performance across diverse domains, including mathematical reasoning and text-to-image generation. However, their reliance on sample-level rewards introduces a key limitation as uniform credit assignment across all tokens fails to capture fine-grained, token-level contributions. To address this issue, we propose Guidance Contrastive Policy Optimization (GCPO), a novel algorithm that enables per-token credit assignment by contrasting model predictions under positive and negative prompts. Rather than uniformly broadcasting sample-level advantages, GCPO assigns token-level advantages proportional to the difference between these contrastive predictions, allowing more precise and informative learning signals. Empirically, we find that GCPO emphasizes semantically relevant regions such as visual areas aligned with textual prompts in text-to-image generation, and critical keywords within reasoning traces for chain-of-thought tasks. Through extensive experiments, GCPO consistently outperforms GRPO and DAPO baselines on both text-to-image generation and chain-of-thought reasoning benchmarks, demonstrating its effectiveness as a general and scalable optimization strategy for discrete policy learning.