信任區域 Q 伴隨匹配

摘要

對於預訓練流策略的離策略強化學習，由於多步採樣過程導致的優化不穩定性，仍然具有挑戰性。近期，透過將問題重新表述為具有學習評論家的無記憶隨機最優控制問題，Q學習與伴隨匹配（QAM）解決了這一問題。然而，QAM繼承了評論家引導改進的一個基本脆弱性：當評論家的條件不佳時，微小的評論家誤差會被放大，常常導致模型崩潰。本文介紹了信賴區域Q伴隨匹配（TRQAM），這是一種穩定的離策略微調算法，通過投影對偶下降自適應地控制與預訓練流策略之間的路徑空間KL散度。具體而言，我們優化SOC動態中的信賴區域參數λ，並在理論上證明路徑空間KL散度可以用λ的閉式函數來表示。因此，我們的方法可以精確控制與預訓練流策略的實際偏差，從而實現穩定的離策略強化學習。透過在50個OGBench任務上的實驗，TRQAM在離線強化學習和離線到線上強化學習中均持續優於先前的方法。特別是，TRQAM在離線強化學習中達到了68%的總體成功率，相對於最強的基線方法（46%）有顯著提升。

English

Off-policy reinforcement learning of pretrained flow policies remains challenging due to the instability of optimization arising from the multi-step sampling process. Recently, Q-learning with Adjoint Matching (QAM) addressed this issue by reformulating into a memoryless stochastic optimal control (SOC) problem with a learned critic. However, QAM inherits a fundamental fragility of critic-guided improvement: small critic errors are amplified when critics are ill-conditioned, often leading to model collapse. This paper introduces Trust Region Q-Adjoint Matching (TRQAM), a stable off-policy fine-tuning algorithm that adaptively controls the path-space KL with pretrained flow policies through projected dual descent. Specifically, we optimize the trust-region parameter λ in SOC dynamics, and theoretically show that the path-space KL can be represented by a closed-form function of λ. As a result, our method can precisely control the exact deviation from pretrained flow policies, achieving stable off-policy RL. Through experiments on 50 OGBench tasks, TRQAM consistently outperforms prior arts in both offline RL and offline-to-online RL. In particular, TRQAM achieves an overall success rate of 68% in offline RL, substantially improves the strongest baseline at 46%.