動態潛在路由

摘要

我們研究時變獎勵函數的馬可夫決策過程（MDP）中子策略的時間序列組合。我們提出通用迪傑斯特拉搜索（GDS），並證明全局最優的目標達成策略可透過中間最優子策略的時間組合來還原。受GDS中「搜索、選擇、更新」原則的啟發，我們提出動態潛在路由（DLR），這是一種語言模型的後訓練方法，透過單一訓練階段中的動態搜索，共同學習離散潛在編碼、路由策略與模型參數。在低資料微調場景中，DLR在四個資料集與六個模型上匹配或超越監督式微調，平均提升+6.6個百分點，而先前的離散潛在基線方法則 consistently 表現不及SFT。機制分析與目標式程式碼消融實驗顯示，DLR學習到具有明確因果角色的結構化路由行為。

English

We investigate the temporal concatenation of sub-policies in Markov Decision Processes (MDP) with time-varying reward functions. We introduce General Dijkstra Search (GDS), and prove that globally optimal goal-reaching policies can be recovered through temporal composition of intermediate optimal sub-policies. Motivated by the "search, select, update" principle underlying GDS, we propose Dynamic Latent Routing (DLR), a language-model post-training method that jointly learns discrete latent codes, routing policies, and model parameters through dynamic search in a single training stage. In low-data fine-tuning settings, DLR matches or outperforms supervised fine-tuning across four datasets and six models, achieving a mean gain of +6.6 percentage points, while prior discrete-latent baselines consistently underperform SFT. Mechanistic analyses and targeted code ablations show that DLR learns structured routing behaviors with distinct causal roles.