ARIA：以意圖驅動的獎勵聚合訓練語言代理

摘要

大型語言模型（LLMs）使智能體能夠通過自由形式的語言互動執行複雜的推理和決策。然而，在開放式語言行動環境中（例如，談判或提問遊戲），行動空間可以表示為詞元的聯合分佈，從而形成一個指數級大的行動空間。在這樣的空間中採樣行動可能導致極端的獎勵稀疏性，這會帶來巨大的獎勵方差，阻礙有效的強化學習（RL）。為了解決這個問題，我們提出了ARIA，這是一種在意圖空間中聚合獎勵的方法，以實現高效且有效的語言智能體訓練。ARIA旨在將自然語言行動從高維的詞元聯合分佈空間投影到低維的意圖空間，在該空間中語義相似的行動被聚類並分配共享的獎勵。這種意圖感知的獎勵聚合通過密集化獎勵信號來減少獎勵方差，促進更好的策略優化。大量實驗表明，ARIA不僅顯著降低了策略梯度方差，還在四個下游任務中平均帶來了9.95%的顯著性能提升，始終優於離線和在線RL基線。

English

Large language models (LLMs) have enabled agents to perform complex reasoning and decision-making through free-form language interactions. However, in open-ended language action environments (e.g., negotiation or question-asking games), the action space can be formulated as a joint distribution over tokens, resulting in an exponentially large action space. Sampling actions in such a space can lead to extreme reward sparsity, which brings large reward variance, hindering effective reinforcement learning (RL). To address this, we propose ARIA, a method that Aggregates Rewards in Intention space to enable efficient and effective language Agents training. ARIA aims to project natural language actions from the high-dimensional joint token distribution space into a low-dimensional intention space, where semantically similar actions are clustered and assigned shared rewards. This intention-aware reward aggregation reduces reward variance by densifying reward signals, fostering better policy optimization. Extensive experiments demonstrate that ARIA not only significantly reduces policy gradient variance, but also delivers substantial performance gains of an average of 9.95% across four downstream tasks, consistently outperforming offline and online RL baselines.

ARIA：以意圖驅動的獎勵聚合訓練語言代理

ARIA: Training Language Agents with Intention-Driven Reward Aggregation

摘要

Support