FLAG-Trader：融合LLM代理與基於梯度的強化學習的金融交易系統

FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading

February 17, 2025

作者: Guojun Xiong, Zhiyang Deng, Keyi Wang, Yupeng Cao, Haohang Li, Yangyang Yu, Xueqing Peng, Mingquan Lin, Kaleb E Smith, Xiao-Yang Liu, Jimin Huang, Sophia Ananiadou, Qianqian Xie

cs.AI

摘要

基於多模態金融數據微調的大型語言模型（LLMs）在各種金融任務中展現了令人印象深刻的推理能力。然而，在互動金融市場（如交易）中，面對多步驟、目標導向的場景時，這些模型往往表現不佳，這類場景需要複雜的代理方法來提升決策質量。為此，我們提出了FLAG-Trader，這是一個統一架構，將語言處理（通過LLMs）與基於梯度的強化學習（RL）策略優化相結合，其中部分微調的LLM作為策略網絡，既利用預訓練知識，又通過參數高效的微調適應金融領域。通過由交易獎勵驅動的策略梯度優化，我們的框架不僅提升了LLM在交易中的表現，還改善了其他金融領域任務的結果。我們提供了廣泛的實證證據來驗證這些改進。

English

Large language models (LLMs) fine-tuned on multimodal financial data have demonstrated impressive reasoning capabilities in various financial tasks. However, they often struggle with multi-step, goal-oriented scenarios in interactive financial markets, such as trading, where complex agentic approaches are required to improve decision-making. To address this, we propose FLAG-Trader, a unified architecture integrating linguistic processing (via LLMs) with gradient-driven reinforcement learning (RL) policy optimization, in which a partially fine-tuned LLM acts as the policy network, leveraging pre-trained knowledge while adapting to the financial domain through parameter-efficient fine-tuning. Through policy gradient optimization driven by trading rewards, our framework not only enhances LLM performance in trading but also improves results on other financial-domain tasks. We present extensive empirical evidence to validate these enhancements.