使用ART微调多模态大语言模型：基于艺术的强化训练

摘要

针对大型语言模型（LLMs），主要有两种参数高效微调（PEFT）技术。低秩适应（LoRA）在LLM层间引入额外权重，而软提示（Soft Prompting）则向LLM输入中添加特定于微调的原始标记。然而，这两种方法均需修改预编译、预优化的LLM计算图，因此在高吞吐量引擎（如vLLM）中无法获得完整支持。我们提出基于艺术强化训练（ART）的微调方法。该方法通过仅优化冻结多模态大语言模型（MLLM）的原始视觉输入来注入信息，从而在预编译计算图上实现软标记方法。它依赖于梯度反向传播至纯像素阵列，因此支持任何微调目标。此外，优化后的视觉输入可被风格化为与任务相关的计算艺术作品。该方法在流行开源Qwen架构的不同规模模型上，以及多个文本基准测试中均验证了其有效性。具体而言，ART在数学和结构化工具使用基准测试中达到了与LoRA相当的准确率。

English

There are two main Parameter-Efficient Fine-Tuning (PEFT) techniques for Large Language Models (LLMs). While Low-Rank Adaptation (LoRA) introduces additional weights between the LLM layers, Soft Prompting introduces additional fine-tuning-specific raw tokens to an LLM input. However, both require modification to the computational graphs of precompiled, preoptimized LLMs. As a result, neither is fully supported in high-throughput engines like vLLM. We propose fine-tuning with ART (Art-based Reinforcement Training). The method injects information into a frozen Multimodal Large Language Model (MLLM) by optimizing only its raw visual input, thus enabling the soft-token approach on pre-compiled computational graphs. It relies on backpropagation of gradients back into a plain pixel array and thus supports any fine-tuning objective. Moreover, the optimized visual input can be stylized as task-relevant computational artworks. The approach's effectiveness is confirmed for different sizes of a popular open Qwen architecture and for several textual benchmarks. Specifically, ART reaches accuracy competitive with LoRA across mathematics and structured-tool-use benchmarks.