DaMo：面向手机智能体的多模态大模型微调中的数据混合优化器

摘要

移动终端智能体（MPAs）因其在多样化场景中的广泛应用潜力，已成为一个极具前景的研究方向。尽管多模态大语言模型（MLLMs）构成了MPAs的基础，但它们在同时处理多项移动终端任务时的效能仍显不足。虽然多任务监督微调（SFT）被广泛用于多任务学习，现有方法在确定最佳训练数据组合以实现峰值性能方面面临挑战。为解决这一难题，我们提出了DaMo（数据混合优化器）——一种创新解决方案，它采用可训练网络预测最优数据混合比例，通过预估任何给定数据集比例下的下游任务表现来实现。为支持全面评估，我们推出了PhoneAgentBench，这是首个专门评估MLLMs在多模态移动终端任务上表现的基准，包含1235个问答对，覆盖多样化的现实工业移动应用场景。在小规模试点实验中，DaMo展现出强大的预测能力（R^2=0.81），能有效推断出最优数据混合配置。我们的结果显示，与替代方法相比，DaMo在PhoneAgentBench上实现了3.38%的性能提升。此外，在包括BFCL-v3、MME-Reasoning、MME-Perception和OCRBench在内的多个成熟基准上的广泛实验表明，DaMo具有卓越的泛化能力，在平均得分上优于其他方法2.57%。当仅用于优化BFCL-v3任务中的MLLM时，DaMo使指标提升了12.47%，显著优于其他方法。值得注意的是，DaMo保持了良好的可扩展性，在应用于其他模型架构时仍能保持其有效性。代码和数据集已公开于https://github.com/OPPO-Mente-Lab/DaMo.git。

English

Mobile Phone Agents (MPAs) have emerged as a promising research direction due to their broad applicability across diverse scenarios. While Multimodal Large Language Models (MLLMs) serve as the foundation for MPAs, their effectiveness in handling multiple mobile phone tasks simultaneously remains limited. Although multitask supervised fine-tuning (SFT) is widely adopted for multitask learning, existing approaches struggle to determine optimal training data compositions for peak performance. To address this challenge, we propose DaMo (Data Mixture Optimizer) - a novel solution employing a trainable network that predicts optimal data mixtures by forecasting downstream task performance for any given dataset ratio. To support comprehensive evaluation, we introduce PhoneAgentBench, the first specialized benchmark to evaluate MLLMs on multimodal mobile phone tasks, comprising 1235 QA pairs spanning diverse real-world industrial mobile application scenarios. Demonstrating strong predictive capability (R^2=0.81) in small-scale pilot experiments, DaMo efficiently extrapolates optimal data mixing configurations. Our results show DaMo achieves a 3.38% performance improvement on PhoneAgentBench compared to alternative methods. Furthermore, extensive experiments across established benchmarks including BFCL-v3, MME-Reasoning, MME-Perception, and OCRBench reveal DaMo's superior generalization, outperforming other approaches by 2.57% in terms of average score. When used solely for MLLM optimization on the BFCL-v3 task, DaMo improves the metrics by 12.47% than other methods. Notably, DaMo maintains robust scalability, preserving its effectiveness when applied to other model architectures. The code and dataset are available at https://github.com/OPPO-Mente-Lab/DaMo.git

DaMo：面向手机智能体的多模态大模型微调中的数据混合优化器

DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents

摘要

Support