DaMo：面向手機代理的多模態大語言模型微調中的數據混合優化器

摘要

移动电话代理（MPAs）因其在多样化场景中的广泛应用而成为一个极具前景的研究方向。尽管多模态大语言模型（MLLMs）构成了MPAs的基础，但它们在同时处理多项移动电话任务时的效能仍显不足。虽然多任务监督微调（SFT）被广泛用于多任务学习，现有方法在确定最佳训练数据组合以实现峰值性能方面仍面临挑战。为解决这一难题，我们提出了DaMo（数据混合优化器）——一种创新解决方案，它采用可训练网络，通过预测任何给定数据集比例下的下游任务性能来优化数据混合。为支持全面评估，我们引入了PhoneAgentBench，这是首个专门评估MLLMs在多模态移动电话任务上的基准，包含1235个问答对，覆盖多样化的现实工业移动应用场景。在小规模试点实验中，DaMo展现出强大的预测能力（R^2=0.81），能够高效外推最佳数据混合配置。我们的结果显示，与替代方法相比，DaMo在PhoneAgentBench上实现了3.38%的性能提升。此外，在包括BFCL-v3、MME-Reasoning、MME-Perception和OCRBench在内的多个成熟基准上的广泛实验表明，DaMo具有卓越的泛化能力，在平均得分上超越其他方法2.57%。当仅用于优化BFCL-v3任务中的MLLM时，DaMo将指标提升了12.47%。值得注意的是，DaMo保持了强大的可扩展性，在应用于其他模型架构时仍能保持其有效性。代码和数据集可在https://github.com/OPPO-Mente-Lab/DaMo.git获取。

English

Mobile Phone Agents (MPAs) have emerged as a promising research direction due to their broad applicability across diverse scenarios. While Multimodal Large Language Models (MLLMs) serve as the foundation for MPAs, their effectiveness in handling multiple mobile phone tasks simultaneously remains limited. Although multitask supervised fine-tuning (SFT) is widely adopted for multitask learning, existing approaches struggle to determine optimal training data compositions for peak performance. To address this challenge, we propose DaMo (Data Mixture Optimizer) - a novel solution employing a trainable network that predicts optimal data mixtures by forecasting downstream task performance for any given dataset ratio. To support comprehensive evaluation, we introduce PhoneAgentBench, the first specialized benchmark to evaluate MLLMs on multimodal mobile phone tasks, comprising 1235 QA pairs spanning diverse real-world industrial mobile application scenarios. Demonstrating strong predictive capability (R^2=0.81) in small-scale pilot experiments, DaMo efficiently extrapolates optimal data mixing configurations. Our results show DaMo achieves a 3.38% performance improvement on PhoneAgentBench compared to alternative methods. Furthermore, extensive experiments across established benchmarks including BFCL-v3, MME-Reasoning, MME-Perception, and OCRBench reveal DaMo's superior generalization, outperforming other approaches by 2.57% in terms of average score. When used solely for MLLM optimization on the BFCL-v3 task, DaMo improves the metrics by 12.47% than other methods. Notably, DaMo maintains robust scalability, preserving its effectiveness when applied to other model architectures. The code and dataset are available at https://github.com/OPPO-Mente-Lab/DaMo.git

DaMo：面向手機代理的多模態大語言模型微調中的數據混合優化器

DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents

摘要

Support