MobilePA-Bench：面向复杂现实世界任务的移动规划智能体基准评估

摘要

随着设备端大语言模型智能体演化为个人助手，移动操作系统已成为这一范式的重要测试平台，对其能力进行严谨评估至关重要。然而，现有基准测试分为两类，且各有其关键盲区：以图形用户界面为中心的基准测试仅检验表层屏幕操作，却忽视了后台工具使用和长程规划能力；而静态函数调用基准则依赖离线API匹配，脱离了真实的运行时约束。为弥合这一差距，我们提出了MobilePA-Bench——一个交互式、有状态、以工具为核心的基准测试，用于评估移动规划智能体的工具调用与规划能力。MobilePA-Bench运行于可执行的沙箱环境中，该沙箱维护实时应用数据库并返回结构化反馈，涵盖13个功能领域和212个逼真的移动工具。除基础工具使用外，它还从一个中央规划智能体的角度，沿三个高级维度进行评估：(1)~子智能体协作——将复杂任务分解并委派给具备专门能力的子智能体；(2)~记忆使用——回忆已存储的记忆、用户画像和过往偏好以解析隐含请求；(3)~技能使用——调用预封装的复合技能，而非从零规划每一步。大量实验表明，当前前沿大语言模型在移动场景中仍不可靠：在严格的工具排序、权限限制和意外运行时错误条件下，性能急剧下降。通过将交互式函数调用沙箱与基于证据的验证相结合，MobilePA-Bench既可作为实用的诊断基准，也可作为智能体强化学习的交互式基础——加速开发可靠的移动智能体。

English

As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capability evaluation essential. Yet existing benchmarks fall into two camps, each with a critical blind spot: GUI-centric benchmarks test surface-level screen manipulation while overlooking background tool use and long-horizon planning, whereas static function-calling benchmarks rely on offline API matching that is detached from real runtime constraints. To close this gap, we present MobilePA-Bench, an interactive, stateful, and tool-centric benchmark for evaluating the tool-calling and planning abilities of mobile planning agents. MobilePA-Bench runs on an executable sandbox that maintains live application databases and returns structured feedback, spanning 13 functional domains and 212 realistic mobile tools. Beyond basic tool use, it evaluates a central planning agent along three advanced dimensions: (1)~Sub-agent Collaboration---decomposing a complex task and delegating specialized work to capable sub-agents; (2)~Memory Usage---recalling stored memories, user profiles, and past preferences to resolve implicit requests; and (3)~Skill Usage---invoking pre-packaged composite skills instead of planning every step from scratch. Extensive experiments show that current frontier LLMs remain unreliable in mobile settings: performance drops sharply under strict tool ordering, permission limits, and unexpected runtime errors. By pairing an interactive function-calling sandbox with evidence-based verification, MobilePA-Bench serves as both a practical diagnostic benchmark and an interactive foundation for agentic reinforcement learning---accelerating the development of dependable mobile agents.