MobilePA-Bench：針對複雜真實世界任務的移動規劃智能體基準測試

摘要

隨著裝置端 LLM 代理演進為個人助手，行動作業系統已成為此一典範的關鍵測試場域，嚴謹的能力評估因而至關重要。然而，現有基準分屬兩大陣營，各具關鍵盲點：以圖形介面為中心的基準測試表面層級的螢幕操作，卻忽略背景工具使用與長程規劃；而靜態函式呼叫基準則依賴於脫離真實執行時期約束的離線 API 比對。為填補此缺口，我們提出 MobilePA-Bench——一個互動式、具狀態且以工具為中心的基準，用於評估行動規劃代理的工具呼叫與規劃能力。MobilePA-Bench 在可執行沙盒中運行，維護即時應用程式資料庫並回傳結構化回饋，涵蓋 13 個功能領域與 212 個真實行動工具。除基本工具使用外，它沿三個進階維度評估中央規劃代理：(1) 子代理協作——分解複雜任務並將專業工作委派給具備能力的子代理；(2) 記憶使用——回憶已儲存的記憶、使用者設定檔與過往偏好，以解決隱含請求；以及 (3) 技能使用——呼叫預先封裝的複合技能，而非從零開始規劃每個步驟。大規模實驗顯示，當前前沿 LLM 在行動環境中仍不夠可靠：在嚴格的工具順序、權限限制與意外執行時期錯誤下，效能急遽下降。藉由將互動式函式呼叫沙盒與基於證據的驗證相結合，MobilePA-Bench 既作為實用的診斷基準，亦為代理式強化學習的互動基礎——加速可靠行動代理的開發。

English

As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capability evaluation essential. Yet existing benchmarks fall into two camps, each with a critical blind spot: GUI-centric benchmarks test surface-level screen manipulation while overlooking background tool use and long-horizon planning, whereas static function-calling benchmarks rely on offline API matching that is detached from real runtime constraints. To close this gap, we present MobilePA-Bench, an interactive, stateful, and tool-centric benchmark for evaluating the tool-calling and planning abilities of mobile planning agents. MobilePA-Bench runs on an executable sandbox that maintains live application databases and returns structured feedback, spanning 13 functional domains and 212 realistic mobile tools. Beyond basic tool use, it evaluates a central planning agent along three advanced dimensions: (1)~Sub-agent Collaboration---decomposing a complex task and delegating specialized work to capable sub-agents; (2)~Memory Usage---recalling stored memories, user profiles, and past preferences to resolve implicit requests; and (3)~Skill Usage---invoking pre-packaged composite skills instead of planning every step from scratch. Extensive experiments show that current frontier LLMs remain unreliable in mobile settings: performance drops sharply under strict tool ordering, permission limits, and unexpected runtime errors. By pairing an interactive function-calling sandbox with evidence-based verification, MobilePA-Bench serves as both a practical diagnostic benchmark and an interactive foundation for agentic reinforcement learning---accelerating the development of dependable mobile agents.