SPIEval：評估大型語言模型作為行動助手在分散個人資訊情境下的表現

摘要

大型語言模型（LLMs）日益被部署為行動助理，其中一項關鍵挑戰在於善用分散於多個應用程式（apps）中的個人資訊來完成使用者指令。然而，由於缺乏專用基準，其能力至今仍未被充分理解。為填補此一缺口，我們提出SPIEval，一個以五項認知能力（即推理、消歧、整合、偏好推斷與多意圖分解）為基礎的人工策劃基準。SPIEval包含250項任務，涵蓋分散於10個應用程式中的4,335筆個人記錄，並透過21種工具支援多輪互動。分析顯示，此基準具備多樣化情境、具挑戰性的任務、分散的資訊、可控的環境與可驗證的結果。我們評估了九個具代表性的LLM，發現仍有大幅改善空間。表現最佳的模型GPT-5.5（xhigh）僅達到57.3%的準確率，而表現最弱者僅達16.4%。進一步分析揭示，79%的失敗源於資訊定位不精確，因為LLM往往執著於看似合理但錯誤的資訊，而非持續檢索以求驗證。我們亦發現，少於2%的檢索動作採用進階搜尋方法，並觀察到各模型在搜尋效率上存在顯著差異。這些發現揭露了當前基於LLM的行動助理之根本限制，並激勵了此一方向的未來研究。資料與程式碼可在https://huggingface.co/datasets/Junjie-Ye/SPIEval取得。

English

Large language models (LLMs) are increasingly deployed as mobile assistants, where a key challenge is leveraging personal information scattered across multiple applications (apps) to complete user instructions. However, due to the lack of dedicated benchmarks, their capabilities remain poorly understood. To address this gap, we introduce SPIEval, a human-curated benchmark grounded in five cognitive capabilities (i.e., reasoning, disambiguation, integration, preference inference, and multi-intent decomposition). SPIEval comprises 250 tasks spanning 4,335 personal records distributed across 10 apps and supports multi-turn interaction through 21 tools. Analysis shows that the benchmark exhibits diverse scenarios, challenging tasks, scattered information, controllable environments, and verifiable outcomes. We evaluate nine representative LLMs and find substantial room for improvement. The best-performing model, GPT-5.5 (xhigh), achieves only 57.3% accuracy, while the weakest achieves just 16.4%. Further analysis reveals that 79% of failures stem from inaccurate information localization, as LLMs often commit to plausible but incorrect information instead of continuing retrieval for verification. We also find that fewer than 2% of retrieval actions employ advanced search methods and observe substantial variation in search efficiency across models. These findings expose fundamental limitations of current LLM-based mobile assistants and motivate future research in this direction. Data and code are available at https://huggingface.co/datasets/Junjie-Ye/SPIEval.