SPIEval：评估大型语言模型在分散个人信息场景下作为移动助手的表现

摘要

大语言模型（LLMs）正越来越多地被部署为移动助手，其中的一个关键挑战是如何利用分散在多个应用程序（app）中的个人信息来完成用户指令。然而，由于缺乏专门的基准测试，它们的能力仍未得到充分理解。为弥补这一空白，我们提出了SPIEval——一个基于五种认知能力（即推理、消歧、整合、偏好推断和多意图分解）的人工策划基准测试。SPIEval包含250个任务，涵盖分布在10个应用中的4,335条个人记录，并通过21个工具支持多轮交互。分析表明，该基准具有场景多样、任务具有挑战性、信息分散、环境可控和结果可验证等特点。我们评估了九个代表性LLM，发现仍有很大的改进空间。表现最佳的模型GPT-5.5（xhigh）仅达到57.3%的准确率，而表现最弱的模型仅为16.4%。进一步分析显示，79%的失败源于信息定位不准确——LLM往往固守看似合理但错误的信息，而不是继续检索以进行验证。我们还发现，不到2%的检索操作采用了高级检索方法，且不同模型间的检索效率差异显著。这些发现揭示了当前基于LLM的移动助手的根本性局限性，并为该方向的未来研究提供了动力。数据和代码可在https://huggingface.co/datasets/Junjie-Ye/SPIEval获取。

English

Large language models (LLMs) are increasingly deployed as mobile assistants, where a key challenge is leveraging personal information scattered across multiple applications (apps) to complete user instructions. However, due to the lack of dedicated benchmarks, their capabilities remain poorly understood. To address this gap, we introduce SPIEval, a human-curated benchmark grounded in five cognitive capabilities (i.e., reasoning, disambiguation, integration, preference inference, and multi-intent decomposition). SPIEval comprises 250 tasks spanning 4,335 personal records distributed across 10 apps and supports multi-turn interaction through 21 tools. Analysis shows that the benchmark exhibits diverse scenarios, challenging tasks, scattered information, controllable environments, and verifiable outcomes. We evaluate nine representative LLMs and find substantial room for improvement. The best-performing model, GPT-5.5 (xhigh), achieves only 57.3% accuracy, while the weakest achieves just 16.4%. Further analysis reveals that 79% of failures stem from inaccurate information localization, as LLMs often commit to plausible but incorrect information instead of continuing retrieval for verification. We also find that fewer than 2% of retrieval actions employ advanced search methods and observe substantial variation in search efficiency across models. These findings expose fundamental limitations of current LLM-based mobile assistants and motivate future research in this direction. Data and code are available at https://huggingface.co/datasets/Junjie-Ye/SPIEval.