OmegaUse-OfficeVal: 面向具有经济基础的长期办公套件任务的LLM代理基准测试

摘要

大语言模型（LLM）智能体越来越多地被期望协助用户完成任务。然而，现有的基准测试对于评估智能体能否以合理成本完成办公套件工作流程所提供的支持有限。我们提出了OmegaUse-OfficeVal，一个用于评估LLM智能体在长期办公套件任务上的基准，该基准具备任务级经济基础。该基准包含100个任务，这些任务源自实践者提出的办公套件需求，并经过隐私保护流程调整。平均而言，每个任务需要2.32小时的人类劳动力完成。该基准的一个关键特性是每个任务都配有两个经济信号：人类劳动时间和任务价格代理。这些信号使得人类成本与LLM推理成本之间能够直接比较，并支持价值加权评估。为了确保稳定的评估，我们基于细粒度评分细则开发了基于代码的验证器。我们评估了几个前沿的LLM以及一个人工基线。尽管所有被评估的LLM在成本和速度上均显著优于人类工作者，但它们尚未达到人类级别的可交付质量。代码和数据集已完全开源，更多信息可在我们的项目网站上获取：https://omegause-officeval.github.io。

English

Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io.