OmegaUse-OfficeVal：在經濟依據下對長程辦公套件任務的LLM智能體進行基準測試

摘要

大型語言模型（LLM）代理日益被期望協助用戶完成任務。然而，現有的基準測試在評估代理能否以合理成本執行辦公套件工作流程方面所提供的支援有限。我們推出 OmegaUse-OfficeVal，這是一個用於評估 LLM 代理在長期辦公套件任務上的基準測試，並具備任務層級的經濟基礎。該基準測試包含 100 項任務，這些任務源自從業者提出的辦公套件請求，並經過隱私保護流程進行改編。平均而言，完成這些任務需要 2.32 小時的人力工時。該基準測試的一個重要特點是每個任務都配備了兩個經濟信號：人力工時與任務價格代理。這些信號能夠直接比較人力成本與 LLM 推論成本，並進行加權價值評估。為支持穩定的評估，我們根據細粒度評分標準開發了基於程式碼的驗證器。我們評估了數個前沿 LLM，同時搭配人類基準。儘管所有受評 LLM 的成本與速度都遠優於人類工作者，但它們尚未達到人類等級的交付品質。程式碼與資料集已完全開源，更多資訊請參閱我們的專案網站：https://omegause-officeval.github.io。

English

Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io.