SceneActBench：智能體能否在其所見的3D場景中行動？

摘要

視覺語言模型（VLM）智能體越來越多地使用工具來對3D場景進行操作，而非僅止於描述。現有的3D基準測試僅針對文字回應或單物體操作進行評分，導致對智能體在完整多物體3D場景中的行動評估不足。我們提出SceneActBench，這是一個在統一智能體-環境循環下，針對五項3D任務的視覺條件行動基準。在給予PNG圖像或取樣視訊幀以及（若適用）提供的3D資產後，智能體對3D環境執行操作。我們使用任務專屬的幾何指標，將每個最終輸出與隱藏的真實標註進行比較評估。SceneActBench由210個來源實例建構的五項任務組成，產生520個任務案例，其中包括成對輸入條件。每個任務皆經由一個固定的智能體循環執行，以確保比較的公平性。在十一種專有VLM配置中，總體分數範圍為38.6至50.2，且沒有任何一種配置能在所有任務中表現一致良好。我們進一步分析了失敗的表現位置與方式。

English

Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them. Existing 3D benchmarks score textual responses or single-object operations, leaving agent action on complete multi-object 3D scenes under evaluated. We present SceneActBench, a benchmark for visually conditioned action across five 3D tasks under a unified agent-environment loop. Given PNG images or sampled video frames and, where applicable, supplied 3D assets, an agent acts on a 3D environment. We evaluate each final output against hidden ground truth with task-specific geometric metrics. SceneActBench comprises five tasks built from 210 source instances, yielding 520 task cases including paired input conditions. Every task runs through one fixed agent loop to keep the comparison fair. Across eleven proprietary VLM configurations, Overall scores span 38.6-50.2, and none performs consistently well across tasks. We further analyse where and how failures manifest.