基於概念的任意場景凝視目標估計

摘要

從非受控環境圖像中估計人類注視目標是一項重要且艱巨的任務。現有方法主要採用脆弱的多階段流程，需要顯式輸入（如頭部邊界框和人體姿態）來識別注視分析的對象。因此，檢測錯誤可能會級聯並導致失敗。此外，這些先前的工作缺乏透過自然語言提示來指定注視分析任務的靈活性，而這種方法在其他圖像分析任務中已被證明在便利性和可擴展性方面具有顯著優勢。為了克服這些限制，我們引入了可提示式注視目標估計（Promptable Gaze Target Estimation, PGE）任務，這是一種新的端到端、概念驅動的注視分析範式。PGE 以靈活的用戶文字或視覺提示（例如「穿紅衣服的男孩」或「位於 [0.52, 0.48] 點的人」）為條件來進行注視預測，以識別特定的注視分析對象。這種方法整合了對象定位與注視估計，並消除了對中間分析階段的剛性依賴。我們開發了一個可擴展的數據引擎，用以生成 Gaze-Co（帶有概念的注視估計），這是一個包含 120K 對高質量、提示標註圖像對的數據集和基準。我們還提出了 GazeAnywhere，這是第一個專為 PGE 設計的模型。GazeAnywhere 使用基於 Transformer 的檢測器，融合來自凍結編碼器的特徵，並同時解決對象定位、畫面內外存在性以及注視目標熱圖估計。GazeAnywhere 在多個 PGE 基準上實現了最先進的性能，即使在困難的跨領域真實世界臨床數據集上，也為這個新問題樹立了強勁的基線。GazeAnywhere 已在 github.com/IrohXu/GazeAnywhere 開源。

English

Estimating human gaze targets from images in-the-wild is an important and formidable task. Existing approaches primarily employ brittle, multi-stage pipelines that require explicit inputs, like head bounding boxes and human pose, in order to identify the subject of gaze analysis. As a result, detection errors can cascade and lead to failure. Moreover, these prior works lack the flexibility of specifying the gaze analysis task via natural language prompting, an approach which has been shown to have significant benefits in convenience and scalability for other image analysis tasks. To overcome these limitations, we introduce the Promptable Gaze Target Estimation (PGE) task, a new end-to-end, concept-driven paradigm for gaze analysis. PGE conditions gaze prediction on flexible user text or visual prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject for gaze analysis. This approach integrates subject localization with gaze estimation, and eliminates the rigid dependency on intermediate analysis stages. We develop a scalable data engine to generate Gaze-Co (Gaze Estimation with Concepts), a dataset and benchmark of 120K high-quality, prompt-annotated image pairs. We also propose GazeAnywhere, the first model designed for PGE. GazeAnywhere uses a transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation. GazeAnywhere achieves state-of-the-art performance on multiple PGE benchmarks, setting a strong baseline for this new problem even on a difficult out-of-domain, real-world clinical dataset. GazeAnywhere is open-sourced in github.com/IrohXu/GazeAnywhere.