利用概念的任意场景注视目标估计

摘要

从真实场景图像中估计人类注视目标是一项重要且艰巨的任务。现有方法主要采用脆弱的多阶段流程，需要诸如头部边界框和人体姿态等显式输入，以识别注视分析的主体。因此，检测错误可能级联传播并导致失败。此外，这些先前的工作缺乏通过自然语言提示来指定注视分析任务的灵活性，而这种方法已被证明在其他图像分析任务中在便利性和可扩展性方面具有显著优势。为了克服这些限制，我们提出了可提示注视目标估计（PGE）任务，这是一种新的端到端、概念驱动的注视分析范式。PGE 根据灵活的用户文本或视觉提示（例如“穿红衬衫的男孩”或“位于点[0.52, 0.48]的人”）来调节注视预测，以识别用于注视分析的特定主体。该方法将主体定位与注视估计相结合，并消除了对中间分析阶段的刚性依赖。我们开发了一个可扩展的数据引擎，用于生成 Gaze-Co（基于概念的注视估计），这是一个包含 12 万对高质量、带提示标注图像对的数据集和基准。我们还提出了 GazeAnywhere，这是首个为 PGE 设计的模型。GazeAnywhere 使用基于 Transformer 的检测器来融合来自冻结编码器的特征，并同时解决主体定位、是否在画面内以及注视目标热图估计的问题。GazeAnywhere 在多个 PGE 基准上取得了最先进的性能，即使是在困难的跨域真实世界临床数据集上，也为这一新问题树立了强大的基线。GazeAnywhere 已在 github.com/IrohXu/GazeAnywhere 开源。

English

Estimating human gaze targets from images in-the-wild is an important and formidable task. Existing approaches primarily employ brittle, multi-stage pipelines that require explicit inputs, like head bounding boxes and human pose, in order to identify the subject of gaze analysis. As a result, detection errors can cascade and lead to failure. Moreover, these prior works lack the flexibility of specifying the gaze analysis task via natural language prompting, an approach which has been shown to have significant benefits in convenience and scalability for other image analysis tasks. To overcome these limitations, we introduce the Promptable Gaze Target Estimation (PGE) task, a new end-to-end, concept-driven paradigm for gaze analysis. PGE conditions gaze prediction on flexible user text or visual prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject for gaze analysis. This approach integrates subject localization with gaze estimation, and eliminates the rigid dependency on intermediate analysis stages. We develop a scalable data engine to generate Gaze-Co (Gaze Estimation with Concepts), a dataset and benchmark of 120K high-quality, prompt-annotated image pairs. We also propose GazeAnywhere, the first model designed for PGE. GazeAnywhere uses a transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation. GazeAnywhere achieves state-of-the-art performance on multiple PGE benchmarks, setting a strong baseline for this new problem even on a difficult out-of-domain, real-world clinical dataset. GazeAnywhere is open-sourced in github.com/IrohXu/GazeAnywhere.