ReferTrack：面向具身视觉跟踪的先指代后跟踪方法

摘要

具身视觉追踪（EVT）要求一个移动智能体仅依靠机载视觉，持续跟随自然语言描述的特定目标。尽管近期视觉-语言-动作（VLA）策略统一了目标识别与轨迹规划，但其思维链（CoT）推理往往在抽象的潜在空间中运行，难以监督且与显式的图像空间检测对齐较弱。为解决这一问题，我们提出ReferTrack——一种先指代后追踪的范式，利用单目前向摄像头实现具身视觉追踪。该模型首先从一组索引化的边界框中选择目标，然后基于这一图像锚定决策解码追踪航点。为保留随时间变化的目标运动线索，ReferTrack维护了一个先前选中的边界框的滑动窗口队列，并通过时空视点-边界框指示（TVBI）令牌将几何特征注入视觉历史。此外，我们通过在自定义Refer-QA数据集上进行联合训练来增强目标识别能力。在EVT-Bench上，ReferTrack在单目标、干扰项和模糊追踪三个子任务中分别取得了89.4%、73.3%和74.1%的成功率，达到最先进的单视图性能，在侧重于身份识别的任务上甚至与多摄像头基线持平或超越。最后，在足式机器人和人形机器人上的实际部署验证了其鲁棒的仿真到现实迁移能力。代码已开源：https://github.com/MedlarTea/referTrack。

English

Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language using only onboard vision. While recent vision-language-action (VLA) policies unify target identification and trajectory planning, their chain-of-thought (CoT) reasoning often operates in abstract spatial latents that are difficult to supervise and weakly aligned with explicit image-space detections. To address this, we introduce ReferTrack, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera. Our model first selects the target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on this image-grounded decision. To preserve target motion cues over time, ReferTrack maintains a sliding-window queue of previously selected bounding boxes, injecting their geometric features into the visual history via temporal-viewpoint-bbox indicator (TVBI) tokens. We further enhance target identification by co-training on a custom Refer-QA dataset. On EVT-Bench, ReferTrack achieves state-of-the-art single-view performance with success rates of 89.4%, 73.3%, and 74.1% on the single-target, distracted, and ambiguity tracking splits, respectively -- matching or even surpassing several multi-camera baselines on identification-heavy tasks. Finally, real-world deployments on legged and humanoid robots validate its robust sim-to-real transfer capabilities. Code is available at https://github.com/MedlarTea/referTrack.