鸟类的时间一致性三维重建

Temporally-consistent 3D Reconstruction of Birds

August 24, 2024

作者: Johannes Hägerlind, Jonas Hentati-Sundberg, Bastian Wandt

cs.AI

摘要

本文讨论了最近引起环境科学家关注的海鸟的三维重建，它们被视为环境变化的有价值的生物指示物。这种三维信息有助于分析鸟类的行为和生理形态，例如通过跟踪运动、形状和外观变化。从计算机视觉的角度来看，鸟类特别具有挑战性，因为它们的运动往往是快速且非刚性的。我们提出了一种方法，从单眼视频中重建特定品种海鸟——普通海雀的三维姿势和形状。我们的方法包括完整的检测、跟踪、分割和时间一致的三维重建流程。此外，我们提出了一种时间损失，将当前单图像三维鸟类姿势估计器扩展到时间域。此外，我们提供了一个真实世界数据集，平均包含10000帧视频观测，同时捕捉九只鸟，包括各种运动和互动，还包括一个带有鸟类特定关键点标签的较小测试集。通过我们的时间优化，我们在数据集中具有挑战性的序列中实现了最先进的性能。

English

This paper deals with 3D reconstruction of seabirds which recently came into focus of environmental scientists as valuable bio-indicators for environmental change. Such 3D information is beneficial for analyzing the bird's behavior and physiological shape, for example by tracking motion, shape, and appearance changes. From a computer vision perspective birds are especially challenging due to their rapid and oftentimes non-rigid motions. We propose an approach to reconstruct the 3D pose and shape from monocular videos of a specific breed of seabird - the common murre. Our approach comprises a full pipeline of detection, tracking, segmentation, and temporally consistent 3D reconstruction. Additionally, we propose a temporal loss that extends current single-image 3D bird pose estimators to the temporal domain. Moreover, we provide a real-world dataset of 10000 frames of video observations on average capture nine birds simultaneously, comprising a large variety of motions and interactions, including a smaller test set with bird-specific keypoint labels. Using our temporal optimization, we achieve state-of-the-art performance for the challenging sequences in our dataset.