鳥類的時間一致性三維重建

Temporally-consistent 3D Reconstruction of Birds

August 24, 2024

作者: Johannes Hägerlind, Jonas Hentati-Sundberg, Bastian Wandt

cs.AI

摘要

本文討論了海鳥的三維重建，最近成為環境科學家關注的焦點，作為環境變化的有價值生物指標。這樣的三維信息有助於分析鳥類的行為和生理形態，例如通過跟踪運動、形狀和外觀變化。從計算機視覺的角度來看，鳥類特別具有挑戰性，因為它們的運動往往是快速的且非剛性的。我們提出了一種方法，從單眼視頻中重建特定品種海鳥 - 普通海鸚鵡的三維姿態和形狀。我們的方法包括檢測、跟踪、分割和時間一致的三維重建完整流程。此外，我們提出了一種時間損失，將當前的單幅圖像三維鳥類姿態估計擴展到時間域。此外，我們提供了一個現實世界的數據集，平均包含10000幀視頻觀測，同時捕捉九只鳥，包括各種運動和互動，還有一個帶有鳥類特定關鍵點標籤的較小測試集。使用我們的時間優化，我們在數據集中具有挑戰性的序列中實現了最先進的性能。

English

This paper deals with 3D reconstruction of seabirds which recently came into focus of environmental scientists as valuable bio-indicators for environmental change. Such 3D information is beneficial for analyzing the bird's behavior and physiological shape, for example by tracking motion, shape, and appearance changes. From a computer vision perspective birds are especially challenging due to their rapid and oftentimes non-rigid motions. We propose an approach to reconstruct the 3D pose and shape from monocular videos of a specific breed of seabird - the common murre. Our approach comprises a full pipeline of detection, tracking, segmentation, and temporally consistent 3D reconstruction. Additionally, we propose a temporal loss that extends current single-image 3D bird pose estimators to the temporal domain. Moreover, we provide a real-world dataset of 10000 frames of video observations on average capture nine birds simultaneously, comprising a large variety of motions and interactions, including a smaller test set with bird-specific keypoint labels. Using our temporal optimization, we achieve state-of-the-art performance for the challenging sequences in our dataset.