DaTaSeg：馴服通用多數據集多任務分割模型

摘要

觀察到全景、語義和實例分割任務之間密切的關係，我們提出訓練通用多數據集多任務分割模型：DaTaSeg。我們為所有任務使用共享表示（具有類別預測的遮罩提議）。為了應對任務差異，我們採用不同的合併操作和後處理方式來處理不同的任務。我們還利用弱監督，使我們的分割模型能夠從更便宜的邊界框標註中受益。為了跨數據集共享知識，我們使用與分類器相同的語義嵌入空間中的文本嵌入，並在數據集之間共享所有網絡參數。我們在ADE語義、COCO全景和Objects365檢測數據集上訓練DaTaSeg。DaTaSeg在所有數據集上提高了性能，特別是在小規模數據集上，實現了ADE語義的54.0 mIoU和COCO全景的53.5 PQ。DaTaSeg還實現了對ADE全景和Objects365實例分割的弱監督知識轉移。實驗表明，DaTaSeg隨著訓練數據集的增加而擴展，並通過直接轉移實現了開放詞彙的分割。此外，我們標註了一組包含1,000張圖像的Objects365實例分割集，並將其作為公共基準發布。

English

Observing the close relationship among panoptic, semantic and instance segmentation tasks, we propose to train a universal multi-dataset multi-task segmentation model: DaTaSeg.We use a shared representation (mask proposals with class predictions) for all tasks. To tackle task discrepancy, we adopt different merge operations and post-processing for different tasks. We also leverage weak-supervision, allowing our segmentation model to benefit from cheaper bounding box annotations. To share knowledge across datasets, we use text embeddings from the same semantic embedding space as classifiers and share all network parameters among datasets. We train DaTaSeg on ADE semantic, COCO panoptic, and Objects365 detection datasets. DaTaSeg improves performance on all datasets, especially small-scale datasets, achieving 54.0 mIoU on ADE semantic and 53.5 PQ on COCO panoptic. DaTaSeg also enables weakly-supervised knowledge transfer on ADE panoptic and Objects365 instance segmentation. Experiments show DaTaSeg scales with the number of training datasets and enables open-vocabulary segmentation through direct transfer. In addition, we annotate an Objects365 instance segmentation set of 1,000 images and will release it as a public benchmark.

DaTaSeg：馴服通用多數據集多任務分割模型

DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model

摘要

Support