DaTaSeg：驯服通用多数据集多任务分割模型

摘要

观察到全景、语义和实例分割任务之间密切的关系，我们提出训练通用多数据集多任务分割模型：DaTaSeg。我们为所有任务使用共享表示（具有类别预测的掩码提议）。为了解决任务差异，我们采用不同的合并操作和后处理方式来处理不同的任务。我们还利用弱监督，使我们的分割模型能够从更便宜的边界框注释中受益。为了跨数据集共享知识，我们使用来自与分类器相同的语义嵌入空间的文本嵌入，并在数据集之间共享所有网络参数。我们在ADE语义、COCO全景和Objects365检测数据集上训练DaTaSeg。DaTaSeg在所有数据集上提高了性能，特别是在小规模数据集上，实现了ADE语义上的54.0 mIoU和COCO全景上的53.5 PQ。DaTaSeg还实现了在ADE全景和Objects365实例分割上的弱监督知识转移。实验表明，DaTaSeg随着训练数据集数量的增加而扩展，并通过直接转移实现了开放词汇的分割。此外，我们标注了一个包含1,000张图像的Objects365实例分割数据集，并将其发布为公共基准。

English

Observing the close relationship among panoptic, semantic and instance segmentation tasks, we propose to train a universal multi-dataset multi-task segmentation model: DaTaSeg.We use a shared representation (mask proposals with class predictions) for all tasks. To tackle task discrepancy, we adopt different merge operations and post-processing for different tasks. We also leverage weak-supervision, allowing our segmentation model to benefit from cheaper bounding box annotations. To share knowledge across datasets, we use text embeddings from the same semantic embedding space as classifiers and share all network parameters among datasets. We train DaTaSeg on ADE semantic, COCO panoptic, and Objects365 detection datasets. DaTaSeg improves performance on all datasets, especially small-scale datasets, achieving 54.0 mIoU on ADE semantic and 53.5 PQ on COCO panoptic. DaTaSeg also enables weakly-supervised knowledge transfer on ADE panoptic and Objects365 instance segmentation. Experiments show DaTaSeg scales with the number of training datasets and enables open-vocabulary segmentation through direct transfer. In addition, we annotate an Objects365 instance segmentation set of 1,000 images and will release it as a public benchmark.

DaTaSeg：驯服通用多数据集多任务分割模型

DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model

摘要

Support