语义学：一种可适应图像条件扩散模型

Semantica: An Adaptable Image-Conditioned Diffusion Model

May 23, 2024

作者: Manoj Kumar, Neil Houlsby, Emiel Hoogeboom

cs.AI

摘要

我们研究了将图像生成模型适应不同数据集的任务，而无需微调。为此，我们引入了Semantica，一种基于图像的扩散模型，能够根据条件图像的语义生成图像。Semantica仅在网络规模的图像对上进行训练，即接收来自网页的随机图像作为条件输入，并对同一网页中的另一随机图像进行建模。我们的实验突显了预训练图像编码器的表达能力以及在实现高质量图像生成中必须进行基于语义的数据过滤。一旦训练完成，它可以通过简单地使用该数据集中的图像作为输入，自适应地生成新图像。我们研究了Semantica在ImageNet、LSUN教堂、LSUN卧室和SUN397上的迁移特性。

English

We investigate the task of adapting image generative models to different datasets without finetuneing. To this end, we introduce Semantica, an image-conditioned diffusion model capable of generating images based on the semantics of a conditioning image. Semantica is trained exclusively on web-scale image pairs, that is it receives a random image from a webpage as conditional input and models another random image from the same webpage. Our experiments highlight the expressivity of pretrained image encoders and necessity of semantic-based data filtering in achieving high-quality image generation. Once trained, it can adaptively generate new images from a dataset by simply using images from that dataset as input. We study the transfer properties of Semantica on ImageNet, LSUN Churches, LSUN Bedroom and SUN397.