基于无配对数据学习的轻量级智能手机图像信号处理器

Learned Lightweight Smartphone ISP with Unpaired Data

May 15, 2025

作者: Andrei Arhire, Radu Timofte

cs.AI

摘要

图像信号处理器（ISP）是现代智能手机相机中的核心组件，负责将RAW传感器图像数据转换为RGB图像，并着重于提升感知质量。近期研究凸显了深度学习方法在捕捉细节方面的潜力，其质量正日益接近专业相机水平。然而，在开发可学习ISP的过程中，获取像素级对齐的配对数据——即从智能手机相机传感器捕获的RAW图像映射到高质量参考图像——是一个既困难又昂贵的步骤。本研究中，我们针对这一挑战提出了一种新颖的可学习ISP训练方法，该方法无需RAW图像与内容匹配的真实数据之间的直接对应关系。我们的无配对方法采用了一种多项目标损失函数，通过对抗训练引导，利用多个判别器处理来自预训练网络的特征图，以在学习目标RGB数据集的色彩和纹理特征的同时保持内容结构。我们以适用于移动设备的轻量级神经网络架构为骨干，在苏黎世RAW到RGB和富士UltraISP数据集上评估了我们的方法。与配对训练方法相比，我们的无配对学习策略展现出强大的潜力，并在多项评估指标上实现了高保真度。代码及预训练模型已发布于https://github.com/AndreiiArhire/Learned-Lightweight-Smartphone-ISP-with-Unpaired-Data。

English

The Image Signal Processor (ISP) is a fundamental component in modern smartphone cameras responsible for conversion of RAW sensor image data to RGB images with a strong focus on perceptual quality. Recent work highlights the potential of deep learning approaches and their ability to capture details with a quality increasingly close to that of professional cameras. A difficult and costly step when developing a learned ISP is the acquisition of pixel-wise aligned paired data that maps the raw captured by a smartphone camera sensor to high-quality reference images. In this work, we address this challenge by proposing a novel training method for a learnable ISP that eliminates the need for direct correspondences between raw images and ground-truth data with matching content. Our unpaired approach employs a multi-term loss function guided by adversarial training with multiple discriminators processing feature maps from pre-trained networks to maintain content structure while learning color and texture characteristics from the target RGB dataset. Using lightweight neural network architectures suitable for mobile devices as backbones, we evaluated our method on the Zurich RAW to RGB and Fujifilm UltraISP datasets. Compared to paired training methods, our unpaired learning strategy shows strong potential and achieves high fidelity across multiple evaluation metrics. The code and pre-trained models are available at https://github.com/AndreiiArhire/Learned-Lightweight-Smartphone-ISP-with-Unpaired-Data .