基於無配對數據學習的輕量級智能手機圖像信號處理器

Learned Lightweight Smartphone ISP with Unpaired Data

May 15, 2025

作者: Andrei Arhire, Radu Timofte

cs.AI

摘要

图像信号处理器（ISP）是现代智能手机相机中的核心组件，负责将RAW传感器图像数据转换为RGB图像，并着重于提升感知质量。近期研究强调了深度学习方法在捕捉细节方面的潜力，其质量日益接近专业相机水平。开发学习型ISP时，一个既困难又昂贵的步骤是获取像素级对齐的配对数据，这些数据将智能手机相机传感器捕获的原始图像映射到高质量参考图像。在本研究中，我们通过提出一种新颖的学习型ISP训练方法来解决这一挑战，该方法无需原始图像与内容匹配的真实数据之间的直接对应关系。我们的非配对方法采用了一种多术语损失函数，通过对抗训练引导，利用多个判别器处理来自预训练网络的特征图，以在学习目标RGB数据集的颜色和纹理特征的同时保持内容结构。我们以适用于移动设备的轻量级神经网络架构为骨干，在苏黎世RAW到RGB和富士胶片UltraISP数据集上评估了我们的方法。与配对训练方法相比，我们的非配对学习策略展现了强大的潜力，并在多项评估指标上实现了高保真度。代码及预训练模型可在https://github.com/AndreiiArhire/Learned-Lightweight-Smartphone-ISP-with-Unpaired-Data 获取。

English

The Image Signal Processor (ISP) is a fundamental component in modern smartphone cameras responsible for conversion of RAW sensor image data to RGB images with a strong focus on perceptual quality. Recent work highlights the potential of deep learning approaches and their ability to capture details with a quality increasingly close to that of professional cameras. A difficult and costly step when developing a learned ISP is the acquisition of pixel-wise aligned paired data that maps the raw captured by a smartphone camera sensor to high-quality reference images. In this work, we address this challenge by proposing a novel training method for a learnable ISP that eliminates the need for direct correspondences between raw images and ground-truth data with matching content. Our unpaired approach employs a multi-term loss function guided by adversarial training with multiple discriminators processing feature maps from pre-trained networks to maintain content structure while learning color and texture characteristics from the target RGB dataset. Using lightweight neural network architectures suitable for mobile devices as backbones, we evaluated our method on the Zurich RAW to RGB and Fujifilm UltraISP datasets. Compared to paired training methods, our unpaired learning strategy shows strong potential and achieves high fidelity across multiple evaluation metrics. The code and pre-trained models are available at https://github.com/AndreiiArhire/Learned-Lightweight-Smartphone-ISP-with-Unpaired-Data .