[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Next Article in Journal
Phase Transition and Controlled Zirconia Implant Patterning Using Laser-Induced Shockwaves
Previous Article in Journal
A Deep Curriculum Learning Semi-Supervised Framework for Remote Sensing Scene Classification
Previous Article in Special Issue
Hybrid Frequency–Spatial Domain Learning for Image Restoration in Under-Display Camera Systems Using Augmented Virtual Big Data Generated by the Angular Spectrum Method
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 15
Issue 1
10.3390/app15010361
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

sRrsR-Net: A New Low-Light Image Enhancement Network via Raw Image Reconstruction

by
Zhiyong Hong
,
Dexin Zhen
,
Liping Xiong
*,
Xuechen Li
and
Yuhan Lin
College of Electronic and Information Engineering, Wuyi University, Jiangmen 529000, China
*
Author to whom correspondence should be addressed.
Appl. Sci. 2025, 15(1), 361; https://doi.org/10.3390/app15010361
Submission received: 1 December 2024 / Revised: 26 December 2024 / Accepted: 30 December 2024 / Published: 2 January 2025
(This article belongs to the Special Issue Advances in Image Enhancement and Restoration Technology)
Browse Figures
Figure 1
<p>Experimental results on four datasets. This is a comprehensive comparison chart of results in both sRGB and raw domains, with detailed data available in <a href="#sec4-applsci-15-00361" class="html-sec">Section 4</a>. The horizontal and vertical axes of the chart represent PSNR and SSIM, respectively. The better the performance, the further right and up the model is on the chart. It can be seen that our model achieves excellent performance.</p> "> Figure 2
<p>Flowchart of sRrsR-Net integrating Sampler, RGB-iRGB, RAWFormer, and RRAW-sRGB modules.</p> "> Figure 3
<p>The structure of the RGB-iRGB module.</p> "> Figure 4
<p>The structure of the RAWFormer module.</p> "> Figure 5
<p>The structure of the RRAW-sRGB module.</p> "> Figure 6
<p>Visual comparison results on the LOL-v1 and LOL-v2 datasets. The magnified portion has already been marked with a red box. The following are the same.</p> "> Figure 7
<p>Visual comparison of raw domain image reconstruction results using sRrsR-Net and six other methods.</p> "> Figure 8
<p>sRrsR-Net’s visualization results on the VE-LOL test set. From top to bottom, the images represent real and synthetic scenarios. From left to right, the input, output, and ground-truth images are depicted.</p> "> Figure 9
<p>Comparison of running times across different datasets. The left side compares the average running times of other methods, while the right side shows our method’s running time compared to other state-of-the-art methods.</p> "> Figure 10
<p>Visualization results of ablation study.</p> ">
Versions Notes

Abstract

:
Most existing low-light image enhancement (LIE) methods are primarily designed for human-vision-friendly image formats, such as sRGB, due to their convenient storage and smaller file sizes. In addition, raw images provide greater detail and a wider dynamic range, which makes them more suitable for LIE tasks. Despite these advantages, raw images, the original format captured by cameras, are larger and less accessible and are hard to use in methods of LIE with mobile devices. In order to leverage both the advantages of sRGB and raw domains while avoiding the direct use of raw images as training data, this paper introduces sRrsR-Net, a novel framework with the image translation process of sRGB–raw–sRGB for LIE task. In our approach, firstly, the RGB-to-iRGB module is designed to convert sRGB images into intermediate RGB feature maps. Then, with these intermediate feature maps, to bridge the domain gap between sRGB and raw pixels, the RAWFormer module is proposed to employ global attention to effectively align features between the two domains to generate reconstructed raw images. For enhancing the raw images and restoring them back to normal-light sRGB, unlike traditional Image Signal Processing (ISP) pipelines, which are often bulky and integrate numerous processing steps, we propose the RRAW-to-sRGB module. This module simplifies the process by focusing only on color correction and white balance, while still delivering competitive results. Extensive experiments on four benchmark datasets referring to both domains demonstrate the effectiveness of our approach.

1. Introduction

Low-light image enhancement (LIE) is a crucial task in the field of computer vision (CV). LIE aims to improve the contrast of images captured under low-light conditions. By extracting more detailed information from the enhanced images, it can increase the accuracy of many downstream visual tasks, such as image segmentation and object detection [1]. Recently, with the rapid development of deep learning and the increasing availability of training data, many efficient LIE models, based on deep neural networks (DNNs), have been proposed [2,3,4]. These models have demonstrated excellent performance on many CV tasks. Among these models, researchers have designed various modules to solve different challenges in LIE, for instance, generative adversarial networks [5,6], diffusion models [7,8], and attention mechanisms [9].
However, these low-light image enhancement models are mostly suitable for standard RGB images (sRGB). This preference is due to sRGB’s convenient storage and smaller file size. In contrast, pixel values in raw images have a linear relationship with the scene’s radiance, which typically have higher bit depth, offering richer image information [10]. Additionally, degradation in actual images mainly occurs in the raw domain, so enhancing images based on the raw format has the advantage of effectively restoring shadows and highlights without introducing noise associated with high ISO levels [11]. Consequently, there has been a growing interest in performing image enhancement directly in the raw domain. For instance, Xu et al. [12] uses a demosaicing module with a Gated Feedback Module and Frequency Selection Module for image demoiréing. Conde et al. [13] introduces the BSRaw model to address sensor noise in super-resolution. Recently, for LIE tasks, there are many researchers using raw images as input; this can more accurately restore the true brightness and color information and further improve the effectiveness and precision of these tasks [1,14].
While raw images offer superior results in LIE, as seen in [12], the widespread use of sRGB images in real-world applications still cannot be ignored. Consequently, integrating the advantages of both raw and sRGB images has, therefore, become a key direction for LIE. However, original raw images often present challenges such as limited accessibility, larger file sizes, and complex data structures, making them difficult to use directly in many applications. This necessitates developing techniques to reconstruct raw images from sRGB inputs, thereby leveraging the benefits of raw image processing without directly relying on raw image training data. This process, however, is non-trivial due to the substantial differences between sRGB and raw formats. A key challenge lies in designing an intermediate RGB feature map to facilitate smoother raw reconstruction while establishing effective global attention to link sRGB and raw pixel relationships.
To address these issues, we propose a novel method, sRrsR-Net, which efficiently integrates the advantages of reconstructed raw and sRGB images through a three-stage framework: sRGB–raw–sRGB. This model is designed to enhance low-light images while avoiding the direct use of raw images as training data. The main contributions of this work are three innovative modules, which are the RGB-iRGB, RAWFormer, and RRAW-sRGB modules. Firstly, the RGB-iRGB module converts sRGB to an intermediate RGB feature map, enabling smoother raw reconstruction. This module uses metadata to make full use of the rich details in the raw image and improve the enhancement effectiveness. Secondly, during the raw reconstruction process, there is a significant difference in the pixel distribution between raw and sRGB images. To establish global attention among sRGB and raw pixels, the RAWFormer module is proposed. It employs a global attention mechanism to transform the low-light input image from R-G-B to the R-G-G-B Bayer format. Lastly, to improve the visual experience and enhance LIE performance, the RRAW-sRGB module is designed to effectively simulate camera ISP operations, such as white balance and color correction.
As depicted in Figure 1, the proposed method, sRrsR-Net, outperforms state-of-the-art methods across sRGB and raw domain datasets, such as sRGB domains LOL-v1, LOL-v2-real, LOL-v2-synthetic, and raw domain SID. Our contributions can be summarized as follows:
  • We propose a novel LIE framework, sRrsR-Net, integrating feature information of both sRGB and raw domains; in our network, by using a translation process of sRGB–raw–sRGB, the training data in the raw domain are avoided;
  • In the translation process of sRGB–raw–sRGB, three simple and efficient modules are proposed; they are RGB-iRGB, RAWFormer, and RRAW-sRGB, respectively;
  • Quantitative and qualitative experiments demonstrate that our model sRrsR-Net outperforms existing state-of-the-art methods in both raw and sRGB image domains.

2. Related Work

2.1. Low-Light Image Enhancement

In traditional LIE approaches, methods like histogram equalization and gamma correction [15] directly enhance the visibility and contrast of low-light images. However, these techniques often overlook the illumination intensity, which makes those processed images deviate from the real-world lighting conditions. With the rapid development of deep learning, Convolutional Neural Networks (CNNs) have been widely applied to the LIE area with the sRGB domain. For instance, Guo et al. [16] propose a method that employs global curves for brightness adjustment. Additionally, Zamir et al. [17] propose MIR-Net to integrate multi-scale contextual information while preserving high-resolution spatial details. Wang et al. introduces network DeepUPE [18], an advanced neural network for photo enhancement that incorporates intermediate illumination layers which leads to the recovery of clear details. Li et al. [19] introduce LightDiff to enhance the quality of low-light images for autonomous driving applications. Despite the advancements, these CNN-based methods face big challenges in capturing long-distance connections between image pixels. To address this issue, the network Restormer [20] is proposed to incorporate a Transformer structure into the U-Net architecture to improve global connections between features. Cai et al. propose an innovative one-stage framework [9] for low-light image enhancement, named Retinexformer, which integrates the strengths of Retinex theory with the Transformer architecture. Although this novel framework exhibits great performance across various datasets, its computational complexity is high, particularly for high-resolution images. For structural light weighting, Zhang et al. [21] introduce a structure-aware lightweight Transformer for image enhancement tasks. Cui et al. [1] present a compact model with only 90k parameters, incorporating a global attention module to simulate ISP processes for LIE.
However, these deep learning models are primarily designed for sRGB images, which creates a domain gap when applied to raw images and reduces the models’ generalization performance. Therefore, Huang et al. [22] proposed REENet which converts sRGB images to the linear raw domain when enhancing the images. Recently, in the raw domain, Xu et al. [4] proposed SNR, which dynamically adapts long-range operations for noisy areas and short-range operations for clearer regions, enhancing the overall image quality effectively. However, the reliance on the SNR prior may affect the accuracy of enhancement across different images. Chen et al. [23] introduces a raw domain low-light image dataset and directly enhance the raw data based on end-to-end deep learning pipeline. Nonetheless, methods leveraging raw images for LIE generally require raw images as training data. This poses challenges, as raw images are often difficult to obtain and inconvenient for training. To address these issues, our study introduces a novel model that leverages sRGB–raw–sRGB transformation before LIE. This approach allows the model to operate effectively in both raw and sRGB domains, mitigating the limitations associated with raw image acquisition while achieving high performance across both domains.

2.2. Raw Image Reconstruction with Metadata

Raw image reconstruction refers to the process of converting an sRGB image to a raw format. Liu et al. [24] propose three improved Convolutional Neural Networks (CNNs) for learning the mapping from Low Dynamic Range to High Dynamic Range. However, these methods are tailored to specific cameras, constraining the generalization capability. Moreover, incorporating nonlinear operations, such as tone mapping, in the reconstruction process leads to inevitable information loss. To enhance the quality of the original image reconstruction, it is crucial to utilize metadata which contain essential information to guide the reconstruction process. Afifi et al. [24] store a small portion of image information as metadata to assist in raw reconstruction. Nam et al. [25] develop a joint training approach for Sampler and reconstruction network modules based on the U-Net architecture to improve the efficiency of both sampling and recovery processes. However, training multiple deep neural networks simultaneously demands substantial computational resources and memory. To address this, we have developed a new RGB-iRGB module, which efficiently utilizes max-pooling to simplify the process.

2.3. Image Signal Processing with Learning

Image Signal Processing is an essential phase in the camera imaging pipeline. It includes various modules such as demosaicing, denoising, color correction, white balance, and so on. Traditional camera ISP methods often create a mapping function for individual color channels [26]. However, these kinds of methods can not accurately reflect the full complexity of camera ISP, which involves multiple nonlinear mappings. Recent ISP models have dealt with this challenge with DNNs. They transform raw sensor data into the sRGB format and significantly impact image quality. Cui et al. [1] design a neural network for fitting ISP rendering to enhance low-light images. Additionally, Xing et al. [27] introduce a reversible ISP network to learn the reciprocal mapping between sRGB space and raw space. Zamir et al. [28] propose a recursive ISP for mutual conversion between sRGB and raw. It can denoise both sRGB and raw images separately.
However, these end-to-end methods rely on specific camera ISP parameters, which means that each camera requires enough training data for modeling, demanding significant computational resources for model generalization. Additionally, implementing an end-to-end ISP pipeline inevitably adds operations that have minimal impact on light adjustment, such as tone mapping and gamma correction. In contrast, our research just focuses on key ISP steps related to image quality; they are color correction and white balance.

3. Methods

In this section, firstly, we present the overall architecture of our model sRrsR-Net which follows the image translation process of sRGB–raw–sRGB and avoids raw images as training data; then, we explain its main four modules (Sampler, RGB-iRGB, RAWFormer, and RRAW-sRGB) in detail.

3.1. Overall Architecture

As illustrated in Figure 2, our low-light image enhancement model, sRrsR-Net, consists of four primary modules: Sampler, RGB-iRGB, RAWFormer, and RRAW-sRGB. The Sampler module first extracts the main metadata M (i.e., M e t a d a t a ( I ) ) from the input image I by a max-pooling operation. These metadata can assist in the reconstruction of the raw image of the original sRGB image I . Then, the RGB-iRGB module takes both the extracted metadata M and the initial low-light sRGB image I as inputs and produces an RGB intermediate feature map F ^ r . This feature map is subsequently processed by RAWFormer, a Transformer-based module, to produce the reconstructed raw image I r . Finally, the reconstructed raw image I r is fed into the last RRAW-sRGB module, which can partly simulate ISP to generate our ultimate enhanced low-light sRGB image I ^ .

3.2. Metadata Sampler and RGB-iRGB Modules

In our approach with sRrsR-Net, in order to leverage the rich details of raw images, before performing low-light enhancement, it is necessary to convert the sRGB image to the raw format. However, generally, the raw image reconstruction is tailored to specific camera parameters and may lead to inaccurate reconstruction. Moreover, the incorporation of nonlinear operations, such as tone mapping, in the reconstruction process inevitably leads to information loss. Therefore, incorporating the key information of sRGB image I , named M (i.e., m e t a d a t a ( I ) ), into the raw reconstruction process is essential. Here, M represents metadata of one image which contain its key image-specific information. Inspired by [25], sRrsR-Net introduces the Sampler module to sample I to assist in raw image reconstruction. However, the joint training of the proposed Sampler consumes more computational resources. Therefore, in the sampling process, this study only adopts a max-pooling operation [29], a downsampling technique that reduces the spatial dimensions of input data while retaining the most salient features. By this simple sampling process, our method can maintain a balance between the desired effects and overall performance. Concretely, the sampling process of the Sampler module is initiated by inputting low-light image I into a max-pooling layer. The sampled metadata M of image I are computed as follows:
M = M a x P o o l i n g ( I )
M a x P o o l i n g is a max-pooling method. Then, these features M are concatenated with the original low-light image I to obtain rich RGB intermediate feature maps of image I .
As shown in Figure 3, the RGB-iRGB module takes the concatenated data c o n c a t ( I ,   M ) as input, which is in the RGB three-channel format, and outputs the RGB intermediate feature map F ^ r , also in the RGB three-channel format. The process of this network can be mathematically expressed as follows:
F ^ r = f R G B - i R G B ( c o n c a t ( I ,   M ) )
Here, f R G B - i R G B is assumed to be a universal function for the RGB-iRGB module. Since there are significant changes in the arrangement of image pixels during the reconstruction process, to facilitate a smoother and more effective reconstruction, this intermediate feature map F ^ r is introduced.
As illustrated in Figure 3, the architecture of the RGB-iRGB network follows the U-Net framework. Firstly, the original input low-light image I ∈ R H × W × R G B undergoes convolution to obtain low−level features F 0 ∈ R H × W × C , where H × W represents the feature map size, and C is the channel number. Following this, the downsampling operation will downsample F 0 to obtain deeper features F d 1 ∈ R H 2 × W 2 × 2 C and F d 2 ∈ R H 4 × W 4 × 4 C . In the final step, a combination of transposed convolution for upsampling and convolution operations is applied. This process increases the number of feature channels by half while simultaneously doubling the spatial dimensions of the feature map. The upsampling feature map is then passed through a 3 × 3 convolution, resulting in the final output feature map F ^ r ∈ R H × W × R G B . The convolutional kernel size in the RGB-iRGB network is 3 × 3, and downsampling is accomplished through max-pooling with a 2 × 2 pooling kernel. Conversely, upsampling utilizes 2 × 2 transposed convolution kernels.

3.3. RAWFormer Module

The design of the RAWFormer module is inspired by the Bayer pattern of R-G-G-B color image arrangement. In digital photography arrangements, the green (G) channel in raw images is twice the number of the red (R) and blue (B) channels due to the human eye’s greater sensitivity to green. To accommodate the pixel distribution, each pixel has R and B occupying 1/4 and G occupying 1/2.
Based on a Transformer architecture, the RAWFormer module is designed to establish global connections between the RGB domain and the raw domain. Concretely, this module is proposed to transform the input RGB intermediate feature map F ^ r into raw Bayer array format image I r ∈ R H × W × R G G B . The “RGB” denotes the red, green, and blue channels, respectively. And the “RGGB” channels include an additional green channel. The transform function for the RAWFormer module is the function 3.
I r = f R A W F o r m e r ( F ^ r )
where f R A W F o r m e r represents the Bayer array fitting function, and I r represents the reconstructed raw image of our approach, which is the output for RAWFormer module.
The detailed structure of the RAWFormer module is depicted in Figure 4. This process transforms the pixels from the pixel arrangement of the RGB domain to the Bayer array in the raw image.
The RAWFormer module utilizes the self-attention (SA) mechanism [30] to capture relationships among global pixels. Firstly, the input feature F ^ r , divided into its three RGB channels with the G channel remaining unchanged, undergoes downsampling through 3 × 3 convolutional kernels, resulting in a corresponding red feature map F r and blue feature map F b , each occupying half of the G channel. This configuration aligns with the pixel distribution of the R-G-G-B pattern.
F r = D s F ^ r ,   F b = D s F ^ r ,   F g = F ^ r
Here, the downsampling operation D s reduces the size of the input feature map by half, by utilizing a Pixel-Unshuffle module [31].
Next, as illustrated in Figure 4, we consider the Multi-head Self-Attention mechanism (MSA) [30]. In MSA, firstly, three linear transformation matrices W q , W k , and W v (corresponding to the linear layers in the diagram) are applied to generate queries ( Q ), keys ( K ), and values ( V ). After obtaining Q ,   K ,   V for each RGB channel, the next step is to compute self-attention weights individually. The self-attention mechanism calculates the similarity between each feature and all other features by taking the dot product of the Q ,   K matrices. This process produces attention weights for RGB channels feature. These weights are then used to perform a weighted sum for each element by multiplying them with the V matrix. The final outcome is the attention vector, corresponding to the Scaled Dot-Product Attention ( S A ).
S A = s o f t m a x ( Q K T d k ) V
In Equation (5), the parameter d k is a learnable scaling factor used to control the magnitude of the dot product of Q and K before applying the softmax function. The softmax function converts a vector of values into a probability distribution. The implementation of self-attention follows previous recommendations, incorporating the use of Depthwise Separable Convolution methods [20,32]. By employing 1 × 1 Pointwise Convolution (PW) and 3 × 3 Depthwise Separable Convolution (DW), this approach not only reduces parameters and computational costs but also aggregates information between channels and local features.
The MSA combines information learned from different “heads” to better capture inter-sequence information. RAWFormer divides the number of self-attention channels into N “heads” to simultaneously learn different attention maps. The allocation of “heads” for the RGB channels can follow the standard MSA approach, assigning an equal number of “heads” to each RGB channel. However, increasing the number of “heads” results in higher computational costs. To improve efficiency, the number of “heads” are divided into three groups: N r H for red, N b H for blue, and N g H for green. This allocation aligns with the pixel distribution of the R-G-G-B pattern, the final allocation of “heads” is as follows:
N r H = N r H = N g H 2 ,                 N = N r H + N r H + N g H
Finally, attention is learned independently for each allocated “head,” and the results are then concatenated:
M S A = c o n c a t [ S A N r H F r ,   S A N g H F g ,   S A N b H F b ] W 0
The parameter W 0 denotes a learnable matrix, and the concat operation combines the outcomes from each RGB channel. The RAWFormer module is specifically designed to align with the R-G-G-B pixel distribution of Bayer arrays, ensuring the establishment of global attention across pixels. The F r , F g , and F b feature maps processed by MSA are then concatenated together, fusing into feature map F r g g b . This fusion enhances the bit depth information contained within the feature maps. After obtaining the feature map F r g g b , a linear layer is used to transform the feature map into the reconstructed raw image I r , bridging the gap between deep feature representation and the final image output.

3.4. RRAW-sRGB Module

The RRAW-sRGB module is designed to convert the reconstructed raw image data generated from the RAWFormer module back into the sRGB format, aiming to enhance the quality of the initial low-light image. This module employs an encoder–decoder architecture, with ISP processing functions such as white balance and color correction introduced to improve image brightness. The detailed structure of the RRAW-sRGB network is illustrated in Figure 5. The RRAW-sRGB module accepts the reconstructed raw image from the RAWFormer module as input. After a downsampling process, it generates deep-level feature maps F d 1 ∈ R H 2 × W 2 × 2 C . These features then pass through a WB block. The output feature map is subsequently merged with the upsampling feature map from the Decoder. A similar process is applied to the deeper feature maps F d 2 ∈ R H 4 × W 4 × 4 C , which undergoe color correction through the CCM block before being fused. Both the WB and CCM blocks are designed to make the output more representative of real-world scenes. The feature maps processed through WB and CCM blocks are ultimately fused to produce the enhanced normal-light sRGB image I ^ .
Many end-to-end cameras’ ISP process typically integrate numerous image processing operations, but some of these operations, such as tone mapping and gamma correction often offer limited benefits for LIE. In contrast, the RRAW-to-sRGB module simplifies the process by focusing on white balance and color correction which have a substantial impact on color and brightness. During the white balance process, the intensities of the R, G, and B color channels are adjusted to ensure that white appears consistent with human perception. This process is commonly represented by a white balance gains matrix D w b .
D w b = G r 0 0 0 G g 0 0 0 G b
The white balance operation is achieved by multiplying the sensor’s red, green, and blue color channels with their respective gain coefficients G r , G g , and G b . These gain coefficients can be learned through the parameters of a neural network. The WB blocks are designed with the understanding that they involve 3 × 3 matrices. To approximate these matrices, a Depthwise Separable Convolution [33] with a 3 × 3 kernel is used. Additionally, a 1 × 1 Pointwise Convolution (PW) is applied to expand the number of channels in the feature map, mapping it into a higher-dimensional space. After that, the ReLU activation function performs nonlinear filtering on the batch normalized (BN) features. Finally, another 1 × 1 PW is applied to restore the features to their original dimension, enhancing the feature representation and inter-feature connections.
In addition, the human eye and camera sensors respond differently to various colors, leading to color shifts in captured images. To correct for these shifts, this study employs a CCM block which uses a 3 × 3 matrix to harmonize the image colors, as shown in the formula:
I ^ = F d 2 · C m
where C m represents the learnable parameters of the CCM block, and it is implemented by a 3 × 3 convolution operation. The output, I ^ , is the final enhanced sRGB image.

3.5. Loss Function

The loss function of our LIE approach sRrsR-Net is defined based on two aspects: the reconstruction of image pixel details and the quality of generated images. To address the reconstruction of pixel details, we incorporate the L 1 loss. Additionally, to enhance the brightness and perceptual quality of the reconstructed images, we employ perceptual loss ( L v g g ) as guidance.
L 1 , also known as the mean absolute error, is applied to minimize the sum of absolute differences between corresponding pixel values in the enhanced image and the ground-truth image. Its goal is to recover image details effectively. On the other hand, L v g g relies on features extracted from a pretrained VGG network [34]. The input to the VGG network includes both the image to be enhanced and the ground-truth image. By comparing the feature representations of these two images, the enhancement process is guided to achieve semantic and stylistic similarity. The perceptual loss function is computed as the Euclidean distance between the feature representations of the enhanced and ground-truth images. The overall loss function of sRrsR-Net, L t o t a l , is expressed as follows:
L t o t a l = λ 1 L 1 + λ 2 L v g g
where λ 1 and λ 2 are weight coefficients. We consider both perceptual quality and reconstruction of pixel details equally important, so λ 1 and λ 2 are both set to 0.5.

4. Experiments

4.1. Datasets, Metrics, and Implementation Details

Datasets. In this paper, the experiments were conducted on four public datasets to assess the performance of our low-light image enhancement model sRrsR-Net. They are LOL-v1 [35], LOL-v2 [36], VE-LOL [37], and SID [23]. Among them, the former three belong to sRGB domains, and the last belongs to the raw domain. Specially, the LOL-v2 dataset is further divided into real and synthetic subsets, named LOL-v2-real and LOL-v2-synthetic. All these datasets consist of pairs of low-light and normal-light images. The distribution of training and test sets for LOL-v1, LOL-v2-real, LOL-v2- synthetic, VE-LOL, and SID are the ratios of 485:15, 689:100, 900:100, 800:200, and 800:200, respectively. For the raw dataset SID [23], there are 2688 pairs of short-long exposure raw images captured by Sony cameras.
Each dataset presents unique challenges and features that influence the evaluation of our model. The LOL-v1 dataset offers a starting point for research in real-world low-light conditions but may lack the diversity needed for robust model training due to its relatively small size. The LOL-v2-real subset expands upon LOL-v1’s foundation by providing a larger dataset (789 pairs).
Metrics. Here, three metrics for the quantitative evaluation of sRrsR-Net model performance are selected, they are the Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM) [38], and Learned Perceptual Image Patch Similarity (LPIPS) [39]. Both PSNR and SSIM are objective metrics, and LPIPS can better conform to human visual perception. Specially, the PSNR measures the maximum signal power-to-noise power ratio, primarily focusing on global quality aspects such as brightness and contrast; the SSIM mainly delves into the visual impact of image similarity by considering luminance, contrast, and structural information, offering a more perceptually aligned evaluation; and the LPIPS is a deep learning-based metric that is capable of capturing the subtle perceptual differences in image quality that the human visual system can discern.
Implementation Details. In our experiments, the model sRrsR-Net was implemented by PyTorch and trained on a single NVIDIA A100 GPU with a batch size of 32 for 120 epochs. The Adam optimizer was employed for parameter tuning, with an initial learning rate of 1 × 10−4 and weight decay set to 0. During the training phase, the low-light images for input were resized to 224 × 224, and they underwent flipping and rotation as part of the data augmentation process. In addition, other comparative models also applied the same image preprocessing methods as the sRrsR-Net model. In the following experiments, many low-light image enhancement approaches were considered, such as Zero-DCE, DeepUPE, EnGAN, UFormer, MIRNet, Sparse, Retinexformer, RQ, SNR, SID, EEMEFN, FIDE, LLNet, RRDNet, EnlightenGAN, KinD, KinD++, RUAS, LLFlow, and so on.
Firstly, to evaluate these models, experiments were conducted on both the sRGB datasets (LOL-v1, LOL-v2-real, and LOL-v2-synthetic) and the raw domain dataset (SID), aiming to enhance low-light conditions in various scenarios. Secondly, for generalization performance, several models were trained on the LOL-v1 dataset, and then they were tested on the VE-LOL dataset. In terms of computational performance, comparisons were made among different model architectures, specifically those based on Transformer, CNN, and U-Net frameworks. Moreover, a lot of ablation experiments were performed to validate the effectiveness of three modules (i.e., RGB-iRGB, RAWFormer, and RRAW-sRGB) in the model sRrsR-Net.

4.2. Comparison Analysis on the sRGB Domain Datasets

In this part, three sRGB domain datasets were considered: LOL-v1, LOL-v2-real, and LOL-v2-synthetic, each presenting unique challenges. The main quantitative experimental results for ten models are shown in Table 1. Our proposed method achieves state-of-the-art performance on most datasets, demonstrating excellent results in terms of the PSNR and SSIM metrics.
The LOL-v1 dataset, with its small size and high dynamic range lighting conditions, poses challenges for generalization. Despite this, our model achieved a PSNR of 29.49 dB and an SSIM of 0.861, surpassing the second-best method, SNR, by 2.77 dB in PSNR and 0.221 in SSIM, effectively handling the diverse and challenging real-world scenes. For the LOL-v2-real dataset, which contains a wider variety of lighting environments, our model achieved a PSNR of 29.48 dB and an SSIM of 0.889, improving by 2.27 dB in PSNR and 0.018 in SSIM over the second-best method, showcasing robustness in realistic conditions.
On the LOL-v2-synthetic dataset, our method delivered a 2.13 dB improvement in PSNR compared to the competitive method SNR, but the SSIM dropped slightly by 0.008. The slight decrease in SSIM can be attributed to the design of our ISP enhancement module, which prioritizes improvements in color and lighting. This may trade off some structural consistency for better subjective visual quality. This trade-off likely contributed to the slight decrease in SSIM. Nonetheless, the overall performance demonstrates the robustness and effectiveness of our method across diverse datasets.
Qualitatively, here we offer a visual comparison of the proposed model’s performance against four mainstream low-light enhancement methods on the LOL-v1 and LOL-v2 datasets, they are MIRNet, SNR, Retinexformer, and Zero-DCE, respectively. The enhancement results of five image examples from these two datasets are illustrated in Figure 6 highlights how our model, sRrsR-Net, produces enhanced images with more consistent color representation and natural brightness compared to the other methods. For instance, in the enlarged portions of the images (line 4 in Figure 6), our model sRrsR-Net effectively eliminates shadows caused by uneven lighting, while maintaining the natural color of the green lake water, closely resembling the real-world scene. This is due to the RAWFormer module’s sensitivity to green tones, which helps in producing more natural colors during image conversion. In contrast, Zero-DCE tends to produce images with a grayish tint due to its lightweight nature. Retinexformer does not perform as well in maintaining color fidelity, leading to less natural-looking images. The SNR model exhibits inconsistent color reproduction, particularly in the enlarged lake region, where darker patches in the lower-right corner give the appearance of fragmentation.

4.3. Comparison Analysis on the Raw Domain Datasets

In this part, many experiments were conducted with raw domain dataset SID; firstly, four channels, R-G-G-B, in images from SID are fused into three channels, R-G-B; then, the generated images are viewed as input images of 12 LIE models in Table 2. According to the experimental results of these 12 models, it is demonstrated that within the raw domain of SID, our method maintains state-of-the-art performance regarding both PSNR and SSIM. Specifically, compared to the second-best approach, SNR, our method exhibits enhancements of 0.23 dB and 0.019 in PSNR and SSIM on the SID dataset, respectively. These quantitative results illustrate the effectiveness of our low-light image enhancement technique in the raw domain.
Compared with other six approaches (EDSR, ABF, Retinexformer, SID, RED, and SNR), Figure 7 clearly illustrates that the reconstructed images by sRrsR-Net can retain more image details with the assistance of RGB-iRGB metadata. As shown in the enlarged apple appearing in the input image, i.e., the left-top image in Figure 7, our method better ensures that the contour and the details inside are also well restored. In contrast, the compared methods exhibit severe content distortion and artifacts due to their limited information for image reconstruction. For instance, as shown in the lower left corner of the image with the “marker pen”, our model restores normal lighting conditions while preserving the color and outline of the pen consistent with the real scene. In contrast, the SNR and Retinexformer exhibit ghosting and blurring artifacts. However, our model has certain limitations in some extremely low-light conditions. In the same figure, when reconstructing the apple in the above image, the “apple” generated by our model produces a darker result compared to the SID model. Nevertheless, the contour of the apple in our model is more clearly defined compared to other methods, and the SID model’s contour is already blurred. Therefore, while our model performs well in most low-light conditions, it still faces challenges in extremely low-light scenarios, indicating areas for future improvement.

4.4. Generalization Performance Evaluation

To assess the generalization performance of our model sRrsR-Net, it is firstly trained on the LOL-v1 dataset and then evaluated for its performance on the VE-LOL [37] test set, which includes both real-captured and synthetic low-light images. To effectively simulate human perception of image similarity, the LPIPS evaluation metric was used here. This metric reflects the quality of low-light images enhanced by the model, aiming to assess the model’s generalization at the perceptual level in real scenes.
To evaluate the generalization ability of various models, we considered 10 low-light enhancement models. In Table 3, a vertical comparison reveals the quality of image enhancement across different models. For our low-light enhancement model sRrsR-Net, the higher SSIM value indicates more effective preservation of the original images’ structural information and excellent structural similarity. And the experimental indicator LPIPS shows a perceptual difference between the generated images and the ground truth, indicating that the model’s generated images are visually closer to the real images. Considering the SSIM, PSNR, and LPIPS results in Table 3, it is evident that sRrsR-Net performs well in LIE tasks. As shown in Table 3, LLFlow exhibits better performance in certain aspects. Specifically, on the LOL-v1 and VE-LOL datasets, LLFlow outperforms sRrsR-Net in SSIM metrics by 0.011 and 0.04, respectively.
Moreover, when comparing the model’s performance across different datasets, the horizontal analysis reveals no significant decline in the test metrics, indicating that our model indeed possesses robust generalization capabilities. But from Table 3, for other methods, there are enormous gaps between the results for PSNR, SSIM, and LPIPS in LOL-V1 and those in VE-LOL.
Here, Figure 8 presents visual results from the generalization test. It is evident that the effects of images enhanced by our model are the best on both the LOL-v1 and VE-LOL datasets, displaying consistent color restoration and retaining complete image details.

4.5. Computational Performance

To evaluate computational efficiency and inference speed, this study also compares our model with another eight models from three categories: Transformer, CNN, and U-Net-like. The inference speed is measured using PyTorch’s serial scheduler on a GPU, with each model tested 20 times on the respective datasets and with the average values taken. All images underwent the same preprocessing, and the input sizes are the same.
The results of the running time, shown in Table 4 and Figure 9, indicate that our model sRrsR-Net outperforms the average and exhibits excellent performance. Among the Transformer-based models, the Retinexformer method reduces running time by processing images using a one-stage retinex-based framework. For the U-Net-like models, the Zero-DCE algorithm uses a specific curve to estimate the task, which helps reduce running time and leads to its best performance.
Although our method performs well in terms of PSNR, SSIM, and LPIPS, it has some limitations when regarding computational cost. Specifically, our method has a high FLOPs (floating point operations) value. This is due to the integration of the RAWFormer module, which is based on Transformer architectures. When calculating global self-attention, the computational complexity increases quadratically with the size of the input image. For an input image comprising N pixels, the number of operations required for self-attention can be approximated as O(N2). This quadratic scaling implies that even a slight increase in image resolution can significantly escalate the computational demands.

4.6. Ablation Study

To assess the individual impacts of the three modules RGB-iRGB, RAWFormer, and RRAW-sRGB on our low-light image enhancement model, seven ablation experiments were conducted on the LOL-v1 dataset. The baseline model is established without these three modules but keeps the U-Nets unchanged. As for the other models in Table 5, when the RGB-iRGB or RRAW-sRGB module are not used, it means that just the U-Nets in these two modules are kept unchanged. Table 5 presents the quantitative results of these ablation experiments, and, qualitatively, Figure 10 gives the visual results of these experiments.
Firstly, compared with the baseline model, there exist notable improvements in PSNR, SSIM, and LPIPS metrics on models A, B, and C, with only one module {RGB-iRGB, RAWFormer, or RRAW-sRGB}, especially for the PSNR metric. Concretely, with the RGB-iRGB module added, model A resulted in notable improvements in the PSNR, SSIM, and LPIPS metrics by 5.91 dB, 0.048, and 0.046, respectively, over the baseline model; model B resulted in notable improvements in the PSNR, SSIM, and LPIPS metrics by 5.31 dB, 0.04, and 0.048, respectively; model C resulted in notable improvements in the PSNR, SSIM, and LPIPS metrics by 6.45 dB, 0.06, and 0.095, respectively. Obviously, among these three modules, model C with just the RRAW-sRGB module achieved the best results with respect to all three metrics.
Secondly, as illustrated in Table 5, when combining RGB-iRGB and RRAW-sRGB, or combining RAWFormer and RRAW-sRGB, the experiments in the corresponding models D and E achieved better results in any metric than the models A, B, and C with only one module, except in the SSIM metric for model E. For instance, compared with model C, model D has an improvement of 1.62 dB in PSNR, about 0.008 in SSIM, and 0.027 in LPIPS.
Lastly, when combing all three modules, model E achieved the best result in all three metrics. Specifically, compared with model D, model F exhibits further improvement, about +0.89 dB in PSNR, +0.002 in SSIM, and −0.007 in LPIPS, respectively.
According to the above quantitative analysis, compared with the baseline model, each module {RGB-iRGB, RAWFormer, and RRAW-sRGB} can help improve the model performance for low-light images. Moreover, combining these modules can achieve better results.
As illustrated in Figure 10, just with the ISP blocks in the enhancement process, model A’s output images provide a more realistic color representation than the baseline model. With the RAWFormer module, model B exhibits fewer blurry artifacts compared to the baseline, and its color tones are more consistent with real-world scenes. For instance, in the enlarged section of the red rope shown in Figure 10, the color closely resembles the reference image. With the RRAW-sRGB module, comparing the outputs of models C with A and B in Figure 10 reveals that model C produces images with more detailed textures. As shown in the enlarged silverware, its patterns are clearer, and the details are richer. This suggests that RGB-iRGB can leverage raw images with more image details and is beneficial for enhancing low-light images. The effectiveness of raw details is further highlighted when comparing models D and E.

5. Conclusions

In this paper, we propose a novel deep neural network sRrsR-Net for enhancing low-light images, by converting them from sRGB to raw and then back to sRGB under normal lighting conditions. During this transformation, the Sampler module extracts metadata to assist the RGB-iRGB module in better reconstructing raw images. Specifically, we introduce the RAWFormer module to map the arrangement of pixels to better align with the R-G-G-B Bayer format. Additionally, we have also designed a new RRAW-sRGB module with ISP blocks to guide the color enhancement of low-light images. Extensive experiments in both the sRGB and raw domains demonstrate the effectiveness and robustness of the proposed method, particularly in retaining detailed image information and enhancing color consistency. Compared to traditional methods like Retinex-based techniques and Gamma correction, sRrsR-Net offers superior detail preservation and color accuracy. While deep learning methods such as Zero-DCE and EnlightenGAN also improve low-light images, they may have drawbacks like introducing artifacts or color distortion. Our sRrsR-Net addresses these issues and demonstrates excellent performance in generalization evaluation.
However, it is important to acknowledge the limitations inherent in our approach. Firstly, while sRrsR-Net has shown excellent generalization ability, it may not be robust against some extreme low-light conditions. Secondly, the deep neural network model sRrsR-Net demands more computational resources than some traditional methods, such as Retinex. Our future work will aim to address these limitations and focus on improving our current method to handle a broad range of lighting conditions and to reduce computational costs without compromising the quality of enhancement.

Author Contributions

Z.H.: Conceptualization, Methodology, Resources, Supervision; D.Z.: Conceptualization, Methodology, Software, Investigation, Formal Analysis, Writing—Original Draft; L.X.: Investigation, Formal Analysis, Writing—Review and Editing. X.L.: Writing—Review and Editing; Y.L.: Writing—Review and Editing. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Doctoral Research Initiation Project of Wuyi University (Grant No. BSQD2304).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

All data that support the findings of this study are included within the article.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Cui, Z.; Li, K.; Gu, L.; Su, S.; Gao, P.; Jiang, Z.; Qiao, Y.; Harada, T. You Only Need 90K Parameters to Adapt Light: A Light Weight Transformer for Image Enhancement and Exposure Correction. arXiv 2022. [Google Scholar] [CrossRef]
  2. Tran, D.Q.; Aboah, A.; Jeon, Y.; Shoman, M.; Park, M.; Park, S. Low-Light Image Enhancement Framework for Improved Object Detection in Fisheye Lens Datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 16–22 June 2024; pp. 7056–7065. [Google Scholar]
  3. Zhu, A.; Zhang, L.; Shen, Y.; Ma, Y.; Zhao, S.; Zhou, Y. Zero-Shot Restoration of Underexposed Images via Robust Retinex Decomposition. In Proceedings of the 2020 IEEE International Conference on Multimedia and Expo (ICME), London, UK, 6–10 July 2020; pp. 1–6. [Google Scholar]
  4. Xu, X.; Wang, R.; Fu, C.-W.; Jia, J. SNR-Aware Low-Light Image Enhancement. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 17693–17703. [Google Scholar]
  5. Jiang, Y.; Chang, S.; Wang, Z. TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up. arXiv 2021. [Google Scholar] [CrossRef]
  6. Marnissi, M.A.; Fathallah, A. GAN-Based Vision Transformer for High-Quality Thermal Image Enhancement. In Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Vancouver, BC, Canada, 17–24 June 2023; pp. 817–825. [Google Scholar]
  7. Zhou, D.; Yang, Z.; Yang, Y. Pyramid Diffusion Models For Low-Light Image Enhancement. arXiv 2023, arXiv:2305.10028v1. [Google Scholar]
  8. Fei, B.; Lyu, Z.; Pan, L.; Zhang, J.; Yang, W.; Luo, T.; Zhang, B.; Dai, B. Generative Diffusion Prior for Unified Image Restoration and Enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 17–24 June 2023; pp. 9935–9946. [Google Scholar]
  9. Cai, Y.; Bian, H.; Lin, J.; Wang, H.; Timofte, R.; Zhang, Y. Retinexformer: One-Stage Retinex-Based Transformer for Low-Light Image Enhancement. arXiv 2023. [Google Scholar] [CrossRef]
  10. Zhang, Z.; Wang, H.; Liu, M.; Wang, R.; Zhang, J.; Zuo, W. Learning RAW-to-sRGB Mappings with Inaccurately Aligned Supervision. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 10–17 October 2021; pp. 4328–4338. [Google Scholar]
  11. Wang, Y.; Yu, Y.; Yang, W.; Guo, L.; Chau, L.-P.; Kot, A.; Wen, B. Raw Image Reconstruction with Learned Compact Metadata. In Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 17–24 June 2023. [Google Scholar]
  12. Xu, S.; Song, B.; Chen, X.; Liu, X.; Zhou, J. Image Demoireing in RAW and sRGB Domains. arXiv 2024. [Google Scholar] [CrossRef]
  13. Conde, M.V.; Vasluianu, F.; Timofte, R. BSRAW: Improving Blind RAW Image Super-Resolution. In Proceedings of the 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 3–8 January 2024; pp. 8500–8510. [Google Scholar]
  14. Wei, K.; Fu, Y.; Yang, J.; Huang, H. A Physics-Based Noise Formation Model for Extreme Low-Light Raw Denoising. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 9–13 June 2020; pp. 2758–2767. [Google Scholar]
  15. Rahman, S.; Rahman, M.M.; Abdullah-Al-Wadud, M.; Al-Quaderi, G.D.; Shoyaib, M. An Adaptive Gamma Correction for Image Enhancement. J. Image Video Proc. 2016, 2016, 35. [Google Scholar] [CrossRef]
  16. Guo, C.; Li, C.; Guo, J.; Loy, C.C.; Hou, J.; Kwong, S.; Cong, R. Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; pp. 1780–1789. [Google Scholar]
  17. Zamir, S.W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F.S.; Yang, M.-H.; Shao, L. Learning Enriched Features for Real Image Restoration and Enhancement. In Computer Vision–ECCV 2020; Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M., Eds.; Lecture Notes in Computer Science; Springer International Publishing: Cham, Switzerland, 2020; Volume 12370, pp. 492–511. ISBN 978-3-030-58594-5. [Google Scholar]
  18. Wang, R.; Zhang, Q.; Fu, C.-W.; Shen, X.; Zheng, W.-S.; Jia, J. Underexposed Photo Enhancement Using Deep Illumination Estimation. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019; pp. 6842–6850. [Google Scholar]
  19. Li, J.; Li, B.; Tu, Z.; Liu, X.; Guo, Q.; Juefei-Xu, F.; Xu, R.; Yu, H. Light the Night: A Multi-Condition Diffusion Framework for Unpaired Low-Light Enhancement in Autonomous Driving. In Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 16–22 June 2024; pp. 15205–15215. [Google Scholar]
  20. Zamir, S.W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F.S.; Yang, M.-H. Restormer: Efficient Transformer for High-Resolution Image Restoration. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 5728–5739. [Google Scholar]
  21. Zhang, Z.; Jiang, Y.; Jiang, J.; Wang, X.; Luo, P.; Gu, J. STAR: A Structure-Aware Lightweight Transformer for Real-Time Image Enhancement. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 10–17 October 2021; pp. 4106–4115. [Google Scholar]
  22. Huang, H.; Yang, W.; Hu, Y.; Liu, J.; Duan, L.-Y. Towards Low Light Enhancement With RAW Images. IEEE Trans. Image Process. 2022, 31, 1391–1405. [Google Scholar] [CrossRef] [PubMed]
  23. Chen, C.; Chen, Q.; Xu, J.; Koltun, V. Learning to See in the Dark. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 3291–3300. [Google Scholar]
  24. Afifi, M.; Punnappurath, A.; Finlayson, G.; Brown, M.S. As-Projective-as-Possible Bias Correction for Illumination Estimation Algorithms. J. Opt. Soc. Am. A 2019, 36, 71. [Google Scholar] [CrossRef] [PubMed]
  25. Nam, S.; Punnappurath, A.; Brubaker, M.A.; Brown, M.S. Learning sRGB-to-Raw-RGB De-Rendering with Content-Aware Metadata. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 17683–17692. [Google Scholar]
  26. Debevec, P.E.; Malik, J. Recovering High Dynamic Range Radiance Maps from Photographs. In Proceedings of the ACM SIGGRAPH 2008 Classes, Los Angeles, CA, USA, 11 August 2008; pp. 1–10. [Google Scholar]
  27. Xing, Y.; Qian, Z.; Chen, Q. Invertible Image Signal Processing. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021; pp. 6283–6292. [Google Scholar]
  28. Zamir, S.W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F.S.; Yang, M.-H.; Shao, L. CycleISP: Real Image Restoration via Improved Data Synthesis. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–9 June 2020; pp. 2693–2702. [Google Scholar]
  29. Boureau, Y.-L.; Ponce, J.; Lecun, Y. A Theoretical Analysis of Feature Pooling in Visual Recognition. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel, 21–24 June 2010; pp. 111–118. [Google Scholar]
  30. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention Is All You Need. arXiv 2017. [Google Scholar] [CrossRef]
  31. Shi, W.; Caballero, J.; Huszár, F.; Totz, J.; Aitken, A.P.; Bishop, R.; Rueckert, D.; Wang, Z. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
  32. Li, K.; Wang, Y.; Zhang, J.; Gao, P.; Song, G.; Liu, Y.; Li, H.; Qiao, Y. UniFormer: Unifying Convolution and Self-Attention for Visual Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 12581–12600. [Google Scholar] [CrossRef]
  33. Chollet, F. Xception: Deep Learning with Depthwise Separable Convolutions. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 1800–1807. [Google Scholar]
  34. Simonyan, K.; Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv 2015. [Google Scholar] [CrossRef]
  35. Wei, C.; Wang, W.; Yang, W.; Liu, J. Deep Retinex Decomposition for Low-Light Enhancement. arXiv 2018. [Google Scholar] [CrossRef]
  36. Yang, W.; Wang, W.; Huang, H.; Wang, S.; Liu, J. Sparse Gradient Regularized Deep Retinex Network for Robust Low-Light Image Enhancement. IEEE Trans. Image Process. 2021, 30, 2072–2086. [Google Scholar] [CrossRef] [PubMed]
  37. Liu, J.; Xu, D.; Yang, W.; Fan, M.; Huang, H. Benchmarking Low-Light Image Enhancement and Beyond. Int. J. Comput. Vis. 2021, 129, 1153–1184. [Google Scholar] [CrossRef]
  38. Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef]
  39. Zhang, R.; Isola, P.; Efros, A.A.; Shechtman, E.; Wang, O. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 586–595. [Google Scholar]
  40. Jiang, Y.; Gong, X.; Liu, D.; Cheng, Y.; Fang, C.; Shen, X.; Yang, J.; Zhou, P.; Wang, Z. EnlightenGAN: Deep Light Enhancement Without Paired Supervision. IEEE Trans. Image Process. 2021, 30, 2340–2349. [Google Scholar] [CrossRef] [PubMed]
  41. Wang, Z.; Cun, X.; Bao, J.; Zhou, W.; Liu, J.; Li, H. Uformer: A General U-Shaped Transformer for Image Restoration. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–4 June 2022; pp. 17683–17693. [Google Scholar]
  42. Liu, Y.; Huang, T.; Dong, W.; Wu, F.; Li, X.; Shi, G. Low-Light Image Enhancement with Multi-Stage Residue Quantization and Brightness-Aware Attention. In Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 1–6 October 2023. [Google Scholar]
  43. Gu, S.; Li, Y.; Van Gool, L.; Timofte, R. Self-Guided Network for Fast Image Denoising. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 2511–2520. [Google Scholar]
  44. Dong, X.; Xu, W.; Miao, Z.; Ma, L.; Zhang, C.; Yang, J.; Jin, Z.; Teoh, A.B.J.; Shen, J. Abandoning the Bayer-Filter to See in the Dark. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 17410–17419. [Google Scholar]
  45. Zhu, M.; Pan, P.; Chen, W.; Yang, Y. EEMEFN: Low-Light Image Enhancement via Edge-Enhanced Multi-Exposure Fusion Network. AAAI 2020, 34, 13106–13113. [Google Scholar] [CrossRef]
  46. Maharjan, P.; Li, L.; Li, Z.; Xu, N.; Ma, C.; Li, Y. Improving Extreme Low-Light Image Denoising via Residual Learning. In Proceedings of the 2019 IEEE International Conference on Multimedia and Expo (ICME), Shanghai, China, 8–12 July 2019; pp. 916–921. [Google Scholar]
  47. Xu, K.; Yang, X.; Yin, B.; Lau, R.W.H. Learning to Restore Low-Light Images via Decomposition-and-Enhancement. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; pp. 2278–2287. [Google Scholar]
  48. Lamba, M.; Mitra, K. Restoring Extremely Dark Images in Real Time. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021; pp. 3486–3496. [Google Scholar]
  49. Lore, K.G.; Akintayo, A.; Sarkar, S. LLNet: A Deep Autoencoder Approach to Natural Low-Light Image Enhancement. Pattern Recognition 2017, 61, 650–662. [Google Scholar] [CrossRef]
  50. Zhang, Y.; Zhang, J.; Guo, X. Kindling the Darkness: A Practical Low-Light Image Enhancer. In Proceedings of the 27th ACM International Conference on Multimedia, Nice, France, 15 October 2019; pp. 1632–1640. [Google Scholar]
  51. Zhang, Y.; Guo, X.; Ma, J.; Liu, W.; Zhang, J. Beyond Brightening Low-Light Images. Int. J. Comput. Vis. 2021, 129, 1013–1037. [Google Scholar] [CrossRef]
  52. Liu, R.; Ma, L.; Zhang, J.; Fan, X.; Luo, Z. Retinex-Inspired Unrolling with Cooperative Prior Architecture Search for Low-Light Image Enhancement. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021; pp. 10556–10565. [Google Scholar]
  53. Wang, Y.; Wan, R.; Yang, W.; Li, H.; Chau, L.-P.; Kot, A.C. Low-Light Image Enhancement with Normalizing Flow. arXiv 2021. [Google Scholar] [CrossRef]
  54. Zhou, S.; Li, C.; Change Loy, C. LEDNet: Joint Low-Light Enhancement and Deblurring in the Dark. In Proceedings of the Computer Vision–ECCV 2022; Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T., Eds.; Springer Nature: Cham, Switzerland, 2022; pp. 573–589. [Google Scholar]
Applsci 15 00361 g001
Figure 1. Experimental results on four datasets. This is a comprehensive comparison chart of results in both sRGB and raw domains, with detailed data available in Section 4. The horizontal and vertical axes of the chart represent PSNR and SSIM, respectively. The better the performance, the further right and up the model is on the chart. It can be seen that our model achieves excellent performance.
Figure 1. Experimental results on four datasets. This is a comprehensive comparison chart of results in both sRGB and raw domains, with detailed data available in Section 4. The horizontal and vertical axes of the chart represent PSNR and SSIM, respectively. The better the performance, the further right and up the model is on the chart. It can be seen that our model achieves excellent performance.
Applsci 15 00361 g001
Applsci 15 00361 g002
Figure 2. Flowchart of sRrsR-Net integrating Sampler, RGB-iRGB, RAWFormer, and RRAW-sRGB modules.
Figure 2. Flowchart of sRrsR-Net integrating Sampler, RGB-iRGB, RAWFormer, and RRAW-sRGB modules.
Applsci 15 00361 g002
Applsci 15 00361 g003
Figure 3. The structure of the RGB-iRGB module.
Figure 3. The structure of the RGB-iRGB module.
Applsci 15 00361 g003
Applsci 15 00361 g004
Figure 4. The structure of the RAWFormer module.
Figure 4. The structure of the RAWFormer module.
Applsci 15 00361 g004
Applsci 15 00361 g005
Figure 5. The structure of the RRAW-sRGB module.
Figure 5. The structure of the RRAW-sRGB module.
Applsci 15 00361 g005
Applsci 15 00361 g006
Figure 6. Visual comparison results on the LOL-v1 and LOL-v2 datasets. The magnified portion has already been marked with a red box. The following are the same.
Figure 6. Visual comparison results on the LOL-v1 and LOL-v2 datasets. The magnified portion has already been marked with a red box. The following are the same.
Applsci 15 00361 g006
Applsci 15 00361 g007
Figure 7. Visual comparison of raw domain image reconstruction results using sRrsR-Net and six other methods.
Figure 7. Visual comparison of raw domain image reconstruction results using sRrsR-Net and six other methods.
Applsci 15 00361 g007
Applsci 15 00361 g008
Figure 8. sRrsR-Net’s visualization results on the VE-LOL test set. From top to bottom, the images represent real and synthetic scenarios. From left to right, the input, output, and ground-truth images are depicted.
Figure 8. sRrsR-Net’s visualization results on the VE-LOL test set. From top to bottom, the images represent real and synthetic scenarios. From left to right, the input, output, and ground-truth images are depicted.
Applsci 15 00361 g008
Applsci 15 00361 g009
Figure 9. Comparison of running times across different datasets. The left side compares the average running times of other methods, while the right side shows our method’s running time compared to other state-of-the-art methods.
Figure 9. Comparison of running times across different datasets. The left side compares the average running times of other methods, while the right side shows our method’s running time compared to other state-of-the-art methods.
Applsci 15 00361 g009
Applsci 15 00361 g010
Figure 10. Visualization results of ablation study.
Figure 10. Visualization results of ablation study.
Applsci 15 00361 g010
Table 1. Quantitative results on the LOL-v1, LOL-v2-real, and LOL-v2-synthetic datasets, where the best results are highlighted in bold, and the second-best results are underlined. The ‘↑’ symbol indicates that higher values are better. The following are the same.
Table 1. Quantitative results on the LOL-v1, LOL-v2-real, and LOL-v2-synthetic datasets, where the best results are highlighted in bold, and the second-best results are underlined. The ‘↑’ symbol indicates that higher values are better. The following are the same.
MethodsLOL-v1LOL-v2-RealLOL-v2-Synthetic
PSNR ↑SSIM ↑PSNR ↑SSIM ↑PSNR ↑SSIM ↑
Zero-DCE [16]14.860.56218.060.58017.840.730
DeepUPE [18]14.380.44613.270.45215.080.623
EnGAN [40]17.480.65018.230.61716.570.734
UFormer [41]16.360.77118.820.77119.660.871
MIRNet [17]24.140.83020.020.82021.940.876
Sparse [36]17.200.64020.060.81622.050.905
Retinexformer [9]25.160.84522.800.84025.670.930
RQ [42]25.240.85522.370.85425.940.941
SNR [4]26.720.85127.210.87127.790.928
Ours29.490.86129.480.88929.920.933
Table 2. Model comparison results on the raw domain dataset. The ‘↑’ symbol indicates that higher numbers are better, while bold indicates the best result and underlined text shows the second-best outcome. The following are the same.
Table 2. Model comparison results on the raw domain dataset. The ‘↑’ symbol indicates that higher numbers are better, while bold indicates the best result and underlined text shows the second-best outcome. The following are the same.
MethodsPSNR ↑SSIM ↑MethodsPSNR ↑SSIM ↑
DeepUPE [18]29.130.792SGN [43]28.910.789
SID [23]28.880.787ABF [44]29.650.797
EEMEFN [45]29.600.795DID [46]28.410.780
Zero-DCE [16]26.530.730EDSR [46]28.410.780
FIDE [47]29.560.799RED [48]28.660.790
SNR [4]29.750.812Ours29.980.831
Table 3. Quantitative analysis results of the model on the VE-LOL dataset’s generalization test. The ‘↓’ symbol indicates that the smaller values are better.
Table 3. Quantitative analysis results of the model on the VE-LOL dataset’s generalization test. The ‘↓’ symbol indicates that the smaller values are better.
MethodsLOL-v1VE-LOL
PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓
LLNet [49]17.960.7130.36017.570.7390.402
RRDNet [3]11.330.5340.36514.310.6200.283
RetinexNet [35]16.770.5590.47414.680.5250.642
Zero-DCE [16]14.860.5890.33521.120.7710.248
EnlightenGAN [40]17.480.6770.32220.430.7920.242
KinD [50]20.870.8020.17018.420.7660.288
KinD++ [51]21.300.8230.16017.630.7990.226
RUAS [52]18.230.7170.35414.360.6710.337
LLFlow [53]25.190.8720.11323.850.8880.146
SNR [4]24.610.8420.15123.520.8740.216
Ours29.490.8610.10429.050.8840.119
Table 4. Compares the runtime (in seconds) and computational complexity (FLOPs) of the four datasets. The best results are in bold.
Table 4. Compares the runtime (in seconds) and computational complexity (FLOPs) of the four datasets. The best results are in bold.
TypeMethodsFLOPs (G)LOL-v1LOL-v2-RealLOL-v2-SyntheticSID
TransformerRestormer [20]144.250.200.250.300.28
Retinexformer [9]11.33 0.110.150.200.18
UFormer [41]15.850.150.200.250.22
CNNRetinexNet [35]587.460.280.300.300.30
KinD [50]35.010.260.280.270.26
U-Net-likeLEDNet [54]35.900.190.220.230.21
Zero-DCE [16]4.810.120.120.130.12
EnlightenGAN [40]526.230.250.260.270.25
MixedAverage170.110.190.220.250.23
Ours40.950.180.210.240.22
Table 5. Ablation experiment results on the LOL-v1 dataset. “✓” indicates the respective module has been utilized.
Table 5. Ablation experiment results on the LOL-v1 dataset. “✓” indicates the respective module has been utilized.
ModulesRGB-iRGBRAWFormerRRAW-sRGBLOL-v1
PSNR ↑SSIM ↑LPIPS ↓
Baseline 20.530.7850.234
A✓ 26.440.8330.188
B ✓ 25.840.8250.186
C ✓26.980.8510.139
D✓ ✓28.600.8590.112
E ✓✓28.520.8420.136
F✓✓✓29.490.8610.105
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Hong, Z.; Zhen, D.; Xiong, L.; Li, X.; Lin, Y. sRrsR-Net: A New Low-Light Image Enhancement Network via Raw Image Reconstruction. Appl. Sci. 2025, 15, 361. https://doi.org/10.3390/app15010361

AMA Style

Hong Z, Zhen D, Xiong L, Li X, Lin Y. sRrsR-Net: A New Low-Light Image Enhancement Network via Raw Image Reconstruction. Applied Sciences. 2025; 15(1):361. https://doi.org/10.3390/app15010361

Chicago/Turabian Style

Hong, Zhiyong, Dexin Zhen, Liping Xiong, Xuechen Li, and Yuhan Lin. 2025. "sRrsR-Net: A New Low-Light Image Enhancement Network via Raw Image Reconstruction" Applied Sciences 15, no. 1: 361. https://doi.org/10.3390/app15010361

APA Style

Hong, Z., Zhen, D., Xiong, L., Li, X., & Lin, Y. (2025). sRrsR-Net: A New Low-Light Image Enhancement Network via Raw Image Reconstruction. Applied Sciences, 15(1), 361. https://doi.org/10.3390/app15010361

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop