Open AccessReview

Compression for Bayer CFA Images: Review and Performance Comparison

Kuo-Liang Chung

Hsuan-Ying Chen

Tsung-Lun Hsieh

and

Yen-Bo Chen

Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology, Taipei 10672, Taiwan

Author to whom correspondence should be addressed.

Sensors 2022, 22(21), 8362; https://doi.org/10.3390/s22218362

Submission received: 26 September 2022 / Revised: 28 October 2022 / Accepted: 28 October 2022 / Published: 31 October 2022

(This article belongs to the Special Issue Advances in Coding, Sensing, and Processing for CFA Images, Light Field Images, and Point Cloud)

Download

Browse Figures

Versions Notes

Abstract

Bayer color filter array (CFA) images are captured by a single-chip image sensor covered with a Bayer CFA pattern which has been widely used in modern digital cameras. In the past two decades, many compression methods have been proposed to compress Bayer CFA images. These compression methods can be roughly divided into the compression-first-based (CF-based) scheme and the demosaicing-first-based (DF-based) scheme. However, in the literature, no review article for the two compression schemes and their compression performance is reported. In this article, the related CF-based and DF-based compression works are reviewed first. Then, the testing Bayer CFA images created from the Kodak, IMAX, screen content images, videos, and classical image datasets are compressed on the Joint Photographic Experts Group-2000 (JPEG-2000) and the newly released Versatile Video Coding (VVC) platform VTM-16.2. In terms of the commonly used objective quality, perceptual quality metrics, the perceptual effect, and the quality–bitrate tradeoff metric, the compression performance comparison of the CF-based compression methods, in particular the reversible color transform-based compression methods and the DF-based compression methods, is reported and discussed.

Keywords:

Bayer color filter array (CFA) images; compression-first-based compression; chroma subsampling-then-luma modification; demosaicing-first-based compression; joint photographic experts group-2000 (JPEG-2000); quality–bitrate tradeoff; reversible color transform-based compression; Versatile Video Coding (VVC)

1. Introduction

To save hardware costs, most color digital cameras employ single-sensor technologies with Bayer color filter array (CFA) patterns to capture real-world scenes. The four widely used 2 × 2 Bayer CFA patterns [1], namely

P a t_{1}

= [

G_{1}

R_{2}

B_{3}

G_{4}

P a t_{2}

= [

G_{1}

B_{2}

R_{3}

G_{4}

P a t_{3}

= [

R_{1}

G_{2}

G_{3}

B_{4}

], and

P a t_{4}

= [

B_{1}

G_{2}

G_{3}

R_{4}

], are shown in Figure 1a–d, respectively. For convenience, the captured Bayer CFA raw image is denoted by I

^{B a y e r}

, in which each pixel has only one R (red), G (green), or B (blue) color value and I

^{B a y e r}

consists of 25% R, 50% G, and 25% B color values. To fully utilize the limited device resources of cameras, such as the limited storage space and transmission capacity, prior to storing or transmitting Bayer CFA images, compressing Bayer CFA images is necessary. Without the loss of generality, in our discussion for compressing

I^{B a y e r}

, we only consider the first Bayer CFA pattern in Figure 1a, but our discussion is also applicable to the other three CFA patterns. During the past two decades, many compression methods for

I^{B a y e r}

have been developed and they can be roughly divided into two schemes, namely the compression-first-based (CF-based) scheme in Figure 2a and the demosaicing-first-based (DF-based) scheme in Figure 3a. In both compression schemes, at the client side, besides the reconstructed Bayer CFA image

I^{r e c, B a y e r}

, the reconstructed RGB full-color image, which is obtained by demosaicing

I^{r e c, B a y e r}

, is also used to evaluate the quality and quality–bitrate performance of the related compression methods. It is noticeable that the input Bayer CFA image

I^{B a y e r}

and the demosaiced RGB full-color image

I^{d e m o, R G B}

at the server side of Figure 2a and Figure 3a are used as the ground-truth Bayer CFA image and the ground-truth reconstructed RGB full-color image, respectively.

1.1. The Related CF-Based Compression Methods

At the server side of Figure 2a, the input Bayer CFA image

I^{B a y e r}

is first decorrelated to some subimages. In Figure 2b, the decorrelated subimages could be the reversible color transform-based (RCT-based) subimages [2,3,4,5,6,7,8,9], the wavelet transform-based (WT-based) subbands [10,11,12,13,14], or the prediction-based residuals [10,15].

In the RCT-based decorrelated subimages, the four RCT-based formats, namely the

Y_{1} C r_{2} C b_{3} Y_{4}

format proposed by Lee and Ortega [2] and implemented on JPEG [16], the

Y D_{g} C_{o} C_{g}

format proposed by Malvar and Sullivan [3] and implemented on JPEG-XR [17] and JPEG-2000 [18,19], the

Y L M N

format proposed by Mohammed et al. [4] and implemented on Golomb–Rice codec [20] and JPEG-2000, and the

Y Δ C_{b} C_{r}

format proposed by Richter and Fößel [8] and implemented on JPEG-XS [21], have received growing attention. For the

Y D_{g} C_{o} C_{g}

representation of

I^{B a y e r}

, Suzuki [12] proposed a lossless WT-based spectral–spatial transformation (WSST) approach to improve the bitrate performance. To improve WSST, Suzuki [14] proposed a weighted version by taking the edge directions into account. For improving the compression performance, Richter et al. [9] not only performed a nonlinear gamma correction on

I^{B a y e r}

but also deployed two white-balance constants into the two luma components in the Star-Tetrix transformation-based representation which was implemented on JPEG-XS [21].

Zhang and Wu [10] proposed a lossless merge- and residual-based method for compressing

I^{B a y e r}

. As a result, the rectangular compact green subimage, the red residual subimage, and blue residual subimage are fed into the codec. Chung and Chan [15] proposed a lossless context matching-based prediction method, in which when predicting the current pixel, the neighboring pixels of the current pixel are ranked, to obtain more accurate residual red and blue subimages, achieving a better compression performance relative to the method [10] on the Rice encoder. Zhang and Wu [10] performed a Mallat wavelet transform [22] on

I^{B a y e r}

, and then, the Golomb–Rice encoder [20] was utilized to encode the transformed wavelet coefficients. Lee et al. [11] proposed a camera-aware multi-resolution analysis (CAMRA) framework for compressing

I^{B a y e r}

. They leveraged the decorrelated wavelet coefficients and the image pipeline techniques at the server side. Later, Lee and Hirakawa [13] proposed a new shift-and-decorrelate lifting method to improve the compression performance of CAMRA.

We now explain why we only consider the RCT-based compression methods in the above-mentioned CF-based compression methods and select JPEG-2000 as the compression platform. In the literature, the RCT-based compression methods often served as the main comparative methods. Among these codecs used to evaluate the compression performance of the considered compression methods, JPEG-2000 is most favored. In addition, the prediction-based residual approaches [10,15] are lossless; the WT-based approaches [10,11,12,13,14] involve different codecs, such as the Golomb–Rice encoder and JPEG-2000, various WT-based computations, such as Haar, 5/3, and 9/7 wavelet transforms, varying matrix chain multiplications, and lifting operations. Accordingly, we take the RCT-based methods as the representatives of the CF-based methods. Section 2 will introduce the above-mentioned four RCT-based methods in detail. Furthermore, JPEG-2000 is adopted to evaluate the compression performance of the considered compression methods.

1.2. The Related DF-Based Compression Methods

In Figure 3a, at the server side of the DF-based compression scheme, the input Bayer CFA image

I^{B a y e r}

is first demosaiced to an RGB full-color image

I^{d e m o, R G B}

which also serves as the ground-truth RGB full-color image. Section 3.1.1 will introduce how to demosaic

I^{B a y e r}

I^{d e m o, R G B}

. Next,

I^{d e m o, R G B}

is converted to a

Y C b C r

image

I^{Y C b C r}

. Section 3.1.2 will introduce an RGB-to-YCbCr transformation. Then, a chroma 4:2:0 subsampling method is performed on the chroma image

I^{C b C r}

to obtain a subsampled CbCr image

I^{s u b, C b C r}

whose size is a quarter of the original chroma image

I^{C b C r}

. In Section 3.2, two kinds of chroma 4:2:0 subsampling approaches [23,24,25,26,27,28,29,30,31,32], namely the Bayer CFA pattern-independent approach and the Bayer CFA pattern-dependent approach in Figure 3b, will be introduced. Furthermore, based on the subsampled CbCr image

I^{s u b, C b C r}

and the luma image

I^{Y}

, a luma modification method is performed on

I^{Y}

to obtain a modified luma image

I^{m o d, Y}

. Section 3.3 will introduce the two related luma modification methods [33,34].

As a result, the subsampled CbCr image

I^{s u b, C b C r}

and the modified luma image

I^{m o d, Y}

are fed into the encoder. At the client side, a chroma upsampling method is first performed on the decompressed subsampled CbCr image to construct the upsampled CbCr image. Finally, a YCbCr-to-RGB transformation is performed on the upsampled

Y C b C r

image to obtain the reconstructed Bayer CFA image

I^{r e c, B a y e r}

, and then as mentioned before,

I^{r e c, B a y e r}

is demosaiced to an RGB full-color image which serves as the reconstructed RGB full-color image. Among the codecs used to evaluate the compression performance of the considered DF-based compression methods, the Versatile Video Coding (VVC) platform [35] is most favored.

1.3. Motivation and Contribution

To date, in the literature, no review article for the above-mentioned CF-based compression scheme and the DF-based compression scheme for Bayer CFA images has been published. Besides that, no compression performance comparison of the two compression schemes has been reported. Therefore, it motivated us to review the related CF-based compression works, in particular the related RCT-based compression works and the DF-based works. In addition, it motivated us to compare and discuss the compression performance for the considered compression methods on JPEG-2000 and the newly released VVC platform VTM-16.2.

In the RCT-based compression methods, the four main methods, namely the

Y_{1} Y_{4} C b_{3} C r_{2}

method [2], the

Y D_{g} C_{o} C_{g}

method [3], the

Y L M N

method [4], and the

Y Δ C_{b} C_{r}

method [8], are introduced in detail. To evaluate the compression performance of these compression methods on JPEG-2000 and VTM-16.2, considering simplicity and effectiveness, the three methods, namely the

Y D_{g} C_{o} C_{g}

method [3], the

Y L M N

method [4], and the

Y Δ C_{b} C_{r}

method [8], are selected as the representatives.

In the DF-based compression scheme, besides the demosaicing part, the other macro-part is called the “chroma subsampling-then-luma modification (CSLM)” macro-part which consists of the chroma subsampling part and the luma modification part. In the chroma subsampling part, we first introduce eight Bayer CFA pattern-independent chroma subsampling methods: 4:2:0(A), 4:2:0(L), 4:2:0(R), 4:2:0(DIRECT), 4:2:0(MPEG-B) [23], the Anchor method [24], the interpolation-dependent image downsampling (IDID) method [25], and the joint chroma downsampling and upsampling (JCDU) method [26]. In Figure 3b, we further introduce five Bayer CFA pattern-dependent chroma subsampling methods: the direct mapping (DM) method [27], the COPY-based distortion minimization (CDM) method [28], the improved CDM (ICDM) method [29], the modified 4:2:0(A) method [30], and the bilinear interpolation-based distortion minimization (BIDM) method [31]. In the luma modification part, we introduce two luma modification methods: the Bayer CFA pattern-independent LM method [33] and the optimal Bayer CFA pattern-dependent LM (OLM) method [34]. Considering the effectiveness, the three CSLM methods, namely CDM-OLM, “modified 4:2:0(A)”-OLM, and BIDM-OLM, are selected as the representatives to evaluate the compression performance of the DF-based compression scheme.

Based on the testing Bayer CFA images created from the Kodak, IMAX, screen content images (SCI), Videos, and classical images (CI) datasets, thorough experiments have been carried out for the above-mentioned representatives of the CF-based and DF-based compression schemes on JPEG-2000 and VTM-16.2. When setting the quantization parameter (QP) to 0 for VTM-16.2 and setting the compression ratio (CR) to 1 for JPEG-2000, in terms of the three popular quality metrics, namely the peak signal-to-noise ratio (PSNR), the structure similarity (SSIM) [36], and the feature similarity (FSIM) [37], the

Y D_{g} C_{o} C_{g}

method achieves the best PSNR performance, and the BIDM-OLM method is ranked second; the BIDM-OLM method achieves the best SSIM and FSIM performance, and the

Y D_{g} C_{o} C_{g}

method is ranked second. On JPEG-2000, in terms of the widely used quality–bitrate tradeoff metric, namely the Bjøntegaard delta (BD)-PSNR [38], the BIDM-OLM method always achieves the best BD-PSNR performance. On VTM-16.2, the

Y D_{g} C_{o} C_{g}

method achieves the best BD-PSNR performance under the high bitrate circumstance, while under the middle and low bitrate circumstances, the BIDM-OLM method achieves the best BD-PSNR performance. In addition, the perceptual quality comparison and the execution time requirement comparison are also made. Finally, some future works are addressed.

The remainder of this article is organized as follows. In Section 2, the related RCT-based compression works for

I^{B a y e r}

are introduced. In Section 3, the related DF-based compression works for

I^{B a y e r}

are introduced. In Section 4, the compression performance comparison between the two compression schemes are provided. In Section 5, some concluding remarks and future works are addressed.

2. The Reversible Color Transform-Based (RCT-Based) Compression Works for Bayer CFA Images

For

I^{B a y e r}

, we mainly introduce the four RCT-based compression methods: the

Y D_{g} C_{o} C_{g}

method [3], the

Y L M N

method [4], and the

Y Δ C_{b} C_{r}

method [8]. Figure 4 depicts the relation between the 2 × 2 Bayer CFA block and the four RCT-based formats.

2.1. The $Y_{1} C r_{2} C b_{3} Y_{4}$ Method

Lee and Ortega [2] proposed a

Y_{1} C r_{2} C b_{3} Y_{4}

method to decorrelate the input Bayer CFA image

I^{B a y e r}

to four subimages, namely the

Y_{1}

C r_{2}

C b_{3}

, and

Y_{4}

subimages such that

I^{Y_{1} C r_{2} C b_{3} Y_{4}}

can be better compressed by JPEG. First, each 2 × 2 Bayer CFA block

B^{B a y e r}

is converted to a 2 × 2

Y_{1} C r_{2} C b_{3} Y_{4}

block in Figure 4b by using the following formula:

[\begin{matrix} Y_{1} \\ Y_{4} \\ C b_{3} \\ C r_{2} \end{matrix}] = [\begin{matrix} 0.587 & 0 & 0.114 & 0.299 \\ 0 & 0.587 & 0.114 & 0.299 \\ - 0.170 & - 0.170 & 0.511 & - 0.172 \\ - 0.214 & - 0.214 & - 0.083 & 0.511 \end{matrix}] [\begin{matrix} G_{1} \\ G_{4} \\ B_{3} \\ R_{2} \end{matrix}] + [\begin{matrix} 0 \\ 0 \\ 128 \\ 128 \end{matrix}]

(1)

where

Y_{1}

and

Y_{4}

denote the two converted luma components, and

C b_{3}

and

C b_{2}

denote the converted chroma components.

On the other hand, by Equation (1), one input Bayer CFA image

I^{B a y e r}

is converted into a

Y_{1} C r_{2} C b_{3} Y_{4}

image

I^{Y_{1} C r_{2} C b_{3} Y_{4}}

which contains four subimages, namely the two luma subimages,

I^{Y_{1}}

and

I^{Y_{4}}

, constituting a quincunx-located luma image in Figure 5a, and the two chroma subimages,

I^{C b_{3}}

and

I^{C r_{2}}

. In order to produce a compact luma image containing

I^{Y_{1}}

and

I^{Y_{4}}

, every luma value in each even column of the quincunx-located image is shifted left to the odd column, and then, each even column is removed. Figure 5b shows the rectangular compact luma image. Next, the rectangular compact luma image is rotated 45 degree clockwise to produce a rhombic compact luma image which is located on the center of the resultant compact luma image using a mirroring method. Finally, the resultant compact luma image is compressed by a shape-oriented JPEG [39], in which only the meaningful luma values of the resultant compact luma image are compressed. Similarly, the compact subimage

I^{C b_{3}}

and the compact subimage

I^{C r_{2}}

are compressed using the same codec.

At the client side, the decompressed

Y_{1} C r_{2} C b_{3} Y_{4}

image is converted to the constructed Bayer CFA image

I^{r e c, B a y e r}

by the inverse of Equation (1).

2.2. The $Y D_{g} C_{o} C_{g}$ Method

Extending from the

Y C_{o} C_{g} R

method [40], Malvar and Sullivan [3] proposed an effective

Y D_{g} C_{o} C_{g}

method. Considering each 2 × 2 Bayer CFA block

B^{B a y e r}

in Figure 1a,

B^{B a y e r}

is converted to a 2 × 2

Y D_{g} C_{o} C_{g}

block where the luma value Y is the average of four Bayer CFA pixel-values in

B^{B a y e r}

. The chroma value

D_{g}

equals the difference between two green pixel-values in

B^{B a y e r}

. The chroma value

C_{o}

equals the difference between the red pixel-value and the blue pixel-value, and the chroma value

C_{g}

is equal to the difference between the average of two green pixel-values and the average of the red pixel-value and the blue pixel-value. In the

Y D_{g} C_{o} C_{g}

method, the

G_{1} G_{4} R_{2} B_{3}

-to-

Y D_{g} C_{o} C_{g}

transformation is expressed as

[\begin{matrix} Y \\ D_{g} \\ C_{o} \\ C_{g} \end{matrix}] = [\begin{matrix} \frac{1}{4} & \frac{1}{4} & \frac{1}{4} & \frac{1}{4} \\ - 1 & 1 & 0 & 0 \\ 0 & 0 & 1 & - 1 \\ \frac{1}{2} & \frac{1}{2} & - \frac{1}{2} & - \frac{1}{2} \end{matrix}] [\begin{matrix} G_{1} \\ G_{4} \\ R_{2} \\ B_{3} \end{matrix}]

(2)

The inverse of the above transformation in Equation (2), i.e., the

Y D_{g} C_{o} C_{g}

-to-

G_{1} G_{4} R_{2} B_{3}

transformation, is expressed as

[\begin{matrix} G_{1} \\ G_{4} \\ R_{2} \\ B_{3} \end{matrix}] = [\begin{matrix} 1 & - \frac{1}{2} & 0 & \frac{1}{2} \\ 1 & \frac{1}{2} & 0 & \frac{1}{2} \\ 1 & 0 & \frac{1}{2} & - \frac{1}{2} \\ 1 & 0 & - \frac{1}{2} & - \frac{1}{2} \end{matrix}] [\begin{matrix} Y \\ D_{g} \\ C_{o} \\ C_{g} \end{matrix}]

(3)

Because the transformation in Equation (2) and the inverse transformation in Equation (3) only have entries 0, 1, −1,

\frac{1}{2}

, −

\frac{1}{2}

, and

\frac{1}{4}

, according to Equation (3) in [3], only right-shift operations for integer values are needed, leading to a low computational cost benefit. On JPEG-2000, experimental results indicated the compression performance superiority of the

Y D_{g} C_{o} C_{g}

method over the “G channel merging plus color differences” method [10] which outperformed the method by compressing the Bayer CFA image directly.

2.3. The $Y L M N$ Method

Mohammed et al. [4] proposed a

Y L M N

method for compressing

I^{B a y e r}

. Considering a 2 × 2 Bayer CFA Block

B^{B a y e r}

, based on the generalized S-transform [41], the lifting-based reversible color transformation [42], and the 2 × 2 unimodular matrix A with

A = [\begin{matrix} α & 1 - α \\ 1 & - 1 \end{matrix}]

(4)

for 0 <

α

< 1, the best value of

α

is determined as 1/2 and it yields

A = [\begin{matrix} \frac{1}{2} & \frac{1}{2} \\ 1 & - 1 \end{matrix}]

(5)

Based on the differential pulse code modulation (DPCM) and the median edge prediction principle, the

G_{1} R_{2} G_{4} B_{3}

-to-

W_{r} D_{r} W_{b} D_{b}

transformation is expressed as

\begin{matrix} [\begin{matrix} W_{r} \\ D_{r} \end{matrix}] = A [\begin{matrix} G_{1} \\ R_{2} \end{matrix}] \\ [\begin{matrix} W_{b} \\ D_{b} \end{matrix}] = A [\begin{matrix} G_{4} \\ B_{3} \end{matrix}] \end{matrix}

(6)

To reduce the correlation between

W_{r}

and

W_{b}

, the

W_{r} D_{r} W_{b} D_{b}

format in Equation (6) is transformed to the following

Y L M N

format:

\begin{matrix} [\begin{matrix} Y \\ L \end{matrix}] = A [\begin{matrix} W_{r} \\ W_{b} \end{matrix}] \\ [\begin{matrix} M \\ N \end{matrix}] = [\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix}] [\begin{matrix} D_{r} \\ D_{b} \end{matrix}] \end{matrix}

(7)

Later, Mohammed and Wahid [5] slightly modified the

Y L M N

format in Equation (7) by taking the floor function into account. According to the results proposed by Khan and Wahid [43], Rahman et al. [6] proposed a modified DPCM-based representation for compressing

I^{B a y e r}

in wireless capsule endoscopy applications.

2.4. The $Y Δ C_{b} C_{R}$ Method

Richter and Fößel [8] proposed a

Y Δ C_{b} C_{r}

method by modifying the

\tilde{g} d B_{3} R_{2}

method [7]. After performing the 4 × 4 Haar wavelet transform on the four channels,

\tilde{g}

, d,

B_{3}

, and

R_{2}

, the

Y Δ C_{b} C_{r}

format is expressed as

\begin{matrix} Y = ⌊ \frac{R_{2} + 2 ⌊ \frac{G_{1} + G_{4}}{2} ⌋ + B_{3}}{4} ⌋ \\ Δ = G_{1} - G_{4} \\ C_{B} = B_{3} - ⌊ \frac{G_{1} + G_{4}}{2} ⌋ \\ C_{R} = R_{2} - ⌊ \frac{G_{1} + G_{4}}{2} ⌋ \end{matrix}

(8)

In Equation (8), the Y component is the average of the four Bayer CFA pixels of

B^{B a y e r}

Δ

denotes the difference between

G_{1}

and

G_{4}

C_{B}

and

C_{R}

equal the blue pixel

B_{3}

and red pixel

R_{2}

minus the average of

G_{1}

and

G_{4}

, respectively.

Experimental results indicated that prior to encoding

I^{B a y e r}

, as a pre-processing step [8], performing a nonlinear gamma correction on

I^{B a y e r}

can achieve better compression performance when using the

Y Δ C_{b} C_{r}

method.

To evaluate the compression performance of the RCT-based compression scheme for

I^{B a y e r}

, considering effectiveness, the three RCT-based methods, namely, the

Y D_{g} C_{o} C_{g}

method [3], the

Y L M N

method [4], and the

Y Δ C_{b} C_{r}

method [8], are selected as the representatives.

3. The Demosaicing-First-Based (DF-Based) Compression Works for Bayer CFA Images

For

I^{B a y e r}

, we introduce the related DF-based compression methods, and in particular, the related CSLM (chroma subsampling-then-luma modification) methods are introduced in more detail. We first introduce how to demosaic

I^{B a y e r}

to an RGB full-color image

I^{d e m o, R G B}

, and then, we introduce how to convert

I^{d e m o, R G B}

to a

Y C b C r

image,

I^{Y C b C r}

. Finally, three representative methods are selected to evaluate the compression performance of the DF-based compression scheme.

3.1. Demosaicing $I^{B a y e r}$ to $I^{D e m o, R g b}$ and Then Converting $I^{D e m o, R g b}$ to $I^{Y c b c r}$

In the DF-based compression scheme for encoding

I^{B a y e r}

, as depicted at the server side of Figure 3a,

I^{B a y e r}

is first demosaiced to an RGB full-color image

I^{d e m o, R G B}

3.1.1. Demosaicing $I^{B a y e r}$ to $I^{d e m o, R G B}$

To demosaic

I^{B a y e r}

I^{d e m o, R G B}

, a demosaicing method is performed on

I^{B a y e r}

to estimate the other two color channels of each Bayer CFA pixel [44,45,46,47].

Bilinear interpolation [48] is the simplest demosaicing method in which the unknown two-color channels of each Bayer CFA pixel are estimated by averaging its proper adjacent pixels. Kimmel [49] proposed a color difference-based demosaicing method using a template matching approach. Gunturk et al. [50] proposed a demosaicing method using an alternating projection approach. Pei and Tam [51] proposed a demosaicing method using a color correlation approach. Lu and Tan [52] proposed a demosaicing method using the spatial and spectral correlation among the neighboring pixels of each Bayer CFA pixel. Wu and Zhang [53] proposed a demosaicing method using the edge direction information and a soft-decision framework. Lukac and Plataniotis [54] proposed a demosaicing method using normalized color-ratio information.

Hirakawa and Parks [55] proposed an adaptive homogeneity-directed demosaicing method. Chung et al. [56] proposed a demosaicing method using several gradient edge-detection masks and an adaptive heterogeneity-projection technique. Using a generic variational approach, Condat [57] proposed a general demosaicing method for arbitrary CFA patterns. Yang et al. [58] proposed a color difference- and edge sensing-based demosaicing method for arbitrary CFA patterns. Zhang et al. [59] proposed a demosaicing method using local directional interpolation and nonlocal adaptive thresholding.

Kiku et al. [60] proposed a residual interpolation-based demosaicing method. In their method, the missing green values are first estimated by using a bilateral interpolation. Next, a window-based linear relation with two parameters, in which the number of equations is larger than 2, between the estimated green values and the collocated ground-truth red values are constructed. Then, a linear regression technique is applied to solve the two parameters involved in the linear relation. Using the solved two parameters, the missing red values are thus reconstructed. In the same argument, the missing blue values are constructed. To alleviate the spot artifact problem in [60], based on a multiple-window approach, Ye et al. [61] first constructed multiple linear systems, and then the average 2-parameter solution was used to estimate the missing red and blue values, leading to a better smoothing effect. In [62,63], the convolutional neural networks (CNN) based demosaicing methods were proposed. Considering the fact that the green channel has twice the sampling rate and better quality than the red and blue channels in

I^{B a y e r}

, Guo et al. proposed a green channel prior-NET-based joint denoising and demosaicing method. Based on a progressive collaborative representation framework, Ni et al. [64] proposed multiple training-and-refining steps to improve the demosaicing performance.

Due to simplicity and effectiveness, as the first step of the DF-based compression method for

I^{B a y e r}

, Kiku et al.’s demosaicing method is adopted to demosaic the input Bayer CFA image

I^{B a y e r}

to an RGB full-color image

I^{d e m o, R G B}

. In the next subsection, the conversion from the demosaiced RGB full-color image

I^{d e m o, R G B}

to a

Y C b C r

image

I^{Y C b C r}

is introduced.

3.1.2. Converting $I^{d e m o, R G B}$ to $I^{Y C b C r}$

After demosaicing

I^{B a y e r}

I^{d e m o, R G B}

I^{d e m o, R G B}

is further transformed to

I^{Y C b C r}

by using the BT.601-5 color conversion [65]:

[\begin{matrix} Y_{i} \\ C b_{i} \\ C r_{i} \end{matrix}] = [\begin{matrix} 0.257 & 0.504 & 0.098 \\ - 0.148 & - 0.291 & 0.439 \\ 0.439 & - 0.368 & - 0.071 \end{matrix}] [\begin{matrix} R_{i} \\ G_{i} \\ B_{i} \end{matrix}] + [\begin{matrix} 16 \\ 128 \\ 128 \end{matrix}]

(9)

where for

1 \leq i \leq 4

, (

R_{i}

G_{i}

B_{i}

) and (

Y_{i}

C b_{i}

C r_{i}

) denote the triple-values of the ith RGB pixel and the ith

Y C b C r

pixel in each 2 × 2 RGB block

B^{d e m o, R G B}

and the collocated converted 2 × 2

Y C b C r

block

B^{Y C b C r}

, respectively.

Because the human visual system is less sensitive for chroma differences than for luminance, the luma image

I^{Y}

and the chroma image

I^{C b C r}

are decorrelated from the converted

Y C b C r

image

I^{Y C b C r}

. Therefore, chroma subsampling on each 2 × 2

C b C r

block

B^{C b C r}

is naturally included prior to encoding the

Y C b C r

image [66].

3.2. Chroma Subsampling

In this subsection, eight Bayer CFA pattern-independent chroma subsampling methods and five Bayer CFA pattern-dependent chroma subsampling methods are introduced.

3.2.1. The Bayer CFA Pattern-Independent Chroma Subsampling Methods

For each 2 × 2 chroma block

B^{C b C r}

, 4:2:0(A) averages the four (

C b

C r

)-pair of

B^{C b C r}

as the subsampled chroma pairs of

B^{C b C r}

. 4:2:0(L) and 4:2:0(R) calculate their chroma pairs by averaging the chroma pairs in the first and second columns of

B^{C b C r}

, respectively. 4:2:0(DIRECT) takes the top-left (

C b

C r

)-pair of

B^{C b C r}

as the subsampled (

C b

C r

)-pair. Figure 6 depicts the four chroma subsampling methods: 4:2:0(A), 4:2:0(L), 4:2:0(R), and 4:2:0(DIRECT). 4:2:0(MPEG-B) calculates the subsampled (

C b

C r

)-pair of

B^{C b C r}

by running the 13-tap filter [2, 0, −4, −3, 5, 19, 26, 19, 5, −3, −4, 0, 2]/64 on the top-left position of

B^{C b C r}

The Anchor method first performs a 3-tap filter [1, 6, 1]/8 at the leftmost location of each row of

B^{C b C r}

, and then, it performs a 3-tap filter

{([0, 4, 4] / 8)}^{T}

at the top-left location of

B^{C b C r}

, where “T” denotes a transpose operator. Based on the new edge-directed interpolation (NEDI) [67], Zhang et al. [25] proposed an IDID chroma subsampling method, and at the client side, NEDI is adopted as the chroma upsampling process. Inspired by the palette mode used for screen content images (SCI) [68], in which each SCI has only a few dominant colors in the background, Wang et al. [26] proposed a JCDU chroma subsampling method, and the bicubic convolution interpolation (BCI) [69] is adopted as the upsampling process at the client side.

However, because the above eight Bayer CFA pattern-independent chroma subsampling methods do not take the Bayer CFA pattern into account, their compression performance is limited. On the other hand, there is room to improve their compression performance.

3.2.2. The Bayer CFA Pattern-Dependent Chroma Subsampling Methods

In this subsection, we introduce the five state-of-the-art Bayer CFA pattern-dependent chroma subsampling methods: the direct mapping (DM) method [27], the COPY-based distortion minimization (CDM) method [28] and the two variants [29,30], and the bilinear interpolation-based distortion minimization (BIDM) method [31].

The Direct Mapping (DM) Method [27]

Before presenting the DM method [27], the YCbCr-to-RGB transformation, which is the reverse of the RGB-to-YCbCr transformation in Equation (9), is defined by

[\begin{matrix} R_{i} \\ G_{i} \\ B_{i} \end{matrix}] = [\begin{matrix} 1.164 & 0 & 1.596 \\ 1.164 & - 0.391 & - 0.813 \\ 1.164 & 2.018 & 0 \end{matrix}] [\begin{matrix} Y_{i} - 16 \\ C b_{i} - 128 \\ C r_{i} - 128 \end{matrix}]

(10)

Chen et al. [27] first observed that the R-color value is dominated by the luma value and the

C b

value, and the B-color value is dominated by the luma value and the

C r

value. In addition, from the 3 × 3 coefficient matrix in Equation (10), the

C b

component has more influence on reconstructing the B pixel than on reconstructing the G pixel. In the same argument, the

C r

component has more influence on reconstructing the R pixel than on reconstructing the G pixel. Consequently, the subsampled (

C b

C r

)-pair of

B^{C b C r}

is set to (

C b_{3}

C r_{2}

), where the relation between the subsampled chroma pair of

B^{C b C r}

, i.e., (

C b_{3}

C r_{2}

), and the 2 × 2 Bayer CFA pattern is depicted in Figure 7.

The COPY-Based Distortion Minimization (CDM) Method and the Two Variants

Lin et al. [28] first adopted the upsampling process “COPY”, which is called the nearest neighbor (NN) upsampling process supported by some compression standard such as VVC [35], to duplicate the subsampled (

C b

C r

)-parameter of

B^{C b C r}

, denoted by (

C b_{s}

C r_{s}

), as the four estimated chroma pairs of

B^{C b C r}

at the server side. Next, they proposed a COPY-based 2 × 2 Bayer CFA block-distortion function to measure the distortion between

B^{C b C r}

and the 2 × 2 estimated chroma block

B^{e s t, C b C r}

, and the block-distortion is defined by

\begin{matrix} D^{B a y e r} (C b_{s}, C r_{s}) & = {(G_{1} - G_{1}^{e s t})}^{2} + {(R_{2} - R_{2}^{e s t})}^{2} + {(B_{3} - B_{3}^{e s t})}^{2} + {(G_{4} - G_{4}^{e s t})}^{2} \\ = \sum_{i = 1}^{4} {[(1.164 Y_{i} + a_{i} C b_{i} + b_{i} C r_{i}) - (1.164 Y_{i} + a_{i} C b_{i}^{e s t} + b_{i} C r_{i}^{e s t})]}^{2} \\ = \sum_{i = 1}^{4} [a_{i} (C b_{i} - C b_{s}) + b_{i} (C r_{i} - C r_{s}))]^{2} \end{matrix}

(11)

with

\begin{matrix} a_{i} = \{\begin{matrix} 0 & for i = 2 \\ - 0.391 & for i = 1 or 4 \\ 2.018 & for i = 3 \end{matrix} \\ b_{i} = \{\begin{matrix} 1.596 & for i = 2 \\ - 0.813 & for i = 1 or 4 \\ 0 & for i = 3 \end{matrix} \end{matrix}

(12)

Applying the differentiation technique to Equation (11), in the real domain, the solution of

(C b_{s}, C r_{s})

is expressed as

\begin{matrix} C b_{s} = \frac{(\sum_{i = 1}^{4} b_{i}^{2}) (\sum_{i = 1}^{4} a_{i}^{2} C b_{i} + a_{i} b_{i} C r_{i}) - (\sum_{i = 1}^{4} a_{i} b_{i}) (\sum_{i = 1}^{4} b_{i}^{2} C r_{i} + a_{i} b_{i} C b_{i})}{(\sum_{i = 1}^{4} a_{i} b_{i})^{2} - (\sum_{i = 1}^{4} a_{i}^{2}) (\sum_{i = 1}^{4} b_{i}^{2})} \\ C r_{s} = \frac{(\sum_{i = 1}^{4} a_{i}^{2}) (\sum_{i = 1}^{4} b_{i}^{2} C r_{i} + a_{i} b_{i} - C b_{i}) - (\sum_{i = 1}^{4} a_{i} b_{i}) (\sum_{i = 1}^{4} a_{i}^{2} C b_{i} + a_{i} b_{i} C r_{i})}{(\sum_{i = 1}^{4} a_{i} b_{i})^{2} - (\sum_{i = 1}^{4} a_{i}^{2}) (\sum_{i = 1}^{4} b_{i}^{2})} \end{matrix}

(13)

In Lin et al.’s copy-based block-distortion minimization (CDM) method [28], Equation (13) is used to determine the subsampled chroma pair of each 2 × 2 chroma block

B^{C b C r}

; experimental data indicated that the CDM method achieves better compression performance relative to most Bayer CFA pattern-independent chroma subsampling methods. Furthermore, according to the convex function definition in [70], Chung et al. [29] proved that the COPY-based 2 × 2 Bayer CFA block-distortion function in Equation (11) is a convex function because the determinant of the Hessian matrix of

D^{B a y e r} (C b_{s}, C r_{s})

in Equation (11) is equal to 66.1412 (>0). Then, using this convex function property, an iterative CDM (ICDM) method was proposed to obtain a better subsampled (

C b

C r

)-pair of

B^{C b C r}

when compared with the CDM method [28].

Based on the same differentiation technique used in [28] but considering the 2 × 2 demosaiced RGB full-color block-distortion function, Lin et al. [30] derived that the subsampled chroma pair of

B^{C b C r}

equals the average chroma pair of the four chroma entries of

B^{C b C r}

. Furthermore, they proposed a “modified 4:2:0(A)” chroma subsampling method that selects the best case among the four average subsampled chroma pairs of

B^{C b C r}

by considering the four combinations of the ceiling operation-based 4:2:0(A) and the floor operation-based 4:2:0(A). At the client side, the “modified 4:2:0(A)” method adopts the three neighboring (TN) reference pixels-based upsampling process [71]. However, in our experiment, the “modified 4:2:0(A)-COPY” method, where “COPY” denotes the copy interpolation, outperforms the “modified 4:2:0(A)-TN” method. Therefore, the “modified 4:2:0(A)-COPY” method is included in the comparative method instead of the modified 4:2:0(A)-TN method.

The Bilinear Interpolation-Based Distortion Minimization (BIDM) Method

To improve the accuracy of COPY-based block-distortion function in Equation (11), Chung et al. [31] proposed a more effective bilinear interpolation-based (BI-based) 2 × 2 Bayer CFA block-distortion function. For simplicity, we just introduce it for each 2 × 2 Cb block

B^{C b}

. For convenience, let the subsampled Cb parameter of

B^{C b}

, denoted by

C b_{s}

, be located at (1, 0) in Figure 8. At the server side, we now describe how to express the estimated top-left entry of

B^{C b}

, denoted by

C b_{1}^{e s t}

, as a function with the parameter

C b_{s}

. The functions for the other three estimated entries of

B^{C b}

, namely

C b_{2}^{e s t}

C b_{3}^{e s t}

, and

C b_{4}^{e s t}

, can be derived similarly. After estimating the four entries of

B^{C b C r}

, the BI-based 2 × 2 Bayer CFA block-distortion function can be derived.

To estimate

C b_{1}^{e s t}

which is located at (3/4, 1/4), the subsampled chroma parameter

C b_{s}

and the three neighboring subsampled Cb values of

B^{C b}

, namely

C b_{1, 1}

located at (1, 1),

C b_{0, 1}

located at (0, 1), and

C b_{0, 0}

located at (0, 0), are referred to. Because the BI-based distortion minimization (BIDM) method is performed on each 2 × 2 chroma block in a raster scanning order, the three reference subsampled

C b

values were obtained in advance. To estimate

C b^{e s t, 2}

C b^{e s t, 3}

, and

C b^{e s t, 4}

, some future neighboring reference subsampled

C b

values are unknown, but they can be calculated by 4:2:0(A) or the CDM chroma subsampling method in which neither method needs to reference any neighboring subsampled

C b

values of the current 2 × 2

C b

block. In our experiment, we adopt 4:2:0(A) to calculate the future subsampled Cb values.

Following the notations in Figure 8 and using the above BI-based approach [31],

C b_{1}^{e s t}

is estimated as

C b_{1}^{e s t} = \frac{9}{16} C b_{s} + \frac{1}{16} C b_{0, 1} + \frac{3}{16} C b_{1, 1} + \frac{3}{16} C b_{0, 0}

(14)

In general, the estimation of

C b_{i}^{e s t}

1 \leq i \leq 4

, is expressed as

C b_{i}^{e s t} = \frac{9}{16} C b_{s} + \bar{C b_{i}}

(15)

with

\begin{matrix} \bar{C b_{1}} = \frac{1}{16} C b_{0, 1} + \frac{3}{16} C b_{1, 1} + \frac{3}{16} C b_{0, 0} \\ \bar{C b_{2}} = \frac{1}{16} C b_{2, 1} + \frac{3}{16} C b_{1, 1} + \frac{3}{16} C b_{2, 0} \\ \bar{C b_{3}} = \frac{1}{16} C b_{0, - 1} + \frac{3}{16} C b_{1, - 1} + \frac{3}{16} C b_{0, 0} \\ \bar{C b_{4}} = \frac{1}{16} C b_{2, - 1} + \frac{3}{16} C b_{1, - 1} + \frac{3}{16} C b_{2, 0} \end{matrix}

After estimating the four chroma pairs of

B^{C b C r}

, the estimated 2 × 2

C b C r

block

B^{e s t, C b C r}

, denoted by

B^{e s t, C b C r}

, the collocated 2 × 2 luma block

B^{Y}

, the Bayer CFA pattern in Figure 1a with

P a t_{1}

= [

G_{1}

R_{2}

B_{3}

G_{4}

], and Equation (10) are utilized together to reconstruct the estimated 2 × 2 Bayer CFA block

B^{e s t, B a y e r}

(= [

G_{1}^{e s t}

R_{2}^{e s t}

B_{3}^{e s t}

G_{4}^{e s t}

]) at the server side. By Equations (9) and (15), the block-distortion

D^{B a y e r} (C b_{s}, C r_{s})

between each 2 × 2 Bayer CFA block and the corresponding 2 × 2 estimated Bayer CFA block is expressed as

\begin{matrix} D^{B a y e r} (C b_{s}, C r_{s}) & = {(G_{1} - G_{1}^{e s t})}^{2} + {(R_{2} - R_{2}^{e s t})}^{2} + {(B_{3} - B_{3}^{e s t})}^{2} + {(G_{4} - G_{4}^{e s t})}^{2} \\ = \sum_{i = 1}^{4} {[(1.164 Y_{i} + a_{i} C b_{i} + b_{i} C r_{i}) - (1.164 Y_{i} + a_{i} C b_{i}^{e s t} + b_{i} C r_{i}^{e s t})]}^{2} \\ = \sum_{i = 1}^{4} [a_{i} (C b_{i} - (\frac{9}{16} C b_{s} + \bar{C b_{i}})) + b_{i} (C r_{i} - (\frac{9}{16} C r_{s} + \bar{C r_{i}}))]^{2} \end{matrix}

(16)

where

a_{i}

and

b_{i}

have been defined in Equation (12).

Furthermore, the determinant of the Hessian matrix of

D^{B a y e r} (C b_{s}, C r_{s})

in Equation (16) equals 6.6216 (>0) [31], and it deduces the convex property of the positive definite block-distortion function

D^{B a y e r} (C b_{s}, C r_{s})

in Equation (16). Using the differentiation technique, it yields that in the real domain, the solution of the subsampled (

C b

C r

)-pair of

B^{C b C r}

, denoted by (

C b_{s}^{(0)}

C r_{s}^{(0)}

), is expressed as

\begin{matrix} C b_{s}^{(0)} = \frac{(\sum_{i = 1}^{4} b_{i}^{2}) [\sum_{i = 1}^{4} a_{i}^{2} (\bar{C b_{i}} - C b_{i}) + a_{i} b_{i} (\bar{C r_{i}} - C r_{i})] - (\sum_{i = 1}^{4} a_{i} b_{i}) [\sum_{i = 1}^{4} b_{i}^{2} (\bar{C r_{i}} - C r_{i}) + a_{i} b_{i} (\bar{C b_{i}} - C b_{i})]}{\frac{9}{16} [(\sum_{i = 1}^{4} a_{i} b_{i})^{2} - (\sum_{i = 1}^{4} a_{i}^{2}) (\sum_{i = 1}^{4} b_{i}^{2})]} \\ C r_{s}^{(0)} = \frac{(\sum_{i = 1}^{4} a_{i}^{2}) [\sum_{i = 1}^{4} b_{i}^{2} (\bar{C r_{i}} - C r_{i}) + a_{i} b_{i} (\bar{C b_{i}} - C b_{i})] - (\sum_{i = 1}^{4} a_{i} b_{i}) [\sum_{i = 1}^{4} a_{i}^{2} (\bar{C b_{i}} - C b_{i}) + a_{i} b_{i} (\bar{C r_{i}} - C r_{i})]}{\frac{9}{16} [(\sum_{i = 1}^{4} a_{i} b_{i})^{2} - (\sum_{i = 1}^{4} a_{i}^{2}) (\sum_{i = 1}^{4} b_{i}^{2})]} \end{matrix}

(17)

In the integer domain, the subsampled chroma solution in Equation (17) is taken as the initial solution of the iterative BIDM method for obtaining a better subsampled chroma solution of

B^{C b C r}

. In the (k + 1)th iteration of BIDM, if the previous subsampled chroma solution can be replaced by a better solution among the eight neighboring subsampled chroma candidates of (

C b_{s}^{(k), B a y e r}

C r_{s}^{(k), B a y e r}

), we set k = k + 1 and repeat the above iterative solution-refinement process; otherwise, we stop the iterative method and report (

C b_{s}^{(k), B a y e r}

C r_{s}^{(k), B a y e r}

) as the final subsampled (

C b

C r

)-pair of

B^{C b C r}

. Because the BIDM chroma subsampling method heavily involves the bilinear interpolation, naturally, BIDM selects the bilinear interpolation (BI) as the chroma upsampling process at the client side. Experimental results indicate that the combination BIDM-BI [31] outperforms 4:2:0(A)-BI, CDM-COPY [28], and ICDM-BI [29].

Considering the effectiveness of the above Bayer CFA pattern-dependent chroma subsampling methods, the CDM method, the “modified 4:2:0(A)” method, and the BIDM method are selected as the representatives to evaluate the compression performance of the DF-based compression scheme.

3.3. Luma Modification

After introducing the related chroma subsampling works in the CSLM methods, in this subsection, the optimal Bayer CFA pattern-dependent luma modification (OLM) method [33] is first introduced, and then, the difference between the OLM method and Chiu et al.’s non-optimal method [34] is highlighted. For easy exposition, the BIDM-OLM method is used to assist the introduction of the OLM method. After performing the BIDM method on each 2 × 2 chroma block

B^{C b C r}

, the goal of the OLM method is to determine the best modified luma value

Y_{i}^{'}

1 \leq i \leq 4

, for the corresponding 2 × 2 luma block

B^{Y}

such that the 1 × 1 Bayer CFA pixel-distortion can be minimized, achieving better quality of the reconstructed Bayer CFA image.

After performing the iterative BIDM method on the chroma block

B^{C b C r}

, let the subsampled (

C b

C r

)-pair of

B^{C b C r}

be denoted by (

C b^{B a y e r}

C r^{B a y e r}

). Let the two chroma variables

C b_{i}

and

C r_{i}

1 \leq i \leq 4

, in Equation (10) be replaced by

C b^{B a y e r}

and

C r^{B a y e r}

, respectively. It is intractable to search for a unique modified luma value of

Y_{i}

1 < = i < = 4

, which satisfies the three equations in Equation (10) simultaneously. To derive the search interval for determining the best modified luma value

Y_{i}^{'}

1 \leq i \leq 4

, for each 2 × 2 luma block

B^{Y}

, we consider

Y_{i}^{R}

Y_{i}^{G}

, and

Y_{i}^{B}

, which satisfy

\{\begin{matrix} R_{i} = 1.164 (Y_{i}^{R} - 16) + 1.596 (C r^{B a y e r} - 128) & for i = 2 \\ G_{i} = 1.164 (Y_{i}^{G} - 16) - 0.391 (C b^{B a y e r} - 128) - 0.813 (C r^{B a y e r} - 128) & for i = 1 or 4 \\ B_{i} = 1.164 (Y_{i}^{B} - 16) + 2.018 (C b^{B a y e r} - 128) & for i = 3 \end{matrix}

(18)

Solving each equation in Equation (18), it yields

\{\begin{matrix} Y_{i}^{R} = \frac{(R_{i} - 1.596 (C r^{B a y e r} - 128))}{1.164} + 16 & for i = 2 \\ Y_{i}^{G} = \frac{(G_{i} + 0.391 (C b^{B a y e r} - 128) + 0.813 (C r^{B a y e r} - 128))}{1.164} + 16 & for i = 1 or 4 \\ Y_{i}^{B} = \frac{(B_{i} - 2.018 (C b^{B a y e r} - 128))}{1.164} + 16 & for i = 3 \end{matrix}

(19)

By using the contradiction method, it has been proved that the modified luma value

Y_{i}^{'}

can be found in the smaller interval [

L o w_{i}

H i g h_{i}

] where

\begin{matrix} L o w_{i} = \{\begin{matrix} ⌊ Y_{1}^{G} ⌋ & for i = 1 \\ ⌊ Y_{2}^{R} ⌋ & for i = 2 \\ ⌊ Y_{3}^{B} ⌋ & for i = 3 \\ ⌊ Y_{4}^{G} ⌋ & for i = 4 \end{matrix} \\ H i g h_{i} = \{\begin{matrix} ⌈ Y_{1}^{G} ⌉ & for i = 1 \\ ⌈ Y_{2}^{R} ⌉ & for i = 2 \\ ⌈ Y_{3}^{B} ⌉ & for i = 3 \\ ⌈ Y_{4}^{G} ⌉ & for i = 4 \end{matrix} \end{matrix}

(20)

where “

⌊ \cdot ⌋

” and “

⌈ \cdot ⌉

” denote the floor function and ceiling function, respectively. For

1 \leq i \leq 4

, it can be verified that the condition “(

H i g h_{i}

−

L o w_{i}

) = 2” holds, and it thus indicates that the best modified luma value

Y_{i}^{'}

can be determined in constant time such that the pixel-distortion value is minimal, where the pixel-distortion (PD) function is defined by

\begin{matrix} P D (Y_{i}) = \{\begin{matrix} {(G_{1} - 1.164 (Y_{i}^{G} - 16) - 0.391 (C b^{B a y e r} - 128) - 0.813 (C r^{B a y e r} - 128))}^{2} & for i = 1 \\ {(R_{2} - 1.164 (Y_{i}^{R} - 16) + 1.596 (C r^{B a y e r} - 128))}^{2} & for i = 2 \\ {(B_{3} - 1.164 (Y_{i}^{B} - 16) + 2.018 (C b^{B a y e r} - 128))}^{2} & for i = 3 \\ {(G_{4} - 1.164 (Y_{i}^{G} - 16) - 0.391 (C b^{B a y e r} - 128) - 0.813 (C r^{B a y e r} - 128))}^{2} & for i = 4 \end{matrix} \end{matrix}

(21)

For

I^{B a y e r}

, the previous method proposed by Chiu et al. [34] determined the modified luma value

Y_{i}^{'}

by using the appropriate equality in Equation (19) directly, but it cannot guarantee that the determined modified luma value is the best. Experimental data revealed that the CDM-OLM method [33] can achieve at least 10 dB quality improvement when compared with the pure CDM chroma subsampling method [28].

To evaluate the compression performance of the DF-based compression scheme for

I^{B a y e r}

, considering the effectiveness, the three selected representatives are CDM-OLM, “modified 4:2:0(A)”-OLM, and BIDM-OLM.

4. Experimental Results

Based on the ground-truth Bayer CFA images collected from the five datasets, namely Kodak, IMAX, SCI, Videos, and CI datasets, the quality and quality–bitrate tradeoff comparison of the reconstructed Bayer CFA and RGB full-color images by using the DF-based compression scheme and the RCT-based compression scheme for encoding Bayer CFA images are demonstrated. In addition, the execution time comparison is also provided.

All considered methods for compressing

I^{B a y e r}

are implemented on a computer with an Intel Core i7-6700 CPU 3.4 GHz and 24 GB RAM. The operating system is the Microsoft Windows 10 64-bit operating system. The program development environment is Visual C++ 2019. The compression standards used to evaluate the compression performance of the considered methods are JPEG-2000 [18,19] and the VVC platform VTM-16.2 [35].

In order to redo the compression experiments for Bayer CFA images, the C++ source codes of the three DF-based compression combinations, namely CDM-OLM-COPY [28,33], “modified 4:2:0(A)”-OLM-COPY [30,33], and BIDM-OLM-BI [31,33], where “COPY” and “BI” denote the upsampling processes used at the client side, can be accessed from the website https://github.com/shuanme/DF-based (accessed on 25 September 2022). The C++ source codes of the three RCT-based compression methods, namely the

Y D_{g} C_{o} C_{g}

method [3], the

Y L M N

method [4], and the

Y Δ C_{b} C_{r}

method [8], can be accessed from the website https://github.com/shuanme/RCT-based (accessed on 25 September 2022).

Because it is difficult to access the testing RGB full-color image datasets with real image pipelining parameters, such as the gamma correction coefficients and white balance parameters, at the client side, the reconstructed RGB full-color image is obtained by demosaicing the decompressed reconstructed Bayer CFA image. As mentioned before, the demosaiced Bayer CFA image

I^{d e m o, B a y e r}

obtained at the server side is used as the ground-truth RGB full-color image for evaluating the quality of the reconstructed RGB full-color image obtained at the client side.

4.1. Quality Comparison and Discussion

When setting QP to zero for VTM-16.2 and setting CR to 1 for JPEG-2000, we first compare the quality performance of the reconstructed Bayer CFA and RGB full-color images between the DF-based compression scheme and the CF-based compression scheme for

I^{B a y e r}

. Secondly, the discussion of the compression comparison of the two compression schemes is provided.

4.1.1. Quality Comparison and Discussion

Three popular objective quality metrics, namely PSNR, SSIM [36], and FSIM [37], are used to compare the quality performance of the reconstructed Bayer CFA images and the reconstructed RGB full-color images by using all the considered compression methods for Bayer CFA images. PSNR is used to evaluate the average quality of one reconstructed Bayer CFA image, and it is defined by

P S N R = \frac{1}{N} \sum_{n = 1}^{N} 10 {log}_{10} \frac{255^{2}}{M S E}

(22)

where denotes the number of the Bayer CFA images in one dataset; MSE (mean square error) equals

\frac{1}{X Y} \sum_{i = 1}^{X} \sum_{i = 1}^{Y} {(I^{B a y e r} (i, j) - I^{r e c, B a y e r} (i, j))}^{2}

, where

I^{B a y e r}

denotes the ground-truth Bayer CFA image,

I^{r e c, B a y e r}

denotes the reconstructed Bayer CFA image, and XY denotes the image size. First, the PSNR value of each dataset is calculated. Next, the average PSNR values of the five considered datasets are calculated as the average PSNR value of one reconstructed Bayer CFA image. By using Kiku et al.’s demosaicing method [60] to demosaic each reconstructed Bayer CFA image

I^{r e c, B a y e r}

to a reconstructed RGB full-color image

I^{r e c, R G B}

, the CPSNR of

I^{r e c, R G B}

is defined by

C P S N R = \frac{1}{N} \sum_{n = 1}^{N} 10 {log}_{10} \frac{255^{2}}{C M S E}

(23)

where CMSE equals

\frac{1}{X Y} \sum_{i = 1}^{X} \sum_{i = 1}^{Y} {(I^{d e m o, R G B} (i, j) - I^{r e c, B a y e r} (i, j))}^{2}

, where

I^{d e m o, R G B}

denotes the ground-truth-demosaiced RGB full-color image obtained at the server side.

SSIM is expressed as the product of the luminance mean similarity, the contrast similarity, and the structure similarity between

I^{B a y e r}

and

I^{r e c, B a y e r}

. To measure the color SSIM (CSSIM) of

I^{r e c, R G B}

, the SSIM value of each c-color,

c \in {R, G, B}

, the image of

I^{r e c, R G B}

is calculated, and then, the average SSIM value of the three calculated SSIM values is used as the CSSIM value of

I^{r e c, R G B}

. FSIM utilizes the phase consistency and gradient magnitude to weight the local quality maps for obtaining a feature quality score of

I^{r e c, B a y e r}

. The color FSIM (CFSIM) value of

I^{r e c, R G B}

is defined as the average FSIM value of the three calculated SSIM values of the three color images of

I^{r e c, R G B}

. Interested readers are suggested to refer to the original papers [36,37] for the detailed definitions of SSIM and FSIM, respectively.

Based on the five testing datasets, when setting QP = 0 and CR = 1 for VTM-16.2 and JPEG-2000, respectively, Table 1 tabulates the average PSNR, SSIM, and FSIM performance of the reconstructed Bayer CFA images by using the three representative DF-based compression methods and the three representative RCT-based compression methods, where the individual results on VTM-16.2 and JPEG-2000 are tabulated in different rows. From Table 1, we observe that on VTM-16.2 and JPEG-2000, the

Y D_{g} C_{o} C_{g}

method always achieves the highest PSNR, and the BIDM-OLM method is always ranked second; the BIDM-OLM method achieves the highest SSIM and FSIM, and the

Y D_{g} C_{o} C_{g}

method is ranked second.

The average CPSNR, CSSIM, and CFSIM values of the reconstructed RGB full-color images are tabulated in Table 2. From Table 2, we observe that on VTM-16.2 and JPEG-2000, the

Y D_{g} C_{o} C_{g}

method always achieves the highest CPSNR, and the BIDM-OLM method is ranked second; the BIDM-OLM method always achieves the highest CSSIM and CFSIM, and the

Y D_{g} C_{o} C_{g}

method is always ranked second.

4.1.2. Execution Time Requirement Comparison and Discussion

For each image, the average execution time (in seconds) to transform the input Bayer CFA image to the subsampled

Y C b C r

image for the DF-based compression method or the RCT-based format for the CF-based compression method is tabulated in Table 3. Besides the average execution time of one image for each testing dataset, the average execution time of one image for all five datasets is listed in the last column of Table 3, namely “AVG”. From Table 3, we observe that the dataset “Kodak” takes more time than the other four datasets because of its high resolution. Furthermore, the compression methods in the DF-based compression scheme take more time than the methods in the CF-based compression scheme. Because the average execution time of one image is always less than one second, it can be neglected when compared with the practical encoding time. It is noticeable that each DF-based method (or RCT-based method) can be realized by the usage of the GPU-based parallel computation to reduce the time requirement because each method can be decomposed into many independent subtasks.

4.2. Quality–Bitrate Tradeoff Comparison and Discussion

Under different QP and CR intervals, in terms of the BD-PSNR metric, the quality–bitrate tradeoff comparison for all considered compression methods for

I^{B a y e r}

is reported and discussed. To compare the visual effects for the considered compression methods for

I^{B a y e r}

, the decompressed Bayer CFA images are further demosaiced by using Kiku et al.’s method to produce the RGB full-color images.

4.2.1. The Quality–Bitrate Tradeoff Comparison

In order to show the average BD-PSNR performance comparison among the considered DF-based compression methods and RCT-based compression methods, we take the method by encoding the testing Bayer CFA images directly as the baseline compression method. According to the reconstructed Bayer CFA images using one considered compression method, under the same bitrate requirement, the quality–bitrate tradeoff metric “BD-PSNR” [38] is used to report the average PSNR gain of the considered compression method over the baseline compression method.

On VTM-16.2, five QP intervals, namely [4, 20], [12, 28], [20, 36], [28, 44], and [36, 51], are used to evaluate the BD-PSNR gains of the reconstructed Bayer CFA images by using the considered methods over the baseline method. On JPEG-2000, five CR intervals, namely [5, 20], [15, 30], [20, 35], [25, 40], and [30, 45], are used to evaluate the BD-PSNR gains of the reconstructed Bayer CFA images by using the considered methods over the baseline method. In Table 4, we observe that on JPEG-2000, the BIDM-OLM method always achieves the best BD-PSNR performance for the five CR intervals. In the same table, on VTM-16.2, the

Y D_{g} C_{o} C_{g}

method achieves the best BD-PSNR gains for the two QP intervals, namely [4, 20] and [12, 28]; the BIDM-OLM method achieves the best BD-PSNR gains for the two QP intervals, namely [20, 36] and [28, 44], and the modified 4:2:0(A)-OLM method achieves the best BD-PSNR gain for the QP interval [36, 51]. It is noticeable that in the QP interval [36, 51], the BD-PSNR gain of the BIDM-OLM method is quite competitive with that of the modified 4:2:0(A)-OLM method, and the difference is only 0.0059 (=3.6112 − 3.6063).

On VTM-16.2, the same five QP intervals are used to evaluate the BD-CPSNR gains of the reconstructed RGB full-color images by using the considered methods over the same baseline method. On JPEG-2000, the same five CR intervals are used to evaluate the BD-CPSNR gains of the reconstructed RGB full-color images by using the considered methods over the baseline method. In Table 5, we observe that the BIDM-OLM method always has the highest BD-CPSNR gains for the five CR intervals on JPEG-2000, and it has the highest BD-CPSNR gains for the three QP intervals, namely [20, 36], [28, 44], and [36, 51], on VTM-16.2. In Table 5, we observe that the

Y D_{g} C_{o} C_{g}

method has the highest BD-CPSNR gains for the two QP intervals, [4, 20] and [12, 28], on VTM-16.2.

In summary, the similar conclusions in Table 4 and Table 5 reveal that on JPEG-2000, the BIDM-OLM method always has the highest BD-PSNR and BD-CPSNR gains for the five CR intervals; on VTM-16.2, the

Y D_{g} C_{o} C_{g}

method has the highest BD-PSNR and BD-CPSNR gains for low and middle QP intervals, and the BIDM-OLM method has the highest BD-PSNR and BD-CPSNR gains for middle and high QP intervals.

4.2.2. The Visual Effect Comparison

This subsection shows the visual effect comparison among the considered compression methods for

I^{B a y e r}

. The testing image example in Figure 9a is taken from the 13th ground-truth IMAX image. After performing the CDM-OLM, modified 4:2:0(A)-OLM, BIDM-OLM,

Y D_{g} C_{o} C_{g}

Y L M N

, and

Y Δ C_{b} C_{r}

methods on the Bayer CFA image of Figure 9b which is cut off from Figure 9a, the demosaiced RGB full-color images are illustrated in Figure 9c–h, respectively, where for each method, the two demosaiced images on the left and on the right are under VTM-16.2 for QP = 44 and under JPEG-2000 for CR = 35, respectively.

Under VTM-16.2, for each method, from the two images on the left in Figure 9c–h, we clearly observe that the DF-based compression methods, particularly the BIDM-OLM method, have better color and texture preservation effects than the RCT-based compression methods. Under JPEG-2000, from the right two images of Figure 9c–h, we observe that the DF-based compression methods still outperform the RCT-based compression methods. The above visual effect observations indicate the visual effect merit of the DF-based methods, particularly the BIDM-OLM method, for one middle and high QP/CR case.

We now consider one low and middle QP/CR case: VTM-16.2 for QP = 24 and JPEG-2000 for CR = 20. After performing the six considered compression methods on the Bayer CFA image of Figure 9b, the demosaiced RGB full-color images are illustrated in Figure 10a–f, where for each method, the two demosaiced RGB full-color images on the left and on the right are under VTM-16.2 for QP = 24 and JPEG-2000 for CR = 20, respectively. For each method, from the two images on the right in Figure 10, under JPEG-2000, we observe that the DF-based compression methods, particularly the BIDM-OLM method, have better color and texture preservation effects in the tree, roof, and chimney parts when compared with the RCT-based compression methods.

Under VTM-16.2 for each method, as shown in the two images on the left in Figure 10, we observe that the DF-based compression methods are quite competitive with the RCT-based compression methods. Note that in the roof part, the BIDM-OLM method has a better visual effect than the

Y D_{g} C_{o} C_{g}

method.

For low QP/CR cases, the visual effect comparison is omitted because the CPSNR values of demosaiced RGB full-color images of the considered compression methods are too high to be visually distinguished. For example, under VTM-16.2 for

Q P \leq 20

, the CPSNR values are often larger than or equal to 40. Under the two codecs, VTM-16.2 and JPEG-2000 for different QP and CR values, it is suggested that the readers refer to the related experimental results in the above-mentioned two github websites.

5. Conclusions and Future Works

We have introduced the compression-first-based compression methods, in particular the reversible color transform-based (RCT-based) compression methods, and the demosaicing-first-based (DF-based) compression methods for Bayer CFA images. Based on five datasets, thorough experiments have been carried out to compare the quality and quality of bitrate tradeoff performance of the RCT-based compression methods and the DF-based compression methods. To the best of our knowledge, this is the first time that such a compression performance comparison has been reported for the two compression approaches for

I^{B a y e r}

. Experimental results demonstrated that on JPEG-2000, the BIDM-OLM method always has the highest BD-PSNR and BD-CPSNR gains for different CR intervals. On VTM-16.2, the

Y D_{g} C_{o} C_{g}

method has the highest BD-PSNR and BD-CPSNR gains for low and middle QP intervals, and the BIDM-OLM method has the highest BD-PSNR and BD-CPSNR gains for middle and high QP intervals.

Some future works are addressed below. The first future work is to deploy some image pipelining techniques, such as denoising, gamma correction, and white balancing, into the reconstructed Bayer CFA image at the client side to produce the reconstructed RGB full-color images for the above-mentioned two compression schemes for

I^{B a y e r}

. After that, the compression performance is examined. The second future work is to extend the results of this article to RGBW CFA images [72,73,74], which have been widely used in consumer markets and can receive more luminance in the low illumination condition than that of Bayer CFA images [75]. In this future work, the demosaicing method for RGBW images can be adopted from the methods reported in [57,76,77]. The third future work is to take the latest SSIM variants [78,79] into account for enhancing the quality comparison of the considered compression methods for Bayer CFA images.

Author Contributions

Conceptualization, K.-L.C. and H.-Y.C.; methodology, K.-L.C. and H.-Y.C.; software, H.-Y.C., T.-L.H. and Y.-B.C.; validation, K.-L.C. and H.-Y.C.; formal analysis, K.-L.C., H.-Y.C. and T.-L.H.; investigation, K.-L.C., H.-Y.C. and T.-L.H.; resources, H.-Y.C., T.-L.H. and Y.-B.C.; data curation, H.-Y.C., T.-L.H. and Y.-B.C.; writing—original draft preparation, K.-L.C.; writing—review and editing, K.-L.C., H.-Y.C. and T.-L.H.; visualization, H.-Y.C. and T.-L.H.; supervision, K.-L.C.; project administration, K.-L.C. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the contracts MOST-110-2221-E-011-088-MY3 and MOST-108-2221-E-011-077-MY3 of the Ministry of Science and Technology, Taiwan.

Data Availability Statement

Kodak data were obtained from https://www.math.purdue.edu/~lucier/PHOTO_CD/BMP_IMAGES/ (accessed on 10 March 2020), IMAX data were obtained from https://www4.comp.polyu.edu.hk/~cslzhang/CDM_Dataset.htm (accessed on 20 December 2018), SCI data were obtained from ftp://140.118.175.164/SCI (accessed on 26 March 2021), Video data were obtained from ftp://140.118.175.164/CFASS/ (accessed on 25 August 2020), CI data were obtained from https://doi.org/10.1109/TMM.2017.2749162 (accessed on 23 August 2019).

Acknowledgments

The authors appreciate the proofreading help of C. Harrington and the comments provided by the two reviewers to improve this article.

Conflicts of Interest

The authors declare no conflict of interest.

References

Bayer, B.E. Color Imaging Array. U.S. Patent 3,971,065, 20 July 1976. [Google Scholar]
Lee, S.Y.; Ortega, A. A Novel Approach of Image Compression in Digital Cameras with a Bayer Color Filter Array. In Proceedings of the 2001 International Conference on Image Processing (Cat. No. 01CH37205), Thessaloniki, Greece, 7–10 October 2001; IEEE: Piscataway, NJ, USA, 2001; Volume 3, pp. 482–485. [Google Scholar] [CrossRef]
Malvar, H.S.; Sullivan, G.J. Progressive-to-Lossless Compression of Color-Filter-Array Images Using Macropixel Spectral-Spatial Transformation. In Proceedings of the 2012 Data Compression Conference, Snowbird, UT, USA, 10–12 April 2012; IEEE: Piscataway, NJ, USA, 2012; pp. 3–12. [Google Scholar] [CrossRef]
Mohammed, S.K.; Wahid, K.A. Lossless Compression in Bayer Color Filter Array for Capsule Endoscopy. IET Image Process. 2017, 5, 13823–13834. [Google Scholar] [CrossRef]
Mohammed, S.K.; Wahid, K.A. Lossless and Reversible Colour Space Transformation for Bayer Colour Filter Array Images. IET Image Process. 2018, 12, 1485–1490. [Google Scholar] [CrossRef]
Rahman, K.M.M.; Mohammed, S.K.; Vedaei, S.S.; Mohebbian, M.R.; Chafjiri, F.S.; Wahid, K.A. A Low Complexity Lossless Bayer CFA Image Compression. Signal Image Video Process. 2021, 15, 1767–1775. [Google Scholar] [CrossRef]
Hernández-Cabronero, M.; Marcellin, M.W.; Blanes, I.; Serra-Sagristà, J. Lossless Compression of Color Filter Array Mosaic Images with Visualization via JPEG 2000. IEEE Trans. Image Process. 2018, 20, 257–270. [Google Scholar] [CrossRef] [Green Version]
Richter, T.; Fößel, S. Bayer Pattern Compression with JPEG XS. In Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 22–25 September 2019; pp. 3177–3181. [Google Scholar] [CrossRef]
Richter, T.; Fößel, S.; Descampe, A.; Rouvroy, G. Bayer CFA Pattern Compression with JPEG XS. IEEE Trans. Image Process. 2021, 30, 6557–6569. [Google Scholar] [CrossRef] [PubMed]
Zhang, N.; Wu, X. Lossless Compression of Color Mosaic Images. IEEE Trans. Image Process. 2006, 15, 1379–1388. [Google Scholar] [CrossRef]
Lee, Y.; Hirakawa, K.; Nguyen, T.Q. Camera-Aware Multi-Resolution Analysis for Raw Image Sensor Data Compression. IEEE Trans. Image Process. 2018, 27, 2806–2817. [Google Scholar] [CrossRef] [PubMed]
Suzuki, T. Wavelet-Based Spectral–Spatial Transforms for CFA-Sampled Raw Camera Image Compression. IEEE Trans. Image Process. 2019, 29, 433–444. [Google Scholar] [CrossRef]
Lee, Y.; Hirakawa, K. Shift-And-Decorrelate Lifting: CAMRA for Lossless Intra Frame CFA Video Compression. IEEE Signal Process. Lett. 2020, 27, 461–465. [Google Scholar] [CrossRef]
Huang, L.; Suzuki, T. Weighted Wavelet-Based Spectral-Spatial Transforms for CFA-Sampled Raw Camera Image Compression Considering Image Features. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Singapore, 22–27 May 2022. [Google Scholar] [CrossRef]
Chung, K.H.; Xiaolin, W. A Lossless Compression Scheme for Bayer Color Filter Array Images. IEEE Trans. Image Process. 2008, 17, 134–144. [Google Scholar] [CrossRef] [Green Version]
Pennebaker, W.B.; Mitchell, J.L. JPEG: Still Image Data Compression Standard; Springer Science & Business Media: Berlin/Heidelberg, Germany, 1992. [Google Scholar]
Dufaux, F.; Sullivan, G.J.; Ebrahimi, T. The JPEG XR Image Coding Standard [Standards in a Nutshell]. IEEE Signal Process. Mag. 2009, 26, 195–204. [Google Scholar] [CrossRef] [Green Version]
Skodras, A.; Christopoulos, C.; Ebrahimi, T. The Jpeg 2000 Still Image Compression Standard. IEEE Signal Process. Mag. 2001, 18, 36–58. [Google Scholar] [CrossRef]
Rabbani, M. Book Review: JPEG2000: Image Compression Fundamentals, Standards and Practice; Kluwer Academic Publishers: Dordrecht, The Netherlands, 2002. [Google Scholar]
Malvar, H.S. Adaptive Run-length/Golomb-Rice Rncoding of Quantized Generalized Gaussian Sources with Unknown Statistics. In Proceedings of the Data Compression Conference (DCC’06), Snowbird, UT, USA, 28–30 March 2006; IEEE: Piscataway, NJ, USA, 2006; pp. 23–32. [Google Scholar]
JPEG XS; Standard, ISO/IEC 21122. ISO: Geneva, Switzerland, 2019.
Mallat, S. A Wavelet Tour of Signal Processing, 3rd ed.; Academic Press: Cambridge, MA, USA, 2008. [Google Scholar]
Sullivan, G.; Sun, S. Spatial Scalability Filters. In Proceedings of the Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG, 17th Meeting, Nice, FR, USA, 14–21 October 2005. [Google Scholar]
Luthra, A.; Francois, E.; Husak, W. Call for Evidence (CfE) for HDR and WCG Video Coding; ISO/IEC JTC1/SC29/WG11 MPEG2014 N; ISO: Geneve, Switzerland, 2015; Volume 15083. [Google Scholar]
Zhang, Y.; Zhao, D.; Zhang, J.; Xiong, R.; Gao, W. Interpolation-Dependent Image Downsampling. IEEE Trans. Image Process. 2011, 20, 3291–3296. [Google Scholar] [CrossRef] [PubMed]
Wang, S.; Gu, K.; Ma, S.; Gao, W. Joint Chroma Downsampling and Upsampling for Screen Content Image. IEEE Trans. Circuits Syst. Video Technol. 2015, 26, 1595–1609. [Google Scholar] [CrossRef]
Chen, H.; Sun, M.; Steinbach, E. Compression of Bayer-Pattern Video Sequences Using Adjusted Chroma Subsampling. IEEE Trans. Circuits Syst. Video Technol. 2009, 19, 1891–1896. [Google Scholar] [CrossRef]
Lin, C.H.; Chung, K.L.; Yu, C.W. Novel Chroma Subsampling Strategy Based on Mathematical Optimization for Compressing Mosaic Videos with Arbitrary RGB Color Filter Arrays in H. 264/AVC and HEVC. IEEE Trans. Circuits Syst. Video Technol. 2016, 26, 1722–1733. [Google Scholar] [CrossRef]
Chung, K.L.; Lee, Y.L.; Chien, W.C. Effective Gradient Descent-Based Chroma Subsampling Method for Bayer CFA Images in HEVC. IEEE Trans. Circuits Syst. Video Technol. 2019, 29, 3281–3290. [Google Scholar] [CrossRef]
Lin, T.L.; Yu, Y.C.; Jiang, K.H.; Liang, C.F.; Liaw, P.S. Novel Chroma Sampling Methods for CFA Video Compression in AVC, HEVC and VVC. IEEE Trans. Circuits Syst. Video Technol. 2019, 30, 3167–3180. [Google Scholar] [CrossRef]
Chung, K.L.; Chen, S.N. An Effective Bilinear Interpolation-Based Iterative Chroma Subsampling Method for Color Images. Multimed. Tools Appl. 2022, 81, 32191–32213. [Google Scholar] [CrossRef]
Chung, K.L.; Leung, T.C.; Liu, T.Y.; Tseng, Y.C. A Cubic Convolution Interpolation-Based Chroma Subsampling Method for Bayer and RGBW CFA Raw Images. IEEE Access 2022, 10, 22687–22699. [Google Scholar] [CrossRef]
Chung, K.L.; Hsu, T.C.; Huang, C.C. Joint Chroma Subsampling and Distortion-Minimization-Based Luma Modification for RGB Color Images with Application. IEEE Trans. Image Process. 2017, 26, 4626–4638. [Google Scholar] [CrossRef] [PubMed]
Chiu, Y.H.; Chung, K.L.; Lin, C.H. An Improved Universal Subsampling Strategy for Compressing Mosaic Videos with Arbitrary RGB Color Filter Srrays in H. 264/AVC. J. Vis. Commun. Image Represent. 2014, 25, 1791–1799. [Google Scholar] [CrossRef]
VTM-16.2. 2022. Available online: https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/tree/VTM-16.2 (accessed on 19 May 2022).
Wang, Z.; Bovik, A.; Sheikh, H.; Simoncelli, E. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Zhang, L.; Zhang, L.; Mou, X.; Zhang, D. FSIM: A Feature Similarity Index for Image Quality Assessment. IEEE Trans. Image Process. 2011, 20, 2378–2386. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Bjontegaard, G. Calculation of Average PSNR Differences between RD-curves. In Proceedings of the 13th VCEG Meeting, Austin, TX, USA, 2–4 April 2001. [Google Scholar]
Stasinski, R.; Konrad, J. A New Class of Fast Shape-adaptive Orthogonal Transforms and Their Application to Region-based Image Compression. IEEE Trans. Image Process. 1999, 9, 16–34. [Google Scholar] [CrossRef] [Green Version]
Malvar, H.S.; Sullivan, G.J. YCoCg-R: A Color Space with RGB Reversibility and Low Dynamic Range; ITU-T/ISO/IEC JVT Document JVT-I014; ISO: Geneve, Switzerland, 2003. [Google Scholar]
Adams, M.D.; Kossentini, F.; Ward, R.K. Generalized S Transform. IEEE Trans. Signal Process. 2002, 50, 2831–2842. [Google Scholar] [CrossRef]
Malvar, H.S.; Sullivan, G.J.; Srinivasan, S. Lifting-based Reversible Color Transformations for Image Compression. SPIE 2008, 7073, 707307. [Google Scholar] [CrossRef]
Khan, T.H.; Wahid, K.A. Low-complexity Colour-space for Capsule Endoscopy Image Compression. Electron. Lett. 2011, 47, 1217–1218. [Google Scholar] [CrossRef]
Gunturk, B.K.; Glotzbach, J.; Altunbasak, Y.; Schafer, R.W.; Mersereau, R.M. Demosaicking: Color Filter Array Interpolation. IEEE Signal Process. Mag. 2005, 22, 44–54. [Google Scholar] [CrossRef]
Li, X.; Gunturk, B.; Zhang, L. Image Demosaicing: A Systematic Survey. In Proceedings of the Visual Communications and Image Processing 2008, San Jose, CA, USA, 29–31 January 2008; SPIE: Bellingham, WA, USA, 2008; Volume 6822, pp. 489–503. [Google Scholar] [CrossRef]
Mihoubi, S.; Lapray, P.J.; Bigué, L. Survey of Demosaicking Methods for Polarization Filter Array Images. Sensors 2018, 18, 3688. [Google Scholar] [CrossRef] [Green Version]
Wang, Y.; Cao, R.; Guan, Y.; Liu, T.; Yu, Z. A Deep Survey in the Applications of Demosaicking. In Proceedings of the 2021 3rd International Academic Exchange Conference on Science and Technology Innovation (IAECST), Guangzhou, China, 10–12 December 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 596–602. [Google Scholar] [CrossRef]
Sakamoto, T.; Nakanishi, C.; Hase, T. Software Pixel Interpolation for Digital Still Cameras Suitable for a 32-bit MCU. IEEE Trans. Consum. Electron. 1998, 44, 1342–1352. [Google Scholar] [CrossRef]
Kimmel, R. Demosaicing: Image Reconstruction from Color CCD Samples. IEEE Trans. Image Process. 1999, 8, 1221–1228. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Gunturk, B.K.; Altunbasak, Y.; Mersereau, R.M. Color Plane Interpolation Using Alternating Projections. IEEE Trans. Image Process. 2002, 11, 997–1013. [Google Scholar] [CrossRef] [PubMed]
Pei, S.C.; Tam, I.K. Effective Color Interpolation in CCD Color Filter Arrays Using Signal Correlation. IEEE Trans. Circuits Syst. Video Technol. 2003, 13, 503–513. [Google Scholar] [CrossRef]
Lu, W.; Tan, Y.P. Color Filter Array Demosaicking: New Method and Performance Measures. IEEE Trans. Image Process. 2003, 12, 1194–1210. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Wu, X.; Zhang, N. Primary-Consistent Soft-Decision Color Demosaicking for Digital Cameras (Patent Pending). IEEE Trans. Image Process. 2004, 13, 1263–1274. [Google Scholar] [CrossRef]
Lukac, R.; Plataniotis, K.N. Normalized Color-Ratio Modeling for CFA Interpolation. IEEE Trans. Consum. Electron. 2004, 50, 737–745. [Google Scholar] [CrossRef]
Hirakawa, K.; Parks, T.W. Adaptive Homogeneity-Directed Demosaicing Algorithm. IEEE Trans. Image Process. 2005, 14, 360–369. [Google Scholar] [CrossRef]
Chung, K.L.; Yang, W.J.; Yan, W.M.; Wang, C.C. Demosaicing of Color Filter Array Captured Images Using Gradient Edge Detection Masks and Adaptive Heterogeneity-Projection. IEEE Trans. Image Process. 2008, 17, 2356–2367. [Google Scholar] [CrossRef]
Condat, L. A Generic Variational Approach for Demosaicking from an Arbitrary Color Filter Array. In Proceedings of the 2009 16th IEEE International Conference on Image Processing (ICIP), Cairo, Egypt, 7–10 November 2009; IEEE: Piscataway, NJ, USA, 2009; pp. 1625–1628. [Google Scholar] [CrossRef] [Green Version]
Yang, W.J.; Chung, K.L.; Yang, W.N.; Lin, L.C. Universal Chroma Subsampling Strategy for Compressing Mosaic Video Sequences With Arbitrary RGB Color Filter Arrays in H.264/AVC. IEEE Trans. Circuits Syst. Video Technol. 2013, 23, 591–606. [Google Scholar] [CrossRef]
Zhang, L.; Wu, X.; Buades, A.; Li, X. Color Demosaicking by Local Directional Interpolation and Nonlocal Adaptive Thresholding. J. Electron. Imaging 2011, 20, 023016. [Google Scholar] [CrossRef] [Green Version]
Kiku, D.; Monno, Y.; Tanaka, M.; Okutomi, M. Residual Interpolation for Color Image Demosaicking. In Proceedings of the 2013 IEEE International Conference on Image Processing, Melbourne, Australia, 15–18 September 2013; IEEE: Piscataway, NJ, USA, 2013; pp. 2304–2308. [Google Scholar] [CrossRef]
Ye, W.; Ma, K.K. Color Image Demosaicing Using Iterative Residual Interpolation. IEEE Trans. Image Process. 2015, 24, 5879–5891. [Google Scholar] [CrossRef] [PubMed]
Tan, D.S.; Chen, W.Y.; Hua, K.L. DeepDemosaicking: Adaptive Image Demosaicking via Multiple Deep Fully Convolutional Networks. IEEE Trans. Image Process. 2018, 27, 2408–2419. [Google Scholar] [CrossRef] [PubMed]
Syu, N.S.; Chen, Y.S.; Chuang, Y.Y. Learning Deep Convolutional Networks for Demosaicing. arXiv 2018, arXiv:1802.03769. [Google Scholar] [CrossRef]
Ni, Z.; Ma, K.K.; Zeng, H.; Zhong, B. Color Image Demosaicing Using Progressive Collaborative representation. IEEE Trans. Image Process. 2020, 29, 4952–4964. [Google Scholar] [CrossRef]
ITU-R BT.601-5; Studio Encoding Parameters of Digital Television for Standard 4:3 and Wide-Screen 16:9 Aspect Ratios. International Telecommunication Union (ITU): Geneva, Switzerland, 2011; p. 624.
Winkler, S.; Kunt, M.; Branden Lambrecht, C.J. Vision and Video: Models and Applications. In Vision Models and Applications to Image and Video Processing; Springer: Berlin/Heidelberg, Germany, 2001; pp. 201–229. [Google Scholar] [CrossRef]
Li, X.; Orchard, M.T. New Edge-Directed Interpolation. IEEE Trans. Image Process. 2001, 10, 1521–1527. [Google Scholar] [CrossRef] [Green Version]
Lu, Y.; Li, S.; Shen, H. Virtualized Screen: A Third Element for Cloud-Mobile Convergence. IEEE Multimed. 2011, 18, 4–11. [Google Scholar] [CrossRef] [Green Version]
Bradski, G.; Kaehler, A. Learning OpenCV: Computer Vision with the OpenCV Library; O’Reilly Media, Inc.: Sebastopol, CA, USA, 2008; Available online: https://books.google.com.tw/books?hl=zh-TW&lr=&id=seAgiOfu2EIC&oi=fnd&pg=PR3&dq=opencv+library&ots=hUL28niGSa&sig=2f-MT4LGYlFPnyfkVd6xLIo27VM&redir_esc=y#v=snippet&q=bilinear&f=false (accessed on 25 September 2022).
Magnus, J.R.; Neudecker, H. Matrix Differential Calculus with Applications in Statistics and Econometrics; John Wiley & Sons: Hoboken, NJ, USA, 2019. [Google Scholar]
Yu, Y.C.; Jhang, J.W.; Wei, X.; Tseng, H.W.; Wen, Y.; Liu, Z.; Lin, T.L.; Chen, S.L.; Chiou, Y.S.; Lee, H.Y. Chroma Upsampling for YCbCr 420 Videos. In Proceedings of the 2017 IEEE International Conference on Consumer Electronics—Taiwan (ICCE-TW), Taipei, Taiwan, 12–14 June 2017; pp. 163–164. [Google Scholar] [CrossRef]
Yamagami, T.; Sasaki, T.; Suga, A. Image Signal Processing Apparatus Having a Color Filter with Offset Luminance Filter Elements. U.S. Patent 5,323,233, 21 June 1994. [Google Scholar]
Tachi, M. Image Processing Device, Image Processing Method, and Program Pertaining to Image Correction. U.S. Patent 8,314,863, 20 November 2012. [Google Scholar]
Compton, J.T.; Hamilton, J.F. Image Sensor with Improved Light Sensitivity. U.S. Patent 8 139 130 B2, 20 March 2012. [Google Scholar]
ho Lee, S.; Oh, P.; Kang, M.G. Three Dimensional Colorization Based Image/Video Reconstruction from White-dominant RGBW Pattern Images. Digit. Signal Process. 2019, 93, 87–101. [Google Scholar] [CrossRef]
Su, P.H.; Chen, P.C.; Chen, H.H. Compensation of Spectral Mismatch to Enhance WRGB Demosaicking. In Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada, 27 September–1 October 2015; pp. 68–72. [Google Scholar] [CrossRef]
Chung, K.L.; Chan, T.H.; Chen, S.N. Effective Three-Stage Demosaicking Method for RGBW CFA Images Using the Iterative Error-Compensation Based Approach. Sensors 2020, 20, 3908. [Google Scholar] [CrossRef]
Bakurov, I.; Buzzelli, M.; Schettini, R.; Castelli, M.; Vanneschi, L. Structural Similarity Index (SSIM) Revisited: A Data-Driven Approach. Expert Syst. Appl. 2022, 189, 116087. [Google Scholar] [CrossRef]
Venkataramanan, A.K.; Wu, C.; Bovik, A.C.; Katsavounidis, I.; Shahid, Z. A Hitchhiker’s Guide to Structural Similarity. IEEE Access 2021, 9, 28872–28896. [Google Scholar] [CrossRef]

Figure 1. Four 2 × 2 Bayer CFA patterns. (a)

P a t_{1}

= [

G_{1}

R_{2}

B_{3}

G_{4}

]. (b)

P a t_{2}

= [

G_{1}

B_{2}

R_{3}

G_{4}

]. (c)

P a t_{3}

= [

R_{1}

G_{2}

G_{3}

B_{4}

]. (d)

P a t_{4}

= [

B_{1}

G_{2}

G_{3}

R_{4}

Figure 1. Four 2 × 2 Bayer CFA patterns. (a)

P a t_{1}

= [

G_{1}

R_{2}

B_{3}

G_{4}

]. (b)

P a t_{2}

= [

G_{1}

B_{2}

R_{3}

G_{4}

]. (c)

P a t_{3}

= [

R_{1}

G_{2}

G_{3}

B_{4}

]. (d)

P a t_{4}

= [

B_{1}

G_{2}

G_{3}

R_{4}

Figure 2. The CF-based compression scheme for

I^{B a y e r}

. (a) The server side and the client side. (b) In terms of the decorrelated subimages, the graphical representation of the presented CF-based methods.

Figure 2. The CF-based compression scheme for

I^{B a y e r}

. (a) The server side and the client side. (b) In terms of the decorrelated subimages, the graphical representation of the presented CF-based methods.

Figure 3. The DF-based compression scheme for

I^{B a y e r}

. (a) The server side and the client side. (b) The graphical representation of chroma subsampling, particularly the Bayer CFA pattern-independent chroma subsampling approach and the Bayer CFA pattern-dependent chroma subsampling approach.

Figure 3. The DF-based compression scheme for

I^{B a y e r}

Figure 4. The relation between the 2 × 2 Bayer CFA block and the four RCT-based formats. (a) The 2 × 2 Bayer CFA block. (b) The four RCT-based formats.

Figure 5. The construction of the rectangular compact luma image. (a) The quincunx-located luma image. (b) The rectangular compact luma image.

Figure 6. The depiction of 4:2:0(A), 4:2:0(L), 4:2:0(R), and 4:2:0(DIRECT).

Figure 7. The relation between the subsampled chroma pair (

C b_{3}

C r_{2}

) and the 2 × 2 Bayer CFA pattern [27].

Figure 7. The relation between the subsampled chroma pair (

C b_{3}

C r_{2}

) and the 2 × 2 Bayer CFA pattern [27].

Figure 8. The notations used in the BI-based method for estimating

C b_{1}^{e s t}

at the server side.

Figure 8. The notations used in the BI-based method for estimating

C b_{1}^{e s t}

at the server side.

Figure 9. The visual effect comparison for the 13th ground-truth IMAX image, where for each method, the two demosaiced RGB full-color images on the left and on the right are under VTM-16.2 for QP = 44 and JPEG-2000 for CR = 35, respectively. (a) The 13th ground-truth IMAX image. (b) The amplified subimages. (c) CDM-OLM. (d) Modified 4:2:0(A)-OLM. (e) BIDM-OLM. (f)

Y D_{g} C_{o} C_{g}

. (g)

Y L M N

. (h)

Y Δ C_{b} C_{r}

Y D_{g} C_{o} C_{g}

. (g)

Y L M N

. (h)

Y Δ C_{b} C_{r}

Figure 10. The visual effect comparison for the 13th ground-truth IMAX image, where for each method, the two demosaiced RGB full-color images on the left and on the right are under VTM-16.2 for QP = 24 and JPEG-2000 for CR = 20, respectively. (a) CDM-OLM. (b) Modified 4:2:0(A)-OLM. (c) BIDM-OLM. (d)

Y D_{g} C_{o} C_{g}

. (e)

Y L M N

. (f)

Y Δ C_{b} C_{r}

Y D_{g} C_{o} C_{g}

. (e)

Y L M N

. (f)

Y Δ C_{b} C_{r}

Table 1. When setting QP = 0 and CR = 1 for VTM-16.2 and JPEG-2000, respectively, the quality performance comparison of the reconstructed Bayer CFA images.

Method	Platform	PSNR	SSIM	FSIM
CDM-OLM	VTM-16.2	55.5654	0.99968	0.99988
modified 4:2:0(A)-OLM	VTM-16.2	55.2914	0.99966	0.99987
BIDM-OLM	VTM-16.2	56.8909	0.99977	0.99991
$Y D_{g} C_{o} C_{g}$	VTM-16.2	58.0274	0.99972	0.99970
$Y L M N$	VTM-16.2	50.7565	0.99928	0.99964
$Y Δ C_{b} C_{r}$	VTM-16.2	53.3100	0.99941	0.99966
CDM-OLM	JPEG-2000	55.5956	0.99968	0.99988
modified 4:2:0(A)-OLM	JPEG-2000	55.4404	0.99967	0.99987
BIDM-OLM	JPEG-2000	56.9200	0.99976	0.99991
$Y D_{g} C_{o} C_{g}$	JPEG-2000	58.0296	0.99972	0.99990
$Y L M N$	JPEG-2000	50.7568	0.99928	0.99964
$Y Δ C_{b} C_{r}$	JPEG-2000	53.3106	0.99941	0.99966

Table 2. When setting QP = 0 and CR = 1 for VTM-16.2 and JPEG-2000, respectively, the quality performance comparison of the reconstructed RGB full-color images.

Method	Platform	CPSNR	CSSIM	CFSIM
CDM-OLM	VTM-16.2	53.5576	0.99866	0.99983
modified 4:2:0(A)-OLM	VTM-16.2	53.3690	0.99862	0.99983
BIDM-OLM	VTM-16.2	54.4657	0.99892	0.99987
$Y D_{g} C_{o} C_{g}$	VTM-16.2	55.2310	0.99888	0.99984
$Y L M N$	VTM-16.2	49.4787	0.99746	0.99959
$Y Δ C_{b} C_{r}$	VTM-16.2	51.6591	0.99786	0.99959
CDM-OLM	JPEG-2000	53.5863	0.99866	0.99983
modified 4:2:0(A)-OLM	JPEG-2000	53.5154	0.99865	0.99983
BIDM-OLM	JPEG-2000	54.4885	0.99892	0.99987
$Y D_{g} C_{o} C_{g}$	JPEG-2000	55.2337	0.99888	0.99984
$Y L M N$	JPEG-2000	49.4793	0.99746	0.99959
$Y Δ C_{b} C_{r}$	JPEG-2000	51.6602	0.99786	0.99959

Table 3. The execution time requirement (in seconds) comparison.

	KODAK	IMAX	SCI	Videos	CI	AVG
CDM-OLM	0.4371	0.0179	0.0307	0.0074	0.0209	0.1028
modified 4:2:0(A)-OLM	2.1914	0.0937	0.1396	0.0381	0.1016	0.5129
BIDM-OLM	0.8362	0.0420	0.0711	0.0151	0.0429	0.2015
$Y D_{g} C_{o} C_{g}$	0.0378	0.0017	0.0028	0.0006	0.0018	0.0089
$Y L M N$	0.0364	0.016	0.0028	0.0007	0.0016	0.0086
$Y Δ C_{b} C_{r}$	0.0547	0.0022	0.0038	0.0010	0.0028	0.0129

Table 4. The BD-PSNR gains of the considered methods for

I^{r e c, B a y e r}

Table 4. The BD-PSNR gains of the considered methods for

I^{r e c, B a y e r}

Method	Platform	QP [4, 20]	QP [12, 28]	QP [20, 36]	QP [28, 44]	QP [36, 51]
CDM-OLM	VTM-16.2	−3.4486	−0.2175	2.1060	3.4628	3.5398
modified 4:2:0(A)-OLM	VTM-16.2	−3.5834	−0.2550	2.1785	3.5687	3.6122
BIDM-OLM	VTM-16.2	−3.0984	0.1299	2.4742	3.6255	3.6063
$Y D_{g} C_{o} C_{g}$	VTM-16.2	0.8212	1.4656	2.1304	2.8782	2.7864
$Y L M N$	VTM-16.2	−1.9614	0.3535	1.5443	2.6012	2.6668
$Y Δ C_{b} C_{r}$	VTM-16.2	−0.3723	1.2455	2.1641	3.0196	2.8695
		CR [5, 20]	CR [15, 30]	CR [20, 35]	CR [25, 40]	CR [30, 45]
CDM-OLM	JPEG-2000	1.5667	3.3552	3.7033	3.9106	4.0223
modified 4:2:0(A)-OLM	JPEG-2000	1.5707	3.3740	3.7248	3.9297	4.0396
BIDM-OLM	JPEG-2000	1.6314	3.3979	3.7335	3.9336	4.0401
$Y D_{g} C_{o} C_{g}$	JPEG-2000	1.3451	2.7924	3.1319	3.3665	3.5208
$Y L M N$	JPEG-2000	1.1450	2.7286	3.1177	3.3823	3.5621
$Y Δ C_{b} C_{r}$	JPEG-2000	1.3662	2.9438	3.3218	3.5802	3.7508

Table 5. The BD-CPSNR gains of the considered methods for

I^{r e c, R G B}

Table 5. The BD-CPSNR gains of the considered methods for

I^{r e c, R G B}

Method	Platform	QP [4, 20]	QP [12, 28]	QP [20, 36]	QP [28, 44]	QP [36, 51]
CDM-OLM	VTM-16.2	−2.7093	0.0106	2.2170	3.6186	3.8182
modified 4:2:0(A)-OLM	VTM-16.2	−2.8379	−0.1073	2.1726	3.6543	3.8561
BIDM-OLM	VTM-16.2	−2.4553	0.2697	2.4895	3.7681	3.8773
$Y D_{g} C_{o} C_{g}$	VTM-16.2	0.4310	1.0292	1.8324	2.8669	3.0045
$Y L M N$	VTM-16.2	−1.8610	0.0021	1.2083	2.5099	2.8323
$Y Δ C_{b} C_{r}$	VTM-16.2	−0.4328	0.8685	1.8585	2.9811	3.0826
		CR [5, 20]	CR [15, 30]	CR [20, 35]	CR [25, 40]	CR [30, 45]
CDM-OLM	JPEG-2000	2.0816	3.8981	4.2366	4.4344	4.5438
modified 4:2:0(A)-OLM	JPEG-2000	2.0211	3.8834	4.2443	4.4536	4.5677
BIDM-OLM	JPEG-2000	2.1005	3.9354	4.2773	4.4794	4.5889
$Y D_{g} C_{o} C_{g}$	JPEG-2000	1.8285	3.4338	3.7969	4.0440	4.2071
$Y L M N$	JPEG-2000	1.5357	3.2973	3.7119	3.9886	4.1804
$Y Δ C_{b} C_{r}$	JPEG-2000	1.8189	3.5772	3.9777	4.2471	4.4271

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

Share and Cite

MDPI and ACS Style

Chung, K.-L.; Chen, H.-Y.; Hsieh, T.-L.; Chen, Y.-B. Compression for Bayer CFA Images: Review and Performance Comparison. Sensors 2022, 22, 8362. https://doi.org/10.3390/s22218362

AMA Style

Chung K-L, Chen H-Y, Hsieh T-L, Chen Y-B. Compression for Bayer CFA Images: Review and Performance Comparison. Sensors. 2022; 22(21):8362. https://doi.org/10.3390/s22218362

Chicago/Turabian Style

Chung, Kuo-Liang, Hsuan-Ying Chen, Tsung-Lun Hsieh, and Yen-Bo Chen. 2022. "Compression for Bayer CFA Images: Review and Performance Comparison" Sensors 22, no. 21: 8362. https://doi.org/10.3390/s22218362

APA Style

Chung, K.-L., Chen, H.-Y., Hsieh, T.-L., & Chen, Y.-B. (2022). Compression for Bayer CFA Images: Review and Performance Comparison. Sensors, 22(21), 8362. https://doi.org/10.3390/s22218362

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Menu

Compression for Bayer CFA Images: Review and Performance Comparison

Abstract

1. Introduction

1.1. The Related CF-Based Compression Methods

1.2. The Related DF-Based Compression Methods

1.3. Motivation and Contribution

2. The Reversible Color Transform-Based (RCT-Based) Compression Works for Bayer CFA Images

2.1. The Y 1 C r 2 C b 3 Y 4 Method

2.2. The Y D g C o C g Method

2.3. The Y L M N Method

2.4. The Y Δ C b C R Method

3. The Demosaicing-First-Based (DF-Based) Compression Works for Bayer CFA Images

3.1. Demosaicing I B a y e r to I D e m o , R g b and Then Converting I D e m o , R g b to I Y c b c r

3.1.1. Demosaicing I B a y e r to I d e m o , R G B

3.1.2. Converting I d e m o , R G B to I Y C b C r

3.2. Chroma Subsampling

3.2.1. The Bayer CFA Pattern-Independent Chroma Subsampling Methods

3.2.2. The Bayer CFA Pattern-Dependent Chroma Subsampling Methods

The Direct Mapping (DM) Method [27]

The COPY-Based Distortion Minimization (CDM) Method and the Two Variants

The Bilinear Interpolation-Based Distortion Minimization (BIDM) Method

3.3. Luma Modification

4. Experimental Results

4.1. Quality Comparison and Discussion

4.1.1. Quality Comparison and Discussion

4.1.2. Execution Time Requirement Comparison and Discussion

4.2. Quality–Bitrate Tradeoff Comparison and Discussion

4.2.1. The Quality–Bitrate Tradeoff Comparison

4.2.2. The Visual Effect Comparison

5. Conclusions and Future Works

Author Contributions

Funding

Data Availability Statement

Acknowledgments

Conflicts of Interest

References

Share and Cite

Article Metrics

Article Access Statistics

Further Information

Guidelines

MDPI Initiatives

Follow MDPI

2.1. The $Y_{1} C r_{2} C b_{3} Y_{4}$ Method

2.2. The $Y D_{g} C_{o} C_{g}$ Method

2.3. The $Y L M N$ Method

2.4. The $Y Δ C_{b} C_{R}$ Method

3.1. Demosaicing $I^{B a y e r}$ to $I^{D e m o, R g b}$ and Then Converting $I^{D e m o, R g b}$ to $I^{Y c b c r}$

3.1.1. Demosaicing $I^{B a y e r}$ to $I^{d e m o, R G B}$

3.1.2. Converting $I^{d e m o, R G B}$ to $I^{Y C b C r}$