Dual-Tree Complex Wavelet Transform and Image Block Residual-Based Multi-Focus Image Fusion in Visual Sensor Networks

3.2.1. Image Block Residual

In multi-focus images, the focused region is more informative [28]. It is easy to find that blocks in the focus area have greater similarity to the blocks of the initial fused image. Therefore, the image block residual is proposed to test the similarity, which is defined as:

The block residual can be assumed as a similarity measure in image processing applications. Constructing an initial decision map by:

3.2.2. Consistency Verification

In a scene, we assume most blocks of a region are in the depth of focus of one source image. According to the theory of imaging, all the blocks in such region must be chosen from the image [5]. However, there can be some erroneous selection of some blocks due to noise or undesired effects. To remove these defects, consistency verification [37] is employed. If the center block comes from one image while the majority of the surrounding blocks come from the other one, the center sample is simply switched to the corresponding block in the other image. Here, consistency verification is applied in a 3 × 3 neighborhood window as in [7]. After consistency verification to M₁(i₀, j₀), we get the modified decision map CV(i₀, j₀) for boundary processing.

3.2.3. Boundary Processing

Assume one block contains both clear areas and blurred areas, so it is meaningless to judge the block is in the focus area or not. According to the theory of imaging, these blocks are always on the boundary of focused region. Therefore, we could choose these blocks by the equations below:

If M₂(i₀, j₀) = 1, the block at (i₀, j₀) from image A is from the focus area, and if M₂(i₀, j₀) = −1, the block at (i₀, j₀) from image B is from the focus area. Equation (11) means if one of M₂(i₀, j₀), M₂(i₀ + 1, j₀), M₂(i₀, j₀ + 1) and M₂(i₀ + 1, j₀ + 1) equals to 1 and another one equals to −1, the block at (i₀, j₀) is on the boundary of focused region.

As we know, the initial fused image has extracted the most characteristics of the source images to some extent. On the boundary of the focused region, the blocks may contain both clear pixels and blurred pixels. To reduce the loss of useful information, we should choose the blocks of the initial fused image on the boundary of the focused region. Then, we can use the Equation (7) to get the fusion result.

4.1.1. Objective Evaluation of Results of Referenced Images

(1)

Root Mean Square Error (RMSE): The RMSE is the cumulative squared error between the fused image and the referenced image and it is defined by:

$$\text{RMSE}=\sqrt{\frac{\text{∑}_{m=-M}^M\text{∑}_{n=-N}^N{\left[F\left(i+m,j+n\right)-R\left(i+m,j+n\right)\right]}^2}{MN}}$$

(13)

where M × N is the size of fused image, F(i, j) and R(i, j) are the pixel value of the fused image and referenced image at the position (i, j), respectively. The lower the value of RMSE, the better the fusion effect.

(2)

Peak Signal to Noise Ratio (PSNR): The PSNR is a ratio between the maximum possible power of a signal and the power of noise that affects the fidelity [41]. The PSNR is formulated as:

$$\text{PSNR}=10lg\left|\frac{L^2}{{\mathit{\text{RMSE}}}^2}\right|$$

(14)

where L is the dynamic range of the pixel values. Here, L = 255 is adopted. The greater the value of PSNR, the better the fusion effect.

(3)

Structural Similarity-based Metric (SSIM): The SSIM is designed by modeling any image distortion as the combination of the loss of correlation and radiometric and contrast distortion [42]. SSIM between the fused image and the referenced image is defined as follows:

$$\mathit{\text{SSIM}}\left(R,F\right)=\frac{\left[2u_R{u}_F-{\left(K_{1}L\right)}^2\right]\left[2{\sigma }_{RF}+{\left(K_{2}L\right)}^2\right]}{\left[{u_R}^2+{u_F}^2+{\left(K_{1}L\right)}^2\right]\left[{{\sigma }_R}^2+{{\sigma }_F}^2+{\left(K_{2}L\right)}^2\right]}$$

(15)

in which u_R, u_F are mean intensity of the standard image and the fused image, respectively. σ_R, σ_F, σ_RF are the variances and covariance, respectively [43]. K₁ and K₂ are small constants, and L is the dynamic range of the pixel values. Here, the default parameters: K₁ = 0.01, K₂ = 0.03, L = 255 is adopted [44]. The greater the value of SSIM, the better the fusion effect.

4.1.2. Objective Evaluation of Results of Non-Referenced Images

(1)

Standard deviation: The standard deviation can be used to estimate how widely spread the gray values in an image and defined as [45]:

$$\text{Std}=\sqrt{\left(\frac{1}{M\times N}\text{∑}_{i=1}^M\text{∑}_{j=1}^N{\left(F(i,j)-\widehat{\mu }\right)}^2\right)},$$

(16)

where μ̂ is the mean value of the fused image. The larger the standard deviation, the better the result.

(2)

Mutual Information (MI): The MI can indicate how much information the fused image conveys about the source images [33]. MI between fusion image and the source images is defined as follows [46]:

$$\text{MI}={\text{MI}}^{AF}+{\text{MI}}^{BF},$$

(17)

in which:

$${\text{MI}}^{AF}=\text{∑}_{f=0}^L\text{∑}_{a=0}^L{p}^{AF}(a,f){log}_2\left(\frac{p^{AF}(a,f)}{p^A(a)p^F(f)}\right),$$

(18)

$${\text{MI}}^{BF}=\text{∑}_{f=0}^L\text{∑}_{b=0}^L{p}^{BF}(b,f){log}_2\left(\frac{p^{BF}(b,f)}{p^B(b)p^F(f)}\right),$$

(19)

where MI^AF and MI^BF denote the normalized MI between the fused image and the source image A and B; a, b and f ε [0, L]. P^A(a), p^B(B) and p^F(f) are the normalized gray level histograms of source images and fused image. p^AF(a, f) and p^BF(b, f) are the joint gray level histograms between the fused image and the source image A and B. The greater the value of MI, the better the fusion effect.

(3)

Edge Based Similarity Measure (Q^AB/F): Q^AB/F which proposed by Xydeas and Petrovic [47] gives the similarity between the edges transferred from the source images to the fused image. The definition is given as:

$${\text{Q}}^{AB/F}=\frac{{\text{∑}}_{i=1}^M{\text{∑}}_{j=1}^N[{\text{Q}}^{AF}(i,j)w^A(i,j)+{\text{Q}}^{BF}(i,j)w^B(i,j)]}{{\text{∑}}_{i=1}^M{\text{∑}}_{j=1}^N[w^A(i,j)+w^B(i,j)]},$$

(20)

in which:

$${\text{Q}}^{AF}(i,j)={\text{Q}}_a^{AF}(i,j)Q_g^{AF}(i,j),$$

(21)

$${\text{Q}}^{BF}(i,j)={\text{Q}}_a^{BF}(i,j){\text{Q}}_g^{BF}(i,j),$$

(22)

where w^A(i, j) and w^B(i, j) are the corresponding gradient strengths for images A and B, respectively. $Q_a^{xF}(i,j)$ and $Q_g^{xF}(i,j)$ are the Sobel edge strength and orientation preservation values at location (i, j) for each source image [48]. The larger the value, the better the fusion effect result.

4.2.1. Fusion of Referenced Multi-Focus Images

The first experiment is conducted using an extensive set of artificially generated images. Figure 3 shows the referenced Lena image, and two blurred artificial images that are generated by convolving the referenced image with an averaging filter centered on the left part and the middle part, respectively [51]. The fusion results of different images are shown in Figure 4a–g. For a clearer comparison, this paper has given the residual images [28] between the results and Figure 3b, and performed the following transform on each pixel of each residual image:

For the focused areas, the residual values between the fused images and the source image should be zero. According to the literature [9], in residual images, the lower residue the features, the more detail information in the focus region has been transferred to fused images and the better the corresponding method. From Figure 4, as mentioned above, the residual image between the result of the proposed method and Figure 3b is almost zero in the clear region. That is to say, the proposed method has extracted the most information from the clear region of Figure 3b.

RMSE and SSIM are the metrics that measure the error or similarity between a referenced image and a fused image. Using these metrics the effect of the methods in the fusion of referenced multi-focus images can be measured well. In addition, MI and Q^AB/F are the state-of-the-art fusion performance metrics [25,40], which can be used to measure the fusion of no-reference multi-focus images. Figure 5 depicts the comparison on quantitative metrics of different methods; it is easy to find that the SSIM value of the proposed method is the largest (0.9912) and RMSE value is the smallest (2.9507). Both of them are the optimal values. That is to say, compared with other methods, the fused image of the proposed method is the closest to the referenced image. In addition, the MI and Q^AB/F values of the proposed method results also outperform the other methods.

For further comparison, we experimented with more standard images. Figure 6 shows the standard test image database. The database is the standard grayscale test images that have been downloaded from [52]. The sizes of the test images are all 512 × 512. Each standard test image has been convolved with an averaging filter centered on the left part and right part, respectively. Figure 7 shows the typical examples of the fusion of artificial multi-focus images, the results of different methods and the corresponding residual images. From Figure 7, the residual images show that the proposed method results have extracted the most information from the focus region. Figure 8 shows the comparison on RMSE and SSIM of different methods for artificial multi-focus images. From Figure 8, it is seen that the results of the proposed method almost represent the referenced images with SSIM values approaching 1. From Figure 8, it is easy to find that in 30 sets of images, the SSIM and RMSE values of the proposed method results outperform other methods.

4.2.2. Fusion of Images in a Noisy Environment

To evaluate the noise-robustness of the proposed framework, the second experiment is conducted on two blurred “Lena” images as shown in Figure 3b–c additionally corrupted with different Gaussian noise, with a standard deviation of 1%, 3%, 5%, 7% and 9%, respectively. Figure 9 shows the blurred images with 5% noise and the corresponding results of different methods. A quantitative evaluation comparison of the different methods for artificial images with different noise is shown in Figure 10. From Figure 10a, the PSNR values of the proposed method results are all the highest of the six methods. That is to say, in different noise environments, the results of the proposed method are closer to the referenced image than other methods. From Figure 10b,c, the MI and Q^AB/F values of the proposed method also outperform other methods. From the above analysis, we can observe that the proposed method provides the best performance and outperforms the other algorithms, even in noisy environments.

4.2.3. Fusion of Non-Referenced Multi-Focus Images

The third experiment is conducted on four sets of non-referenced multi-focus test images. These images are shown in Figure 11. The size of Figure 11a,b is 256 × 256, while the others are 480 × 640. The fusion results are shown in Figure 12a1–f1, a2–f2 and their sharpened residual images between the fused result and image focused on background are shown in Figure 12g1–l1,g2–l2. From the residual images, it is easy to find in the focused background areas that the proposed method has extracted more information than other methods and almost all of the useful information in the focus area of the source images has been transferred to the fused image.

These experiments have no referenced images. Therefore, we used standard deviation, MI and Q^AB/F to evaluate how much clear or detailed information of the source images is transferred to the fused images. The comparison of the quantitative evaluation of different methods for non-referenced multi-focus images is shown in Figure 13. Comparing the average values of the standard deviation, MI and Q^AB/F of DTCWT 1 method and the proposed method, we can find that their corresponding values have been increased 1.37%, 29.50% and 8.26%, respectively. From Figure 13, the values of standard deviation, MI and Q^AB/F of the proposed method result are all better than those of the other methods. From the above analysis, we can also observe that the proposed scheme provides the best performance and outperforms the other algorithms.

Finally, we compare the time-consumption of the proposed method with the DTCWT- and NSCT- based methods. These methods can provide both good shift invariance and directional selectivity. A comparison of the execution times of the DTCWT-, NSCT-based methods and the proposed method for the Pepsi image sets is presented in Table 1.

The execution times are calculated by running all the codes in MATLAB R2012a on the same computer. From Table 1, it is easy to find that the NSCT-based methods (including NSCT-1 and NSCT-2 methods) are time-consuming and of high complexity, and the efficiency of the proposed method and the DTCWT-based method is higher than that of the NSCT-based methods. Although the proposed method needs a little more execution time than the DTCWT method, the fusion performance including the above qualitative and quantitative evaluations of our method have been proved better than the DTCWT method. Hence, from the overall assessments, we can conclude that the proposed method is effective and suitable for real time applications.