WO2014078985A1

WO2014078985A1 - Method and apparatus for image regularization

Info

Publication number: WO2014078985A1
Application number: PCT/CN2012/084886
Authority: WO
Inventors: Wenfei JIANG; Hengbin CUI; Zhibo Chen
Original assignee: Thomson Licensing
Priority date: 2012-11-20
Filing date: 2012-11-20
Publication date: 2014-05-30

Abstract

Methods and apparatuses for image regularization are described. The image regularization is performed through total variation (TV) lθ regularization, wherein 0<θ<1. In one embodiment, an image is divided into image blocks. For each image block, directions are determined for gradient block calculation by identifying the directions that match the orientation of the content of the image block. TV lθ regularization is then performed for each image block using the calculated gradient blocks. The processed is performed iteratively to alleviate the problem of non-regularized neighboring blocks affecting the regularization of the current block. The TV lθ regularization is applicable to applications such as image denoising, video compression, and exposure fusion.

Description

METHOD AND APPARATUS FOR IMAGE REGULARIZATION

TECHNICAL FIELD

The present invention generally relates to image regularization. More particularly, it relates to total variation (TV)image regularization.

BACKGROUND OF THE INVENTION

Total variation (TV) has been applied in various areas such as image de-noising, compression and super resolution. It is a widely-used measure of intensity continuity. TV regularization is based on an inherent property of image data wherein the difference between adjacent pixels is often small. Thus, reducing the TV of an image subject to a close match to the original signal removes noise while preserving real details.

Denote an M*M distorted image by a 2-dimensional matrix f and its horizontal and vertical gradient images byV_xf and V_yf, respectively, where

V_xf(*, y) = f(* + l, y) - f(*, y)

V_yfO, y) = fO, y + l) - fO, y)-

The TV of the image is defined as the sum of the II or 12 norm of the gradient images for all the pixels, formulated as:

TV ) =∑_w(|V_xf(U)| + |V_yf(U)|) (1)

TV_l2 (f ) =∑_{tj x}f(i,jY + V_yf(/ )² (2) where i and j are the coordinates of the pixels in the image.

Studies on TV minimization based image regularization mainly focus on solving a constrained TV minimization problem:

f = arg min_f 7T(f) , s. t. ||u - <p(f) ll < ε² (3) where u and f are the input image and output image; andcp(f)is a measuring function that maps the output image to a certain domain to compare with the input image. For example, when representing the input and output images in 1 -D signals u and / respectively, (p(f)can take the form of a random matrix or a specially designed matrix<¾>, e.g. a full or a partial DCT matrix, multiplying the output image/ i.e. Φ/, and the constraint becomes \\u— Φ/|| < ε². The term \\u— Φ/||ίη the constraint measures the difference of certain projections of the signals rather than the signal itself, ε is a parameter which controls the regularization intensity.

In many applications, there are two desired features of image regularization. First, the real edges are smoothed as little as possible; second, the high-gradient edges are preferably preserved over the low-gradient detail or noise. Unfortunately, traditional TV 11112 regularization as defined in Eqn. (1) or Eqn. (2) does not meet the requirement well. First, it can be inferred from its definition that the textures and edges are inevitably blurred. In particular, the oblique edges are likely to be significantly smoothed since both vertical and horizontal gradient regularizations tend to reduce the gradient across oblique edges. Second, strong edges are not particularly protected during TV 11112 regularization; and TV /2regularization even gives higher priority to keeping trivial details rather than the strong edges. This can be seen from following.

Consider a pixel f(i, j) in the image f whose horizontal and vertical gradient is(V_xf V_yf Computing the partial derivative of TV II and TV 12 with respect to the gradient of f (i, ), namely, | V_fc f |, yields

From Eqn. (5) it can be seen that in TV 12 regularization, the direction with larger gradient has more contribution to the TV 12 reduction for each pixel. In other words, the high gradient details are preferred to be smoothed rather than the low gradient edges in TV 12 regularization. Regarding TV II , Eqn. (4) shows that the gradient of all pixels along all directions has equivalent effect on the TV II reduction.

Therefore, there is a need to design a TV regularization that overcomes these problems. Prior solutions have not adequately been established in the art.

SUMMARY OF THE INVENTION

This invention is directed to methods and apparatuses for image regularization.

According to an aspect of the present invention, there is provided a method and an apparatus for regularizing an image. The method comprises calculating a plurality of gradient images for said image; and regularizing said image by modifying pixel values of said image to minimize a function of said image defined as a weighted sum of absolute values of said calculated gradient images, each raised by a power that is less than 1 and greater than 0.

According to another aspect of the present invention, there is provided a method and an apparatus for regularizing an image. The method comprises dividing the image into a plurality of image blocks; for each of said plurality of image blocks, calculating a plurality of gradient blocks; and regularizing said image by modifying pixel values of said each image block to minimize a function of said image block defined as a weighted sum of absolute values of said calculated gradient blocks, each raised by a power that is less than 1 and greater than 0.

BRIEF DESCRIPTION OF THE DRAWINGS

The above features of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

Figure 1 illustrates an image regularizer according to an embodiment of the present invention.

Figure 2 illustrates an exemplary process of a block or patch based image regularization according to another embodiment of the present invention.

Figure 3 shows a set of example candidate directions and the corresponding directional gradients for gradient image/block calculation.

Figure 4 shows an example image to illustrate the advantages of the TV W image regularization according to one embodiment of the present invention: (a) regularization results of TV II and TV 12; (b) regularization results of TV/½.

Figure 5 depicts a block schematic diagram of a system in accordance with the present principles for accomplishing TV W image regularization.

DETAILED DESCRD7TION

In the present invention, a solution to image regularization is provided. By designing new functions such as total variations, effective and efficient image regularization is achieved. An image described in the present application can be a still image, a frame of a video sequence or a block/patch of an image. According to the principles of the present invention, a plurality of gradient images for an image is calculated. The regularization of the image can be performed by modifying pixel values of the image to minimize a function of the image. The function can be any function that measures changes of the image. For example, the function of the image can be defined as a weighted sum of the absolute values of calculated gradient image coefficients. Each of the absolute values of the coefficients is raised by a power that is less than 1 and greater than 0. Different weights can be assigned to different gradient images, or even different gradient coefficients. The image can be represented in forms such as a ID signal or a 2D matrix. The power for each of the coefficients can be the same or different.

In one embodiment, the function of the image is a Total Variation TV W . For an image f, K gradient images V_fcf, k = 1 , ... , K, are calculated and the total variation TV W can be defined as:

TV_w (f) =∑_{i >k} w_k \ /_kf(i,j) \^e (6) where V_fcfis the k-th calculated gradient image, V_fcf(i,y) is a value of V_fcf at position w_fcis a weight for the k-th calculated gradient image, and O<0<l .The K gradient images represent different orientations that are considered in the calculation of TV W . In one embodiment, { V_fcf }={ V_xf , V_yf } is considered wherein only edges along horizontal and vertical directions are considered. In a different embodiment, { V_fcf } can also include diagonal and anti-diagonal directions. {w_k} assigns different weights to different directions. In one embodiment, all {w_k} are set to 1 to put equal weights on all directions.

The image f can be regularized by modifying pixel values of said image to minimize the total variation of the image using the calculated gradient images as formulated in Eqn. (6). In one embodiment, the minimization problem can be formulated by a constrained TV minimization problem

f = arg min_f TV_W (f ) , s. t. γ (u, f ) < ε² (7) which can be transformed into an unconstrained optimization problem:

f = argmin_f 7V_ifl (f) + ¾ y(u, f) (8) where u and f are the images before and after the modifying step, respectively; y(u, f)is a measurement of closeness between u and f ; and 1 is a parameter which controls the regularization intensity.In one embodiment, y(u, f) = ||u— f || =∑i (u(i,_/)— f(i,/))²when the images before and after the modifying step are represented in a 2D form u and f and y(u, f) = \\u— f\\ =∑_q(u(q)— /(q))²when the images are represented in a ID formu and .In both cases, u and fare compared pixel by pixel. In the following, unless otherwise indicated or required by context, a bold italic letter, such as / , is the one dimensional representation (vector) of the same image represented by the same letter in bold font, such as f, in two dimensional matrix.

Similarly, the total variation TV Woi Eqn. (6) can also be defined on the ID form of the image /. In one embodiment, such a definition can be TV_W (/) =∑^_fc w_k \V_kf(q \^e , where V_fc/is the k-th calculated gradient image represented in ID form, V_fc/(qⁱ) is a value of V_fc/ at position q , w_fcis a weight for the k-th calculated gradient image, and O<0<1. For the same image represented as f and /, TV_W (/) = TV_W (f).

In a different embodiment, rather than directly comparing u and f , y(u, f) may compare^ (u) and,g (f), where $(·) is a function or a transform applied to its argument, or compare g(u— f) with a given value, or may take the form of ||u— φ(ί) || as shown in Eqn. (3).

Fig. 1 illustrates an image regularizer 100, which comprises a gradient image calculator 1 10 and a minimizer 120. The gradient image calculator calculates a plurality of gradient images along different directions. The calculated gradient images and the input image are input to the minimizer 120 to adjust/modify the image pixel value so that the minimization of a function of the image, such as the TV as formulated in Eqn. (7), is achieved. The output is the processed regulated image.

In a different embodiment, the image can be regularized patch by patch, or block by block. Fig. 2 illustrates an exemplary process of such a block or patch based image regularization. In step 220, the input image is divided into image blocks/patches. For each of the image blocks (step 230), gradient images of the image block along several directions, called gradient blocks, are calculated in step 240. The image pixel values of each image block are then adjusted/modified in step 250 by minimizing a function of said image block defined as a weighted sum of absolute values of said calculated gradient block coefficients. Each of the absolute value of the coefficients is raised by a power that is less than 1 and greater than 0. In one embodiment, the function is the TV of the image block which is defined based on the calculated gradient blocks in the corresponding image block as: TV_w (f_t) =∑ _k w_t,k \ /_kf_t(i,j) \^e (9) where V_fcf_tis the k-th calculated gradient block for t-th image block,w_{t fc}is a weight for the k- th calculated gradient block, V_fcf_t(i, ) is a value of V_fcf_t at position (i,y^')and O<0<1.

In one implementation, the image regularization of one image block may be affected by the pixel values of its neighboring blocks. For example, the calculation of the gradient at the border of an image block may require the pixel values of its neighbor blocks. In such a scenario, the steps 230 to 250 may be repeated iteratively until a termination condition is met at step 260. The terminating condition can be, for example, the improvement of current iteration over the last iteration is smaller than a given threshold. In a different embodiment, an optional step of pre-processing the input image 210 can be introduced to place the image in a better condition for regularization, for example, by roughly removing noise. The preprocessing step can be performed by the pre-processor 130 shown in Fig. 1.

The K gradient image/block can be determined by selecting K gradient directions for an image or an image block. For example, several initial directions can be pre-selected, among which gradient direction(s) that most likely match the orientation of the content of the image or the image block are selected as the set of directions for calculating gradient images/blocks.

In the following, a detailed implementation of the present invention is disclosed. For simplicity of presentation, Θ = 0.5 and y(u, f) = \\u— Φ || ^Γβ used as an example. The same principles apply to other Θ values and other forms of y(u, f). Then TV regularization problem becomes:

f = argmin_f TV_ll/2 (f) + ~₂ \\u - Φ[\\² ₂ (10) where:

TV_ll/2 (f) =∑ _k w_k \V_kf(i,j) \^ (11) and / is the ID representation of the same image f.

Determine the directions for gradient image/block calculation

One embodiment of the present invention is to perform TV /½ regularization along the real edge directions. Fig. 3 shows a set of example candidate directions and the corresponding directional gradients. The directional gradients are computed by, (V_afO,y) = f(*,y) -f(*-l,y)

V_bfO,y) = f(x,y) — f(x— 2,y - 1)

v_cf(*,y) = f(*,y) -f(x- l,y- 1)

V_df(*,y) = f(*,y) -f(x-l,y- 2)

(12)

' V_ef(*,y) = f(*,y) -f(*,y-l)

V_ff(*,y) = f(*,y) - f 0 + 1, y - 2)

V_gf(*,y) = f(x,y) -f(x + l,y- 1)

hf(x,y) = f(*,y) - i(x + 2,y - 1)

In one implementation, two most significant directions are picked for the regularization purpose. In different implementations, more directions can be taken into account. The sum of the /½ norm of the gradient for each direction can be calculated for an image or an image block/patc

where kE{a, b, c, d, e, f, g, h}. The directions that match the orientations of the content usually have a small value of E_k, so the determined orientations are, for K=2,

(α,β = argmin2_fe£_fe, (14) where arg min2 _k E_k returns the indices of the two smallest E_k .

Given the orientations of the content in each patch/block, the value of the TV /½ for the patch/block f_fcan be calculated by,

TV_1¾ (f _t) =∑_tj(∑_k w_t,_feVlv_fcf_t(U)l). (15)

For example, if the directions a and c are detected as edges, TV /½ is computed by,

TV_l¾(f_t) + w_t,_cVl _cf_ta;)l. (16)

The weights w_t,k are adaptively calculated by normalizing the harmonic mean of Ek,

Eqns. (16) and (17) indicate that it is preferred to smooth the image along the small-norm- directions with higher intensity.

It should be noted that the TV /½ calculation may not be patch-independent. To calculate the gradients of the pixels on the boundary of a patch, the information of its adjacent patches may be utilized. For pixels near the boundary of the image, the image is extended in advance, e.g. by repeating the boundary pixels or by symmetrically copying the boundary pixels, in order to calculate the boundary gradients. The global TV /½ of the can be calculated as the sum of the TV /½ of all patches as formulated in Eqn. (18), TV_m (f) =∑_t TV_m (f_t). (18)

TV1½ regularization

In a bloclc/patch based TV /½ regularization, Eqn. (1 1) is decomposed into a patch-wise optimization,

f_t = arg min_ft{7V_i¾(f_t) + ½ \\u_t - <¾>/_t || ¾ (19) where X_t is the regularization intensity parameter for the regularization of the input patch f_t.

In the following, a simplified version of Eqn. (19) as shown in Eqn. (20) is first discussed,

f_t = arg min_ft{7V_i½(f_t) + ½ \\u_t - /_t |||}, (20) where the measuring matrix <¾>is set as an identity matrix. The solution for the cases with arbitrary reversible measuring matrix <¾>is briefly discussed later.

Half-Quadratic Splitting Algorithm for Eqn.(20)

Given the two most significant directions a and β of each patch, obtained through Eqn. (14)for example, Eqn. (20) can be reformulated as, f = arg min_f w_a \\ V (21)

I 1

where || V_fcf ||i =∑i_j |V_fcf(i,7) l² - = _> β , and the patch index t is omitted for simplicity

2

hereinafter.

Eqn. (21) can be solved by the Half-Quadratic Splitting method. As in Krishnan, D. and Fergus, R. Fast Image Deconvolution using Hyper-Laplacian Priors, Proc. Neural Information Processing Systems 2009, I _ki is substituted by an auxiliary variables d_¾ which leads to: min, wJldJ! + wp || dp ||! + - ||u - /||!,

2 2

s. t d_a = V_af, dp = V_pf. (22) To weakly enforce the constraints in (22), a penalty term is added as shown in (23).

where μ is a tuning parameter to control the penalization. As μ→∞, the solution of (23) converges to that of (21).

The minimization problem of (22)can be solved by an alternating iterative minimization approach, wherein the minimization problem is decomposed into two subproblems, namely, the f-subproblem and the d subproblem. f-subproblem

The f-subproblem seeks for the optimal f given a fixed d_¾ which is done by minimizing the following cost function:

Γ = argmin_f ||u - f || + f∑_fc=a,p|| d_fc - V_fcf \\_F ². (24) By setting the first derivative of Eqn. (24) to zero,

(II + μ∑_k V/ _fc)f = Au + μ∑_k V^T _kd_k (25) is obtained, where I is an identity matrix.

This linear system can be solved by method such as Conjugate Gradient Square Method described in P. Sonneveld. CGS: a fast Lanczos-type solver for nonsymmetric linear systems. SIAM Journal on Scientific and Statistical Computing, 10:36-52, 1989. d/rsubproblem

The d subproblem seeks for the optimal d_fc given a fixed f which is done by minimizing the following cost function: d^ = argmln_dfc w_fc ||d_fc ||| + f ||d_fc - V_fcf|| 2 (26)

2

In cases when there are multiple k values, such as k = a, β as in Eqn. (21), the same process applies for each k value by assuming the d_fcsfor the remaining k values are fixed. The optimization problem of (26) can be decomposed into 2Mndependent one-dimension /½ regularization problems (Ms the number of pixels in a patch),

d_k ^* = argmin_dfc w_k \d_k |i + (d_k - V_fe )² (27) where d_fcand V_fc are the gradient of one pixel before and after the regularization respectively. According to Z. B. Xu. Data modeling: Visual psychology approach and /% regularization theory. Proceedings of the International Congress of Mathematicians, Hyderabad, India, 20 0,there is a closed-form thresholding formula for the/½ regularization proble

where

The half-quadratic splitting algorithm is implemented by alternatively solving the f- subproblem and the d subproblem until the following stopping criterion is met,

\\iⁿ⁺¹ - iⁿ \\_F ² /\\iⁿ \\_F ² < e

where eis a predefined threshold.fⁿandfⁿ⁺¹are the regulated image lock at n-th iteration and (n+l)-th iteration, respectively.

Solving algorithm for the regularization with a reversible measuring matrixO

For the cases in which the image is measured by a reversible matrix Φ, the problem (19) can be converted into

Likewise, it can be decomposed into the f-subproblem as formulated in Eqn. (30) and the d subproblem as formulated in Eqn. (31),

Γ = argmin_f \\u - Φ/Ui +

(30) ¾ = argmin_dk w_fc ||d_fc ||f + f ||d_fc - V_fcf|| 2 (31) where

(λΦ^τΦ + μ∑_k V/ _fc)/ = 1Φ½ + μ∑_k V^T _kd (32) The algorithms to solve Eqn. (31) and (32) are similar to that of (26) and (25).

Parameter Updating Strategy Considering that the content of an image may vary significantly from part to part, a spatially varying regularization intensity parameter that locally controls the regularization intensity over image regions according to the content can be added. Intuitively, the regularization intensity parameter should positively correlated with the TV value of the intermediate patch, i.e. the patch obtained in between iterations of the above algorithm, and negatively correlated with variance of the noise in the image. = _¾ ^¾ (33) where c^s a constant, σ² is variance of the noise, which can be estimated, for example, using the method by Deepak S. Turaga, Yingwei Chen, Jorge E. Caviedes: No reference PSNR estimation for compressed pictures. Sig. Proc: Image Comm. 19(2): 173-184 (2004).

As to the tuning parameter μ, it can be updated by,

where c₂ is a constant; N is the number of pixels in a patch; and σ is the standard deviation of the noise in the image. Since the value of TV /½ normally decreases from iteration to iteration, the penalty gets stronger and stronger as the regularization process proceeds.

Applying the same analysis of Eqn. (4) and (5) to a TV /1/2 defined using horizontal and vertical gradient images arrives at a7T_I¾(f) ₌ a∑ (V ^ ^ | _yf(U) |) _{= 1}

d |v_fcf(ij) l d | v_kf(ij) | 2V l v_kf(ij) l ^' ^ ' Eqn. (35) is a decreasing function of | V_fcf (ij) |, which means that the value of TV /½ can be more significantly decreased by reducing the gradient with smaller amplitude. In other words, the low gradient details are preferred to be smoothed rather than the high gradient edges in TV /½ regularization.

According to the analysis above, TV /½ regularization not only tends to smooth along the low-gradient direction for one pixel, but also prefers to smooth the low-gradient pixels among all pixels in the image. Fig. 4 shows an example of the smoothing results by TV /½ regularization on a ID signal. The dash line represents the ID signal before the regularization, and the solid line represents the ID signal after the regularization. Fig. 4(a) shows the regularization results of TV II and TV 12; and Fig. 4(b) shows the regularization results of TV /½. Both TV II and 12 regularization have the same result for a ID signal, in which the strong spikes are suppressed more significantly than the trivial spikes. In comparison, with the same difference between the original and the resultant signal (|| u— f || = 140), TV /½ regularization performs better in retaining the main structure and removing the details.

TV /½ regularization can be applied to a wide range of image processing applications, such as image denoising, video compression and exposure fusion.

TVl½ Image Denoising

TV /½ denoising in the spatial domain, as formulated in (19), is discussed here as an example. Nonetheless, the combined TV /½ -wavelet idea, i.e. to use wavelet constraintin the regularization \\u—

can be implemented.

An exemplary image denoising algorithm is summarized in Table I. This framework addresses two problems of applying TV /½ to image denoising.

a. The direction determination for gradient block calculation in the regularization is usually vulnerable to the noise. Thus directly applying the TV regularization in Eqn. (19)may not lead to satisfactory results especially when the input images are seriously distorted. To alleviate the problem, a pre-processing step is introduced to roughly remove the noise of the image. For example, pre-denoising the images with conventional TV regularization, such as TV 11112 as defined in Eqn. (1) and Eqn. (2),can be performed in the pre-processing step.

b. The patch-independent regularization, meaning each patch/block is regularized independently, often generates unpleasant blocking effects. To address this issue, TV /½ takes the pixels in the adjacent patches into account. However, some of these pixels lie in the unprocessed patches, which means the regularization of the current patch possibly uses unreliable noisy pixels. To address this issue, the patch based regularization can be performed iteratively. In later iterations, the pixels from adjacent patches become much more reliable, which could improve the regularization of the current patch. The terminating condition for the iteration can be, for example, ||f(n) _ fOⁱ-ⁱ) || ||_f < thl , wherein f⁽ⁱ⁾ is the processed image afterthe z^'-th

iteration, and thl is a pre-determined threshold. An example value for thl can be 0.001.

TABLE ITV /½ image denoising Input: a noisy image y

Pre-denoising:f ⁽°⁾ = pre_denoise(y);

Repeat: (the n-t iteration f

Update the image f ^ patch by patch.

For Each Patch (f_t ⁽ⁿ⁾)

Update f_t using TV 1112 regularization

End For

Until: ||f⁽ⁿ⁾ - f ^(n_1) ||_F/||f ^(n_1) ||_F < thl

Output: the denoised version f ^

TV1½ Regularization for Video Compression

TV can be used in compressive sensing (CS), which is capable of acquiring and recovering signals that can be sparsely represented by some bases below the Nyquist rate. TV /½ regularization based on such a principle can be applied to video compression. In the context of video compression, the reconstruction / decoding method can be formulated as

f_t = argmin_ft{7V_i½(f_t) + ½ ||z_t - Φ(Λ - /D ID, (37) where z_t is the vector of quantized transform coefficient of the block f_t, and f is the prediction of f_t either by motion estimation or intra prediction. Φ is a partial DCT matrix, which takes a number of low-frequency measurements of the block. With the prior knowledge that the original image should have a small value of TV /½, a number of image blocks can be reconstructed by a subset of the transform coefficients. For those image blocks, the codec defines a CS mode in which the blocks are represented by T low frequency DCT coefficients, and decoded by (37) can be adaptively determined by the estimation of the sparsity. To decode a block compressed in the CS mode, we first obtain a preliminary reconstruction by applying Inverse DCT (IDCT) to the dequantized coefficients, determine the orientation of its content, and then reconstruct the block via DCT-constrained TV /½ regularization (37).

Unfortunately, the sparsity of the image content varies from block to block. Not all blocks can be well reconstructed by (37). It is thus necessary to select the reconstruction mode adaptively. The encoder can choose between a CS mode and the ordinary IDCT mode for each block by rate distortion optimization. For each mode m, the rate distortion (RD) cost is computed with the estimation of the bit cost ?_m and the distortion D_m,

mode_opt = min_m D_m + xR_m (38) wherer is the Lagrangian multiplier derived by the quality parameter of the compression. The optimal mode mode_opt is the mode whose RD cost is lower.

Since a new CS mode is introduced, the encoder has to transmit a mode flag for each block to the decoder. This can be a large overhead to the compression efficiency, especially for low bit rate coding. Thus, in the codec, the mode information can be transmitted covertly in terms of the parity of the number of nonzero coefficients like watermarks. For example, the number of nonzero coefficients after quantization is made to be odd when CS mode is chosen, and be even when IDCT mode is chosen.

The rule above can be easily fulfilled during the quantization step. The encoder simply quantizes the last nonzero coefficient to zero if the parity of the number of nonzero DCT coefficients does not match the reconstruction mode. Considering that the coefficients at the six lowest frequencies have remarkable influence to the visual appearance, they cannot be modified for this mode watermarking. Consequently, a block must be reconstructed in the CS mode if all of its nonzero DCT coefficients are at the 6 lowest frequencies. As will be demonstrated, the compression efficiency benefits a lot from this covert mode communication mechanism.

Exposure fusion

TV /½ regularization can also be used in exposure fusion. Exposure fusion is a process to compute the desired image by keeping only the "best" parts in the multi-exposure image sequence. An exposure fusion process can be guided by a set of quality measures (contrast, saturation and well-exposedness), which is consolidated into a scalar-valued weight map. Then the fused image is obtained by a weighted blending of input images.

The calculation of the saturation S and the well-exposedness E is shown in MERTENS T., KAUTZJ., REETHF. V.: Exposure fusion. In Pacific Graphics (2007), and formulated by (39) and (40), where k denotes the indices of the input images, / denotes channel indices (R, G, B, or Y, U, V, etc.), ij denote the coordinates of the pixels, hi denotes the channel / of the image k, and G_kiJ- denotes the normalized color intensity at the coordinate (/, j) of image k (e.g., the gray-scale values calculated by the R,G,B values),

S_kij = var_l I_klij, (39) iT_fc,y = exp (- ¾¾ (40) Eqn. (39) indicates that all the pixels in one channel of an input image share a saturation weights; Eqn. (40) indicates that each pixel of an input image have one well-exposedness weight for its 3 channels.

Regarding contrast, TV 1½ regularization can be utilized to extract the structure Struct_k and the detail Detail_k of the gray-scale version of an input image, G_k. Then the intensity of each pixels in the detail image can be used as the contrast weight,

Struct_k = min_G TV_ll/2 (G) + - G - G_k \\² ₂ (41) Detail_k = G_k— Struct_k (42) C_kii_j = Detail_kij. (43) The weight map W is computed by the multiplication of (¾, <¾ and Ekiy. Eqn. (44) uses a power function to control the influence of each measure,

W ^vvk, h ,^■]^■ = C ^u/^Wd^ci; S^Ws E^We

Finally, the recomposed image F is obtained by the weighted average of the input images .

Fii_j = W_kli]I_klij. (45) Note that the above process is based on an assumption that the images are perfectly aligned, for example using a registration algorithm.

Fig.5 depicts a system 10 in accordance with the present principles for accomplishing image regularization using TV W regularization in the manner discussed in greater detail hereinafter. The system 10 includes a processor 12, in the form of a computer, which executes software that performs image TV /^regularization. The processor 12 is connected to one or more conventional data input devices for receiving operator input. In practice, such data input devices include a keyboard 14 and a computer mouse 16. Output information generated by the processor undergoes display on a monitor 18. Additionally such output information can undergo transmission to one or more destinations via a network link (not shown).

The processor 12 is connected to a database 22 which can reside on a hard drive or other non-volatile storage device internal to, or separate from, the processor. The database 22 can store raw image information as well as processed image information, in addition to storing software and/or data for processor use.

The system 10 further includes an image acquisition device 24 for supplying the processor 12 with data associated with one or more incoming images. The image acquisition device 24 can take many different forms, depending on the incoming images. For instance, if the incoming images are "live", the image acquisition device 24 could comprise a television camera. In the event the images were previously recorded, the image acquisition device 24 could comprise a storage device for storing such images. Under circumstances where the images might originate from an another location, the image acquisition device 24 could comprise a network adapter for coupling the processor 12 to a network (not shown) for receiving such images. Although Fig. 5 depicts the image acquisition device 24 as separate from the processor, depending on how the images originate, the functionality of the image acquisition device 24 could reside in the processor 12.

It is further to be understood that the present invention may be implemented in various forms of hardware, software, firmware, special purpose processors, or a combination thereof. Preferably, the present invention is implemented as a combination of hardware and software. Moreover, the software is preferably implemented as an application program tangibly embodied on a program storage device. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units (CPU), a random access memory (RAM), and input/output (I/O) interface(s). The computer platform also includes an operating system and microinstruction code. The various processes and functions described herein may either be part of the microinstruction code or part of the application program (or a combination thereof), which is executed via the operating system. In addition, various other peripheral devices may be connected to the computer platform such as an additional data storage device and a printing device.

Although preferred embodiments of the present invention have been described in detail herein, it is to be understood that this invention is not limited to these embodiments, and that other modifications and variations may be effected by one skilled in the art without departing from the scope of the invention as defined by the appended claims.

Claims

1. A method for regularizing an image, comprising:

calculating a plurality of gradient images for said image; and

regularizing said image by modifying pixel values of said image to minimize a function of said image defined as a weighted sum of absolute values of said calculated gradient images, each raised by a power that is less than 1 and greater than

0.

2. The method of claim 1, wherein said image pixels are adjusted to minimize said function in accordance with the relationship

f = argmin_f 7V_i0 (f) + Ay(u, f),

where7V_i6i (f) is said function; u and f arethe images before and after said adjusting step, respectively; y(u, f)is a measurement of closeness between u and f ; and 1 is a parameter which controls the regularization intensity.

3. The method of claim 2, wherein y(u, f) = || u - f || .

4. The method of claim 1, wherein the weights are the same for all the gradient images.

5. The method of claim 1, wherein there are two gradient images and the two gradient images are a horizontal gradient image and a vertical gradient image.

6. The method of claim 1, wherein the power equals 0.5.

7. An apparatus for regularizing an image, comprising:

a gradient image calculator for calculating a plurality of gradient images for said image; and

a minimizer for modifying pixel values of said image to minimize a function of said image defined as a weighted sum of absolute values of said calculated gradient images, each raised by a power that is smaller than 1 and greater than 0.

8. The apparatus of claim 7, wherein said image pixels are adjusted to minimize said function in accordance with the relationship

f = argmin_f 7V_i0 (f) + Ay(u, f),

W ereTV_w (f ) is said function;u and f arethe images before and after said adjusting step, respectively; y(u, f)is a measurement of closeness between u and f ; and 1 is a parameter which controls the regularization intensity.

9. The apparatus of claim 8, wherein y(u, f) = ||u— f || .

10. The apparatus of claim 7, where in the weights are the same for all the gradient images.

11. The apparatus of claim 7, wherein there are two gradient images, and the two gradient images are a horizontal gradient image and a vertical gradient image.

12. The apparatus of claim 7, wherein the power equals 0.5.

13. A method for regularizing an image, comprising:

dividing the image into a plurality of image blocks;

for each of said plurality of image blocks,

calculating a plurality of gradient blocks; and

regularizing said image by modifying pixel values of said each image block to minimize a function of said image block defined as a weighted sum of absolute values of said calculated gradient blocks, each raised by a power that is less than 1 and greater than 0.

14. The method of claim 13, wherein said calculating and regularizing steps are performed iteratively.

15. The method of claim 13, further comprising pre-processing said image.

16. The method of claim 13, wherein calculating a plurality of gradient blocks for each of said plurality of image blocks comprises

selecting gradient directions for said each image block; and

calculating a gradient block along each of selected gradient direction.

17. The method of claim 16, wherein said selecting gradient directions comprises

pre-selecting a plurality of initial directions; and

selecting gradient directions that match orientation of the content of said image block.

18. The method of claim 17, wherein said gradient directions that match orientation of the content of the image block are selected as the directions that have lowest values of E_k, wherein^ =∑i J \V_kf(i,j) \.

19. An apparatus for regularizing an image, comprising:

a gradient image calculator for calculating a plurality of gradient blocks for each of a plurality of image blocks of said image wherein said image is divided into said plurality of image blocks; and

a minimizer for modifying pixel values of said each image block to minimize a function of said image block defined as a weighted sum of absolute values of said calculated gradient blocks, each raised by a power that is less than 1 and greater than 0.

20. The apparatus of claim 19, wherein said gradient image calculator performs said calculating step and said minimizer performs said modifying step iteratively.

21. The apparatus of claim 19, further comprising a pre-processor for pre-processing said image.

22. The apparatus of claim 19, wherein said gradient image calculator calculates said plurality of gradient blocks for each of said plurality of image blocks by selecting gradient directions for said each image block; and

calculating a gradient block along each of selected gradient direction.

23. The apparatus of claim 22, wherein said gradient image calculator selects said gradient directions by

pre-selecting a plurality of initial directions; and

24. The apparatus of claim 23, wherein said gradient directions that match orientation of the content of the image block are selected as the directions that have lowest values