JPH1153534A - Method and device for reparing image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To form a desirable image by removing only scratch noise from an image having large shade by repairing the image by processing noise included in the image by a diffusion method. SOLUTION: To repair the image, the part including the noise of the repair image is designated first (S1). Then the diffusion method is employed to repair shade components of the noise part of the designated image (S2) and the image is divided by frequency areas (S3). Then texture components of the divided images are subjected to a POCS processing by referring to high frequency components of a sample image (S4). The processed image is corrected by referring to a soft mask (S5) and the corrected image and divided shade components are put together (S6). Then clipping is carried out (S7) so that pixel values are within a specific range, and soft mask repair is made (S8) by referring to the source repair image and soft mask.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image restoration method and apparatus for removing noise shadow components from digital images such as digitized photographic images and computer-generated images.

[0002]

BACKGROUND OF THE INVENTION With the proliferation of available channels for television broadcasts and the proliferation of multimedia technology, many old cinema and photographic films have been converted to video and digital format signals for use. More. However, over the years of use and browsing, many old motion picture and photographic films have become scratched and more contaminated with dust and other debris.

[0003] For example, defects such as scratches and debris in film storage are very noticeable to viewers, and signals of high definition television and digital television formats, and signals of other digital multimedia formats. As with signals, a need exists for a method of removing scratches from digital images without degrading or adversely affecting image quality.

[0004] For example, the special effects of a movie require that the actor / object be suspended or supported by wires, cables or rods, etc., in order to raise it high. Generally, the appearance of these wires, cables, rods, etc.
It is removed by optical processing in later production, and in recent years it has become more common to process it by digital processing of film.

[0005] Similarly, desktop publishing often removes unwanted portions of an image or removes scratches and stains from a photograph. Here, the term "scratch" or "scratch noise" includes all the above-mentioned types of noise that occur in digital images.

[0006] Digital image scratches are unwanted pixels in a digital image. Scratch noise generally consists of a number of adjacent pixels, rather than a small independent group or collection of pixels. In the digital image, the scratch noise is constituted by an area of pixels such as 10 pixels × 100 pixels, and is present in an area of the digital image having details such as textures or edges cut by the scratch.

[0007] To remove scratches from a digital image, the pixels that make up the scratch area must represent the original data or be replaced with data comparable to the original data. In order to effectively repair the scratch area, the pixel data must produce an image as sharp as the surrounding area in the scratch area. In addition, sharp edge continuity must be maintained, and the texture generated to replace wire or scratch pixels must match the surrounding texture.

The effect of the prior art in digital technology varies according to the characteristics of the image, which occur regularly or randomly, and the relative size within the image. Examples of regularly occurring features include textures woven into brick walls and cloth. Examples of randomly occurring features include asphalt roads, concrete sidewalks or sandy beaches.

[0009] The effect of the conventional technique also depends on the size and type of the scratch area. A collection of small, isolated noise pixels in a smooth or blurred area of the image can be relatively easily removed by conventional techniques such as filtering, cloning, and painting techniques. However, conventional noise elimination techniques cannot be applied to scratches consisting of a large number of adjacent pixels, or to scratches in textured areas or areas with prominent edges of an image.

Generally, the scratch noise appears at the position of the scratch as a sudden change in the pixel value when compared with the surrounding pixel values. This sudden change appears as a high frequency component in the frequency spectrum of the image.

A method for removing such image noise is proposed in Japanese Patent Application No. 8-204776. This method restores these noisy pixels in a natural way, based on the location of the noisy pixels and the sample image. That is, the user overwrites the noise area with a noise mask, which is a mask covering the noise area, using a tool such as a digital paint brush, so that the position of the noisy pixel and the repair area around the noise can be obtained. A noise-free sample region referred to in repairing the repair region is specified.

For example, the repair area corresponds to A in FIG.
As shown in the figure, the noise is taken in a substantially rectangular area including a noise 6 such as a scratch / wire whose longitudinal direction is the upper left in the figure. Here, in this image, blocks whose longitudinal direction is the horizontal direction in the figure are alternately and regularly arranged.

The noise mask is overwritten with the noise 6 so as to cover the noise 6 in the repair area 31 shown in FIG. The noise 6 is covered by the noise mask 5.

As shown in FIG. 11, the sample area is set to the same shape and the same size as the repair area 1 including the noise mask 5 in the same area as the repair area which does not include noise. Is done. The repair area 1 is repaired by referring to the sample area 29. That is, the noise 5 included in the repair area 1
Is removed using the information on the space and frequency domain of the repair area 1 and the sample area 29.

[0015] Compared with the conventional methods, the popular conventional methods such as copy pasting, painting, median filtering, low-pass filtering, etc.
The noise is removed using only the information of two regions. On the other hand, the results of the above proposed method are better than the results obtained by the known prior art, but are not perfect, taking into account images with large variations in luminance over a wide range. That is, according to this method, it is difficult to process an image having non-uniform shadows as shown in FIG. 12 and remove the non-uniform shadows from the image.

Next, a specific procedure of the method proposed in the above specification will be described with reference to a flowchart shown in FIG.

In the first step S101, a fast Fourier transform (FFT) of the repair image is performed, and the magnitude (M2) and the phase (M2) calculated by the FFT are sent to the next step S102. Here, the repair image is a rectangular image area around the noise area selected by the user.

On the other hand, in step S106, a fast Fourier transform (FFT) of the sample image is performed. FFT
Are the phase and magnitude of the fast Fourier transform of the sample image. Here, the phase is ignored, and the magnitude is used in a later step S102.

Here, the sample image S is a rectangular image region having exactly the same size as the repair image without the user including a noise image region. The selected area is used as a sample image. The relative position of salient lines in the sample image may be different from the position of salient lines in the repair image.

In step S102, the minimum magnitude of the magnitudes M1 and M2 is calculated. The magnitude M1 is a magnitude obtained by FFT of the sample image, and the magnitude M2 is obtained by FFT of the repair image. The calculation of the minimum magnitude is performed for all frequency components other than the DC component. In DC, the value of magnitude M2 is selected. The new magnitude obtained by the calculation of the minimum is sent as the new value of the Fourier transform used in the next step. Phase information P obtained by FFT of repair image
2 is not changed and is sent to the next step S103.

In step S103, an inverse fast Fourier transform (IFFT) is performed using the magnitude and phase outputs obtained in the previous step S102. Then, the result of the IFFT is sent to step S104.

In step S104, the input value is set to a real number. This input value is a matrix, some of which may be complex. Such complex values in the matrix are set to zero. Since the real number may be out of the range of 0 to 255, clipping is performed. That is, when the value is 0 or less, it is set to 0.
If it is greater than 255, it is set to 255, otherwise nothing is done. Then, the next step S105
Proceed to.

In step S105, binary mask restoration is performed. That is, the pixels obtained in the previous step S104 are mixed with the pixels of the original repair image. The value of the pixel at a position distant from the binary mask should be the value of the original repaired image since it is known that there is no noise at this position. In the region where the value of the binary mask is 0, the original repaired image has noise, and since a better value has been calculated, those values should be left as they are.

Here, the binary mask is a mask overwritten (painted) on the noise area by the user. The binary mask is large enough to cover the noise pixels, but need not be exactly the same shape.

Note that the user can select the number of loops to be executed in steps S101 to S105.

[0026]

In the conventional method of restoring an image by applying a low-pass filter to the image to separate the image into a low-frequency component and a high-frequency component, it is difficult to restore an image in which the shading of the restored image changes greatly. Failed to repair the shadow component.

This is because the separation of the shadow component in the frequency domain using the low-pass filter only works when the shadow is sufficiently smooth, and the pixel value of the noise portion is not affected only by applying the low-pass filter. In this case, only the shadow component cannot be accurately separated, and a high-frequency component includes a noise difference between the noise and the surrounding portion. Also, apply a strong filter,
Further, the effect of noise can be reduced by blurring the image, but the shadow component is also separated, so that a good restoration result cannot be obtained.

The present invention has been made in view of the above-mentioned circumstances, and overcomes the drawbacks of the prior art, and actively performs a restoration process on a shadow component, so that only scratch noise can be reduced from an image having a large shadow. It is an object of the present invention to provide a method and an apparatus for restoring a shadow component of an image so as to obtain a desired image by removing the image.

[0029]

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems, a method of restoring an image according to the present invention uses a continuous over-relaxation (SOR; SuR) method for reducing noise contained in an image, which is a method of reducing a boundary value problem.
ccessive Over Relaxation) method, and a second restoration step of restoring noise contained in the image, specifically, to restore a shadow component of noise in the image. is there. Here, the continuous over-relaxation method is a diffusion method.

That is, the above-described image restoration apparatus and method restores a shadow component by performing restoration using a diffusion method on a noise portion in an image to be restored.

For this reason, in the image restoration process, before the restoration process is started, the restoration target portion is restored in advance by using the diffusion method. As a result, it is possible to restore the low-frequency component and to avoid the influence of unnecessary high-frequency components caused by the noise and the luminance difference between the surrounding images, thereby improving the accuracy of the image restoration.

Further, the image restoration method according to the present invention comprises:
When restoring a missing portion of an image, the amount of calculation is reduced by setting an initial value to the missing portion before performing the diffusion method.

That is, the diffusion method has a problem in the number of calculations required for image restoration. In order for the restoration accuracy to be sufficient, the filter must be repeatedly calculated by a value obtained by taking the square root of the number of pixels included in the repair area. Assuming that the size of the repair area is 100 pixels both vertically and horizontally, 100 repeated calculations are required.
Therefore, in order to reduce the number of repetitive calculations, an initial value is given in advance to the inside of the restoration area, thereby terminating the restoration processing at high speed.

Here, before starting calculation of the diffusion method, an initial value is given in advance. As a result, the number of calculations required for restoration can be reduced, and high-speed calculations can be performed.

Further, in the image restoring method according to the present invention, the shape of a filter for performing processing by the diffusion method is changed according to the image. This filter is represented by a matrix corresponding to a linear operation performed on the pixels of the image.

That is, since the diffusion method filter refers to the upper, lower, left, and right pixels, when an image having symmetry is restored by the diffusion method, the colors of other parts are mixed with each other and cannot be restored successfully. There is. Therefore, color blur is avoided by performing calculations while changing the shape of the filter.

Accordingly, since the shape of the filter used in the diffusion method can be changed, it is possible to perform restoration in accordance with the direction of the image.

[0038] In order to solve the above-mentioned problems, an image restoration apparatus according to the present invention includes a diffusion method means for processing noise contained in an image by a diffusion method;
Restoration means for restoring noise included in an image.

[0039]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An image restoration method and apparatus according to the present invention will be described below with reference to the drawings.

As shown in FIG. 1, the image restoration apparatus includes a control section 10 for controlling the entire apparatus, a central processing section 21 for centrally performing data calculations, and a memory 22 for storing data. Have.

The control section 10 is a controller for controlling each section of the apparatus. This control unit 10
The central processing unit 21 and the memory 22 transmit and receive data, and execute a series of procedures related to a method for setting an area in the image integrally with the central processing unit 21 and the memory 22. The control unit 10 receives data from a mouse 23, a pen / tablet 24, and an image input 26, and outputs data to a CRT 25.

The central processing unit 21 is a processor that performs operations relating to data in a concentrated manner. This central processing unit 21
Is connected to the control unit 10 and performs an operation on data provided from the control unit 10 under the control of the control unit 10.

The memory 22 is a storage unit for storing data. The memory 22 is connected to the control unit 10, and writes and reads data under the control of the control unit 10.

The image restoration apparatus has a mouse 23 and a pen / tablet 24 as input means, and a CRT 25 as an output means for outputting an image relating to a process of setting an area in the image. .

The mouse 23 is a pointing device having a casing large enough to be operated by holding it with the palm. The mouse 23 is slid in a desired direction and a desired distance on a flat surface such as a desk, for example, to input a position of a cursor displayed on the CRT 25, for example. Then, the mouse 23 transmits data relating to the input position to the control unit 10.

The pen / tablet 24 has a substantially rectangular parallelepiped flat casing, is provided with a plurality of sensing elements arranged in a matrix on one of the flat main surfaces, and determines the position on the matrix. The input means includes a pair of a tablet to be detected and a pen for inputting a position by tracing a main surface of the tablet with a pen tip. The pen / tablet 24 transmits data relating to the input position to the control unit 10.

The CRT 25 is an output means for displaying data relating to an image as an image on a plurality of pixels arranged on the cathode ray tube surface. The CRT 25 includes the control unit 10
Transmits data relating to an image, and displays the data as an image.

Next, a series of steps relating to a method for restoring an image will be described with reference to a flowchart shown in FIG.

In the first step S1, a noise-containing portion of a repair image containing noise is designated. The portion including the noise is designated by using a pointing device such as the mouse 23 on the CRT 25, for example. Then, the process proceeds to the next step S2.

In step S2, the shadow component of the noise portion of the image specified in step S1 is restored using the diffusion method, and the flow advances to step S3. This diffusion method will be described later in detail.

In step S3, the image is divided in the frequency domain. In other words, the image to be restored is divided into a high-frequency texture component and a low-frequency shadow component according to the frequency. Then, the process proceeds to the subsequent step S4. Step S3 and subsequent steps correspond to a normal noise restoration process in an image.

In step S4, the above step S
The texture component of the image obtained by dividing the image to be repaired in step 4 in the frequency domain is processed by POCS (Projection onto Convex Sets) with reference to the texture component that is the high frequency component of the sample image that is the reference image I do. Then, the process proceeds to the next step S5.

In step S5, the above step S5
The soft mask is corrected by referring to the soft mask (soft noise mask) for the image processed in step 4. Then, the process proceeds to step S6.

In step S6, the above step S
5. The image with the soft mask repaired in step 5
In step 3, a shadow component, which is a low-frequency component divided in the frequency domain, is synthesized. Then, the process proceeds to the next step S7.

In step S7, clipping is performed. For example, when the pixel value is outside the range of 0 to 255, clipping is performed so that the value falls within the above range. Then, the process proceeds to step S8 following this.

In step S8, a soft mask is repaired. That is, the image to which clipping has been performed in step S7 is soft mask repaired by referring to the original repair image and the soft mask. Then, a series of steps relating to the method of restoring the shadow component to this image is completed. This soft mask repair will be described later.

As described above, in the present method, only the shadow component of the noise portion is restored by using the diffusion method at the stage before the frequency division shown in step S3 is performed.
After step S4, the texture component is restored using the conventional method. As a result, both the low frequency component and the high frequency component can be restored, and a desired restoration result can be obtained.

Next, a first specific example of the image restoration method will be described with reference to a flowchart shown in FIG. In this specific example, a repair area containing noise in an image is repaired by using a soft mask at that time by referring to a sample reference area as a reference area.

In the first step S21, the repair image is multiplied by a soft mask (soft noise mask). The noise mask may be a binary mask (binary noise mask) or a soft mask having a soft edge. Here, the repair image is a rectangular image area around the noise area selected by the user.

The binary mask is a mask overwritten by the user on the noise area. The binary mask is large enough to cover the noise pixels, but need not be exactly the same shape.

The soft edge mask generates a soft edge mask from a binary mask overwritten by the user, if necessary. When generating a soft mask, the value of the soft mask is 0 in the binary mask. Outside the binary mask, the value of the soft mask is a Gaussian function between 0 and 1. That is, for pixels outside the binary mask, assuming that the distance from the binary mask is d, the value of the soft mask is 1-exp (-k * d * d), where k is the soft mask Is a constant that can be set to make the soft edge sharper or gentler. In general, the value of the constant k is
0.15. Once the distance d from the binary mask is determined, one value of the soft mask is obtained. We have found that a distance of about 5 pixels is suitable for this purpose.

When a binary mask is used as the noise mask, the actual noise in the repaired image is covered by black areas. The area outside this area is a part without noise. The result of the multiplication is a mask repair image. on the other hand,
When using a soft mask, the soft range of the soft mask is approximately 0 near the noise edge,
By multiplying the repair image by a factor having a value between 0 and 1 at a position a few pixels away from the edge of the soft mask and at a position between them, the original value below is reduced. . The result of the multiplication is called a mask repair image.
Then, the process proceeds to the next step S22.

In step S22, processing is performed using the diffusion method, and the flow advances to step S23. This diffusion method will be described later in detail.

In step S23, the mask repair image is divided into two images, a high-pass filter mask repair image and a low-pass filter mask repair image. The low-pass filter repair image is stored in the next step S2
4, but instead is sent to step S29. Only the high-pass filter mask repair image is sent directly to the next step S24.

In step S24, a fast Fourier transform (FFT) of the high-pass filter mask repair image is calculated, and the magnitude (M2) calculated by the FFT is calculated.
And the phase (P2) are sent to the next step.

On the other hand, in step S31, the sample image is filtered using a low-pass filter to generate a low-pass filter sample image and a high-pass filter sample image. The low-pass filter sample image is ignored, and the high-pass filter sample image is input to the next step S32.

Here, the sample image is a rectangular image region which is similar to the repair image and does not include the noise image region selected by the user and has the same exact size. The selected area is used as a sample image. This sample image may be considerably darker or brighter than the repair image. The relative position of salient lines in the sample image may be different from the position of salient lines in the repair image.

In step S32, a fast Fourier transform (FFT) of the high-pass filter sample image is calculated. The output of the FFT is the phase and magnitude of the Fourier transform of the high-pass sample image. Here, the phase is ignored. The magnitude (M1) is calculated in the next step S2
Sent to 5.

In step S25, the minimum magnitude of the magnitudes (M1) and (M2) is calculated. The magnitude (M1) is the FFT of the sample image
Is the magnitude obtained by
2) FFT of the high-pass filter mask sample image
Is the magnitude obtained in The new magnitude obtained by the calculation of the minimum value is used in the next step S26.
Send as the new value of the inverse Fourier transform used in. The phase information (P2) obtained by FFT of the high-pass filter mask repair image is sent to the next step S26 without being changed.

In step S26, an inverse fast Fourier transform (IFFT) is calculated using the magnitude and phase outputs obtained in the previous step S25. Then, the process proceeds to step S27.

In step S27, the input value is set to a real number. This input value is a matrix, some of which may be complex. Such complex values in the matrix are set to zero. Real number is also 0
Since there is a possibility that the value is out of the range from 255 to 255, clipping is performed. If the value is 0 or less, it is set to 0. That is 2
If it is 55 or more, it is set to 255. Do nothing else. Then, the process proceeds to step S28.

In step S28, step S2
The soft mask restoration of mixing the pixels obtained in step 2 with the pixels of the mask repair image through the low filter in step S23 is performed first. The value of the pixel at a position distant from the noise mask should be the value of the mask repair image because it is known that there is no noise at this position. In the area where the value of the noise mask is 0, the mask repair image has noise and a better value was calculated.
Their values should be left as is. When the soft mask is used, the value of the area where the value of the soft mask is between 0 and 1 is a value obtained by mixing the value calculated in the previous step and the value of the mask repair image. These three operations (the processing of a soft mask value of 0, 1 and an intermediate value between 0 and 1) can be defined as follows. Assuming that the value of the soft mask at a certain position is m, a new pixel at that position is m * rp + (1-p)
* R, where rp is the value of that position in the mask repair image R through the high-pass filter, and r is
The value of that position obtained in the previous step. Note that a binary mask repair using a binary mask instead of the soft mask can also be performed. Then, step S2
Go to 9.

In step S29, the output of step S28 and the low-pass filter mask repair image obtained in step S23 are combined (merged). Merging is performed by simply adding two images.
Then, the process proceeds to the next step S30.

In step S30, clipping is performed. That is, if any of the input pixels in this step is larger than 255, it is set to 255. Otherwise, do nothing.

The user can select the number of loops to be executed in steps S23 to S30. As a result of this series of steps, an image is obtained in which the noise region included in the image has been restored.

Steps S24 to S27 correspond to POCS in step S4 in FIG.

Subsequently, the steps relating to the diffusion method used for restoring the shadow component in the above series of steps are as follows:
This will be described with reference to the flowchart of FIG. The portion related to the restoration of the shadow component in this image corresponds to step S2 in FIG. 2 and step S22 in FIG. 3 which are steps for applying the diffusion method. Here, the diffusion method is a continuous over-relaxation method (SO
R; Successive Overrelaxation (Numerical Recipe)
s in C, William H. Press, etc.).

In the first step S11, the value of the portion to be restored of the image is set to 0. The restoration target portion is, for example, a portion where noise such as scratch has occurred in the image. Then, the process proceeds to the next step S12.

In step S12, the image to be restored is filtered by the diffusion method. The filter of the diffusion method is, for example, a matrix of 3 rows and 3 columns as shown in the following formula, and a first row, second column, and a second row, first column.
Column, second row, third column, and third row, second column have w
/ 4, the element in the second row and second column is 1-w, and the other parts are 0.

[0080]

(Equation 1)

The order in which the filter expressed by this matrix is applied to the image to be restored is as shown in FIG.
In the repair image 31, in the horizontal direction within the frame, the operation direction is from left to right, and this scanning line is applied to each pixel and surrounding pixels in order from top to bottom with respect to the vertical direction of the frame of the repair image. I do.

At this time, the pixel values obtained by the immediately preceding calculation are used for the leftward and upward components of the pixel to be calculated in the figure. The leftward and upward components in these figures correspond to the elements in the first row, second column, and second row, first column of the matrix, respectively. Here, w that appears in the element of the matrix representing the filter is called an over-relaxation parameter, and 0 <
Take a value of w <2. Then, the process proceeds to step S13.

In step S13, the pixel value after restoration is compared with the pixel value before restoration. If the pixel value is equal to or larger than the predetermined threshold value, the process returns to step S12 to repeat the restoration calculation. If the difference is less than the threshold, the series of steps ends.

FIG. 6 shows an example of restoration of an image by the diffusion method. That is, the image shown in A in the figure is a part of the normal part 8 where the luminance value changes continuously,
It has a missing portion 9 whose luminance value is extremely low.

As shown in B and C in the figure, the value of the image having the missing portion 9 is gradually restored in such a manner that the pixel value near the missing portion flows in every time the number of repetitions is repeated. Will be done.

In the above embodiment, the repairing method using the diffusion method has been described. Here, when restoration is performed by the diffusion method, a filter is repeatedly applied, but the number of calculations is a problem. In order for the restoration accuracy to be sufficient, the filter must be repeatedly calculated by a value obtained by taking the square root of the number of pixels included in the repair area. If the size of the repair area is 100
Assuming pixels, 100 repetitive calculations are required.

A method for speeding up the repair calculation by giving an initial value to the inside of the repair area in order to reduce the number of calculations will be described with reference to FIG. A to C in FIG.
This is an image composed of 8 × 8 pixels, and 4 × 4 pixels at the center of this image are missing.

In the conventional method, when the diffusion method is applied to the repair area 2 indicated by A in the figure, the brightness value of the repair area 2 is set to 0 as shown in B in the conventional method. Therefore, the number of repetitions is large and the calculation time is long to sufficiently recover the luminance value. Here, in the diffusion method, if restoration is sufficiently performed, the value is always equal to or more than the lowest value of the surrounding pixels 1 surrounding the area. Therefore, as shown by C in the figure, a pixel having the lowest luminance is selected from the values of the pixels 1 adjacent to the restoration area, and the restoration area 2 is filled using the selected pixel value. Thereafter, repair is performed using a diffusion method. As a result, an increase in the number of times required for the restoration to be completed can be significantly suppressed.

Further, even in the case where a large change in luminance has a pattern like the image shown by A in FIG. 8, the above-described image restoration method can be easily applied. A to C in FIG.
The horizontal direction in the drawing has symmetry such that the same luminance value is obtained. A in the figure includes noise 6 such as a scratch / wire, with the longitudinal direction being substantially the horizontal direction in the figure. The noise 6 is covered by the noise mask 5, and a rectangular area 4 including the noise 67 and the noise mask 5 is set.

When the area 4 is repaired by the diffusion method, the repair result becomes B in FIG. B in this figure
Then, there is a part where the luminance changes rapidly, but since the value of the pixel in the vertical direction is largely different at this part especially at the center part, as a result of the color of the different part flowing in,
Blurring 7 appears in the image. Therefore, unnecessary bleeding 7 can be prevented by adjusting the shape of the filter to the direction of the pattern of the image.

Here, since the image pattern is in the horizontal direction, the shape of the filter is restored using the following matrix. As a result, as shown by C in the figure, the shadow component can be restored without being affected by the pixels in the vertical direction. This matrix has three rows and three columns, a second row and a first column, and a second row and a third row.
The value of the element in the column is w / 2, and the value of the element in the second row and second column is 1-
w, and the values of the other elements are 0.

[0092]

(Equation 2)

Next, a second specific example of the method for restoring the shadow component of the image will be described with reference to the flowchart of FIG. This second specific example is the same as the first specific example except that the filter step of step S23 is omitted. The description of the parts common to the second specific example and the first specific example is omitted for simplicity.

In the first step S31, the repair image is multiplied by a soft mask. As a result, the original noise is covered by the black areas of the soft mask. Then, the process proceeds to the next step S32.

In step S32, a process according to the diffusion method is performed. Since the diffusion method has been described in detail above, the description is omitted here. Then, the process proceeds to the next step S32.

In step S33, fast Fourier transform (FFT) of the mask repair image is performed, and the magnitude (M2) and phase (M2) calculated by FFT are sent to the next step S34.

On the other hand, in step S38, fast Fourier transform (FFT) of the sample image is performed. The output of the FFT is the phase and magnitude of the fast Fourier transform of the sample image. Here, the phase is ignored and the magnitude is used later. The magnitude is
It is sent to step S34.

In step S34, the minimum magnitude of the magnitudes M1 and M2 is calculated. The magnitude M1 is a magnitude obtained by the FFT of the sample image, and the magnitude M2 is a magnitude obtained by the FFT of the mask repair image. The calculation of the minimum magnitude is performed for all frequency components other than the DC component. In DC, the magnitude M2
Is selected. The new magnitude obtained by the calculation of the minimum is sent as the new value of the Fourier transform used in the next step. The phase information P2 obtained by the FT of the mask repair image is not changed and is not changed in the next step S
Sent to 35.

In step S35, inverse fast Fourier transform (IFFT) is performed using the magnitude and phase outputs obtained in the previous step S34. And
The result of the IFFT is sent to the next step S36.

In step S36, the input value is set to a real number. This input value is a matrix, some of which may be complex. Such complex values in the matrix are set to zero. Since the real number may be outside the range of 0 to 255, clipping is performed. If the value is 0 or less, it is set to 0. That is 2
If it is 55 or more, it is set to 255; otherwise, nothing is done. Then, the result is sent to the subsequent step S37.

In step S37, a soft mask restoration for mixing the pixels obtained in the previous step S36 with the pixels of the original repair image is performed.

Here, the user can select the number of loops to be executed in steps S2 to S6. As a result of this series of steps, an image is obtained in which the noise region included in the image has been restored.

The above-described image restoration method and apparatus are not limited to the diffusion method, but may employ various SORs.
Law can be used.

[0104]

As described above, according to the present invention, a method for restoring an image having a large shadow in a noise portion by using the SOR method, for example, the diffusion method, which has been a problem in the conventional method, is described. It was solved by developing. In addition, a method was developed to reduce the number of calculations by giving an initial value to the number of calculations that would be a problem when using the diffusion method. In addition, we developed a method that can provide restoration according to the direction of the shadow by giving the direction of the filter even if the shadow has a direction.

As a result, an area having a shadow component having a large change in luminance value, which has been difficult in the past, can be restored.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a schematic configuration of an image restoration device.

FIG. 2 is a flowchart showing a series of steps relating to an image restoration method.

FIG. 3 is a flowchart showing a series of steps of a first specific example of the repair method.

FIG. 4 is a flowchart showing a main part of a process according to the repair method.

FIG. 5 is a diagram showing the order of processing on an image.

FIG. 6 is a diagram showing a state of restoration of a missing image.

FIG. 7 is a diagram showing a mode in which an initial value is given to a missing portion in an image.

FIG. 8 is a diagram illustrating restoration of a missing portion in an image having symmetry.

FIG. 9 is a flowchart showing a series of steps of a second specific example of the repair method.

FIG. 10 is a diagram showing an image in which a scratch / wire exists.

FIG. 11 is a diagram showing an image in which a repair area and a sample area are set.

FIG. 12 is a diagram showing an image including non-uniform shadows.

FIG. 13 is a flowchart showing a series of steps of an example of a conventional image restoration method.

[Explanation of symbols]

 10 control unit, 21 central processing unit, 22 memory

Claims (8)

[Claims] 1. A method in which noise included in an image is continuously over-relaxed (SOR), which is a method of relaxing a boundary value problem.
A method for restoring an image, comprising: a first restoration step of performing processing according to the n) method; and a second restoration step of restoring noise included in the image. 2. The method according to claim 1, wherein the continuous over-relaxation method is a diffusion method. 3. The second restoration step includes a step of performing a fast Fourier transform on the image processed in the first restoration step to obtain a first fast Fourier coefficient, and an image including noise. Obtaining a second fast Fourier coefficient by performing a fast Fourier transform on an image referred to when the image is restored, and performing an inverse Fourier transform on the Fourier coefficient obtained based on the first and second Fourier coefficients. 3. An image restoration method according to claim 2, further comprising the steps of: acquiring an image by performing the above-mentioned steps; and replacing the image containing noise with the image acquired by the inverse Fourier transform. 4. The restoration of an image according to claim 2, wherein when restoring a missing portion of the image, the amount of calculation is reduced by setting an initial value to the missing portion before performing the diffusion method. Method. 5. The image restoration method according to claim 2, wherein the shape of a filter for performing a process by the diffusion method is changed according to the image. 6. The method according to claim 5, wherein the filter is represented by a matrix corresponding to a linear operation performed on pixels of the image. 7. An image restoration apparatus comprising: a diffusion method unit for processing noise included in an image by a diffusion method; and a restoration unit for restoring noise included in an image. 8. The restoration means performs fast Fourier transform on the image processed by the diffusion means to obtain a first fast Fourier coefficient, and refers to a means for restoring an image containing noise. Means for performing a fast Fourier transform on the image to be obtained to obtain a second fast Fourier coefficient, and obtaining an image by performing an inverse Fourier transform on the Fourier coefficient obtained based on the first and second Fourier coefficients 8. The image restoration apparatus according to claim 7, further comprising: means for performing an operation for replacing an image containing noise with an image obtained by the inverse Fourier transform.