JP2020195127A - Image processing device, image processing method, imaging apparatus, program, and storage medium - Google Patents

Abstract

To provide an image processing device capable of satisfactorily removing fog and haze in an image.SOLUTION: An image processing device includes an acquisition unit for acquiring image data, a first gradation correction unit for acquiring a dark channel value for each area of the image data to estimate the transmittance and correct the gradation, a second gradation correction unit for correcting the gradation of the image data using a tone curve, and a control unit for controlling which gradation correction unit is to be used to perform the gradation correction.SELECTED DRAWING: Figure 2

Description

The present invention relates to a technique for removing fog haze in an image taken in a situation where fog haze is generated to improve visibility.

In a camera used for conventional surveillance and the like, fog haze may occur when shooting outdoors, which may reduce visibility. In order to relieve such an image, there is known a technique of performing a correction process for removing fog haze when it occurs. For the correction processing, a method of correcting the contrast using a tone curve for the entire screen, and a scattering separation method that separates Mie scattering and Rayleigh scattering in the atmospheric model and removes wavelength-independent Mie scattering components such as fog haze. There is a method to adopt.

The scattering separation method is a method in which the transmittance is estimated using the dark channel value for each region of the image and the gradation is controlled to remove the fog haze. Specifically, a technique has been proposed in which the transmittance due to the concentration of fog haze in the atmospheric model is estimated and the gradation is controlled.

In Patent Document 1, an empty region or a backlit region is calculated, a dark channel value in that region is corrected, a transmittance is calculated from the dark channel value, and the fog haze component is removed from an input image using the transmittance. ..

In Patent Document 2, a pixel used for deriving a reference intensity of scattered light is selected in a color image, and a first reference intensity corresponding to a first color component of scattered light and a second reference intensity corresponding to a second color component are selected. Derivation of the reference strength of. The first reference intensity and weight value are used to correct the pixel value of the first color component of the color image, and the second reference intensity and weight value are used to correct the pixel value of the second color component. Generate a corrected image.

JP-A-2017-138647 JP-A-2015-103167

However, in the prior art disclosed in Patent Document 1, although it is possible to correct scattered light by using the dark channel value, halo noise is generated in the contour portion in the image, and the noise can be suppressed. Can not.

Further, in the conventional technique disclosed in Patent Document 2, similarly, the component caused by the concentration of fog haze can be removed and corrected by using the scattered light in the atmospheric model, but the contour portion in the image is also halo. Noise is generated, and the noise cannot be suppressed.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image processing apparatus capable of satisfactorily removing fog haze in an image.

The image processing apparatus according to the present invention includes an acquisition means for acquiring image data and a first gradation correction means for acquiring dark channel values for each region of the image data to estimate transmission and correct gradation. A second gradation correction means for correcting the gradation of the image data using a tone curve, and a control means for controlling which gradation correction means is used for gradation correction are provided. It is a feature.

According to the present invention, it is possible to provide an image processing apparatus capable of satisfactorily removing fog haze in an image.

The figure which shows the structure of the digital camera which is 1st Embodiment of the image processing apparatus of this invention. The block diagram which shows the structure of the image processing circuit 120 in 1st Embodiment. The figure which shows the luminance histogram in the presence or absence of fog haze. The figure which shows the example of the toe curve correction according to the degree of fog haze generation. The flowchart which showed the fog haze removal operation in the 1st Embodiment. The flowchart which showed the removal operation of the fog haze in the 2nd Embodiment. The block diagram which shows the structure of the image processing circuit 121 in 3rd Embodiment. The flowchart which showed the fog haze removal operation in 3rd Embodiment. The flowchart which showed the removal operation of fog haze in 4th Embodiment. The figure which shows the structure of the image pickup system in 5th Embodiment. The flowchart which showed the fog haze removal operation in 5th Embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate description is omitted.

(First Embodiment)
FIG. 1 is a diagram showing a configuration of a digital camera 100, which is a first embodiment of the image processing apparatus of the present invention.

In FIG. 1, the lens unit 102 forms an optical image of a subject on an image pickup unit 101 having an image pickup element including a CCD or a CMOS sensor. Further, the lens driving device 103 performs zoom control, focus control, aperture control, and the like. The image pickup signal processing circuit 104 performs various correction processes, data compression processes, and the like on the image signal output from the image pickup unit 101. Further, the image pickup signal processing circuit 104 has an image processing circuit 120 inside for removing fog haze of the image. The image sensor drive circuit 105 outputs various power sources, shooting mode instruction signals, and various timing signals to the image sensor in the image sensor 101.

The memory unit 106 functions as a memory for temporarily storing image data, and the overall control calculation unit 107 performs various calculations and controls the entire digital camera 100. The recording medium control I / F unit 108 is an interface for recording data on the recording medium or reading data from the recording medium. The recording medium 109 is a detachable semiconductor memory for recording or reading image data. Further, the display unit 110 is a display device such as a liquid crystal display device that displays various information and captured images.

FIG. 2 is a block diagram showing the configuration of the image processing circuit 120 in FIG. The image processing circuit 120 has a function of removing fog haze while suppressing halo noise generated in the contour portion of an image.

In FIG. 2, the image processing circuit 120 includes a first image processing unit 11, a first haze correction unit 12, a second haze correction unit 14, a second image processing unit 15, an image output unit 16, and a histogram acquisition. The analysis unit 13 is provided.

The image data captured by the image pickup unit 101 is input to the image processing circuit 120. The first image processing unit 11 is an image processing unit of the first stage of image data. Specifically, the image data acquired from the image pickup unit 101 is subjected to sensor correction processing, scratch correction processing, conversion processing for converting a Bayer-type RGB signal into a YC signal, contour enhancement processing, noise reduction processing, white balance processing, etc. I do. The first haze correction unit 12 estimates the transmittance (density map) using the dark channel value for each area of the image, and performs the first gradation correction process (gradation correction operation) to control the gradation. This is the first-stage gradation correction unit. The histogram acquisition / analysis unit 13 detects the contrast of the image by acquiring the histogram of the luminance component in the corrected image after the first gradation correction process and calculating the dispersion value thereof. The second haze correction unit 14 is a second-stage gradation correction unit that performs a second gradation correction process (gradation correction operation) that controls contrast and intermediate gradation using a tone curve.

Hereinafter, the details of the first gradation correction processing will be described. First, the image I in which the fog haze is generated and the image J after the fog haze is removed are related by the atmospheric model as in the equation (1).

I (x, y) = J (x, y) · t (x, y) + A (1-t (x, y) equation (1)
x and y are two-dimensional coordinate positions in the horizontal and vertical directions in the image, t is a fog haze density map, and A is ambient light. Here, the density map t (x, y) means attenuation due to the atmosphere. The longer the subject distance from the camera, the larger the attenuation (pixel value), and the closer it is, the smaller the attenuation (pixel value). By estimating the ambient light A and the density map t (x, y) in the equation (1), it is possible to obtain J (x, y) after removing the fog haze.

First, the method of estimating the ambient light A will be described using a mathematical formula. The ambient light A is an average pixel value of the RGB components in the empty region, and is calculated by the equation (2).

The ave function represents a function for calculating the average value in the argument, c represents a color component, and Ωsky represents a local region in an empty region. Here, the empty region can be specified by, for example, a method of calculating based on the distribution of the histogram, a method of using a coordinate position specified in advance, a method of using a position instructed by the user, or the like.

Subsequently, a method of estimating the density map t (x, y) will be described. The first gradation process in the present embodiment is a process of estimating the density map t (x, y) based on the dark channel value, and is at least one component of the RGB component in an outdoor image in which fog haze is not generated. It is assumed that the pixel value of is locally very small. The dark channel value I ^drk is calculated as in Eq. (3).

c represents a color component, and Ω represents a local region including the coordinates of interest (x, y). As shown in the equation (3), the dark channel value is the minimum value of the RGB component in the local region including the pixel of interest. By substituting the calculation formula for the dark channel of the formula (3) into the atmospheric model of the formula (1), the formula (4) for calculating the dark channel value from the atmospheric model can be obtained.

I ^drk = J ^drk (x, y) · t (x, y) + A (1-t (x, y)) Equation (4)
Here, considering the premise that the pixel value of at least one component of the RGB component is locally small in the image in which the fog haze is not generated, the dark channel value J ^drk (the dark channel value of the fog haze removal image in the equation (4)) ( x, y) is a value as close to 0 as possible. Therefore, the equation (4) can be approximated as the equation (5).

I ^drk (x, y) ≒ A (1-t (x, y)) Equation (5)
By modifying the approximate expression of the equation (5), the concentration map t (x, y) can be estimated as in the equation (6).

t (x, y) ^≈1- {ω × I ^drk (x, y) / A} Equation (6)
Here, ω is a parameter that controls the degree of correction of fog haze, and is defined in the range of 0.0 to 1.0. The larger the value, the stronger the correction effect of fog haze can be set. By substituting the airglow A and the concentration map t (x, y) calculated by the above-described equations (2) to (6) into the equation (7), J (x, y) after removing the fog haze can be obtained. Can be sought.

J (x, y) = {(I (x, y) -A) / t (x, y)} + A formula (7)
As shown in the equation (1), the first gradation correction process based on the dark channel takes into consideration the density map that changes according to the subject distance from the camera in addition to the ambient light A, for each pixel. Calculate the correction amount. Therefore, fog haze can be effectively removed not only for a subject in a near view but also for a subject in a distant view. However, as shown by the equation (3), the calculation of the dark channel value does not consider the edge information, and as shown by the equation (4), the density map is obtained by approximation. , Miscorrection occurs around the edge portion, specifically, a phenomenon in which white appears (hereinafter, this phenomenon is referred to as halo noise (white area)) occurs.

Subsequently, the details of the second haze correction unit 14 will be described. First, the histogram acquisition / analysis unit 13 generates a histogram of the input image and calculates a statistical value representing the distribution density of the generated histogram. For example, the variance σ ² of the histogram is obtained as in Eq. (8).

σ ² = {(x1-μ) ² · f1 ＋ (x2-μ) ² · f2 ＋… ＋ (xN−μ) ² } / n Equation (8)
Here, xi is the input luminance level, xN is the maximum value of the input luminance level, f is the frequency, n is the total number of data (total number of pixels), and μ is the average. In the present embodiment, the variance of the histogram is used as the statistical value representing the distribution density, but the present invention is not limited to this.

Further, the histogram acquisition / analysis unit 13 calculates the degree of fog haze generation based on the above-mentioned histogram and statistical values. Generally, in an image in which fog haze is generated, as shown in FIG. 3A, the distribution of the histogram is concentrated in a part of the region, and the image has low contrast. Therefore, when fog haze is generated, the variance value becomes a relatively small value. Therefore, the fog haze correction unit 2 calculates the degree of fog haze generation H by comparing an arbitrary threshold value Th with the variance value σ ² . For example, it is calculated by the formula (9).

0.0 (if σ ² <0)
H = (Th−σ ² ) / Th (if 0 ≦ σ ² ≦ Th)
1.0 (if Th <σ ² ) Equation (9)
Here, the degree of fog haze H represented by the equation (9) is normalized in the range of 0.0 to 1.0, and the larger the value, the stronger the degree of fog haze generation becomes an index.

Subsequently, the second haze correction unit 14 adjusts the toe curve according to the degree of fog haze calculated by the histogram acquisition / analysis unit 13. Specifically, as shown in FIG. 4, a tone curve that lowers the dark part and raises the bright part is applied as the degree of fog haze is stronger. By applying the above-mentioned tone curve, as shown in FIG. 3B, the histogram is smoothly distributed, and fog haze can be removed.

Therefore, the second haze correction unit 14 can remove the fog haze while suppressing harmful effects by improving the contrast of the entire screen according to the distribution of the histogram (the degree of fog haze generation). .. However, since the correction amount in consideration of the subject distance is not calculated, this method is less effective for a distant subject.

In the above, the first haze correction unit 12 (one haze correction unit) performs the first gradation correction process, and then the second haze correction unit 14 (the other haze correction unit) performs the second gradation. Although the correction process is performed, the order may be reversed. That is, the same effect of removing fog haze can be obtained even if the first haze correction unit 12 performs the second gradation correction process and then the second haze correction unit 14 performs the first gradation correction process.

The second image processing unit 15 performs the second stage image processing on the image from which the fog haze has been removed by the first haze correction unit 12 and the second haze correction unit 14. Specifically, contour enhancement processing, noise reduction processing, color correction processing, resolution conversion processing, and the like are performed. Here, the contour enhancement processing, the noise reduction processing, the color correction processing, and the like may be performed by the first image processing unit 11 or the second image processing unit 15. The image output unit 16 outputs an image that has undergone image processing or fog haze removal processing to another image processing unit, display unit 110, or the like that performs image compression processing or the like.

In the first gradation correction process, the correction process is executed so as to gradually obtain the correction effect over time. Specifically, the value of ω in the equation (6) is gradually set from a small value to a large value. Then, when the halo noise is extracted from the contour portion, the first gradation correction process is stopped (stopped). By doing so, it becomes possible to suppress the halo noise of the contour portion that may be generated in the first gradation correction process.

Further, as described above, there are cases where the first haze correction unit 12 performs the first gradation correction processing and the second haze correction unit 14 performs the second gradation correction processing, and gradation correction. It is conceivable that the order of processing is reversed. In each case, the gradation correction process is performed by the following method.

First, when the first haze correction unit 12 performs the first gradation correction process and the second haze correction unit 14 performs the second gradation correction process, the following method is used. That is, the first haze correction unit 12 gradually performs the first gradation correction processing while observing the appearance of the halo noise as described above, and after the gradation correction processing is stopped, the statistic of the image, concretely Specifically, the effect of gradation control (correction result) is judged by looking at the luminance histogram. If the effect of gradation control is insufficient, the second haze correction unit 14 takes a second time until a gradation correction amount (target gradation) at which a sufficient effect can be obtained is reached. Perform gradation correction processing. However, since it is not necessary to consider halo noise, the second gradation correction process does not necessarily have to be performed slowly.

Further, when the first haze correction unit 12 performs the second gradation correction processing and the second haze correction unit 14 performs the first gradation correction processing, the following method is used. That is, the first haze correction unit 12 gradually performs the second gradation correction processing over time while observing the effect of the second gradation correction processing. In the second gradation correction process, the effect of removing fog haze cannot be obtained in the distant view portion, so the process is performed while observing the degree of the effect. However, the second gradation correction process has the advantage that it is easier to see the change in the effect if it is performed over time, but as described above, it is not necessary to consider halo noise, so it is not always necessary to perform it slowly. There is no. Then, if the effect of gradation control is insufficient after the second gradation correction processing is stopped, the second haze correction unit 14 takes time to obtain a sufficient effect. The first gradation correction process is performed until the value reaches. Also in this case, in the first gradation correction process, when the halo noise is extracted from the contour portion, the process is stopped.

By the above method, it is possible to perform a good fog haze removal process in which the generation of halo noise is suppressed.

FIG. 3 is a flowchart showing an operation of removing fog haze so that halo noise does not occur in the contour portion of the image. Note that FIG. 3 shows an example in which the first haze correction unit 12 performs the first gradation correction process and the second haze correction unit 14 performs the second gradation correction process.

When the flow of the fog haze removal processing is started, in step S31, the image acquired by the imaging unit 101 is input to the image processing unit 120. In step S32, the first haze correction unit 12 performs the first gradation correction process. Here, as described above, the correction process is performed within a range in which halo noise does not occur in the contour portion of the image.

In step S33, the overall control calculation unit 107 determines the magnitude of contrast with respect to the image resulting from the first gradation correction process performed in step S32. If the contrast is smaller than the predetermined value, the process proceeds to step S34, and if the contrast is greater than or equal to the predetermined value, the gradation correction process is terminated.

In step S34, in order to remove the fog haze that could not be removed by the first gradation correction processing executed in step S32, the second gradation correction processing is executed and the gradation correction processing is completed.

Note that FIG. 3 shows an example in which the first gradation correction process is performed in step S32 and the second gradation correction process is performed in step S34. However, as already described, the second gradation correction process may be performed in step S32, and the first gradation correction process may be performed in step S34. In either case, in the first gradation correction process, the correction process is performed within a range in which halo noise does not occur.

As described above, according to the first embodiment, it is possible to perform an effective fog haze removal process within a range in which halo noise does not occur.

(Second Embodiment)
Hereinafter, a digital camera according to a second embodiment of the image processing apparatus of the present invention will be described. Since the overall configuration of the digital camera in the second embodiment is the same as the configuration of the first embodiment shown in FIG. 1, the description thereof will be omitted. In this second embodiment, the first gradation correction process and the second gradation correction process described in the first embodiment are weighted and implemented.

FIG. 4 is a flowchart showing an operation of removing fog haze so that halo noise does not occur in the contour portion of the image in the second embodiment.

When the flow of the fog haze removal processing is started, in step S41, the image acquired by the imaging unit 101 is input to the image processing unit 120. In step S42, the first image processing unit 11 acquires a statistic of the brightness value of the input image, specifically, a brightness histogram.

In step S43, the overall control calculation unit 107 determines the magnitude of contrast from the luminance histogram acquired in step S42. If the contrast is smaller than the predetermined value, it is determined that the fog haze is strong and the process proceeds to step S44. If the contrast is equal to or more than the predetermined value, it is determined that the fog haze is not so strong and the process proceeds to step S46.

In step S44, the weight of the first gradation correction process described in the first embodiment is increased. Here, the degree of removal of fog haze is increased even if the halo noise generated in the outline of the image is allowed to some extent. Further, in step S45, in order to remove the fog haze that could not be removed in step S44, the second gradation correction process described in the first embodiment is performed. Then, the fog haze removal process is completed. The same effect can be obtained even if the order of steps S44 and S45 is reversed.

In step S46, since the contrast is not so small and the fog haze is not strongly applied, the weight of the first gradation correction process is reduced so that the halo noise of the outline portion of the image is not generated. In step S47, the weight of the second gradation correction process is increased in order to remove the fog haze that could not be removed in step S46. Then, the fog haze removal process is completed. The same effect can be obtained even if the order of steps S46 and S47 is reversed.

As described above, according to the second embodiment, it is possible to perform an effective fog haze removal treatment according to the strength of the fog haze.

(Third Embodiment)
Hereinafter, a digital camera according to a third embodiment of the image processing apparatus of the present invention will be described. The overall configuration of the digital camera in the third embodiment is also the same as the configuration of the first embodiment shown in FIG. However, in the present embodiment, the configuration of the image processing circuit 121 is different from the configuration of the image processing circuit 120 of the first embodiment. In this third embodiment, the method of removing fog haze is switched and executed according to the region of interest.

FIG. 5 is a diagram showing the configuration of the image processing circuit 121 according to the third embodiment. The image processing circuit 121 specifies (sets) a region of interest, measures the subject distance in that region, switches the gradation control method according to the subject distance, and removes fog haze.

In FIG. 5, the image processing circuit 121 includes a first image processing unit 21, a region of interest designation unit 22, a distance information acquisition unit 23, a first haze correction unit 24, a second haze correction unit 25, and a second image. It includes a processing unit 26 and an image output unit 27.

The image data captured by the image pickup unit 101 is input to the image processing circuit 121. The first image processing unit 21 is an image processing unit of the first stage of image data, and is the same as the first image processing unit 11 described in the first embodiment. The attention area designation unit 22 has a function of designating an area of interest in the image data, and includes a GUI or the like in which the user specifies the area of interest. The distance information acquisition unit 23 measures the distance to the attention area designated by the attention area designation unit 22. Here, the distance to the region of interest is measured using control information of a focus lens or a zoom lens arranged in the lens unit 102.

The first haze correction unit 24 executes the first gradation correction process described in the first embodiment. The second haze correction unit 25 executes the second gradation correction process described in the first embodiment. The second image processing unit 26 performs selection processing or mixing processing of the image corrected by the first haze correction unit 24 and the image corrected by the second haze correction unit 25, and image processing of the second stage. .. Since the first haze correction unit 24 is effective in removing fog haze in a region of interest at a long distance, the first haze correction unit 24 is the first when the distance information acquired by the distance information acquisition unit 23 is farther than the first predetermined distance. The haze correction unit 24 of 1 is selected. Since the second haze correction unit 25 is effective in removing fog haze in a region of interest at a short distance, when the distance information acquired by the distance information acquisition unit 23 is closer than the second predetermined distance, The second haze correction unit 25 is selected. When the distance information acquired by the distance information acquisition unit 23 is less than or equal to the first predetermined distance and greater than or equal to the second predetermined distance, the image corrected by the first haze correction unit 24 and the second haze correction unit The mixed processing (compositing processing) of the images corrected in 25 is performed.

Here, the image composition ratio is determined for each region using the edge information of the subject. Specifically, for the region considered to be an edge, the ratio of the second gradation correction processing described in the first embodiment is set large. On the other hand, with respect to the region considered to be flat, the ratio of the first gradation correction processing described in the first embodiment is set large. By synthesizing at such a ratio, it is possible to effectively remove fog haze while suppressing halo noise at the edge portion. The specific procedure will be described below.

First of all, the edges are detected from the image. As a method of edge detection, edge information can be extracted as a shading image by executing a sobel filter on the input image I in which fog haze is generated.

Subsequently, based on the generated edge information (concentration value F (x, y)), the mixing ratio Blend (x, y) is obtained, for example, as in the formula (10).

Blend (x, y) = F (x, y) / Fmax equation (10)
Here, Fmax indicates the maximum value of the concentration value, and the mixing ratio is normalized by a value of 0.0 to 1.0.

Finally, based on the mixing ratio of the equation (10), the image Dehaze1 (x, y) corrected by the first haze correction unit 24 and the image Dehaze2 (x, y) corrected by the second haze correction unit 25. ) Is synthesized as in the formula (11).

Dehaze _Blend (x, y)
= (1.0-Blend (x, y)) x Dehaze1 (x, y) + Blend (x, y) x Dehaze2 (x, y)
Equation (11)
By following the above procedure, a mixed image Dehaze _Blend (x, y) in which fog haze removal is effectively performed while suppressing halo noise at the edge portion can be obtained.

The second image processing unit 26 is the same as the second image processing unit 15 described in the first embodiment. Each image processing function may be performed by the first image processing unit 21 or may be performed by the second image processing unit 26. The image output unit 27 outputs an image that has undergone image processing or fog haze removal processing to another image processing unit, display unit 110, or the like that performs image compression processing or the like.

FIG. 6 is a flowchart showing an operation of removing fog haze by switching the gradation control method according to the distance of the designated region of interest.

When the flow of the fog haze removal processing is started, in step S61, the image acquired by the imaging unit 101 is input to the image processing unit 121. In step S62, the area that the user wants to pay attention to is designated by GUI or the like. In step S63, the distance information acquisition unit 23 measures the distance to the region of interest designated by the user. The distance to the region of interest is measured using control information of a focus lens or a zoom lens arranged in the lens unit 102.

In step S64, the overall control calculation unit 107 determines whether or not the distance to the subject included in the region of interest is farther than the first predetermined distance. If the distance to the subject is longer than the first predetermined distance, the process proceeds to step S65. If the distance to the subject is not longer than the first predetermined distance, the process proceeds to step S66.

In step S65, the first gradation correction process described in the first embodiment, which is effective for removing the fog haze on a distant subject, is executed.

In step S66, it is determined whether or not the distance to the subject included in the region of interest is closer than the second predetermined distance. If the distance to the subject is closer than the second predetermined distance, the process proceeds to step S67. If the distance to the subject is not closer than the second predetermined distance, the process proceeds to step S68.

In step S67, the second gradation correction process described in the first embodiment, which is effective for removing the fog haze on a nearby subject, is executed. In step S68, based on the synthesis ratio obtained by the formula (10). The image obtained by the first gradation correction process described in the first embodiment and the image obtained by the second gradation correction process are mixed as shown in the equation (11). This step S68 is executed when the distance to the subject included in the region of interest is equal to or less than the first predetermined distance and greater than or equal to the second predetermined distance. After executing step S65, S67, or S68, the fog haze removal operation is terminated.

As described above, according to the third embodiment, it is possible to perform an effective fog haze removal process according to the distance of the subject.

(Fourth Embodiment)
Hereinafter, a digital camera according to a fourth embodiment of the image processing apparatus of the present invention will be described. The configuration of the image processing circuit in the fourth embodiment is the same as the configuration of the third embodiment shown in FIG. However, in the present embodiment, the attention area designation unit 22 detects the subject to be focused on and automatically designates the area in which the subject exists.

FIG. 7 is a flowchart showing an operation of detecting a subject of interest and switching the gradation control method based on the distance to the subject of interest to remove fog haze.

When the flow of the fog haze removal processing is started, in step S71, the image acquired by the imaging unit 101 is input to the image processing unit 121. In step S72, the subject to be noticed is detected, and the area in which the subject exists is automatically specified. In order to detect the subject to be focused on, for example, the user specifies an object or a person to be watched. Then, using object detection, AI technology, etc., the object or person to be noticed is detected, and the area including them is specified. Alternatively, a moving object or person may be detected by using motion detection or the like, and an area including them may be specified.

In step S73, the distance information acquisition unit 23 measures the distance to the area including the designated subject of interest. The distance to the region of interest is measured using control information of a focus lens or a zoom lens arranged in the lens unit 102.

In step S74, the overall control calculation unit 107 determines whether or not the distance to the subject of interest is greater than the first predetermined distance. If the distance to the subject is longer than the first predetermined distance, the process proceeds to step S75. If the distance to the subject is not longer than the first predetermined distance, the process proceeds to step S76.

In step S75, the first gradation correction process described in the first embodiment, which is effective for removing the fog haze on a distant subject, is executed.

In step S76, it is determined whether or not the distance to the subject of interest is closer than the second predetermined distance. If the distance to the subject is closer than the second predetermined distance, the process proceeds to step S77. If the distance to the subject is not closer than the second predetermined distance, the process proceeds to step S78.

In step S77, the second gradation correction process described in the first embodiment, which is effective for removing the fog haze on a nearby subject, is executed. In step S78, the image obtained by the first gradation correction process described in the first embodiment and the image obtained by the second gradation correction process are mixed. This step S78 is executed when the distance to the subject of interest is equal to or less than the first predetermined distance and is equal to or greater than the second predetermined distance. After executing step S75, S77, or S78, the fog haze removal operation is terminated.

As described above, according to the fourth embodiment, it is possible to perform an effective fog haze removal process according to the detected distance of the subject of interest.

(Fifth Embodiment)
Hereinafter, a network camera according to a fifth embodiment of the image processing apparatus of the present invention will be described. As shown in FIG. 10, the configuration of the image processing circuit in the fifth embodiment includes a network camera, a network, and a client device.

In this fifth embodiment, the fog haze removal method is switched and executed according to the recognition / analysis application executed by the client device. In the present embodiment, a face identification application for identifying a person's face and a number of people counting application for counting a subject existing on the screen will be used as an example.

As a feature of each application, since the face identification application performs identification based on the detailed features of the face, it is necessary to suppress the occurrence of artifacts typified by the above-mentioned halo noise as much as possible. On the other hand, the number of people counting application is not a sum that detects based on a detailed structure, but detects a rougher structure and shape as compared with a face recognition application, and counts the number of people. Therefore, when removing fog haze when running the number counting app, prioritizing the fog haze removal effect rather than suppressing halo noise improves the subject detection accuracy. The recognition / analysis application is not limited to this, and may be, for example, an analysis application that heat-maps a person's movement line, an application that tracks a specific person, an application that analyzes the attributes of a subject, and the like.

FIG. 11 is a flowchart showing an operation of removing fog haze in the present embodiment.

When the flow of the fog haze removal processing is started, in step S111, the image acquired by the imaging unit 101 is input to the image processing unit 121. In step S112, the information of the recognition / analysis application executed by the client device is acquired.

In step S113, the overall control calculation unit 107 determines whether or not the acquired recognition / analysis application type is a face identification application. If it is a face recognition application, the process proceeds to step S114. If it is not a face recognition application, the process proceeds to step S115.

In step S114, the second gradation correction process described in the first embodiment, which is effective for removing fog haze, is executed while suppressing adverse effects.

In step S115, it is determined whether or not the acquired recognition / analysis application type is a number counting application. If it is a number counting application, the process proceeds to step S116. If it is neither a face recognition application nor a number counting application, the process proceeds to step S117.

In step S116, the first gradation correction process described in the first embodiment, which is effective for removing the fog haze on a distant subject, is executed.

In step S117, similarly to step S68 described in the third embodiment, the image obtained by the first gradation correction process and the image obtained by the second gradation correction process are mixed based on the edge information. Perform processing. After executing step S114, step S116, or step S117, the fog haze removal operation is terminated.

As described above, according to the fifth embodiment, it is possible to perform an effective fog haze removal process according to the recognition / analysis application executed by the client device.

(Other embodiments)
The present invention also supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads the program. It can also be realized by the processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

100: Digital camera, 101: Image pickup unit, 102: Lens section, 104: Image pickup signal processing circuit, 107: Overall control calculation unit, 120, 121: Image processing circuit

Claims (22)

Acquisition method for acquiring image data and
A first gradation correction means that acquires a dark channel value for each region of the image data, estimates the transmittance, and corrects the gradation.
A second gradation correction means for correcting the gradation of the image data using a tone curve, and
A control means for controlling which gradation correction means is used for gradation correction, and
An image processing device characterized by comprising. The control means causes one of the first gradation correction means and the second gradation correction means to perform the gradation correction of the first stage of the image data, and performs the gradation correction of the first stage. Based on the correction result of the image data after that, the image data after the first stage gradation correction is performed on the other of the first gradation correction means and the second gradation correction means. The image processing apparatus according to claim 1, wherein two-step gradation correction is performed. When the contrast of the image data after performing the gradation correction of the first stage is lower than a predetermined value, the control means is used on the other of the first gradation correction means and the second gradation correction means. The image processing apparatus according to claim 2, wherein the second-stage gradation correction is performed on the image data after the first-stage gradation correction is performed. The image processing apparatus according to claim 3, wherein the contrast is calculated based on a luminance histogram of the image data after the gradation correction of the first stage is performed. The control means causes the other of the first gradation correction means and the second gradation correction means to perform the second stage gradation correction until the contrast reaches a target value. The image processing apparatus according to claim 3 or 4. The image processing apparatus according to any one of claims 2 to 5, wherein the control means causes the first gradation correction means to perform gradation correction in the first stage. The image processing apparatus according to any one of claims 2 to 5, wherein the control means causes the second gradation correction means to perform gradation correction in the first stage. The control means causes the first gradation correction means to gradually perform gradation correction over time, and when a white region is extracted from the outline portion of the image data, the first gradation correction The image processing apparatus according to any one of claims 1 to 7, wherein the gradation correction operation of the means is stopped. A calculation means for calculating a dispersion value of a statistic of the brightness value of the image data is further provided, and when the calculated dispersion value is smaller than a predetermined value, the control means is a floor by the first gradation correction means. The image processing apparatus according to claim 1, wherein the gradation correction is strengthened and the gradation correction by the second gradation correction means is weakened. A calculation means for calculating the variance value of the statistic of the brightness value of the image data is further provided, and when the calculated dispersion value is equal to or more than a predetermined value, the control means is a floor by the first gradation correction means. The image processing apparatus according to claim 1, wherein the gradation correction is weakened and the gradation correction by the second gradation correction means is strengthened. The control means further includes a setting means for setting the subject of interest in the image data and a measuring means for measuring the distance to the subject of interest, and the control means has the first gradation based on the distance to the subject of interest. The image processing apparatus according to claim 1, wherein the gradation correction operation of the correction means and the gradation correction operation of the second gradation correction means are controlled. An analysis means for performing analysis using the image data is further provided, and based on the analysis result of the analysis means, the gradation correction operation of the first gradation correction means and the gradation correction operation of the second gradation correction means The image processing apparatus according to claim 1, wherein the gradation correction operation is controlled. A claim characterized in that the control means causes the first gradation correction means to perform gradation correction of the image data when the distance to the subject of interest is longer than the first predetermined distance. Item 11. The image processing apparatus according to item 11. A claim characterized in that the control means causes the second gradation correction means to perform gradation correction of the image data when the distance to the subject of interest is closer than a second predetermined distance. Item 11. The image processing apparatus according to item 11. When the distance to the subject of interest is equal to or less than the first predetermined distance and is equal to or greater than the second predetermined distance smaller than the first predetermined distance, the control means corrects the first gradation. The image processing apparatus according to claim 11, wherein the result of gradation correction of the image data by the means and the result of gradation correction of the image data by the second gradation correction means are combined. The image processing apparatus according to claim 15, further comprising an edge detecting means for detecting the edge information of the image data, and calculating the ratio of the composition based on the detected edge information. The image processing apparatus according to claim 11, further comprising a designation means for designating the subject of interest by the user. The image processing apparatus according to claim 11, further comprising a designating means for automatically designating the subject of interest. An imaging means that captures the subject,
The image processing apparatus according to any one of claims 1 to 18.
An imaging device characterized by comprising. The acquisition process to acquire image data and
The first gradation correction step of acquiring the dark channel value for each region of the image data, estimating the transmittance, and correcting the gradation,
A second gradation correction step of correcting the gradation of the image data using a tone curve, and
A control process that controls which gradation correction process is used to perform gradation correction,
An image processing method characterized by having. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 18. A computer-readable storage medium that stores a program for causing the computer to function as each means of the image processing apparatus according to any one of claims 1 to 18.

Images