JP2018037857A5 - Image processing apparatus, image processing method, computer program, and storage medium - Google Patents

Description

When imaging what small, depending on imaging conditions, since the depth of field becomes shallow, it may be difficult to focus on the entire object. For this reason, only a blurred image of a part of the subject can be obtained, and there are cases where the image is not satisfactory for the photographer. Alternatively, when taking a picture with a family member or a friend, if all the members do not fall within the depth of field, an image in a state where all the members are in focus cannot be obtained. For this reason, when trying to capture an image focused on everyone, the degree of freedom of arrangement of the person inevitably has been limited.

The present invention provides a contrast value calculating means for calculating a contrast value for each partial region of a plurality of images captured while changing the focus position of the optical system, and the contrast value for each partial region of the plurality of images. There is provided an image processing apparatus comprising: a combining unit configured to combine the plurality of images at a combining ratio set based on the in-focus position when the plurality of images are captured.

The control unit 101 is a computer such as a signal processor such as a CPU or MPU, and controls each part of the digital camera 100 while reading a computer program built in the ROM 105 described later in advance. For example, as will be described later, the control unit 101 instructs the imaging unit 104 described later to start and end imaging. Alternatively, an image processing command is issued to an image processing unit 107 described later based on a program built in the ROM 105. A user command is input to the digital camera 100 by an operation unit 110 described later, and reaches each part of the digital camera 100 through the control unit 101.

The ROM 105 is a read-only non-volatile memory as a recording medium that stores a computer program that can be read by a computer, and stores parameters necessary for the operation of each block in addition to the operation program of each block provided in the digital camera 100. doing. The RAM 106 is a rewritable volatile memory, and is used as a temporary storage area for data output in the operation of each block included in the digital camera 100.

The built-in memory 109 is a place for recording an image captured by the image capturing unit 104, an image obtained by the processing of the image processing unit 107, information on a focus position of the optical system at the time of image capturing, and the like. A memory card or the like may be used instead of the built-in memory.

In step S201, the digital camera 100 captures a plurality of images while automatically changing the in-focus position by a predetermined amount in accordance with an imaging instruction from the user. The imaging unit 104 captures an image of the subject via the optical system 103, saves the image captured by the control unit 101 in the built-in memory 109, and acquires information on the in-focus position of the optical system 103 when each image is captured. For example , the image data is stored in the built-in memory 109 in a state of being included in each image file as metadata . Or you may memorize | store said focusing position information in a file different from each image file. In that case, information indicating which image file corresponds to each in-focus position information is also stored together. It should be noted that the user may manually turn the focus ring so that images are taken a plurality of times while changing the focus position. In this embodiment, the driving unit 102 changes the in-focus position by driving a lens provided in the optical system 103. However, the present invention is not limited to this, and for example, a plurality of digital cameras are used, and each digital camera is different. You may make it image-capture in the state focused on the position. However, in this case, it is necessary to devise so that the optical axes of the respective digital cameras are close to each other so that the parallax between images obtained by a plurality of digital cameras does not increase.

In step S204, the image processing unit 107 calculates a contrast value for a plurality of captured images. As an example of the method for calculating the contrast value, an edge detection filter is applied to the output of the pixel as the partial region , and the result is used as the contrast value. As the edge detection filter, for example, a band pass filter, a Sobel filter, a Laplacian filter, or the like is used.

3 (b) shows the arrangement order of the image, the focus position of the optical system 103 when the captured each image. FIGS. 3C to 3E are enlarged views of the area 303 in FIG. 3A, and each shows a case where the portion 301 is focused, the portion 303 is focused, and the both are combined. It is. FIG. 4A is a diagram showing the contrast value of any one pixel included in the portion 304. The corresponding pixels included in the images in the first to eighth order are indicated by the pixels 411 to 418, respectively. As shown. In FIG. 4A , the contrast value of the pixel 412 included in the portion 304 in the image captured by the digital camera 100 in focus at the in-focus position 312 in FIG. Indicates that it will be the highest. Therefore, the image processing unit 107 sets the composition ratio of the pixels 412 included in the second image in the arrangement order to 100% at the position of the portion 304. At the same time, the image processing unit 107 sets the combination ratio of the first pixel 411 in the arrangement order and the corresponding pixels 413 to 418 of the third to eighth images to 0%. The image processing unit 107 performs such a composition ratio setting for each corresponding pixel of each image. Note that if the composition ratio suddenly changes from 0% to 100% (or changes from 100% to 0%) between adjacent pixels, unnaturalness at the composition boundary becomes conspicuous. Therefore, the image processing unit 107 applies a filter having a predetermined number of pixels (number of taps) to the composite map so that the composite ratio changes stepwise between adjacent pixels.

Further, in step S206, the control unit 101 determines a pixel for which the synthesis ratio is adjusted. FIG. 5 is a flowchart for explaining the synthesis ratio adjustment processing. First, in step S501, as shown in FIG. 4 (a) ~ (c) , the control unit 101, the order of images, from the relationship between the contrast value of the corresponding pixel of each image, the maximum value ( ) Is detected. In the case of FIG. 4A, the contrast value in the corresponding pixel of the image in the second order is the maximum value. In the case of FIG. 4B, the contrast value in the corresponding pixel of the images in the second and seventh order is the maximum value. In the case of FIG. 4C, the contrast values in the corresponding pixels of the second, fourth, and seventh images in the arrangement order are maximum values.

The reason for the above processing will be briefly described with reference to FIG. For example, as shown in FIG. 3A, a subject 301 and a subject 302 exist, FIG. 3C shows an image captured when focusing on the subject 301, and FIG. An image taken when focusing is shown. However, it is assumed that the subject 321 in FIG. 3C and the subject 331 in FIG. 3D are a part (head) of the subject 301, and when the subject 301 is focused, the subject 321 is also focused. Similarly, it is assumed that the subject 322 in FIG. 3C and the subject 322 in FIG. 3D are a part of the subject 302 and that when the subject 302 is focused, the subject 322 is also focused. When focusing on the subject 301, the contrast value of the region corresponding to the subject 321 increases, and when focusing on the subject 302, the contrast value of the region corresponding to the subject 322 increases. Contrast value of the pixels of the subject 301 and the subject 302 and the overlap region 323, when changing the focus position from the focus position 311 in FIG. 3 (b) 318, at two positions of the focus position 312 and 317 Maximum value. When the relationship between the arrangement order based on the focus position and the contrast value is graphed, it has two local maximum values. On the other hand, in the pixel of the region 324 where only one subject exists, two or more local maximum values do not exist. Therefore, in the above-described processing, when there are two or more maximum values, the control unit 201 determines that the pixel needs to be adjusted in the synthesis ratio.

In step S207, the image processing unit 107 corrects the composite map. Here, in step S206, the composition ratio of the pixel determined to be necessary to adjust the composition ratio determined by the control unit 101 is changed. Specifically, the image processing unit 107 increases the combination ratio of the pixels of the closest image having the maximum value with respect to the pixels determined to need adjustment of the combination ratio in step S206. For example, in the case of FIG. 4B, the composition ratio of the second pixel in the arrangement order is set to 100%. Thus, as shown in FIG. 3E, the composition ratio of the subject 321 can be set to 100%, and the subject 322 behind can be completely hidden. Alternatively, although the composition ratio of the subject 321 is increased, the composition ratio of the pixels at the edge portion is not limited to 100% and is slightly suppressed, so that the image after composition can be made a blurred image.

When the processing in step S207 is completed, in step S208, the control unit 101 determines whether or not the composite map correction processing for all the pixels has been completed. If not completed, the image processing unit 107 receives the next pixel from the internal memory 109, the processing from step S206. If completed, the process enters step S210, and the images are combined based on the combination map. Also in this case, similarly to step S205 described above, the image processing unit 107 applies a filter having a predetermined number of pixels to the composite map corrected in step S207 as necessary, and combines the adjacent pixels with a composite ratio. To gradually change.

According to the first embodiment, when synthesizing using a plurality of images picked up by changing the in-focus position, by adjusting the composition ratio between the plurality of images of pixels whose positions overlap at the time of composition, A composite image that correctly reflects the positional relationship before and after can be generated. Here, the configuration for obtaining the contrast value and the composition ratio for each pixel has been described as an example. However, in order to reduce the processing load, the contrast value and the composition ratio may be obtained for each block composed of a plurality of pixels. . In other words, as long as the contrast value and the composition ratio are obtained for each of the plurality of partial areas included in the image, the size of the area can be set to an arbitrary size in consideration of processing load and image quality. It is.

When focusing on one of the front and rear subjects in a pixel that requires adjustment of the composition ratio, the other degree of blur increases as the distance between the front and rear subjects increases. Therefore, the contrast value calculated in step S601 is corrected in consideration of the in-focus position. As an example, a correction coefficient is applied to the contrast value of the pixel of the closest image, and the result is used as a corrected contrast value. In FIG. 7C, the correction value W_near1 is applied to the contrast value C_near1 of the pixel having the second number in the arrangement order to obtain a corrected contrast value C_near1 ′. Similarly, in FIG. 7D, a correction coefficient W_near2 is applied to the contrast value C_near2 of the pixel having the number 5 in the arrangement order to obtain a corrected contrast value C_near2 ′. In FIGS. 7 (c) and (d), the seventh image in the rearrangement order is focused in the rear, but FIG. 7 (c) is a front subject and a rear subject as compared with FIG. 7 (d). The distance to is large. In FIG. 7C, the degree of blurring of the second image in the arrangement order is larger, and the front-rear position of the subject in the composite image is likely to be disturbed. Therefore, it is necessary to increase the correction coefficient. That is, the control unit 101 sets the correction coefficient W_near1 to be larger than the correction coefficient W_near2. Specifically, in FIG. 7C and FIG. 7D, a correction coefficient larger than 1 is given according to the distance from the most infinite pixel among the corresponding pixels having the maximum contrast value. . On the other hand, among the corresponding pixels having the maximum contrast value, 1 is given as a correction coefficient to the most infinite pixel, or no correction is performed.

The reason for the above determination will be described. The in-focus position of the pixel having the local maximum value 702 on the closest side is set to have a larger distance from the rear subject and the correction coefficient described above sufficiently larger than those of other pixels having the local maximum value. As long as a sufficiently large correction coefficient is applied to the contrast value 702 of the corresponding pixel of the front subject image, the contrast value 703 of the corresponding pixel of the image when focusing on the rear subject should be larger. Nevertheless, if the corrected value 712 of the contrast value 702 of the corresponding pixel of the front subject image may be smaller than the contrast value 703 of the corresponding pixel of the rear image, the value due to the influence of noise or the like. it is conceivable that. That is, the contrast value 702 is not focused on the front subject in the first place, and it is considered that the contrast value 802 is the maximum value due to the influence of noise or the like. Therefore, in such a case, adjustment processing of the composition ratio of the pixel being processed is not performed.

(Other embodiments)
The above embodiment has been described based on the implementation with a digital camera, but is not limited to a digital camera. For example, it may be implemented by a portable device such as a smartphone or a tablet with a built-in image sensor , or a network camera that can capture an image.

Claims (13)

A calculation means of the contrast values for calculating a contrast value with the respective partial regions of a plurality of images captured while changing the focus position of the optical system,
Before Kiko contrast value for each of the partial areas of the previous SL plurality of images, and combining means for combining the plurality of images in combination ratio is set based on the focus position at the time of imaging the plurality of images an image processing apparatus characterized by having a.   The synthesizing unit compares the contrast values of the plurality of images for each of the regions, and makes the composition ratio of the image having the highest contrast value among the plurality of images highest for each of the regions. The image processing apparatus according to claim 1. The synthesizing unit has two or more local maximum values of which the contrast value is a predetermined value or more when the contrast values of the plurality of images are arranged in the order of the in-focus position of the optical system for each region. 3. The image processing apparatus according to claim 1, wherein when the image processing is performed, the composition ratio of any one of the plurality of images corresponding to the maximum value is set to be the highest. The combining unit includes a plurality of contrast values corresponding to a maximum value equal to or greater than a predetermined value when the contrast values of the plurality of images are arranged in the order of the in-focus positions of the optical system for each region. of the image, the image processing apparatus according to claim 3, wherein, characterized by the highest setting the mixing ratio of the image positioned at the most front side. The combining means, when the contrast value is the maximum value of the predetermined value or more there are two or more, in claim 4, characterized in that the highest setting the mixing ratio of the image positioned at the most front side The image processing apparatus described. The combining means, the mixing ratio of the images of the plurality of images sac lichen Zureka as 1, any one of claims 1 to 5 the mixing ratio of the other image, characterized in that the 0 An image processing apparatus according to 1. The synthesizing unit corrects one or more maximum values using a correction coefficient when there are two or more maximum values with the contrast value equal to or greater than a predetermined value, and based on the corrected maximum values , have an image processing apparatus according to claim 3, characterized in that the highest setting the mixing ratio of the image Zureka. The combining means, the maximum value after correction of the image positioned at the most front side is greater than the other maximum value is the highest setting the mixing ratio of the previous SL image positioned at the most front side The image processing apparatus according to claim 7. The synthesizing unit sets the correction coefficient based on the in-focus position of an image corresponding to each local maximum value when there are two or more local maximum values with the contrast value equal to or greater than a predetermined value. The image processing apparatus according to claim 7 or 8. The combining means, when the maximum value of the contrast values exist two or more, according to claim 9, the correction factor for the maximum value of the image positioned at the most front side, characterized in that the greatest Image processing device. A calculation step of the contrast values for calculating a contrast value with the respective partial regions of a plurality of images captured while changing the focus position of the optical system,
Before Kiko contrast value for each of the partial areas of the previous SL plurality of images, a synthesizing step of synthesizing the plurality of images in combination ratio is set based on the focus position at the time of imaging the plurality of images an image processing method characterized by having a. A calculation step of the contrast values for calculating a contrast value with the respective partial regions of a plurality of images captured while changing the focus position of the optical system,
Before Kiko contrast value for each of the partial areas of the previous SL plurality of images, a synthesizing step of synthesizing the plurality of images in combination ratio is set based on the focus position at the time of imaging the plurality of images Is executed by a computer. A computer-readable storage medium storing the computer program according to claim 12.