JP2006279546A - Electronic camera, image processing program, and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technique for acquiring information related to the depth of an object field with a simple structure. <P>SOLUTION: An electronic camera includes imaging section, image processing section, imaging control section, deviation detection section and distance map generation section. The imaging section generates an image by imaging an object. The image processing section processes the image imaged by the imaging section. The imaging control section acquires a plurality of images by imaging for a plurality of times by means of the imaging section. The deviation detection section detects a local image deviation width in regard to a plurality of points in a screen by comparing the plurality of images. The distance map generation section groups the inside of the screen based on the image deviation widths of the plurality of points, and generates distance information representing object distance differences in the screen. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an electronic camera, an image processing program, and an image processing method that generate a distance map representing a difference in subject distance within a screen.

(Conventional technology related to camera ranging)
Patent Document 1 below describes a passive ranging method. In this passive ranging method, two imaging devices are arranged apart from each other by a predetermined baseline length. The parallax is obtained for a set of optical images generated by these two imaging devices, and the distance is obtained from the parallax.
Patent Document 2 below also describes a distance measuring technique using a contrast hill climbing method. In this system, the focus position of the photographic lens is scanned and moved in the optical axis direction by driving the photographic lens to focus. In parallel with the focus scan, an image area where the contrast of the captured image is maximum is searched.

(Conventional technology related to continuous shot image composition)
2. Description of the Related Art Conventionally, there has been known a technique for preventing image blurring that occurs in long-time exposure by generating a plurality of images by divided exposure and aligning and synthesizing the plurality of images.
There is also known a technique for preventing gradation skipping and gradation collapse by photographing a plurality of images while changing exposure conditions, and aligning and synthesizing the plurality of images.

JP-A-8-178737 JP-A-9-145318

  The present inventor considered realizing advanced image processing reflecting depth information (perspective difference) of the object scene by adjusting image processing for each area in the screen according to the distance to the subject. .

  Therefore, we examined a technology that easily detects the depth over the entire field without having a special structure. In the passive distance measuring method described above, it is necessary to dispose two or more imaging devices, which causes a problem that the structure becomes complicated. In addition, the above-described contrast climbing method does not require a special structure. However, in order to detect the depth over the entire field, it is necessary to scan the focus position of the taking lens many times, and there is a problem that it takes time to measure the distance.

  By the way, as described above, there is known a technique for suppressing image blurring and gradation collapse by combining a plurality of images. The present inventor has considered to detect depth information by applying such processing of a plurality of images.

  Accordingly, an object of the present invention is to provide a technique for obtaining information related to the depth of the object scene with a simple configuration.

<< 1 >>
The electronic camera of the present invention includes an imaging unit, an image processing unit, an imaging control unit, a deviation detection unit, and a distance map creation unit.
The imaging unit captures a subject and generates an image.
The image processing unit processes an image captured by the imaging unit.
The imaging control unit acquires a plurality of images by performing imaging a plurality of times with the imaging unit.
The deviation detection unit compares a plurality of images and detects a local image deviation width at a plurality of locations in the screen.
The distance map creation unit divides the screen into groups based on the image misalignment widths at a plurality of locations, and creates distance information (hereinafter referred to as “distance map”) representing the difference in subject distance within the screen.

<< 2 >>
Preferably, the deviation detection unit includes a distance acquisition unit that detects the subject distance A for at least one location P in the screen.
On the other hand, the distance map creation unit uses the subject distance B at the location in the screen with the image displacement width m as the reference and the image displacement width n at the location P as a reference.
B = A · n / m (1)
To obtain a distance map representing the difference in subject distance within the screen.

<< 3 >>
Preferably, the image processing unit changes the image processing applied to the image using the distance map as a mask.

<< 4 >>
Preferably, the image processing unit corrects the brightness more brightly as the screen area is far away using the distance map as a mask.

<< 5 >>
Preferably, the image processing unit corrects the brightness darker as the screen area is farther away, using the distance map as a mask.

<< 6 >>
Preferably, the image processing unit has a function of generating a composite image by aligning a plurality of images generated by the imaging control unit.

<< 7 >>
The image processing program according to the present invention provides a computer to which a plurality of images generated by performing imaging a plurality of times by an imaging unit are input, the image processing unit according to any one of the above << 1 >> to << 6 >>, and deviation detection And an image processing program for functioning as a distance map creation unit.

<< 8 >>
According to the image processing method of the present invention, a step of acquiring a plurality of images generated by photographing a plurality of times by an imaging unit and a plurality of images are compared to detect a local image shift width at a plurality of locations in the screen. And a step of grouping the screen based on the image shift widths at a plurality of locations and creating distance information (hereinafter referred to as “distance map”) representing the difference in subject distance within the screen.

[1, 7, 8]
In the present invention, first, a plurality of images generated by a plurality of shootings are acquired. In hand-held shooting, image misalignment due to camera shake or the like occurs in these multiple images. In the present invention, the width of the image shift is detected at a plurality of locations in the screen by image comparison.
When the camera shake includes a parallel movement orthogonal to the photographing optical axis, the image shift width increases as the subject distance is shorter. Therefore, in the present invention, the distance information (hereinafter referred to as “distance map”) indicating the difference in subject distance in the screen is created by grouping the image shift widths at a plurality of locations.
By such processing, in the present invention, information relating to the depth of the object scene can be obtained from a plurality of images obtained by a plurality of shootings without requiring a particularly complicated configuration.

[2]
Furthermore, in the present invention, it is preferable to separately detect the subject distance A for at least one place P of the plurality of places. With this subject distance A as a reference, it is possible to roughly estimate the distances of other parts.
That is, if the image deviation width of this location P is n and the subject distance A, the subject distance B at the location in the screen with the image deviation width m is as follows.
B = A · n / m (1)
It becomes possible to ask for.

[3]
In the present invention, it is preferable to change the image processing by using the distance map as a binary or multi-value mask. By this processing, it is possible to realize image processing reflecting depth information (perspective difference) of the object scene.

[4]
In the image processing using this distance map, the luminance correction may be performed brighter as the image area is farther away. In this case, it is possible to correct the object scene that becomes darker at a distance, such as during flash photography, to a natural luminance distribution.

[5]
Further, in the image processing using this distance map, the luminance correction may be made darker as the image area is farther away. In this case, by making the background dark, brightness correction is realized such that the main subject on the near side is brightly three-dimensionally raised.

[6]
Further, the image processing unit may generate a composite image by aligning a plurality of images.
In this case, image processing using a distance map may be performed for each of a plurality of images, and the images may be combined. Moreover, after generating a composite image, image processing using a distance map may be performed on the composite image.

[Description of Embodiment Configuration]
FIG. 1 is a block diagram showing an electronic camera 11 of the present embodiment.
In FIG. 1, a photographing lens 12 is attached to the electronic camera 11. The lens control unit 12a performs focus driving, aperture control, and the like of the photographing lens 12. In the image space of the photographic lens 12, the light receiving surface of the image sensor 13 is arranged. The imaging element 13 is driven by the imaging control unit 14. Image data output from the image sensor 13 is processed through the signal processing unit 15 and the A / D conversion unit 16 and then temporarily stored in the memory 17 as digital image data.

This memory 17 is connected to a bus 18. The bus 18 is connected to a lens control unit 12a, an imaging control unit 14, a microprocessor 19, an image processing unit 20, a displacement detection unit 21, a recording unit 22, a monitor display unit 23, and a built-in projector 30.
The microprocessor 19 is connected to an operation unit 19a such as a release button. The recording unit 22 is loaded with a recording medium 22a. The built-in projector 30 includes a projection lens 31, a display element 32, and an illumination light source 33, and projects image data onto an external screen or the like. The display element 32 is controlled by a display element driving circuit 34. The illumination light source 33 is driven to be lit by the lighting circuit 35.

[Description of Operation of Embodiment]
FIG. 2 is a flowchart for explaining the operation of the present embodiment. Hereinafter, the operation will be described along the step numbers shown in FIG.

Step S1: When the main power supply of the electronic camera 11 is turned on, the imaging control unit 14 continuously drives the imaging device 13 in a thinning readout mode (draft mode) to capture a moving image for monitoring. The monitor moving image is displayed on the monitor display unit 23. Further, the lens control unit 12a performs contrast hill-climbing focus control based on the monitor moving image, and focuses on a point P in the screen. This location P can be selected by a user operation or the like from a central location or a plurality of surrounding locations.
On the other hand, the microprocessor 19 detects the subject brightness from the moving image and determines the exposure conditions (exposure time, aperture value, imaging sensitivity). Note that a photometric element may be provided separately to perform photometric operation and determine the exposure conditions.

Step S2: The microprocessor 19 acquires information on the focal length and lens position of the taking lens 12 from the lens control unit 12a.

Step S3: The lens controller 12a holds in advance the correspondence between the lens position of the taking lens 12 and the subject distance. The microprocessor 19 refers to the lens position of the photographic lens 12 in this correspondence relationship to obtain the subject distance A to the in-focus location P.

Step S4: The microprocessor 19 obtains a divided exposure time in which the influence of camera shake can be ignored from the focal length of the photographing lens 12 and the subject distance A. This divided exposure time is preferably determined in advance for each condition of focal length and subject distance based on a live-action test or the like.

Step S5: The microprocessor 19 obtains the necessary number of divided exposures by dividing the exposure time determined from the subject luminance into divided exposure times. Thereby, the time schedule of divided exposure is determined.

Step S6: The microprocessor 19 controls the imaging control unit 14 and repeats the imaging operation according to the time schedule of the divided exposure to generate a plurality of images. These multiple images are temporarily stored in the memory 17. The division exposure operation here may be realized by a mechanical shutter (not shown) or by electronic shutter control of the image sensor 13.

Step S7: The deviation detecting unit 21 reads at least two images from the memory 17. It should be noted that it is preferable to select images corresponding to both ends of camera shake vibration as the two images. For example, when picking up a camera shake when pressing the release button, an image of the start of pressing the release button and an image at the time when the downward vibration of the electronic camera 11 stops after the release button is pressed are based on a time interval or the like. You can choose. Alternatively, a vibration sensor (such as an angular velocity sensor) may be mounted on the electronic camera 11 and images corresponding to both ends of the vibration may be selected from the output of the vibration sensor.
The deviation detection unit 21 subdivides one of these images into blocks having a predetermined pixel size. The misalignment detection unit 21 searches the other image for a position where these blocks and the image pattern match well (so-called block matching). By this processing, the image shift for each block can be detected as a vector.

Step S8: The deviation detection unit 21 obtains a common image deviation direction for the image deviation of each block. For example, an average image shift direction may be obtained by vector addition of the image shift of each block. Further, for example, the direction of image shift of each block may be frequency-analyzed, and the most frequent direction may be set as the image shift direction. Further, for example, an image shift with the maximum width may be obtained, and the direction of the image shift may be set as a common image shift direction. In addition, for example, the pressing direction of the release button and the vibration direction of the mirror shock may be set as a common image shift direction.

Step S9: The deviation detection unit 21 calculates the image deviation width in the common image deviation direction for the image deviation of each block.

Step S10: The microprocessor 19 acquires information about the image shift width of each block from the shift detection unit 21. The microprocessor 19 selects from the image shift widths based on the image shift width n at the in-focus position P.

Step S11: The microprocessor 19 calculates the subject distance B for each block by substituting the image shift width m of each block into the following equation.
B = A · n / m (1)
FIG. 3 is a schematic diagram for explaining the meaning of the above equation.
In FIG. 3, a reference subject (reference subject) is located at a position away from the front principal point of the taking lens 12 by a subject distance A in the optical axis direction. Similarly, a subject to be measured (a target for distance measurement) is located at a position separated by the subject distance B.

Here, it is assumed that the position of the electronic camera 11 is displaced by an interval R due to camera shake or the like. In this case, when viewed from the coordinates of the electronic camera 11, both the reference subject and the distance measurement object are relatively moved by the interval R in the object space.
By the way, if the shooting angle of view of the electronic camera 11 is 2θ and the diameter of the light beam due to the entrance pupil is ignored, the shooting width in the object space that fits on the imaging screen is 2Atanθ at the position of the subject distance A. The photographing width at the subject distance B is 2 Btan θ. The shooting width in the object space that changes depending on the subject distance is projected on the shooting screen at an equal width.
From this, the ratio between the image shift width n of the reference subject and the image shift width m of the distance measurement object on the shooting screen is
n: m = (R / 2 Atan θ): (R / 2 Btan θ)
It becomes. By transforming this equation, the above equation (1) is obtained.

Step S12: The microprocessor 19 generates distance map data in the memory 17 by classifying each block according to the image shift width (or subject distance). FIG. 4 is a diagram schematically showing the distance map in an easily understandable manner.
The distance map may be data representing a difference in subject distance. For example, it may be data in which numerical values (for example, numerical values indicating subject distance and perspective) are arranged in block units. Also, a numerical array of pixel units obtained by interpolating numerical values of block units may be used. Moreover, the data which show the area division of the partial image classified by the perspective may be sufficient.

Step S13: The image processing unit 20 aligns and composites a plurality of images obtained by the divided exposure to obtain a composite image. For the alignment here, the information on the image shift obtained in step S7 can be used.

Step S14: Here, the microprocessor 19 branches the process according to the setting of the image processing mode.
For example, in the “bright background mode”, the process branches to step S15.
For example, in the “dark background mode”, the process branches to step S16.
Note that the branching of steps S15 and S16 may be automatically selected based on “image analysis”, “division metering evaluation result”, and “presence / absence of flash emission”. For example, it is preferable to automatically select the dark background mode when it is determined that the backlight photography is performed by image analysis or divided photometry. In addition, when there is flash light emission, it is preferable to automatically select a bright background mode.

Step S15: Here, a bright background mode is selected. In this case, the image processing unit 20 uses the distance map as a binary or multi-value mask to perform brightness correction on the composite image so that the image area with a longer subject distance becomes brighter.
The mask process here can be performed in various ways. For example, the image processing parameters to be applied to the image may be changed (modulated) by the distance information of the distance map or the perspective difference. In addition, the image may be divided into a plurality of divided regions according to the perspective classification of the distance map, and the image processing may be changed for each divided region.
The image for which the brightness correction has been completed in this manner is stored and recorded in 22a via the recording unit 22. Further, the image for which the brightness correction has been completed can be projected onto an external screen or the like via the built-in projector 30. Further, a multi-tone projection image may be obtained by modulating the illumination distribution of the illumination light source 33 using the distance map. With the above operation, the operation of the present embodiment is completed.

Step S16: On the other hand, here, the dark background mode is selected. In this case, the image processing unit 20 uses the distance map as a binary or multi-value mask, and performs luminance correction on the composite image so that the image area with a longer subject distance becomes darker. The image for which the brightness correction has been completed is stored and recorded in 22a via the recording unit 22. Further, the image for which the brightness correction has been completed can be projected onto an external screen or the like via the built-in projector 30. Further, a multi-tone projection image may be obtained by modulating the illumination distribution of the illumination light source 33 using the distance map. With the above operation, the operation of the present embodiment is completed.

[Effects of this embodiment, etc.]
As described above, in this embodiment, a distance map can be created using an imaging unit of a normal electronic camera. Therefore, it is not necessary to provide a plurality of imaging devices as in the prior art, and information regarding the depth of the object scene can be obtained with a simple structure.
Further, in the present embodiment, by separately detecting the subject distance A for one place P, the subject distance B at other places can be obtained.

  Furthermore, in the present embodiment, by performing image processing while reflecting the distance map, it is possible to add a sense of depth of the object scene to the originally flat image. Conversely, it is also possible to correct image nonuniformity (difference between the background and background during flash photography) caused by the depth of the object scene.

[Supplementary items of the embodiment]
In the above-described embodiment, the distance map is created in combination with the image blurring improvement technique based on alignment synthesis. Therefore, it is not necessary to perform multiple exposures only for distance measurement, and there is no waste in the shooting sequence.

  Further, a distance map may be created by using the present invention in combination with a technique that suppresses gradation collapse by image synthesis.

  It should be noted that the present invention can be implemented independently by separately performing multiple exposures for distance measurement.

  In the above-described embodiment, it is preferable that the depth of field is increased by changing the aperture of the aperture to be small when performing multiple exposures. In this case, since the image shift can be accurately detected in a wider distance range, an accurate distance map over the entire object field can be obtained.

  In the embodiment described above, the distance map is used for image processing. However, the present invention is not limited to this. The present invention may be implemented for the purpose of generating a distance map. In this case, the present invention can be applied to a distance measuring device or the like.

  In the above-described embodiment, the distance map may be recorded as a file integrated with the image data or as a separate file. In this case, since the user can appropriately use the distance map as a mask, it is possible to save the trouble of generating the mask by a troublesome work from the image to be processed.

  In the embodiment described above, the distance map is used for luminance correction. However, the present invention is not limited to this. In general, any image processing that can utilize depth information may be used. For example, by reflecting the distance map in the saturation adjustment, it is possible to correct the desaturation of the distant view due to the air layer and obtain an image with good color loss. Conversely, by reducing the saturation of the distant view (or increasing the saturation of the close view), it is possible to obtain a so-called airy image.

  In the above-described embodiment, the case where the subject distance A is obtained from the lens position has been described. However, the present invention is not limited to this. Other known distance measuring means can be used as appropriate as means for obtaining the subject distance A as a reference.

  In addition, since the present invention can be realized if there is at least one image sensor, it can be implemented by any single-lens reflex type or compact type electronic camera.

If information on multiple images and subject distance A is acquired, the same processing as in the embodiment (FIG. 2) can be performed on the computer (image processing program).
Further, the above-described image processing method can be provided using an image server (eg, album server) on the Internet.

  As described above, the present invention is a technique that can be used for image processing and the like.

1 is a block diagram showing an electronic camera 11. FIG. It is a flowchart explaining operation | movement of embodiment. It is a schematic diagram explaining the calculation principle of a subject distance. It is the figure which showed the distance map typically clearly.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Electronic camera, 12 ... Shooting lens, 12a ... Lens control part, 13 ... Imaging device, 14 ... Imaging control part, 15 ... Signal processing part, 16 ... A / D conversion part, 17 ... Memory, 18 ... Bus, 19 ... Microprocessor, 19a. Operation unit, 20 ... Image processing unit, 21 ... Displacement detection unit, 22 ... Recording unit, 22a ... Recording medium, 23 ... Monitor display unit, 30 ... Built-in projector, 31 ... Projection lens, 32 ... Display Element 33 ... Illumination light source 34 ... Display element drive circuit 35 ... Lighting circuit

Claims (8)

An imaging unit that images a subject and generates an image;
An image processing unit for processing an image captured by the imaging unit;
An imaging control unit that acquires a plurality of images by performing imaging a plurality of times with the imaging unit;
A displacement detection unit that compares the plurality of images and detects a local image displacement width for a plurality of locations in the screen;
A distance map creation unit that divides the screen into groups based on the image shift widths at a plurality of locations and creates distance information (hereinafter referred to as a “distance map”) that indicates a difference in subject distance within the screen. And an electronic camera. The electronic camera according to claim 1,
The deviation detection unit includes a distance acquisition unit that acquires or detects the subject distance A for at least one location P in the screen,
The distance map creation unit
The subject distance B at the location in the screen with the image misalignment width m is based on the image misalignment width n at the location P.
B = A · n / m (1)
An electronic camera characterized by creating a distance map that represents the difference in subject distance within the screen. The electronic camera according to claim 1 or 2,
The image processing unit
An electronic camera, wherein the image processing applied to the image is changed using the distance map as a mask. The electronic camera according to claim 3.
The image processing unit
An electronic camera, wherein the distance map is used as a mask and brightness correction is performed brighter for a screen area at a longer distance. The electronic camera according to claim 3.
The image processing unit
An electronic camera, wherein the distance map is used as a mask and brightness correction is made darker as the screen area is far away. The electronic camera according to any one of claims 3 to 5,
The image processing unit
An electronic camera having a function of aligning a plurality of images generated by the imaging control unit and generating a composite image. A computer to which a plurality of images generated by photographing a plurality of times with an imaging unit are input,
The image processing program for functioning as the said image processing part of any one of Claim 1 thru | or 6, the said shift | offset | difference detection part, and the said distance map preparation part. Acquiring a plurality of images generated by shooting a plurality of times in the imaging unit;
Comparing the plurality of images and detecting a local image shift width for a plurality of locations in the screen;
A step of grouping the screen based on the image misalignment widths at a plurality of locations and creating distance information (hereinafter referred to as “distance map”) indicating a difference in subject distance within the screen. Processing method.

Images