JP2013232751A - Imaging apparatus, image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To acquire an arbitrary focus image from a captured image.SOLUTION: The imaging apparatus comprises: an imaging unit for capturing an image of subjects; an extraction unit which extracts an image area corresponding to a subject selected by a user from the captured image; a replacing unit which acquires information representing a subject distance from each subject included in the captured image to the imaging unit, and generates image data in which the subject distance from the selected subject is replaced with a subject distance designated by the user; and a smoothing unit for smoothing the captured image on the basis of the subject distance corresponding to each subject, with respect to the image data generated by the replacing unit.

Description

  The present invention relates to an imaging device, an image processing device, an image processing method, and a program.

2. Description of the Related Art Conventionally, there is a technique for processing an image data and then refocusing it after imaging a subject with a camera. This technique divides the surface of a lens into a plurality of viewpoints and generates one image for each viewpoint. And the parallax which the image for every viewpoint has is correct | amended, and it recombines (for example, refer nonpatent literature 1).
Non-Patent Document 1 Light Field Photography with a Hand-held Plenoptic Camera, Ren Ng, et. Al., Stanford Tech Report CTSR 2005-02

  However, with the conventional method, a pseudo-focused image in which an arbitrary subject is in focus can be obtained, but the subject distance of the subject cannot be changed.

  In the first aspect of the present invention, an extraction unit that extracts an image area corresponding to a selected subject selected by a user from a captured image, and a subject distance to the imaging unit for each subject included in the captured image is shown. Information is obtained, a replacement unit that generates image data in which the subject distance of the selected subject is replaced with the subject distance specified by the user, and the subject distance corresponding to each subject with respect to the image data generated by the replacement unit Based on this, an imaging apparatus is provided that includes a smoothing processing unit that smoothes the captured image.

  In the second aspect of the present invention, an extraction unit that extracts an image region corresponding to the selected subject selected by the user from the captured image of the subject captured by the imaging device, and each subject included in the captured image is captured. Information indicating the subject distance to the image, and a replacement unit that generates image data in which the subject distance of the selected subject is replaced with the subject distance specified by the user, and the image data generated by the replacement unit An image processing apparatus is provided that includes a smoothing unit that smoothes a captured image based on a subject distance corresponding to a subject.

  In the third aspect of the present invention, an extraction stage for extracting an image region corresponding to the selected subject selected by the user from the captured image, and the subject distance to the imaging device for each subject included in the captured image are shown. Information is obtained, and the subject distance of the selected subject is replaced with the subject distance specified by the user, and the subject distance corresponding to each subject is generated for the image data generated in the replacement step. And a smoothing step for smoothing the captured image.

  In a fourth aspect of the present invention, a program for causing a computer to function as the image processing apparatus is provided.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

1 is a block diagram of an imaging apparatus 100 according to an embodiment. It is a flowchart of the image processing method concerning embodiment. It is a flowchart of the image processing method of another example. It is a graph which shows the relationship between a relative distance and the blurring amount. 3 is an example of a distance information image 200 generated by the imaging apparatus 100. The example of the pan focus image imaged with the imaging part 10 is shown. An example of a pseudo-focused image after smoothing processing by the image processing apparatus 11 is shown. An embodiment of the distance measuring unit 12 is shown. The example of the arrangement | sequence of the image pick-up element 30 of the distance measurement part 12 of FIG. 8 is shown. 6 is a graph showing the relationship between the position of the image sensor 30 and the amount of light signal. Another embodiment of the distance measurement part 12 is shown. 6 shows a timing chart of the operation of the TOF image sensor 60. Another embodiment of the distance measurement part 12 is shown. An example of the optical system 80 is shown. It is a block diagram of the computer 1900 concerning embodiment.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 is a block diagram of an imaging apparatus 100 according to an embodiment of the present invention. The imaging device 100 includes an imaging unit 10, a distance measurement unit 12, a memory 14, an image generation unit 16, a display unit 18, an extraction unit 20, a replacement unit 21, a calculation unit 22, and a smoothing processing unit 24. A user interface 25, a main memory 26, a user interface 27, and an output interface 28. The image generation unit 16, the extraction unit 20, the replacement unit 21, the calculation unit 22, and the smoothing processing unit 24 constitute the image processing device 11.

  The imaging unit 10 captures a captured image of a subject. Here, the captured image refers to an image before image processing is performed by the image processing apparatus 11. The captured image of this example is a two-dimensional or three-dimensional pan focus image. A pan-focus image refers to an image that appears to be in focus with respect to each of the two or more subjects even if the subject distance between each of the two or more subjects and the imaging unit 10 is different. A pan-focus image can be acquired by capturing an image with the optical aperture stopped. That is, a pan-focus image can be obtained by shooting in a state where the depth of field is deep. The deep state of depth of field is ∞ and a short distance (for example, 1 m) within the depth of field. Indicates a state in which both of the subjects are included. The imaging unit 10 that captures a pan-focus image has a lens with a relatively large open f value. The open f value may be greater than or equal to F8.

  The distance measuring unit 12 measures a subject distance from the imaging unit 10 to each of two or more subjects. Here, the subject distance refers to an optical distance from the imaging unit 10 to each of two or more subjects. When there is a difference in the distance within the same subject, the distance measuring unit 12 may measure the distance for a plurality of points within the same subject. The distance measurement unit 12 measures distance information with a resolution lower than that of the imaging unit 10, for example, for a subject in the same range as the imaging unit 10. For example, the imaging unit 10 has about several million to tens of millions of pixels. On the other hand, the distance measuring unit 12 may have about 800 × 400 pixels and measure the distance for each pixel. Details of the distance measuring unit 12 will be described later. The distance measuring unit 12 may be a part of the imaging unit 10. Further, the distance measuring unit 12 may be an optical system independent of the imaging unit 10. In the latter case, it is preferable that the distance measuring unit 12 is disposed close to the imaging unit 10.

  The memory 14 is connected to the imaging unit 10 and the distance measuring unit 12 and stores captured image data from the imaging unit 10 and distance information data from the distance measuring unit 12. The memory 14 may be a cache memory. In that case, the main memory 26 stores the captured image data from the imaging unit 10 and the distance information data from the distance measuring unit 12. The memory 14 may hold attribute information such as addresses and flags of the respective data.

  The image generation unit 16 is connected to the memory 14, and based on the distance from the imaging unit 10 to each of the two or more subjects measured by the distance measurement unit 12, between each of the two or more subjects and the imaging unit 10. A distance information image representing the distance is generated. The image generation unit 16 acquires distance information data from the memory 14 or the main memory 26 by accessing the memory 14. Here, the distance information data refers to data representing the optical distance from the imaging unit 10 to each subject. The image generation unit 16 generates a distance information image based on the acquired distance information data. Here, the distance information image indicates an image distinguished for each distance range from the imaging unit 10 to each of two or more subjects. The range of distance is obtained, for example, by distinguishing the distance from the imaging unit 10 to the subject every 50 cm, 80 cm, or 1 m. The predetermined range may be equally spaced or may not be equally spaced. The range of distance may be distinguished by a line or by color. The range of distance may be distinguished by a pattern, a character, or other symbols.

  The display unit 18 acquires distance information image data from the image generation unit 16 and displays the distance information image. The display unit 18 is a liquid crystal color display provided on the surface of the imaging device 100, for example.

  The extraction unit 20 extracts an image area corresponding to the selected subject selected by the user from the image displayed on the display unit 18. When the user selects a selected subject based on an image displayed on the display unit 18, the extraction unit 20 of this example classifies a captured image region corresponding to the selected subject by edge detection. Further, the extraction unit 20 similarly extracts corresponding image regions for non-selected subjects other than the selected subject. The extraction unit 20 may classify each image region corresponding to a plurality of subjects included in the image captured by the imaging unit 10 before the user selects a selected subject. Further, the extraction unit 20 may extract the image area of each subject after the user selects the selected subject.

  The display unit 18 may have a touch panel screen. In this example, the user may specify the selected subject by selecting any one of the distance information images displayed on the display unit 18 by touching with a fingertip or a pen. Further, the extraction unit 20 may include a user interface 25 for the user to select a selected subject by operating a cursor or the like displayed on the distance information image. In another example, the user may select an image area including an image corresponding to a target subject by remote operation via the wireless user interface 25. For example, the user may select an image area including an image corresponding to a target subject from a mobile phone or the like.

  The replacement unit 21 acquires information indicating the subject distance to the imaging unit 10 for each subject included in the captured image, and generates image data in which the subject distance of the selected subject is replaced with the subject distance specified by the user. The replacement unit 21 may generate image data in which only distance data is replaced, or may generate image data in which the size of the subject is changed according to the difference in subject distance before and after replacement. For example, the replacement unit 21 may increase the size of the subject when the subject distance is reduced (closer to the imaging unit 10). In addition, the replacement unit 21 may reduce the size of the subject when the subject distance is increased. In this case, the image data of the area hidden behind the subject before replacement may be interpolated from the image data around the subject. Further, the replacement unit 21 may not change the size of the subject when the subject distance is increased.

  The replacement unit 21 may further specify a desired focal length from the user. The desired focal length refers to a virtual focal length for quasi-focusing the captured image. As the desired focal length, a distance different from the focal length at the time of capturing the captured image can be designated.

  The user can designate a desired focal length by selecting one of the subjects. The replacement unit 21 may set the subject distance of the subject selected by the user to specify the desired focal length as the desired focal length. The replacement unit 21 may set a desired focal length based on a numerical value input by the user. The user interface 27 may receive this information. For example, the user interface 27 receives such information via a touch panel or a keypad. Further, the replacement unit 21 may replace the subject distance of each subject sequentially selected by the user with the subject distance sequentially designated by the user.

  The replacement unit 21 may replace the subject distance of another subject that is the same distance as the selected subject with the same distance as the subject distance of the selected subject. In this case, each of a plurality of subjects belonging to the same distance range belongs to the same virtual distance range.

  Based on the distance measured by the distance measuring unit 12, the calculating unit 22 calculates the relative distance of the subject distance of each subject with respect to the desired focal length specified by the user. Here, the relative distance refers to a difference between the desired focal length and the distance between each of the two or more subjects included in the captured image and the imaging unit 10. The relative distance can take a positive or negative value. That is, for a subject whose optical distance is shorter than the desired focal length, the relative distance is a negative value, and for a subject whose optical distance is longer than the desired focal length, the relative distance is a positive value.

  The smoothing processing unit 24 smoothes the captured image based on the relative distance calculated for each subject. Here, based on the relative distance may include smoothing the captured image by comparing absolute values of the relative distance. In addition, a threshold value of relative distance may be designated in advance, and the captured image may be smoothed by comparing the threshold value with the relative distance of each subject. Furthermore, when the difference in relative distance between subjects is small, the relative distance may be weighted, and the weighted relative distance may be compared with a threshold value to smooth the captured image. The smoothing processing unit 24 creates a smoothing filter according to the relative distance. The smoothing filter may be a low-pass filter that averages pixel levels in the vertical direction and the horizontal direction of the captured image.

  The smoothing processing unit 24 accesses the memory 14 and reads pan focus image data captured by the imaging unit 10. The smoothing processing unit 24 performs a smoothing process on the pan focus image data read from the memory 14. The smoothing processing unit 24 increases the degree of blurring for a captured image region corresponding to a subject having a larger relative distance. Here, the degree of blur refers to the relative strength of smoothing. The smoothing filter filters the image corresponding to the subject having a large relative distance in the pan-focus image so that the smoothing strength is relatively increased and the image is strongly blurred.

  On the other hand, the smoothing filter filters the image corresponding to the subject with a small relative distance so as to make the smoothing intensity relatively weak and weakly blur. By increasing or decreasing the number of times the low-pass filter is applied, the relative strength of blur can be changed. According to the present embodiment, a pseudo-focused image can be generated at an arbitrary selected position without acquiring a plurality of parallax images at the same time, so that a high-resolution image can be obtained. In addition, since the pseudo-focused image with high resolution can be obtained only by the smoothing process, it is not necessary to re-synthesize the image data after the smoothing process. Therefore, the load on the apparatus in image processing can be reduced.

  The main memory 26 is connected to the smoothing processing unit 24 and stores pseudo focused image data. The display unit 18 displays the pseudo focused image data read from the main memory 26 as a pseudo focused image. In addition, the output interface 28 is connected to the main memory 26, and outputs pseudo focused image data stored in the main memory 26 to the outside of the imaging apparatus 100. The imaging device 100 can be connected to an external display device such as a mobile phone, a PAD terminal, a personal computer, or a television via the output interface 28.

  FIG. 2 is a flowchart of the image processing method according to the embodiment. The image processing method according to the present embodiment includes an imaging step S1, a distance measurement step S2, a pan focus image display step S3, an edge detection step S4, an image selection step S5, and a desired focal length designation step S6. Step S7, relative distance calculation step S8, smoothing processing step S9, and determination step S10 are provided.

  First, in imaging step S1, the imaging unit 10 acquires a captured image obtained by imaging a subject. In the present embodiment, the captured image captured by the imaging unit 10 is a pan focus image. Next, in distance measurement step S2, the distance measurement unit 12 measures the distance from the imaging unit 10 to each of two or more subjects, and associates each subject with distance information data. Here, the distance means the shortest distance from the imaging unit 10 to each subject. Details of the distance measuring method will be described later with reference to FIGS.

  Subsequently, in the pan focus image display step S3, the image generation unit 16 accesses the memory 14, and reads the pan focus image captured by the imaging unit 10. Then, the read pan focus image is displayed on the display unit 18.

  Next, in edge detection step S5, edge detection is performed on the pan focus image on which the extraction unit 20 is displayed. For example, edge detection can be performed by performing Gaussian filtering on an image corresponding to each displayed subject to generate a smoothed image, and subtracting the smoothed image from an image region corresponding to each subject. By the edge detection, an image area corresponding to each subject is segmented.

  Subsequently, in an image selection step S5, the user selects an image area corresponding to the target subject. The extraction unit 20 extracts an image area corresponding to the subject selected by the user from the captured image.

  Next, in the desired focal length designation step S6, the user designates the desired focal length. The desired focal length is designated via the user interface 27.

  Subsequently, in replacement step S7, the replacement unit 21 generates image data in which the subject distance of the selected subject is replaced with the subject distance designated by the user. The desired focal length may be used as the subject distance designated by the user in the replacement step S7.

  Next, in the relative distance calculation step S8, the calculation unit 22 sets the relative distance, which is the difference between the desired focal length and the distance between each of the two or more subjects included in the captured image and the imaging unit 10, to each subject. To calculate. In the relative distance calculation step S8, the target subject and other subjects are distinguished in the captured image. The calculation unit 22 associates the subject with the relative distance between the subject. The relative distance to the target subject placed at the desired focal length is zero.

  Subsequently, in the smoothing processing step S9, the smoothing processing unit 24 smoothes the captured image based on the relative distance calculated for each subject. The smoothing processing unit 24 accesses the memory 14 and reads pan focus image data captured by the imaging unit 10. The smoothing process step S9 may be a filtering process using a low-pass filter, for example. In the smoothing processing step S9, processing is performed so as to increase the degree of blurring in the region of the captured image corresponding to the subject having a larger relative distance. The degree of blurring can be controlled, for example, by changing the filter coefficient and the number of taps of the smoothing filter.

  Next, in determination step S10, it is determined whether or not another selected subject is to be selected. When the user determines to select another selected subject, the process returns to the image selection step S5. If it is determined that the user does not select, the image processing ends. In determination step S10, the user may determine whether to select another desired focal length. When the desired focal length is selected, the process returns to the desired focal length designation step S6.

  According to the imaging apparatus 100 and the image processing method according to the present embodiment, the subject distance of an arbitrary subject can be changed to a position different from that at the time of imaging. For example, from an image that has been taken once, generate an image that makes each subject that should be at a different distance appear to be at the same distance, and an image that makes each subject that should be at the same distance appear at a different distance Can be generated. Therefore, the expression in the depth direction can be freely edited from the image once captured. Further, according to the imaging apparatus 100 and the image processing method according to the present embodiment, a plurality of parallax images are not required, and thus a pseudo focused image with high resolution can be acquired. Furthermore, according to the imaging device 100 and the image processing method according to the present embodiment, the image processing can be simplified, so that the load on the device accompanying the image processing can be reduced. Further, according to the imaging apparatus 100 and the image processing method according to the present embodiment, it is possible to increase the speed of image processing and reduce the apparatus cost.

  FIG. 3 is a flowchart of another image processing method. The image processing method according to the present embodiment includes an imaging step S21, a distance measurement step S22, a distance information image creation / display step S23, an image selection step S24, a desired focal length designation step S25, a replacement step S26, Relative distance calculation step S27, smoothing processing step S28, and determination step S29 are provided. Other than the distance information image creation / display step S23 and the image selection step S24 is the same as the image processing method shown in FIG.

  In the distance information image creation / display step S23, the image generation unit 16 generates distance information image data from the distance information data measured in the distance measurement step S22. The distance information image data is transmitted to the display unit 18, and the display unit 18 displays the distance information image. The distance information image displays the distance from the imaging unit 10 to each of the subjects corresponding to each pixel separately for each predetermined range. Each distance range is distinguished by a line, color, pattern, character, symbol, or the like. In the distance information image of the present example, regions where the distance from the imaging unit 10 is within the respective ranges are distinguished by boundary lines.

  Subsequently, in the image selection step S24, the extraction unit 20 extracts an image area of each subject with a boundary where the density of the boundary line in the distance range is higher than a predetermined threshold. In other words, the extraction unit 20 automatically detects an image area indicated by the distance information image in which the gradient of the change in the subject distance is steeper than a predetermined threshold, and classifies the image area as an image area of each subject. Good. After the segmentation, the segment including the image area corresponding to the selected subject is determined based on the user's selection command.

  FIG. 4 is a graph showing the relationship between the relative distance and the intensity of the blur amount. The vertical axis represents the intensity of the blur amount by the smoothing process, and the horizontal axis represents the relative distance of each subject. FIG. 4 shows the relationship between the intensity of the blur amount and the relative distance for each open f value of the imaging lens of the imaging unit 10. There is a relationship of f1> f2> f3 between the three open f values. From the graph, it can be seen that the greater the absolute value of the relative distance, the greater the intensity of the blur amount. The smoothing processing unit 24 may increase the blur amount as the absolute value of the relative distance increases. It can also be seen that this tendency becomes more prominent for lenses with a large open f value.

  FIG. 5 is an example of the distance information image 200 displayed by the display unit 18. In this example, the subject 210 is the upper body of a single human who has both hands protruding forward. The distance information image 200 includes an image of the subject 210 and a pattern chart 220 displayed beside it. The distance information image 200 may be an image having a resolution that allows the outline of the subject 210 to be understood. In the distance information image 200, the surface of the body of the subject 210 is distinguished by being given a different pattern for each predetermined distance range according to the distance from the imaging unit 10. Further, the distance information image distinguishes regions where the distances from the imaging unit 10 are within the respective ranges by boundary lines. In this example, the distance from 1 m to 7 m is distinguished by five types of patterns in five stages.

  For example, since the subject portion distinguished by the distance range a and the subject portion distinguished by the distance range b have different patterns, the user can easily identify that the distance from the imaging unit 10 is different. . A place where the interval between the boundary lines of the pattern is narrow indicates that the amount of change in the distance in the depth direction is large. On the other hand, a place where the distance between the boundary lines of the pattern is wide indicates that the amount of change in the distance in the depth direction is small. The range of the distance may be distinguished by a color, a character, a symbol, or the like other than the pattern. In this example, the case where the subject 210 is the upper body of one person has been described, but the same applies to the case where the subject is composed of two or more objects such as a landscape.

  The extraction unit 20 may extract the image area of each subject using a boundary in the distance information image 200 where the density of the boundary line divided for each predetermined distance range is higher than a predetermined threshold. The smoothing processing unit 24 may perform a smoothing process along the boundary line. That is, the filtering range may be moved along the boundary line so that the blurring amounts of the images on the same boundary line are substantially the same.

  FIG. 6 is an example of a captured image obtained by capturing an image of the outdoors using the imaging unit 10 of the imaging apparatus 100 according to the present embodiment. The captured image of this example includes images corresponding to a plurality of subjects such as a person floating in the central sea, the sea surface behind the sea, the horizon, a yacht floating in the sea, and clouds in the sky. The captured image in this example is a pan-focus image captured with the aperture of the imaging unit 10 being narrowed.

  FIG. 7 is a pseudo-focused image obtained by processing the captured image shown in FIG. 6 by the image processing method according to the present embodiment. The pseudo-focused image of this example shows an image that is pseudo-focused on the person at the center. Among multiple subjects, only the central person is in focus, and the other subjects appear to be strongly blurred. Thus, according to the imaging apparatus 100 according to the present embodiment, an image with a deep depth of field can be shown as an image with a shallow depth of field. That is, according to the imaging apparatus 100 according to the present embodiment, an image as if the position of the desired focal length is in focus can be obtained according to an arbitrary desired focal length designated by the user.

  FIG. 8 shows an embodiment of the distance measuring unit 12 shown in FIG. The distance measuring unit 12 according to the present embodiment includes an image plane phase difference sensor 50 that receives light from the same subject 33 via different optical paths 44 and 46. Here, the image plane phase difference sensor 50 includes an imaging element 30 and an opening holding portion 42 including two openings 38 and 40. The image sensor 30 includes a photodiode 32, a pupil division unit 34, and a microlens 48. The pupil division unit 34 is provided with a slit 36, and the light that has passed through either the optical path 44 or the optical path 46 is incident on the photodiode 32. In the opening holding part 42, the opening 38 and the opening 40 are arranged at an equal distance from the optical axis 37 of the microlens 48.

  The image plane phase difference sensor 50 measures the distance from the imaging unit 10 to each of two or more subjects based on the position where each light is received. The image plane phase difference sensor 50 arranges two types of image pickup elements 30 that have undergone pupil division on a chip so that two or more subjects from the image pickup unit 10 are obtained based on the obtained parallax information. Measure the distance to each. The image plane phase difference sensor 50 can also acquire a normal image at the same time.

  The pupil division unit 34 may be configured by a liquid crystal. The pupil division unit 34 changes the position of the slit 36 between the position A and the position B in accordance with a change in the voltage applied to the liquid crystal. When the slit 36 exists at the position A, the light in the optical path 46 that has passed through the microlens 48 passes through the slit 36 and enters the photodiode 32 corresponding to the position A. When the slit 36 exists at the position B, the light of the optical path 44 that has passed through the microlens 48 passes through the slit 36 and enters the photodiode 32 corresponding to the position B.

  FIG. 9 shows an example of an imaging chip 52 in which the imaging elements 30 are arranged in a matrix. In this example, in the image sensor 30, an image sensor 30-1 having a slit 36 at a position B and an image sensor 30-2 having a slit at a position A are alternately arranged in the x direction at a certain time. A plurality of image sensors 30 having slits 36 are arranged at the same positions as adjacent image sensors in the y direction. The position of the slit 36 may alternate between position A and position B over time. The arrangement of the imaging elements 30-1 and 30-2 in the imaging chip 52 is not limited to this. The imaging elements 30-1 and 30-2 may be spread over the entire surface of the imaging chip 52, but the imaging elements 30-1 and 30-2 may be arranged at a predetermined interval. In this case, the arrangement density of the image sensors 30-1 and 30-2 may be such that the distance to the subject 33 can be predicted.

  FIG. 10 shows the position of the image pickup device 30 in the x direction and the light signals received by the image pickup devices 30-1 and 30-2 at the positions A and B of the image pickup chip 52 of the image plane phase difference sensor 50 shown in FIG. It is a graph which shows the relationship with quantity. A line A indicates the relationship between the signal amount received by the image sensor 30-1 having the slit 36 at the position A and the position of the image sensor 30-1 in the x direction. Line B shows the relationship between the signal amount received by the image sensor 30-2 having the slit 36 at the position B and the position of the image sensor 30-2 in the x direction. That is, the line A indicates an image formed by the image pickup element 30-1 divided at the position A, and the line B indicates an image formed by the image pickup element 30-2 obtained by dividing the pupil at the position B.

  First, the entire image captured by the image plane phase difference sensor 50 is divided into a certain range. Next, only the output of the image plane phase difference sensor is cut out from the image of the divided range, and a pupil division image is extracted. Next, the shift amount between the pupil division pixel at position A and the pupil division pixel at position B in the pupil division image is calculated by shift calculation. Next, the shift amount is calculated for each position of the image and stored as distance difference information. To save data, the method of saving the image and the result after distance information calculation separately, the method of storing the result after distance information calculation in the image and saving it in one file, and as one image, before the distance information calculation There is a method of saving the divided pupil image and the normal image.

  FIG. 11 shows another embodiment of the distance measuring unit 12 shown in FIG. The distance measuring unit 12 according to the present embodiment includes a TOF image sensor 60 that measures the flight time of light. The TOF image sensor 60 measures the distance from the imaging unit 10 to the subject 65 based on the round-trip time of light from the imaging unit 10 to the subject 65. The TOF image sensor 60 includes a light emitting element 64, a light receiving element 62, and a flight time measuring unit 63. The light emitting element 64 emits light 66 toward the subject 65. The light emitting element 64 is, for example, an infrared LED. Infrared LEDs are preferred because they emit infrared rays with long wavelengths that are not easily scattered. The light receiving element 62 receives the light 68 reflected from the subject 65. The light receiving element 62 may be a photodiode. The light receiving element 62 is preferably a device having a photodiode having a high photoelectric conversion speed. The flight time measurement unit 63 is connected to the light emitting element 64 and the light receiving element 62 and measures the flight time of the light 66 and the light 68.

  FIG. 12 shows a timing chart of the operation of the TOF image sensor 60. A waveform 72 indicates a pulse waveform of the light 66 emitted from the light emitting element 64. A waveform 74 indicates a pulse waveform of the light 68 received by the light receiving element 62. Time t1 represents the rise time of the light emission pulse of the light 66, and time t2 represents the rise time of the light reception pulse of the light 68. The flight time measuring unit 63 can accurately measure t2-t1. t2-t1 indicates the light delay time 76. If the distance to the subject is D, the relationship D = (t2−t1) × c / 2 is established. Here, c represents the speed of light.

  FIG. 13 shows another embodiment of the distance measuring unit 12 shown in FIG. The distance measuring unit 12 according to the present embodiment includes an optical system 80 that images the subject 85 from different optical positions. In the optical system 80, the distance from the imaging unit 10 to the subject 85 is measured based on the parallax between the captured images captured from different optical positions. The optical system 80 includes a pair of lenses 86 and 88 and image sensors 82 and 84 corresponding to the lenses 86 and 88, respectively. The light traveling on the optical path 81 from the subject 85 passes through the lens 88 and enters the photodiode at the position m of the image sensor 84. On the other hand, the light traveling on the optical path 83 passes through the lens 86 and enters the photodiode at the position n of the image sensor 82. The photodiode position m of the image sensor 84 and the photodiode position n of the image sensor 82 constitute different optical positions. That is, the optical system 80 images the same subject 85 from different optical positions n and m. A viewpoint distance 87 between the optical position n and the optical position m represents parallax. The optical system 80 is, for example, a binocular camera or a stereoscopic image capture device.

  FIG. 14 shows the optical system 80 when the distance between the subject 85 and the imaging unit 10 is shorter than that in FIG. The configuration and operation of the optical system 80 are the same as those of the optical system 80 shown in FIG. The optical system 80 shown in FIG. 14 images the same subject 85 from different optical positions p and q. The inter-viewpoint distance 89 between the optical position p and the optical position q represents parallax. Here, comparing FIG. 13 and FIG. 14, the inter-viewpoint distance 89 when the subject 85 is closer is longer than the inter-viewpoint distance 87 when the subject 85 is further away. The optical system 80 measures the distance from the imaging unit 10 to the subject 85 based on the difference in parallax caused by the change in the distance from the imaging unit 10 to the subject 85.

  FIG. 15 shows an example of a hardware configuration of a computer 1900 according to this embodiment. A computer 1900 according to this embodiment is connected to a CPU peripheral unit having a CPU 2000, a RAM 2020, a graphic controller 2075, and a display device 2080 that are connected to each other by a host controller 2082, and to the host controller 2082 by an input / output controller 2084. Input / output unit having communication interface 2030, hard disk drive 2040 and CD-ROM drive 2060, and legacy input / output unit having ROM 2010, flexible disk drive 2050 and input / output chip 2070 connected to input / output controller 2084 With.

  The host controller 2082 connects the RAM 2020 to the CPU 2000 and the graphic controller 2075 that access the RAM 2020 at a high transfer rate. The CPU 2000 operates based on programs stored in the ROM 2010 and the RAM 2020 and controls each unit. The graphic controller 2075 acquires image data generated by the CPU 2000 or the like on a frame buffer provided in the RAM 2020 and displays it on the display device 2080. Instead of this, the graphic controller 2075 may include a frame buffer for storing image data generated by the CPU 2000 or the like.

  The input / output controller 2084 connects the host controller 2082 to the communication interface 2030, the hard disk drive 2040, and the CD-ROM drive 2060, which are relatively high-speed input / output devices. The communication interface 2030 communicates with other devices via a network. The hard disk drive 2040 stores programs and data used by the CPU 2000 in the computer 1900. The CD-ROM drive 2060 reads a program or data from the CD-ROM 2095 and provides it to the hard disk drive 2040 via the RAM 2020.

  The input / output controller 2084 is connected to the ROM 2010, the flexible disk drive 2050, and the relatively low-speed input / output device of the input / output chip 2070. The ROM 2010 stores a boot program that the computer 1900 executes at startup and / or a program that depends on the hardware of the computer 1900. The flexible disk drive 2050 reads a program or data from the flexible disk 2090 and provides it to the hard disk drive 2040 via the RAM 2020. The input / output chip 2070 connects the flexible disk drive 2050 to the input / output controller 2084 and inputs / outputs various input / output devices via, for example, a parallel port, a serial port, a keyboard port, a mouse port, and the like. Connect to controller 2084.

  A program provided to the hard disk drive 2040 via the RAM 2020 is stored in a recording medium such as the flexible disk 2090, the CD-ROM 2095, or an IC card and provided by the user. The program is read from the recording medium, installed in the hard disk drive 2040 in the computer 1900 via the RAM 2020, and executed by the CPU 2000.

  A program installed in the computer 1900 and causing the computer 1900 to function as the image processing apparatus 11 includes an image generation module, an extraction module, a replacement module, a calculation module, and a smoothing processing module. These programs or modules work on the CPU 2000 or the like to cause the computer 1900 to function as the image generation unit 16, the extraction unit 20, the replacement unit 21, the calculation unit 22, and the smoothing processing unit 24, respectively.

  The information processing described in these programs is read by the computer 1900, whereby the image generating unit 16, the extracting unit 20, and the replacing unit, which are specific means in which the software and the various hardware resources described above cooperate. 21, the calculation unit 22, and the smoothing processing unit 24. And the specific image processing apparatus 11 according to the intended use is constructed | assembled by implement | achieving the calculation or processing of the information according to the intended use of the computer 1900 in this embodiment by these specific means.

  As an example, when communication is performed between the computer 1900 and an external device or the like, the CPU 2000 executes a communication program loaded on the RAM 2020 and executes a communication interface based on the processing content described in the communication program. A communication process is instructed to 2030. Under the control of the CPU 2000, the communication interface 2030 reads transmission data stored in a transmission buffer area or the like provided on a storage device such as the RAM 2020, the hard disk drive 2040, the flexible disk 2090, or the CD-ROM 2095, and sends it to the network. The reception data transmitted or received from the network is written into a reception buffer area or the like provided on the storage device. As described above, the communication interface 2030 may transfer transmission / reception data to / from the storage device by a DMA (direct memory access) method. Instead, the CPU 2000 transfers the storage device or the communication interface 2030 as a transfer source. The transmission / reception data may be transferred by reading the data from the network and writing the data to the communication interface 2030 or the storage device of the transfer destination.

  The CPU 2000 is all or necessary from among files or databases stored in an external storage device such as a hard disk drive 2040, a CD-ROM drive 2060 (CD-ROM 2095), and a flexible disk drive 2050 (flexible disk 2090). This portion is read into the RAM 2020 by DMA transfer or the like, and various processes are performed on the data on the RAM 2020. Then, CPU 2000 writes the processed data back to the external storage device by DMA transfer or the like. In such processing, since the RAM 2020 can be regarded as temporarily holding the contents of the external storage device, in the present embodiment, the RAM 2020 and the external storage device are collectively referred to as a memory, a storage unit, or a storage device. Various types of information such as various programs, data, tables, and databases in the present embodiment are stored on such a storage device and are subjected to information processing. Note that the CPU 2000 can also store a part of the RAM 2020 in the cache memory and perform reading and writing on the cache memory. Even in such a form, the cache memory bears a part of the function of the RAM 2020. Therefore, in the present embodiment, the cache memory is also included in the RAM 2020, the memory, and / or the storage device unless otherwise indicated. To do.

  In addition, the CPU 2000 performs various operations, such as various operations, information processing, condition determination, information search / replacement, etc., described in the present embodiment, specified for the data read from the RAM 2020 by the instruction sequence of the program. Is written back to the RAM 2020. For example, when performing the condition determination, the CPU 2000 determines whether the various variables shown in the present embodiment satisfy the conditions such as large, small, above, below, equal, etc., compared to other variables or constants. When the condition is satisfied (or not satisfied), the program branches to a different instruction sequence or calls a subroutine.

  Further, the CPU 2000 can search for information stored in a file or database in the storage device. For example, in the case where a plurality of entries in which the attribute value of the second attribute is associated with the attribute value of the first attribute are stored in the storage device, the CPU 2000 displays the plurality of entries stored in the storage device. The entry that matches the condition in which the attribute value of the first attribute is specified is retrieved, and the attribute value of the second attribute that is stored in the entry is read, thereby associating with the first attribute that satisfies the predetermined condition The attribute value of the specified second attribute can be obtained.

  The program or module shown above may be stored in an external recording medium. As the recording medium, in addition to the flexible disk 2090 and the CD-ROM 2095, an optical recording medium such as DVD or CD, a magneto-optical recording medium such as MO, a tape medium, a semiconductor memory such as an IC card, and the like can be used. Further, a storage device such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet may be used as a recording medium, and the program may be provided to the computer 1900 via the network.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

DESCRIPTION OF SYMBOLS 10 Imaging part, 11 Image processing apparatus, 12 Distance measurement part, 14 Memory, 16 Image generation part, 18 Display part, 20 Extraction part, 21 Replacement part, 22 Calculation part, 24 Smoothing process part, 25 User interface, 27 User -Interface, 26 Main memory, 28 Output interface, 30 Image sensor, 32 Photodiode, 33 Subject, 34 Pupil dividing part, 36 Slit, 37 Optical axis, 38 Aperture, 40 Aperture, 42 Aperture holding part, 44 Optical path, 46 optical path, 48 microlens, 50 image plane phase difference sensor, 52 imaging chip, 60 TOF image sensor, 62 light receiving element, 63 flight time measuring unit, 64 light emitting element, 65 subject, 66 light, 68 light, 72 waveform, 74 Waveform, 76 delay time, 80 optical system, 81 optical path, 82 Image element, 83 optical path, 84 image sensor, 85 subject, 86 lens, 87 distance between viewpoints, 88 lens, 89 distance between viewpoints, 100 imaging device, 200 distance information image, 210 subject, 220 pattern chart, 1900 computer, 2000 CPU , 2010 ROM, 2020 RAM, 2030 communication interface, 2040 hard disk drive, 2050 flexible disk drive, 2060 CD-ROM drive, 2070 input / output chip, 2075 graphic controller, 2080 display device, 2082 host controller, 2084 input / output controller , 2090 Flexible disk, 2095 CD-ROM

Claims (17)

An imaging unit that captures a captured image of the subject;
An extraction unit that extracts an image region corresponding to the selected subject selected by the user from the captured image;
For each subject included in the captured image, a replacement unit that acquires information indicating the subject distance to the imaging unit and generates image data in which the subject distance of the selected subject is replaced with the subject distance specified by the user When,
An imaging apparatus comprising: a smoothing unit that smoothes the captured image based on the subject distance corresponding to each subject with respect to the image data generated by the replacement unit. A calculation unit that calculates a relative distance of the subject distance of each subject with respect to a desired focal length specified by the user;
The imaging apparatus according to claim 1, wherein the smoothing unit smoothes the captured image based on the relative distance calculated by the calculation unit. The imaging apparatus according to claim 2, wherein the smoothing unit sets the subject distance designated by the user for the selected subject as the desired focal length. The replacement unit, when replacing the subject distance of the selected subject, changes the size of the selected subject on the image according to a difference between the subject distances before and after replacement. The imaging device according to one item.   The distance measurement part which measures the to-be-photographed object distance from the said imaging part to each of two or more said to-be-photographed objects is further provided, The said calculation part calculates the said relative distance based on the said to-be-photographed object distance. Imaging device.   The distance measuring unit includes an image plane phase difference sensor that receives light from the same subject through different optical paths, and the image plane phase difference sensor is configured to receive the light based on a position at which each light is received. The imaging apparatus according to claim 5, wherein a subject distance from the imaging unit to each of the two or more subjects is measured.   The distance measuring unit includes a TOF image sensor that measures a flight time of light, and based on a round-trip time of light from the imaging unit to each of the two or more subjects, the two or more subjects from the imaging unit. The imaging apparatus according to claim 5, wherein the subject distance to each is measured.   The distance measuring unit includes an optical system that images the subject from different optical positions. In the optical system, two or more of the subjects from the imaging unit are based on parallax between captured images captured from different optical positions. The imaging apparatus according to claim 5, wherein the subject distance to each is measured.   The extraction unit according to any one of claims 1 to 8, wherein when the user selects the selected subject based on the captured image, the region of the captured image corresponding to the selected subject is classified by edge detection. The imaging device described.   A distance information image representing a distance between each of the two or more subjects and the imaging unit is generated based on the subject distance from the imaging unit to each of the two or more subjects measured by the distance measuring unit. The imaging device according to any one of claims 5 to 8, further comprising an image generation unit that performs the operation.   The imaging apparatus according to claim 10, wherein the distance information image displays the distance from the imaging unit to each of the subjects corresponding to each pixel separately for each predetermined range. In the distance information image, a region where the distance from the imaging unit is within the respective range is distinguished by a boundary line,
The imaging apparatus according to claim 11, wherein the extraction unit extracts the image area of each of the subjects with a boundary where a density of the boundary line is higher than a predetermined threshold.   The imaging device according to claim 1, wherein the captured image is a pan focus image.   The imaging apparatus according to claim 2, wherein the smoothing unit increases a degree of blurring for a region of the captured image corresponding to the subject having the larger relative distance. An extraction unit that extracts an image region corresponding to the selected subject selected by the user from the captured image of the subject captured by the imaging device;
For each subject included in the captured image, a replacement unit that acquires information indicating a subject distance to the imaging device and generates image data in which the subject distance of the selected subject is replaced with a subject distance specified by the user When,
An image processing apparatus comprising: a smoothing unit that smoothes the captured image based on the subject distance corresponding to each of the subjects with respect to the image data generated by the replacement unit. Acquiring a captured image obtained by imaging the subject by the imaging device;
An extraction step of extracting an image region corresponding to the selected subject selected by the user from the captured image;
For each subject included in the captured image, a replacement step of acquiring information indicating a subject distance to the imaging device and generating image data in which the subject distance of the selected subject is replaced with a subject distance specified by the user When,
An image processing method comprising: a smoothing step of smoothing the captured image based on the subject distance corresponding to each subject with respect to the image data generated in the replacement step.   A program for causing a computer to function as the image processing apparatus according to claim 15.

Images