JP2007208704A - Imaging apparatus, method for processing image, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain an excellent image by efficiently superposing a plurality of the images acquired from each imaging section, in an imaging apparatus with a plurality of the imaging sections. <P>SOLUTION: Cameras 11a and 11b fitted to a stereocamera 10 are driven simultaneously, and a fixed number of continuous images are obtained in each camera. When these images are superposed and the images inhibiting a hand shaking and noises are prepared, a control unit 17 uses the images first obtained from the cameras 11a and 11b as reference images, and decides a suitability between these reference images. One synthetic image is prepared by successively superposing all images in the case of a high suitability. Then, two kinds of the synthetic images are prepared by discretely superposing each image by a division into the cameras 11a and 11b in the case of a low suitability. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an imaging apparatus that performs image composition using a plurality of imaging units provided in a multi-view camera such as a stereo camera, and an image processing method and program used in the imaging apparatus.

  A so-called stereo camera is known as an imaging apparatus having a plurality of imaging units. This stereo camera has imaging units on the left and right sides, and creates a stereoscopic image using parallax between the imaging units (see, for example, Patent Document 1). It is not so often used when shooting a distant subject where parallax is usually difficult to obtain from such usage.

On the other hand, with recent digital cameras, there is a problem of camera shake along with the reduction in size and weight, and as one of the solutions, a method using an image synthesis technique is considered. In this method, continuous shooting is performed with a short exposure time, and a plurality of images obtained by the continuous shooting are superimposed to obtain a good image with reduced camera shake and noise.
JP-A-8-84353

  However, in order to produce a good image by the method described above, a large number of images are required. In this case, if the continuous shooting time is lengthened, a large number of images can be obtained. However, the possibility that the subject and the camera move during that time increases, so the positional deviation between the continuous shot images increases, making it unsuitable for composition. .

  Accordingly, the present invention provides an imaging apparatus, an image processing method, and a program capable of efficiently superimposing a plurality of images obtained from each imaging unit and obtaining a good image in an imaging apparatus including a plurality of imaging units. The purpose is to provide.

  An imaging apparatus according to a first aspect of the present invention is an imaging apparatus that creates a composite image from a plurality of images continuously shot using at least two imaging units, and is obtained first from each of the imaging units. An image is used as a reference image, and compatibility determination means for determining compatibility between these reference images, and when the compatibility determination means determines that the compatibility is high, the reference images are aligned and overlapped. A first synthesized image creating means for creating a synthesized image by sequentially superimposing and aligning a predetermined number of continuous images following each of the reference images, and the suitability determining means When each of the reference images is used individually, the predetermined number of continuous shot images following each of the reference images are not aligned with any of the reference images. Configured and a second composite image generating means for generating a plurality of synthesized image by al sequentially superimposed.

  According to such a configuration, it is possible to obtain a good image as a composite result by superimposing a plurality of continuous images individually obtained from the respective imaging units and suppressing camera shake and noise. At that time, the first image obtained from each imaging unit is used as a reference image, and the compatibility between these reference images is judged, so that the compatibility is low (when the influence of positional deviation is small) and the compatibility is low. In this case (when the influence of positional deviation is great), it is possible to efficiently superimpose the continuous shot images and create a good image.

  Further, according to a second aspect of the present invention, in the imaging apparatus according to the first aspect, when the suitability judging means determines that the suitability is low, a warning means for warning that effect is provided. It is characterized by.

  According to such a configuration, when the compatibility between the reference images is low, the user can know that the influence of the parallax is large by giving a warning to that effect.

  According to a third aspect of the present invention, in the imaging apparatus according to the first aspect, the first composite image creating means is configured to synthesize a predetermined number of continuous shot images following each of the reference images. It is characterized by superimposing by excluding unsuitable ones.

  According to such a configuration, it is possible to obtain a better image as a synthesis result by not superimposing consecutively shot images but excluding those that are not suitable for synthesis, and superimposing them. Can do.

  According to a fourth aspect of the present invention, in the imaging apparatus according to the first aspect, the second composite image creating unit corresponds to a predetermined number of continuous images following each of the reference images. It is characterized by being superimposed on a reference image.

  According to such a configuration, when the compatibility between the reference images is low, a predetermined number of continuous shot images following each of the reference images are overlapped to be affected by the parallax between the cameras. Can create no composite image.

  Further, according to a fifth aspect of the present invention, in the imaging apparatus according to the first aspect, the second composite image creating unit converts a predetermined number of continuous images following each of the reference images into the reference images. One of them is characterized by being sorted and superimposed on the one with the smallest positional deviation.

  According to such a configuration, when the compatibility between the reference images is low, the predetermined number of continuous shot images that follow each of the reference images are distributed to the one of the reference images with the least positional deviation. By superimposing, for example, when a time-series error occurs due to movement of the subject or camera during continuous shooting, a composite image with less influence can be created.

  An image processing method according to a sixth aspect of the present invention is an image processing method for creating a composite image from a plurality of images continuously shot using at least two imaging units, and each of the imaging units is started first from each of the imaging units. The obtained image is used as a reference image, and a compatibility determination step for determining compatibility between these reference images, and when it is determined that the compatibility is high by this compatibility determination step, the respective reference images are aligned. A first composite image creating step for creating a single composite image by sequentially superimposing and superimposing a predetermined number of continuous images following each of the reference images while aligning; When the determination step determines that the compatibility is low, each of the reference images is used individually, and a predetermined number of continuous images following each of the reference images are used for each of the reference images. And having a second composite image creation step of creating a plurality of synthesized image by sequentially superimposed while aligning the or displacement.

  According to such an image processing method, the same effect as that of the first aspect of the present invention can be achieved by executing the processing according to each step.

  A program according to a seventh aspect of the present invention is a program executed by a computer mounted on an imaging device that creates a composite image from a plurality of images continuously shot using at least two imaging units, the computer In addition, the first image obtained from each of the imaging units is used as a reference image, and a compatibility determination function for determining compatibility between these reference images, and the compatibility determination function determines that the compatibility is high. In this case, the reference images are aligned and overlapped, and a predetermined number of continuous shot images following each of the reference images are sequentially overlapped while being aligned to create one composite image. Each of the reference images is used individually when the suitability is determined to be low by the composite image creation function of 1 and the suitability determination function. Realizing a second composite image creation function for creating a plurality of composite images by sequentially superimposing a predetermined number of continuous images following each of the reference images while aligning them with any of the reference images. It is characterized by.

  Therefore, when the computer executes the program for realizing each function, the same effects as those of the first aspect of the invention can be achieved.

  According to the present invention, a plurality of continuous images individually obtained from each imaging unit provided in a multi-lens camera such as a stereo camera are efficiently overlapped to synthesize a good image with reduced camera shake and noise. As a result.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
In the present embodiment, a digital stereo camera having two left and right cameras (imaging units) is taken as an example, and N images obtained from the left and right cameras are superimposed by N consecutive shooting (continuous shooting), A method for obtaining an image with reduced camera shake and noise as a synthesis result will be described.

  The initial frame images obtained first from the left and right cameras are referred to as “left reference image” and “right reference image”, respectively. In addition, a frame image continuously following the initial frame is referred to as a “continuous shot image”.

(First embodiment)
FIG. 1 is a block diagram showing a configuration when a stereo camera is taken as an example of the imaging apparatus according to the first embodiment of the present invention.

  The stereo camera 10 includes two camera units (imaging units) 11a and 11b. The two camera units 11a and 11b are installed with a predetermined distance in the horizontal direction with the optical axes parallel to each other. Here, it is assumed that the camera unit 11a is disposed on the left side and the camera unit 11b is disposed on the right side as viewed from the photographer side.

  The camera unit 11a includes a photographing lens 12a formed of an optical system lens such as a focus lens, an imaging element 13a such as a CCD (charge coupled device) that converts light incident through the photographing lens 12a into an electric signal, and the imaging device 13a. The signal processing unit 14a and the like that generate digital image data by performing signal processing on the output of the element 13a. The same applies to the camera unit 11b, which includes a photographing lens 12b, an image sensor 13b, a signal processing unit 14b, and the like.

  Note that the camera unit 11a and the camera unit 11b have the same characteristics, and the imaging elements 13a and 13b are simultaneously driven and photographed in accordance with the shutter operation. The photographing lenses 12a and 12b, which are optical systems, are also driven and controlled simultaneously in the optical axis direction via the motor 15.

  The image data obtained by the camera unit 11a is temporarily stored in the buffer memory 16a and then given to the control unit 17. Similarly, image data obtained by the camera unit 11b is temporarily stored in the buffer memory 16b and then given to the control unit 17. The buffer memories 16a and 16b have a capacity capable of storing a predetermined number of image data (still images).

  The control unit 17 includes a microprocessor including a CPU, a ROM, a RAM, and the like, and controls the entire camera according to various programs stored in the program memory 17a. The program memory 17a includes a program for realizing the present invention. The control unit 17 has a function of sequentially reading N images stored in the buffer memories 16a and 16b and superimposing them to create a composite image. The left buffer 18 a and the right buffer 18 b are used for the synthesis process of the control unit 17. The left buffer 18a is a first image composition buffer used when creating a composite image based on the left image, and the right buffer 18b is used when creating a composite image based on the right image. This is a second image composition buffer.

  The stereo camera 10 is provided with an operation unit 19, a display unit 20, and a storage memory 21. The operation unit 19 is for the user to operate the camera. The operation unit 19 is provided with a shutter key 19a for instructing shooting timing, a mode key 19b for setting various modes including a shooting mode / playback mode, and the like. The continuous shooting mode is set by operating the mode key 19b. When the continuous shooting mode is set, when the shutter key 19a is pressed, the left camera unit 11a and the right camera unit 11b are simultaneously driven, and a predetermined number (N) of images are continuously obtained respectively. .

  The display unit 20 is configured by a color liquid crystal panel with a backlight, for example, and displays a through image on the monitor as an electronic viewfinder in the shooting mode, and reproduces and displays the selected image in the playback mode. The storage memory 21 is a memory for storing image data after being compressed by a predetermined method. The storage memory 21 includes, in addition to an internal memory such as a flash memory built in the main body in advance, a memory card that is detachably mounted as a recording medium.

  Next, an image composition processing operation in the first embodiment will be described.

For the sake of simplicity, the coordinate system of the left image is a homogeneous coordinate system obtained by dividing the image coordinates by the focal length and normalizing to the form of X = (x, y, 1) ^T. Similarly, the coordinate system of the right image is X ′ = (x ′, y ′, 1) ^T. From this assumption, the offset vector between the camera center from the left camera to the right camera can be expressed as h = (0, h _y , 0) ^T , and the orthonormal matrix corresponding to the rotation angle can be expressed as R = I.

  In the first embodiment, when the left and right N images obtained by N consecutive shootings (continuous shooting) are superimposed, the compatibility between the left reference image and the right reference image that are obtained first is determined. Here, the determination of compatibility is to determine whether or not the left reference image and the right reference image have a small error due to the parallax between cameras and can be superimposed as one image.

  Specifically, feature points of the left reference image and the right reference image are extracted, and a non-conformity evaluation calculation of the coordinate transformation model is performed based on the position coordinates of the feature points. A single composite image is created by superimposing all the individual images (see FIG. 2). On the other hand, when the evaluation value is high, a composite image in which N left images are superimposed and a composite image in which N right images are superimposed are respectively created (see FIG. 3).

  The evaluation value is an index indicating the degree of error due to the positional deviation (parallax) between the left and right images. The lower the numerical value as the evaluation value, the less the influence of positional deviation and the matching between images. Represents high nature. Conversely, the higher the numerical value as the evaluation value, the greater the influence of misalignment and the lower the compatibility between images.

  FIG. 4 is a flowchart showing image processing (main processing) of the stereo camera 10 in the first embodiment. Note that the processing shown in this flowchart is executed by the control unit 17 being a computer reading a predetermined program. The same applies to other embodiments, and is executed by the control unit 17 reading a program for realizing the processing of each embodiment.

  Now, it is assumed that continuous shooting is performed by pressing the shutter key 19a in a state where the two camera units 11a and 11b provided in the stereo camera 10 are directed toward a certain subject (step A11). With this continuous shooting, N consecutive images on the left and right are obtained simultaneously and stored in the buffer memories 16a and 16b. Since the photographing process is not directly related to the present invention, detailed description thereof is omitted here.

  When the left and right N images are obtained by the continuous shooting, first, the control unit 17 captures the first image, that is, the left reference image among the N images stored in the buffer memory 16a (step A12). The first image among the N images stored in the buffer memory 16b, that is, the right reference image is captured (step A13).

  Next, the control unit 17 performs a feature point extraction process on the left reference image, and extracts a predetermined number of feature points for detecting the motion of the image (step A14). Subsequently, the control unit 17 performs a feature point tracking process on the right reference image, and tracks where the feature point extracted from the reference addition corresponds to the right reference image (step A15). Thus, feature points are associated between the left and right images. Although not an indispensable process, it is better to remove the outlier by RANSAC or the like and to perform the following process using only the inlier feature point in order to eliminate the miscorrespondence of the feature point.

  Next, the control unit 17 evaluates incompatibility with the coordinate transformation model based on the associated feature points (step A16). This is a process for evaluating the compatibility between the left and right images. Specifically, using the coordinate transformation model determined in advance, for example, the parallel movement model or the projective transformation model, the associated feature Whether or not the point can be coordinate-converted accurately between images is determined.

  For this model incompatibility evaluation processing, for example, an algorithm as shown in FIG. 5 or FIG. 6 can be considered. Note that the number of feature points is c.

  FIG. 5 is a flowchart showing the model incompatibility evaluation process. Here, the three-dimensional coordinates of the feature points are estimated, and it is evaluated whether or not it can be said that the three-dimensional coordinates are substantially on the spatial plane.

  That is, if it is substantially flat, both images (that is, the left reference image and the right reference image) can be exactly overlapped by projective transformation. In addition, in models with few degrees of freedom other than projective transformation, the judgment is strict and the calculation may be simplified, but the constraints described here are still necessary conditions, so the optimal calculation in the case of other models Is omitted.

For example, the estimation of the three-dimensional coordinates of the feature points is the estimation of the Z coordinate k. Therefore, the three-dimensional coordinate is obtained as r = kX based on k obtained by the following equation (step B11).

Next, in the evaluation of planarity, the three-dimensional coordinates of feature points are set to r _i , and the covariance matrix estimate is set to V [r _i ] (step B12). Then, if n that minimizes the following functional expression is n _min , the residual J [n _min ] is set as the evaluation value (step B13).

  FIG. 6 is a flowchart showing another model incompatibility evaluation process. Here, a coordinate conversion coefficient is directly obtained by a known linear algorithm, and the sum of coordinate errors when the feature point is coordinate-converted by the coefficient is used as an evaluation value.

That is, when the obtained coordinate transformation coefficient is represented by H (x) (step C11), an evaluation value is obtained by the following equation (step C12). In this case, the same applies to other than projective transformation.

  Returning to FIG. 4, when the evaluation value is calculated in step A16, the control unit 17 compares the evaluation value with a preset reference value, and branches the process according to the comparison result (step). A17).

  That is, when the evaluation value is lower than the reference value (that is, when the positional deviation between the left and right images is small and the compatibility of the synthesis is high), the control unit 17 superimposes the left and right images and synthesizes them into one. Therefore, as the initialization process, composite image initialization 1 as shown in FIG. 7 is executed (step A18). On the other hand, when the evaluation value is equal to or higher than the reference value (that is, when the positional deviation between the left and right images is large and the compatibility of the synthesis is low), the control unit 17 determines that the left and right images are individually synthesized. As initialization processing, composite image initialization 2 as shown in FIG. 8 is executed (step A20).

  If the evaluation value is high, for example, a warning message such as “the number of combined images is reduced due to the influence of parallax” may be displayed on the display unit 20 (step A19). Thus, the user can know in advance that the high-quality image cannot be obtained as a synthesis result in advance of the time-consuming synthesis process, and in some cases, re-photographing can be attempted without waiting for the end of the process.

  Here, as shown in FIG. 7, in composite image initialization 1, first, the left buffer 18a is initialized with the left reference image (step D11). Specifically, the left reference image extracted from the buffer memory 16a is stored in the left buffer 18a that is the first image composition buffer. Subsequently, a coordinate conversion coefficient for converting the right reference image into the left reference image is calculated (step D12), and the right reference image is subjected to coordinate conversion based on the coordinate conversion coefficient and added to the left buffer 18a (step D13). At this time, since the left reference image is already stored in the left buffer 18a, the right reference image is subjected to position correction (coordinate conversion) and synthesized. The right buffer 18b is not used.

  On the other hand, as shown in FIG. 8, in the composite image initialization 2, the left buffer 18a and the right buffer 18b are initialized with respective reference images (steps E11 and E12). More specifically, the left reference image taken out from the buffer memory 16a is stored in the left buffer 18a, which is the first image composition buffer, and also taken out from the buffer memory 16b in order to create a composite image separately on the left and right. The right reference image is stored in the left buffer 18b which is the second image composition buffer.

  Next, the control unit 17 acquires the left and right continuous shot images after two frames from the buffer memories 16a and 16b one by one (Step A21), and according to the evaluation value obtained in Step A16 (Step A22), The continuous shot images are sequentially developed in the buffer memory 16a or the buffer memory 16b to create a composite image.

  Here, when the evaluation value is lower than the reference value (that is, when the positional deviation between the left and right images is small and the compatibility of the synthesis is high), the control unit 17 executes the synthesis process 1 in FIG. (Step A23). On the other hand, when the evaluation value is equal to or higher than the reference value (that is, when the positional deviation between the left and right images is large and the compatibility of the synthesis is low), the control unit 17 executes the synthesis process 2 of FIG. (Step A24).

  In the composition process 1, the left and right continuous images are coordinate-transformed using the left buffer 18a and superimposed on the left reference image.

  Specifically, as shown in FIG. 9, first, the control unit 17 tracks the feature point of the left continuous image from the left reference image (step F11), and obtains a coordinate conversion coefficient from the coordinate position of the feature point (step F11). F12). Then, the left continuous shot image is subjected to coordinate conversion based on the coordinate conversion coefficient and added to the left buffer 18a (step F13). At this time, since the left reference image and the right reference image are synthesized in the left buffer 18a, the right continuous shot image is further position-corrected (coordinate converted) and synthesized.

  Subsequently, the feature point of the right continuous shot image is traced from the left reference image (step F14), and a coordinate conversion coefficient is obtained from the coordinate position of the feature point (step F15). Then, the right continuous shot image is subjected to coordinate conversion based on the coordinate conversion coefficient and added to the left buffer 18a (step F16). At this time, since the left reference image, the right reference image, and the left continuous image are combined in the left buffer 18a, the right continuous image is further position-corrected (coordinate converted) and combined. . Note that the reason why the coordinate conversion coefficient is obtained and the position is corrected each time is that there is a possibility that a position shift occurs due to movement of the subject or the camera during continuous shooting.

  On the other hand, in the synthesizing process 2, both the left buffer 18a and the right buffer 18b are used, and the left and right continuous shot images are coordinate-converted to be individually superimposed on the left reference image and the right reference image.

  Specifically, as shown in FIG. 10, first, the control unit 17 tracks a feature point of the left continuous image from the left reference image (step G11), and obtains a coordinate conversion coefficient from the coordinate position of the feature point (step G11). G12). Then, the left continuous shot image is coordinate-converted based on the coordinate conversion coefficient and added to the left buffer 18a (step G13). At this time, since the left reference image is already stored in the left buffer 18a, the left continuous shot image is subjected to position correction (coordinate conversion) and synthesized.

  Subsequently, the feature point of the right continuous shot image is traced from the right reference image (step G14), and a coordinate conversion coefficient is obtained from the coordinate position of the feature point (step G15). Then, the right continuous shot image is subjected to coordinate conversion based on the coordinate conversion coefficient and added to the right buffer 18b (step G16). At this time, since the right reference image is already stored in the right buffer 18b, the right continuous shot image is subjected to position correction (coordinate conversion) and synthesized.

  Thereafter, the same processing is repeated for all the left and right continuous shot images (step A25). Thereby, in the synthesis process 1, as shown in FIG. 2, N-1 left continuous images and N-1 right continuous images are alternately displayed on the image obtained by superimposing the left and right reference images. A composite image is created by sequentially superimposing the images. Further, in the synthesis process 2, as shown in FIG. 3, a composite image obtained by sequentially superimposing N−1 left continuous shot images on the left reference image and a right continuous image corresponding to N−1 sheets. Two types of composite images are created by sequentially superimposing the copied images.

  When the composite image is obtained in this way, the control unit 17 divides and averages the pixel data of the composite image by the composite number, and then outputs this as a final captured image (step A26). The captured image is displayed on the display unit 20 as necessary, and is compressed in a predetermined format and stored in the storage memory 21.

  When the evaluation value is high (when the compatibility between the left and right images is low), two types of synthesized images are created by the synthesis process 2, but either one of the synthesized images (for example, in the left buffer 18a). (Composite image to be created) may be output, or both composite images may be output and then selected by the user.

  As described above, according to the first embodiment, two image capturing units included in a stereo camera are used, and N images obtained individually from the two image capturing units are superimposed to create a composite image. As a result, the same number of images can be acquired in a continuous shooting time shorter than that of a monocular camera, and a good image with reduced camera shake and noise can be created.

  In addition, since the compatibility is determined by comparing the reference images (the left reference image and the right reference image) that are the initial frames, the calculation load is larger than the method of determining each frame image one by one with a monocular camera. This makes it possible to perform efficient synthesis processing.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. Note that the apparatus configuration is the same as that of the first embodiment (FIG. 1), and therefore processing differences will be described here.

  In the first embodiment, after the compatibility between the left and right reference images is determined, the subsequent continuous images are superimposed as they are without determining whether they can be combined. In the second embodiment, it is determined whether or not synthesis is possible, and those that are inappropriate for synthesis are excluded. The determination material is a deviation (movement distance of the feature point) from the reference image. A continuous shot image that exceeds 1/2 of the left / right parallax (the offset vector h ′ between the left reference image and the right reference image) is excluded because it is affected by the parallax and is inappropriate for synthesis.

  However, as will be described later, in the synthesis process A (a process for superimposing all the continuous shot images on the left reference image) and the synthesis process B (a process for superimposing each of the continuous shot images on the reference image closer to each), The judgment criteria for deviation from the reference image are different. That is, in the synthesis process A, it is determined whether or not the deviation (offset vector) of the continuous shot image to be processed from the midpoint of the left and right reference images is within | h ′ / 2 | If within |, the continuous shot image is superimposed on the left reference image. On the other hand, in the synthesis process B, if the deviation (offset vector) of the continuous shot image to be processed from the midpoint on the left reference image is within | h ′ / 2 |, the continuous shot image is superimposed on the left reference image. In addition, if the deviation (offset vector) of the continuous shot image from the midpoint of the right reference image is within | h ′ / 2 |, the continuous shot image is superimposed on the right reference image.

  Note that the offset vector tracks feature points between each reference image and the continuous shot image, and takes the average of the motion vectors of each pixel obtained from the correspondence of the feature points. In addition, it is better to remove the outlier by RANSAC or the like and perform the calculation using only the feature points of the inlier.

  Also, the distance from the midpoint of the left and right reference images is the offset vector a from the left reference image || a−h ′ / 2 ||, or the offset vector b from the right reference image || b + h ′ / 2. It can be calculated as ||. In this case, the offset vector is ideally the same regardless of whether it is obtained from either of the left and right reference images. Actually, it is better to obtain from a close image to prevent false tracking.

  In calculating the offset vector, instead of the average of the motion vectors of each pixel, for example, one point near the center may be taken or the maximum value of the whole may be taken.

  A specific example is shown in FIG. In the figure, 31 is a left reference image and 41 is a right reference image, which are obtained by simultaneously photographing the same subject (pumpkin in this example) with the left and right imaging units. First, the center positions 32 and 42 of each subject part are determined, and an offset (positional deviation) between them is obtained. In this case, if the z component is ignored, the offset vector on the image plane is represented by h ′.

  Here, in the composition process A, the shift of the continuous shot image to be processed is examined with reference to the midpoint between the center position 32 of the left reference image 31 and the center position 42 of the right reference image 41. On the other hand, in the synthesis process B, the shift of the continuous shot image to be processed is examined with reference to each of the center position 32 of the left reference image 31 and the center position 42 of the right reference image 41.

  FIG. 12 schematically shows the criteria for determining whether or not the synthesis process A and the synthesis process B can be combined. FIG. 12A shows a range Ea of continuously shot images that are determined to be compositable by the compositing process A. Note that Eo in the figure indicates a midpoint between the center position 32 of the left reference image 31 and the center position 42 of the right reference image 41. That is, an image within the range of | h ′ / 2 | with respect to the midpoint Eo is treated as a continuous shot image suitable for synthesis.

  FIG. 12B shows the range of continuous shot images that are determined to be compositable in the synthesis process B. If the continuous shot image to be processed is within the range Eb1, the left reference image and the range Eb2 If there is, it is superimposed on the right reference image. That is, an image within the range of | h ′ / 2 | from the center position 32 of the left reference image 31 or the center position 42 of the right reference image 41 is treated as a continuous shot image suitable for synthesis, and the closest reference image Will be synthesized.

  Hereinafter, a specific processing operation according to the second embodiment will be described with reference to a flowchart.

  The basic processing flow is the same as that in FIG. 4 in the first embodiment, and the synthesis processing 1 in step A23 and the synthesis processing 2 in step A24 are different. That is, in the second embodiment, when the evaluation value obtained by the model non-conformity evaluation process is lower than the reference value (that is, when the misalignment between the left and right images is small and the composite compatibility is high), As an alternative to the synthesis process 1, a synthesis process 1 'is executed. Further, when the evaluation value obtained by the model non-conformity evaluation process is higher than the reference value (that is, when the positional deviation between the left and right images is large and the composite compatibility is low), as an alternative to the synthesis process 2 A synthesis process 2 'is executed.

  As shown in FIG. 13, in the synthesis process 1 ′, the synthesis process A is performed on the left continuous shot image and the right continuous shot image (steps H11 and H12). As shown in FIG. 15, in the synthesis process 2 ′, the synthesis process B is performed on the left continuous shot image and the right continuous shot image (steps J11 and J12).

  FIG. 14 is a flowchart showing the synthesis process A. Note that the image P in the figure is a left continuous shot image or a right continuous shot image to be processed.

  First, the control unit 17 tracks the feature point of the image P from the left reference image (step I11), and calculates the distance from the midpoint of the left and right reference images to the feature point (step I12). If this distance is within a predetermined range, that is, within | h ′ / 2 | (Yes in Step I13), the control unit 17 determines that the image P is a continuous image suitable for synthesis, and the feature point After obtaining the coordinate conversion coefficient from the coordinate position (step I14), the image P is subjected to coordinate conversion based on the coordinate conversion coefficient and added to the left buffer 18a (step I15). At this time, since the left reference image and the right reference image are combined in the left buffer 18a, the image P is further subjected to position correction (coordinate conversion) and combined.

  On the other hand, if the distance from the midpoint of the left and right reference images to the feature point exceeds a predetermined range, that is, exceeds | h ′ / 2 | (No in step I13), an error due to parallax becomes a problem. Therefore, it is determined that the image P is a continuous shot image inappropriate for synthesis, and the synthesis process is not performed.

  FIG. 16 is a flowchart showing the synthesis process B. Note that the image P in the figure is a left continuous shot image or a right continuous shot image to be processed.

  First, the control unit 17 tracks the feature points of the image P from the left reference image, and also tracks the feature points of the image P from the right reference image (Steps K11 and K12), from the midpoint of the left reference image to the feature points. And a second distance from the middle point of the right reference image to the feature point are calculated (steps K13 and K14).

  If the first distance from the middle point of the left reference image to the feature point is within a predetermined range, that is, within | h ′ / 2 | (Yes in step K15), the control unit 17 determines the image P as the image P. In order to synthesize the left reference image, after obtaining a coordinate conversion coefficient from the coordinate position of the feature point (step K16), the image P is coordinate-converted based on the coordinate conversion coefficient and added to the left buffer 18a (step K17). ). At this time, since the left reference image is already stored in the left buffer 18a, the image P is subjected to position correction (coordinate conversion) and synthesized on the left reference image.

  On the other hand, when the first distance exceeds a predetermined range, the second distance from the midpoint of the right reference image to the feature point is determined next. As a result, if the second distance is within a predetermined range, that is, within | h ′ / 2 | (Yes in Step K18), the control unit 17 may combine the image P with the right reference image. After obtaining the coordinate conversion coefficient from the coordinate position of the point (step K19), the image P is subjected to coordinate conversion based on the coordinate conversion coefficient and added to the right buffer 18b (step K20). At this time, since the right reference image is already stored in the right buffer 18b, the image P is subjected to position correction (coordinate conversion) and synthesized on the right reference image.

  In the first embodiment, the left continuous shot image is simply added to the left buffer 18a and the left continuous shot image is added to the right buffer 18b. However, when the camera shake is larger than half of the inter-camera distance, the left continuous shooting is added. Since the image may be closer to the right reference image than to the left reference image, it can be said that it is reasonable to superimpose the image on the closer reference image.

  On the other hand, if | h ′ / 2 | is exceeded from any of the reference images (No in Step K18), an error due to parallax becomes a problem, so that the image P is a continuous shot image inappropriate for synthesis. Judgment is not performed by combining.

  Thereafter, the same processing is repeated for all the left and right continuous shot images. Thereby, in the synthesis process 1 ′, one synthesized image is created using only the left and right continuous shot images suitable for the synthesis. In the synthesis process 2 ', the left and right continuous shot images are divided into an image close to the left reference image and an image close to the right reference image to create two types of composite images. In this case, as the output image, either one of the composite images (for example, the composite image created in the left buffer 18a) may be output, or after both composite images are output, the user may select them. . It is also possible to output a composite image having a larger composite number.

  As described above, according to the second embodiment, when the left and right continuous shot images are superimposed on the reference image, an image inappropriate for synthesis is excluded, so that a better image is synthesized. As a result.

  In addition, the continuous shot images are divided and superimposed on the left and right reference images with the least positional deviation, so that time-series errors occur due to movement of the subject or camera during continuous shooting, for example. If it is, an image with less influence can be obtained as a synthesis result.

  It should be noted that the threshold value in the synthesis process A can be widened to some extent if it conforms to the conversion model. For example, instead of | h ′ / 2 |, it is possible to use | h ′ |. That is, it is possible to set a threshold value by a constant multiple of the distance between images and determine whether or not to synthesize.

  In each of the above embodiments, a stereo camera in which two camera units (imaging units) are provided on the left and right has been described. However, two cameras may be provided in the vertical direction and the like.

  A multi-lens camera in which three or more camera units are integrated may be used. In this case, the composition process is performed by the same method as described above using the images obtained by simultaneously driving the cameras.

  Further, the camera optical axis is not parallel, and the offset between the cameras may not be only in the horizontal direction (because coordinate conversion can be performed if necessary).

  Also, when using a single composite image, the left image composite buffer is used to superimpose each image on the left reference image, but the right image composite buffer is used to Each image may be superimposed on the reference image in order.

  Further, although it has been described that the composition process is performed after the continuous shooting, the composition process may be performed during the continuous shooting. In this case, the composition processing is determined by comparing the left and right reference images obtained first, and then the left and right continuous captured images obtained are overlapped while being distributed to the left buffer or the right buffer according to the determined composition processing. To go.

  In addition to a digital camera, the present invention can be applied to all of electronic devices having a photographing function such as a mobile phone with a camera.

  In short, the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying constituent elements without departing from the scope of the invention in the implementation stage. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the respective embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  Further, the method described in the above-described embodiment is a program that can be executed by a computer, for example, recording on a magnetic disk (flexible disk, hard disk, etc.), optical disk (CD-ROM, DVD-ROM, etc.), semiconductor memory, etc. The program can be written on a medium and applied to various apparatuses, or the program itself can be transmitted through a transmission medium such as a network and applied to various apparatuses. A computer that implements this apparatus reads a program recorded on a recording medium or a program provided via a transmission medium, and performs the above-described processing by controlling operations by this program.

FIG. 1 is a block diagram showing a configuration when a stereo camera is taken as an example of the imaging apparatus according to the first embodiment of the present invention. FIG. 2 is a diagram for explaining a method of creating a composite image when the evaluation value is low in the embodiment. FIG. 3 is a diagram for explaining a method of creating a composite image when the evaluation value is high in the embodiment. FIG. 4 is a flowchart showing image processing (main processing) of the stereo camera in the embodiment. FIG. 5 is a flowchart showing a model incompatibility evaluation process in the embodiment. FIG. 6 is a flowchart showing another model incompatibility evaluation process in the embodiment. FIG. 7 is a flowchart showing the process of composite image initialization 1 in the embodiment. FIG. 8 is a flowchart showing the process of composite image initialization 2 in the embodiment. FIG. 9 is a flowchart showing the process of the synthesis process 1 in the embodiment. FIG. 10 is a flowchart showing the process of the synthesis process 1 in the embodiment. FIG. 11 is a specific example for explaining an image composition method according to the second embodiment of the present invention. FIG. 12 is a diagram for explaining a range of deviation of continuously shot images that are determined to be compositable by the compositing process A and the compositing process B in the embodiment. FIG. 13 is a flowchart showing the synthesis process 1 'in the same embodiment. FIG. 14 is a flowchart showing a synthesis process A included in the synthesis process 1 ′ in the embodiment. FIG. 15 is a flowchart showing the synthesis process 2 'in the same embodiment. FIG. 16 is a flowchart showing a synthesis process B included in the synthesis process 2 ′ in the embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Stereo camera, 11a, 11b ... Camera part, 12a, 12b ... Shooting lens, 13a, 13b ... Image sensor, 14a, 14b ... Signal processing part, 15 ... Motor, 16a, 16b ... Buffer memory, 17 ... Control part, 18a ... Left buffer (first image composition buffer), 18b ... Right buffer (second image composition buffer), 19 ... Operation section, 19a ... Shutter key, 19b ... Mode key, 20 ... Display section, 21 ... Memory for storage.

Claims (7)

An imaging device that creates a composite image from a plurality of images continuously shot using at least two imaging units,
First, an image obtained from each of the imaging units is used as a reference image, and compatibility determination means for determining compatibility between these reference images;
When it is determined that the suitability is high by the suitability judging means, the reference images are aligned and superimposed, and a predetermined number of continuous shot images following each of the reference images are aligned. First composite image creation means for creating one composite image by sequentially superimposing;
When it is determined that the suitability is low by the suitability judging means, each of the reference images is used individually, and a predetermined number of continuous images following each of the reference images are used as any of the reference images. An image pickup apparatus comprising: a second synthesized image creating unit that creates a plurality of synthesized images by sequentially superimposing while aligning.   The imaging apparatus according to claim 1, further comprising a warning unit that warns when the compatibility determination unit determines that the compatibility is low.   2. The imaging according to claim 1, wherein the first synthesized image creating unit superimposes a predetermined number of continuous images following each of the reference images excluding those not suitable for synthesis. apparatus.   The imaging apparatus according to claim 1, wherein the second composite image creating unit superimposes a predetermined number of continuous images following each of the reference images on a corresponding reference image.   The second composite image creating means distributes and superimposes a predetermined number of continuous shot images following each of the reference images to one of the reference images with a smaller positional deviation. The imaging apparatus according to 1. An image processing method for creating a composite image from a plurality of images continuously shot using at least two imaging units,
A compatibility determination step for determining a compatibility between these reference images, with the first image obtained from each of the imaging units as a reference image;
When it is determined that the suitability is high in this suitability judging step, the reference images are aligned and superimposed, and a predetermined number of continuous shot images following each of the reference images are aligned. A first composite image creating step of creating one composite image by sequentially superimposing;
When it is determined that the suitability is low by the suitability judging step, each of the reference images is used individually, and a predetermined number of continuous images following each of the reference images are used as any of the reference images. And a second composite image creating step of creating a plurality of composite images by sequentially superimposing them while aligning. A program executed by a computer mounted on an imaging device that creates a composite image from a plurality of images continuously shot using at least two imaging units,
In the computer,
A compatibility determination function for determining the compatibility between these reference images, using the first image obtained from each of the imaging units as a reference image,
When it is determined that the suitability is high by the suitability judging function, the reference images are aligned and superimposed, and a predetermined number of continuous shot images following each of the reference images are aligned. A first composite image creation function for creating one composite image by sequentially superimposing;
When it is determined that the suitability is low by the suitability judging function, each of the reference images is used individually, and a predetermined number of continuous shot images following each of the reference images are assigned to any of the reference images. A program for realizing a second composite image creation function for creating a plurality of composite images by sequentially superimposing them while aligning them.

Images