JP2007049545A - Image composing apparatus, image composing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a composite image while reducing computational complexity required for image processing and keeping sufficient tracing accuracy. <P>SOLUTION: Pyramid hierarchical layers are configured by stepwise reducing images being composite objects through the use of a prescribed reduction size of an original image for an uppermost layer. Then a feature point is traced sequentially from a lowermost layer toward the uppermost layer. In this case, the motion of the feature point is converged by multiplying a prescribed coefficient with a vector denoting the motion of the feature point. Thus, the computational complexity is reduced by eliminating the need for image processing at the original image size in tracing the feature point. Moreover, the convergence is improved by multiplying the prescribed coefficient with the motion vector of the feature point so as to generate the composite image while keeping sufficient tracing accuracy. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image composition device, an image composition method, and a program used in an imaging device such as a digital camera.

  In recent years, with digital cameras, there is a method of obtaining a clear image with reduced noise by performing continuous shooting with a short exposure time without causing camera shake and overlaying a plurality of images obtained by the continuous shooting. It is considered.

  Here, even if the interval between the frames in the continuous shooting is short, there is a possibility that a positional deviation occurs between the camera and the subject during that time. For this reason, when superimposing and synthesizing the images obtained by continuous shooting, it is necessary to correct the positional deviation and superimpose the images.

  For example, Patent Literature 1 discloses a technique for calculating an optical flow from two images and performing alignment between the two images based on the optical flow. The optical flow is an apparent movement of an object as viewed from the camera.

  One of the optical flow estimation methods is called a gradient method. The gradient method is a method of calculating the motion from the gradient (first derivative) of the pixel value (brightness). In other words, optical flow estimation by the gradient method means that the relation between the spatio-temporal derivative and the optical flow is derived from the assumption that the brightness of the point on the object does not change after movement, and is used to It estimates motion.

  In the image synthesis technique, a determinant H of the projective transformation between the two images is determined by tracking the feature points on the image using such a gradient method, and the positional deviation between the images is determined using the determinant H. Is corrected and superimposed.

  More specifically, for example, as shown in FIG. 8, when image 1 and image 2 are to be combined, first, a predetermined number of feature points are randomly extracted from reference image 1, and these are extracted. Are tracked using the gradient method on the image 2 that is superimposed on the image 1. If P1, P2, and P3 in the figure are feature points extracted from the image 1, the locations of the three feature points P1, P2, and P3 are traced in the image 2.

  When the feature points P1, P2, and P3 can be tracked, a determinant H in the projective transformation is calculated from the relationship between the coordinate positions of the feature points P1, P2, and P3 on the image 2 and the coordinate positions on the image 1. Then, all feature points on the image 1 are shifted based on the determinant H (image 1 ′), and if they match each feature point on the image 2 with a certain degree of accuracy (image 1 ′ + image 2). Then, the determinant H is used to correct the positional deviation between the image 1 and the image 2 and superimpose them.

  It should be noted that in practice, processing such as extraction and tracking of such feature points is repeated several times to finally select a determinant H having the highest evaluation value.

  By the way, when tracking feature points between images, if the feature points are tracked one by one for all the pixels on the image, the amount of calculation increases and it takes a very long time. In view of this, a method is used in which an image with a resolution lowered stepwise in advance is used, and feature points are tracked in order from the layer with the lowest resolution toward the layer with the highest resolution. This is called pyramidization.

  FIG. 9 is a diagram for explaining a conventional method of tracking feature points by pyramidization. K in the figure is a hierarchical parameter and takes a value of k = 0 to m. f represents the original image size, and ∂f / ∂x and ∂f / ∂y represent the first derivative of the point (x, y).

  First, a noise removal filter is applied to the original image, and the image is set as layer 0. Also, layer 1 is subsampled (reduction processing by thinning) with respect to layer 0, and layer 2 is subsampled with respect to layer 1, and a pyramid hierarchy up to layer m is formed in order.

  Here, the tracking of feature points starts with the lowest layer m, applies a differential filter to the image of layer 0 from the layer m to the highest layer 0, and obtains the motion vector V of the feature points by the gradient method described above.

  That is, for example, when the feature point of the image 1 (reference image) is tracked by the image 2 continuous with the image 1, first, the tracking start point Y of the image 2 is set to Y = X in the lowest layer, The motion vector V of the feature point is obtained by the gradient method. Then, the next point is updated to Y = Y + V, and the motion vector V is obtained again from the point by the gradient method. Similarly, the next point is updated to Y = Y + V. This is repeated, and the calculation is terminated when V becomes sufficiently small or when the predetermined number of times is repeated.

  Subsequently, the same tracking is performed in the upper hierarchy. In this case, the image size increases each time the hierarchy is increased by one. If it is assumed that one side of the image doubles for each layer, X and Y obtained in the lower layer are doubled, and feature point tracking (motion vector calculation) is performed using this as an initial value. repeat.

  Finally, the feature points are tracked while the motion vectors are successively inherited up to the highest layer. Then, the determinant H of the projective transformation described above is calculated from the relative relationship between the position of the feature point on the image 2 finally obtained and the position of the feature point on the image 1.

  Specific processing operations are shown in FIGS. FIG. 10 is a flowchart showing the overall flow of a conventional image composition process. FIG. 11 is a flowchart showing a feature point extraction process included in the image composition process, and FIG. 12 is a flowchart showing a motion vector calculation process included in the feature point extraction process.

  Now, a description will be given assuming that a plurality of continuous images are stored in a memory (not shown) by continuous shooting, and each image is sequentially read out from the memory and superimposed to create one composite image.

  In the flowchart of FIG. 10, steps D11 to D19 indicate pyramidal processing for the reference image, and steps D20 to D27 indicate pyramidal processing for the image to be superimposed on the reference image.

  That is, first, an image serving as a reference for image composition is read from the memory (step D11). At that time, as shown in FIG. 9, the layer number k = 0 (step D12), and a noise removal filter is applied to the reference image to generate a layer 0 image (step D13). After extracting feature points from the layer 0 image (step D14), a differential filter is applied to the image to obtain a gradient of pixel values (step D15).

  Subsequently, k is updated by +1 (step D16), and a layer 1 image is created by thinning the layer 0 image into 1/2 size vertically and horizontally by the sub-sampling filter (step D17). Then, a differential filter is applied to the layer 1 image to obtain the gradient of the pixel value (step D18).

  Thereafter, while updating k, the same processing is repeated until k = m (step D19). As a result, as shown in FIG. 9, a hierarchical image with the layer 0 as the highest level and the layer m as the lowest level is created.

  Subsequently, an image to be overlaid on the reference image is read from the memory (step D20), k = 0 is set (step D21), and the same pyramidal process is performed on the image (step D22). To D27), images of layers 0 to m are created.

  When the reference image and the image to be superimposed on it are pyramidal in this way, next, feature point tracking processing is performed between the respective layer images (step D28), and the coordinate position of the feature point obtained by the tracking processing is performed. Based on the above, both images are aligned and synthesized (step D29). By repeating this for a predetermined number of sheets (step D30), one composite image is created.

  Next, the feature point tracking process executed in step D28 will be described.

  As shown in the flowchart of FIG. 11, first, i = 0 is set (step E11). i is the number of the feature point to be processed, and takes a value of 0 to n.

  Now, assuming that the position of the feature point of the reference image is Xi and the position (search position) of the feature point of the image to be superimposed on the reference image (hereinafter referred to as a tracked image) is Yi, the feature point position in layer m is (Step E12).

Xi = Xi / 2 ^m
Yi = Xi
Here, since tracking of feature points starts from the lowest layer, k = m is set (step E13). Also, the number of repetitions (that is, the number of tracking of feature points for each layer) j is set to an initial value 0 (step E14), and the motion vector Vj of the feature point i on the tracked image is calculated (step E15).

  As shown in the flowchart of FIG. 12, the motion vector Vj is calculated from Xi and Yi (step F11). In this case, Yi = Xi and Vj = 0 at the tracking start time. Using this point as the tracking start point, the motion vector Vj is obtained by the gradient method, and the next point is updated to Yi = Yi + Vj (step F12).

  If the amount of motion of Vj obtained at this time has converged to a predetermined value or less (Yes in step F13), the process ends. If it is larger than the predetermined value (No in step F13), the process returns to step F11 and the same calculation is repeated. At that time, the number of repetitions j is updated by +1 (step F15), and when the value reaches the preset maximum number of repetitions (Yes in step F14), the process is terminated.

  Returning to FIG. 11, when the motion vector Vj of the feature point in the layer m is obtained, next, Xi and Yi are doubled as follows in order to perform the feature point tracking process in one layer above (Step E16).

Xi = Xi × 2
Yi = Yi × 2
Then, the value of the hierarchy parameter k is decremented by -1 (step E18), the process returns to step E14, and the same tracking process is performed in the hierarchy one level higher. At this time, the tracking process is performed by taking over the value of the motion vector Vj obtained in the lower layer. When tracing up to the highest layer 1 (Yes in step E17), the same processing is repeated for another feature point. When i = n is reached (Yes in step E19), that is, when the tracking processing for n feature points is completed, the processing is completed.

In the image synthesizing process in step D29, a determinant H of projective transformation is obtained based on the coordinate positions of these feature points, and the two images (reference image and tracked image) are aligned according to the determinant H. Will be synthesized.
JP 2001-167249 A

  As described above, when tracking feature points between images, a method is used in which each image is made into a pyramid and tracked from the layer with the lowest resolution to the layer with the highest resolution in order.

  However, when this method is implemented by software, there is a problem in that the number of filter operations to be applied to a high-resolution image increases, and the amount of processing increases.

  The present invention has been made in view of the above points, and an image composition device, an image composition method, and a program capable of reducing the amount of calculation required for image processing and creating a composite image while maintaining sufficient tracking accuracy. The purpose is to provide.

  An image composition device according to claim 1 of the present invention tracks a feature point extracted from a first image with a second image, and extracts the first image and the second image from the coordinate position of the feature point. An image composition device for aligning and compositing, wherein a pyramidal means for composing a pyramid hierarchy by reducing the first and second images stepwise with a predetermined reduced size of the original image as the highest layer A feature point tracking means for tracking the feature points from the lowest layer to the highest layer in order by the pyramidization means, and a vector indicating the motion of the feature points obtained by the feature point tracking means. And a converging means for converging the movement of the feature points by multiplying the coefficient.

  According to such a configuration, when the feature points are tracked, image processing with the original image size is not necessary, and the amount of calculation is significantly reduced accordingly. Further, by multiplying a motion vector of a feature point by a predetermined coefficient, convergence can be improved, and a composite image can be created while maintaining sufficient tracking accuracy.

  According to a second aspect of the present invention, in the image synthesizing apparatus according to the first aspect, the pyramidizing means steps the first and second images with a quarter size of the original image as the highest layer. It is characterized in that a pyramid hierarchy is formed by reducing the size.

  According to such a configuration, the feature points are tracked in order from the lowest layer, and finally the tracking can be completed with a quarter size of the original image. Can be

  According to a third aspect of the present invention, in the image synthesizing apparatus according to the first aspect, the convergence means increases the convergence in a stepwise manner according to the number of tracking of the feature points in each layer. It is characterized by changing.

  According to such a configuration, motion vectors of feature points can be prevented from diverging and can be stably converged within a predetermined number of times of tracking.

  According to a fourth aspect of the present invention, a feature point extracted from a first image is tracked with a second image, and the first image and the second image are obtained from the coordinate position of the feature point. An image composition method for aligning and compositing, wherein a first reduction step of the original image is used as the highest layer, and the first and second images are reduced stepwise to form a pyramid hierarchy And a second step for tracking feature points from the lowest layer in the first step toward the highest layer in order, and a vector indicating the motion of the feature points obtained by the second step. And a third step of converging the movement of the feature points by multiplying by a predetermined coefficient.

  According to such an image synthesizing method, the same effects as those of the first aspect of the invention can be achieved by executing the processing according to each of the steps.

  The program according to claim 5 of the present invention tracks a feature point extracted from the first image with the second image, and aligns the first image and the second image from the coordinate position of the feature point. A first function for composing a pyramid hierarchy by gradually reducing the first and second images with a predetermined reduced size of the original image as the uppermost layer. A second function for tracking feature points from the lowest layer to the highest layer in order from the lowest layer by the first function, and a vector indicating the motion of the feature points obtained by the second function are predetermined. And a third function for converging the movement of the feature points by multiplying by the coefficient.

  Therefore, when the computer executes the program for realizing each function, the same effects as those of the first aspect of the invention can be achieved.

  According to the present invention, when tracking feature points by pyramidization, it is possible to reduce the amount of calculation required for image processing and create a composite image while maintaining sufficient tracking accuracy.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram showing an external configuration when a digital camera is taken as an example of an image composition device according to an embodiment of the present invention. FIG. 1 (a) is mainly a front configuration, and FIG. It is a perspective view which shows the structure of a back surface.

  The digital camera 1 has a photographing lens 3, a self-timer lamp 4, an optical finder window 5, a strobe light emitting unit 6, a microphone unit 7 and the like on the front surface of a substantially rectangular thin plate-like body 2 on the upper surface (for the user). On the right end side, a power key 8 and a shutter key 9 are provided.

  The power key 8 is a key operated every time the power is turned on / off, and the shutter key 9 is a key for instructing a photographing timing at the time of photographing.

  Also, on the back of the digital camera 1, a shooting mode (R) key 10, a playback mode (P) key 11, an optical viewfinder 12, a speaker unit 13, a macro key 14, a strobe key 15, a menu (MENU) key 16, a ring A key 17, a set (SET) key 18, a display unit 19, and the like are provided.

  The shooting mode key 10 is operated automatically from the power-off state to automatically turn on the power and shift to the still image shooting mode. On the other hand, by repeatedly operating from the power-on state, the still image mode and the moving image mode are switched. Set cyclically. The still image mode is a mode for photographing a still image. The moving image mode is a mode for shooting a moving image.

  The shutter key 9 is commonly used for these photographing modes. That is, in the still image mode, a still image is taken at the timing when the shutter key 9 is pressed. In the moving image mode, shooting of a moving image is started at a timing when the shutter key 9 is pressed, and shooting of the moving image is ended when the shutter key 9 is pressed again.

  When the playback mode key 11 is operated from the power-off state, the playback mode key 11 is automatically turned on to enter the playback mode.

  The macro key 14 is operated when switching between normal shooting and macro shooting in the still image shooting mode. The strobe key 15 is operated when switching the light emission mode of the strobe light emitting unit 6. The menu key 16 is operated when selecting various menu items including the continuous shooting mode. The ring key 17 is integrally formed with item selection keys in the up, down, left, and right directions, and the set key 18 located in the center of the ring key 17 sets the item selected at that time. To operate.

  The display unit 19 is composed of a color liquid crystal panel with a backlight, and displays a through image on the monitor as an electronic viewfinder in the photographing mode, and reproduces and displays the selected image and the like in the reproduction mode.

  Although not shown, the digital camera 1 has a memory card slot for attaching / detaching a memory card used as a recording medium, a serial interface connector for connecting to an external personal computer, etc., for example, USB (Universal). Serial Bus) connector and the like are provided.

  FIG. 2 is a block diagram showing an electronic circuit configuration of the digital camera 1.

  The digital camera 1 includes an optical lens device 21, an image sensor 22, a memory 23, a display device 24, an image processing device 25, an operation unit 26, a computer interface unit 27, an external storage IO device 28, a program code storage device 29, and a CPU 30. A memory card 31 is provided.

  The optical lens device 21 includes a lens optical system including a focus lens and a zoom lens (not shown) constituting the photographing lens 3 and a driving unit thereof, and collects light from the photographing target on the image sensor 22. Light to form an image.

  The image sensor 22 is for capturing a formed image as digitized image data, and is configured by, for example, a CCD (Charge Coupled Device). If the image sensor 22 is controlled by the CPU 30 and the shutter key 9 is not pressed, digital image data having a low preview resolution is generated, and this image data is periodically stored in the memory 23 at intervals of about 30 sheets per second. To send. Further, when the shutter key 9 is pressed, the image sensor 22 generates image data with high resolution and sends the generated image data to the memory 23. Further, the image sensor 22 can set imaging sensitivity (ISO sensitivity) by the CPU 30.

  The memory 23 temporarily stores a low-resolution preview image from the image sensor 22, high-resolution image data, original image data processed by the image processing device 25, and processed image data. The memory 23 sends the temporarily stored image data to the display device 24 or the image processing device 25.

  The display device 24 is for displaying an image on the display unit 19 which is a liquid crystal monitor. The display device 24 displays a low-resolution preview image or a high-resolution image temporarily stored in the memory 23 on the display unit 19.

  The image processing device 25 is for performing image processing such as image data compression on the image data temporarily stored in the memory 23.

  In addition to the shutter key 9, the operation unit 26 includes a power key 8, a shooting mode key 10, a playback mode key 11, a macro key 14, a strobe key 15, a menu key 16, a ring key 17, a set key 18, and the like. Signals associated with these key operations are sent directly to the CPU 30.

  The computer interface unit 27 operates as a USB storage class driver when the digital camera 1 is connected to a computer (not shown). Thus, when the computer is connected to the digital camera 1, the computer handles the memory card 31 as an external storage device of the computer.

  The external storage IO device 28 inputs and outputs image data and the like with the memory card 31. The memory card 31 stores image data and the like supplied from the external storage IO device 28.

  The program code storage device 29 is for storing a program executed by the CPU 30, and is configured by a ROM, a flash memory, or the like.

  The CPU 30 controls the entire system according to a program stored in the program code storage device 29. The memory 23 is also used as a work memory for the CPU 30.

  When operation information is transmitted from the operation unit 26 by pressing a switch key of the operation unit 26, the CPU 30 performs image sensor 22, memory 23, display device 24, and image processing device based on the operation information. 25 etc. are controlled.

  Specifically, when operation information indicating that the shooting mode key 10 is pressed is transmitted from the operation unit 26, the CPU 30 sets each unit to the shooting mode. In this state, if the shutter key 9 is not pressed, the image sensor 22 is set to the preview mode, and if the shutter key 9 is pressed, the high-resolution mode for reading a high-resolution image to be captured is set. At this time, if the continuous shooting mode is set by operating the menu key 16, a predetermined number of image reading processes are executed at predetermined time intervals as the shutter key 9 is pressed.

  When the operation information indicating that the playback mode key 11 is pressed is transmitted, the CPU 30 sets each unit to the playback mode.

  Further, the CPU 30 records preview image and high-resolution image data on the memory card 31 via the external storage IO device 28, and reads the recorded image data from the memory card 31. For example, the CPU 30 records image data compressed in the JPEG format in the memory card 31.

  When the image data is temporarily stored in the memory 23, the CPU 30 records the preview image and the high-resolution image data in different storage areas. Further, the CPU 30 records the image data separately in the image file in the memory card 31.

  Further, the CPU 30 records preview image and high-resolution image data on the memory card 31 via the external storage IO device 28, and reads the recorded image data from the memory card 31. The CPU 30 creates an image file for storing image data in the memory card 31.

  Next, image composition processing by the digital camera 1 having the above-described configuration will be described.

  As described with reference to FIG. 9, in the general pyramidization method, an image obtained by applying a noise removal filter to an original image is set as layer 0, and the subsampled layer 0 is layer 1 and layer 1 Are sub-sampled to layer m in order as layer 2. When tracking feature points between images, tracking is performed from the lowest layer m to the highest layer 0 having the highest resolution in order.

  Here, as a matter of course, all the filtering processes increase as the resolution increases. Therefore, in the method as shown in FIG. 9, the filter processing for the layer 0 which is the original image size is the largest, the processing amount becomes enormous when implemented by software, and the tracking processing takes time. There are problems such as.

  Therefore, in the present embodiment, as shown in FIG. 3, the pyramid hierarchy is configured with the 1/4 size (vertical / horizontal size) of the original image as the highest layer (k = 1), and feature point extraction processing is performed. The tracking process is performed in this pyramid hierarchy. Thus, as is apparent from FIG. 3, the amount of filter processing required to create an image of each layer is greatly reduced, and the tracking processing time is also shortened.

  However, as a price, there is a problem that the accuracy of the optical flow is lowered. This is because each Vj vibrates in the calculation of the motion vector of the feature points described in FIG. 12 (Yi = Xi + V0 + V1 +... + Vj). To solve this problem, instead of simply adding each Vj to Yi, a method is used in which a different coefficient is applied to Vj depending on j (the number of repetitions), and the added value is gradually reduced.

  Specific processing will be described below with reference to the flowcharts of FIGS. The processing shown in the following flowcharts is executed according to the procedure described in the program by the CPU 30 that is a microcomputer reading the program stored in the program code storage device 29.

  FIG. 4 is a flowchart showing an overall flow of image composition processing of the digital camera 1. FIG. 5 is a flowchart showing a feature point extraction process included in the image composition process, and FIG. 6 is a flowchart showing a motion vector calculation process included in the feature point extraction process.

  Now, it is assumed that a plurality of continuous images are stored in the memory 23 shown in FIG. 2 by continuous shooting, and each image is sequentially read from the memory 23 and superimposed to create one composite image. To do.

  In the flowchart of FIG. 4, steps A11 to A19 indicate pyramidal processing for a reference image, and steps A20 to A27 indicate pyramidal processing for an image to be superimposed on the reference image.

  That is, first, the CPU 30 reads an image serving as a reference for image composition from the memory 23 (step A11). At that time, as shown in FIG. 3, the CPU 30 sets the number of hierarchies k = 1 (step A12), applies a noise removal and sub-sampling synthesis filter to the reference image, and ¼ (vertical / horizontal 1 / vertical) of the original image. 2) An image of layer 1 reduced in size is generated (step A13). In the present embodiment, the image size of this layer 1 is the highest level of the pyramid hierarchy.

  After extracting feature points from the layer 1 image (step A14), the CPU 30 applies a differential filter to the image to obtain a gradient of pixel values (step A15).

  Subsequently, the CPU 30 updates k by +1 (step A16), and creates a layer 2 image by thinning the layer 1 image into 1/2 size vertically and horizontally by the sub-sampling filter (step A17). Then, a differential filter is applied to the layer 2 image to obtain the gradient of the pixel value (step A18).

  Thereafter, while updating k, the same process is repeated until k = m (step A19). As a result, as shown in FIG. 3, a hierarchical image having the highest layer 1 and the lowest layer m is created.

  Subsequently, the CPU 30 reads an image to be superimposed on the reference image from the memory 23 (step A20), sets k = 1 (step A21), and performs the same pyramidization process on the image. (Steps A22 to A27), the images of layers 1 to m are created. In this case, as in the case of the reference image, in layer 1, the image is reduced to 1/4 (vertical / horizontal 1/2) size of the original image by applying a noise reduction and sub-sampling synthesis filter to the image. Create

  When the reference image and the image to be superimposed on it are pyramidal in this way, next, feature point tracking processing is performed between the respective layer images (step A28), and the coordinate position of the feature point obtained by the tracking processing is performed. Based on the above, both images are aligned and synthesized (step A29). By repeating this for a predetermined number of sheets (step A30), one composite image is created.

  Next, the feature point tracking process executed in step A28 will be described.

  As shown in the flowchart of FIG. 5, first, i = 0 is set (step B11). i is the number of the feature point to be processed, and takes a value of 0 to n.

  Now, assuming that the position of the feature point of the reference image is Xi and the position (search position) of the feature point of the image to be superimposed on the reference image (hereinafter referred to as a tracked image) is Yi, the feature point position in layer m is (Step B12).

Xi = Xi / ^{2 m}
Yi = Xi
Here, since tracking of feature points starts from the lowest layer, k = m is set (step B13). Also, j (number of repetitions) is set to an initial value 0 (step B14), and a motion vector Vj of the feature point i on the tracked image is calculated (step B15).

  As shown in the flowchart of FIG. 6, the CPU 30 calculates a motion vector Vj from Xi and Yi (step C11). In this case, Yi = Xi and Vj = 0 at the tracking start time. Using this point as a tracking start point, a motion vector Vj is obtained by the gradient method.

  Here, in order to suppress the vibration of the motion vector Vj, the CPU 30 multiplies Vj by a coefficient that differs depending on j (number of repetitions) (steps C12 to C15). Specifically, when the maximum number of iterations is T and the coefficient is α, α is set under the following conditions.

When 0 ≦ j <T × 1/3, α = 1
When T × 1/3 ≦ j <T × 2/3, α = 1/2
When T × 2/3 ≦ j, α = 1/4
That is, for example, if the maximum number of repetitions T is 12, and if j is less than 4, the coefficient α = 1. Therefore, Vj is a value as it is. On the other hand, when j is 4 to 7 times, the coefficient α is set to 1/2. Therefore, Vj = Vj / 2. If j is 8 times or more, the coefficient α = 1/3 is set, and Vj = Vj / 4.

  In this way, after multiplying the motion vector Vj of the feature point by a coefficient corresponding to j, the next point is updated to Yi = Yi + Vj (step C16). Then, the amount of motion of Vj obtained at this time is compared with a predetermined value, and if the amount of motion of Vj has converged below the predetermined value (Yes in step C17), the processing ends. If it is larger than the predetermined value (No in step C17), the process returns to step C11 and the same calculation is repeated. At that time, the number of repetitions j is updated by +1 (step C19), and when the value reaches the preset maximum number of repetitions (Yes in step C18), the process is terminated.

  Returning to FIG. 5, when the motion vector Vj of the feature point in the layer m is obtained, the CPU 30 next sets Xi and Yi as follows in order to perform the feature point tracking process in one layer above Double it (step B16).

Xi = Xi × 2
Yi = Yi × 2
Then, the CPU 30 decrements the value of the hierarchy parameter k by -1 (step B18), returns to step B14, and performs the same tracking process on the next higher hierarchy. At this time, the tracking process is performed by taking over the value of the motion vector Vj obtained in the lower layer. When tracing to the highest layer 1 (Yes in step B17), the same processing is repeated for another feature point. When i = n is reached (Yes in step B19), that is, when the tracking processing for n feature points is completed, the processing is completed.

  In the image composition processing in step A29, a determinant H of projective transformation is obtained based on the coordinate positions of these feature points, and the two images (reference image and tracked image) are aligned according to the determinant H. Will be synthesized.

  7A and 7B are diagrams for explaining the concept of motion vector convergence. FIG. 7A shows a divergent state, and FIG. 7B shows a convergent state.

  When a certain feature point is tracked by repeating the gradient method between two images, the motion vector Vj gradually converges as shown in FIG. 7A, but at the final stage, the motion vector Vj May vibrate and diverge. This example shows that the motion vector Vj has converged gradually from V1 to V7, and then has entered a divergence state. In this case, since the position of the feature point cannot be accurately specified, the accuracy of the determinant H of the projective transformation is affected.

  Therefore, as indicated by the dotted line in FIG. 7B, the vibration is suppressed by multiplying the motion vector Vj by a predetermined coefficient, and is converged at the final stage. In this example, it is shown that the vector components from V6 to V9 are attenuated by the coefficient and finally become converged. In this case, the value of the coefficient is not constant, and is changed stepwise in the direction of increasing the amount of attenuation (that is, the direction of increasing the convergence) according to the number of repetitions at that time. Specifically, as described with reference to FIG. 6, the change is made such that 1 → 1/2 → 1/4.

  As described above, since the feature points are tracked by the pyramid hierarchy with the 1/4 size of the original image as the highest layer, the original image size is compared with the conventional pyramid hierarchy including the original image size. Thus, the amount of calculation is greatly reduced. Further, by multiplying a motion vector of a feature point by a predetermined coefficient, convergence can be improved, and a composite image can be created while maintaining sufficient tracking accuracy.

  In the above-described embodiment, the ¼ size of the original image is set as the highest layer, but a pyramid hierarchy may be configured with a reduced size other than the ¼ size as the highest layer. However, as the image size in the uppermost layer, which is the final stage of tracking, becomes smaller, the motion vector vibrates and becomes more difficult to converge. Therefore, it is preferable to set a coefficient by which the motion vector is multiplied in consideration of this.

  In the above embodiment, the coefficient to be multiplied by the motion vector is changed to two stages as shown in FIG. 6, but it may be changed more finely in stages, and the timing for multiplying the coefficient by the number of tracking of the feature points is also possible. It can be changed as appropriate. In short, it is only necessary to multiply at appropriate timing an attenuation coefficient that can prevent the motion vector from diverging in each layer and can be stably converged.

  In the above embodiment, the digital camera has been described as an example. However, the present invention is not limited to this, and can be similarly applied to an electronic device having an imaging function such as a mobile phone with a camera. It is.

  Furthermore, a plurality of images continuously photographed by a camera in advance may be given to an information processing apparatus such as a PC (personal computer) and the above-described processing may be performed in the information processing apparatus. As a method for providing an image, in addition to a method for storing an image in a removable memory, the image may be provided via a communication network.

  In short, the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  Further, the method described in the above-described embodiment is a program that can be executed by a computer, for example, recording on a magnetic disk (flexible disk, hard disk, etc.), optical disk (CD-ROM, DVD-ROM, etc.), semiconductor memory, etc. The program can be written on a medium and applied to various apparatuses, or the program itself can be transmitted through a transmission medium such as a network and applied to various apparatuses. A computer that implements this apparatus reads a program recorded on a recording medium or a program provided via a transmission medium, and performs the above-described processing by controlling operations by this program.

FIG. 1 is a diagram showing an external configuration when a digital camera is taken as an example of an image composition device according to an embodiment of the present invention. FIG. 1 (a) is mainly a front configuration, and FIG. It is a perspective view which shows the structure of a back surface. FIG. 2 is a block diagram showing an electronic circuit configuration of the digital camera in the embodiment. FIG. 3 is a view for explaining pyramidization processing by the digital camera in the embodiment. FIG. 4 is a flowchart showing an overall flow of image composition processing of the digital camera in the embodiment. FIG. 5 is a flowchart showing a feature point extraction process included in the image composition process. FIG. 6 is a flowchart showing a motion vector calculation process included in the feature point extraction process. 7A and 7B are diagrams for explaining the concept of motion vector convergence. FIG. 7A shows a divergent state, and FIG. 7B shows a convergent state. FIG. 8 is a diagram for explaining an image synthesis technique using optical flow estimation. FIG. 9 is a diagram for explaining a conventional method of tracking feature points by pyramidization. FIG. 10 is a flowchart showing the overall flow of a conventional image composition process. FIG. 11 is a flowchart showing a feature point extraction process included in the image composition process. FIG. 12 is a flowchart showing a motion vector calculation process included in the feature point extraction process.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Digital camera, 2 ... Body, 3 ... Shooting lens, 4 ... Self-timer lamp, 5 ... Optical finder window, 6 ... Strobe light emission part, 7 ... Microphone part, 8 ... Power key, 9 ... Shutter key, 10 ... Photographing Mode key, 11 ... Playback mode key, 12 ... Optical viewfinder, 13 ... Speaker unit, 14 ... Macro key, 15 ... Strobe key, 16 ... Menu (MENU) key, 17 ... Ring key, 18 ... Set key, DESCRIPTION OF SYMBOLS 19 ... Display part, 21 ... Optical lens apparatus, 22 ... Image sensor, 23 ... Memory, 24 ... Display apparatus, 25 ... Image processing apparatus, 26 ... Operation part, 27 ... Computer interface part, 28 ... External storage IO device, 29 ... Program code storage device, 30 ... CPU, 31 ... memory card.

Claims (5)

An image synthesis device that tracks a feature point extracted from a first image with a second image, aligns the first image with the second image from the coordinate position of the feature point, and synthesizes the image,
Pyramidizing means for forming a pyramid hierarchy by reducing the first and second images in stages with a predetermined reduced size of the original image as the top layer;
Feature point tracking means for tracking the feature points in order from the lowest layer to the highest layer by the pyramidal means;
An image synthesizing apparatus comprising: convergence means for converging the motion of the feature points by multiplying a vector indicating the motion of the feature points obtained by the feature point tracking means by a predetermined coefficient.   2. The image according to claim 1, wherein the pyramidization unit forms a pyramid hierarchy by reducing the first and second images in stages with a quarter size of the original image as a top layer. Synthesizer.   The image synthesizing apparatus according to claim 1, wherein the convergence unit changes the predetermined coefficient in a direction in which the convergence is gradually improved in accordance with the number of tracking of feature points in each layer. An image synthesis method for tracking a feature point extracted from a first image with a second image and aligning and synthesizing the first image and the second image from a coordinate position of the feature point,
A first step of forming a pyramid hierarchy by reducing the first and second images in stages, with a predetermined reduced size of the original image as the top layer;
A second step of tracking the feature points from the lowest layer to the highest layer in order from the first step;
And a third step of converging the motion of the feature points by multiplying a vector indicating the motion of the feature points obtained by the second step by a predetermined coefficient. A feature point extracted from the first image is tracked by the second image, and the first image and the second image are aligned and synthesized from the coordinate position of the feature point,
On the computer,
A first function that configures a pyramid hierarchy by gradually reducing the first and second images with a predetermined reduced size of the original image as the top layer;
A second function for tracking the feature points from the lowest layer to the highest layer in order from the first function;
And a third function for converging the motion of the feature point by multiplying a vector indicating the motion of the feature point obtained by the second function by a predetermined coefficient.

Images