JPH09322061A - Image composing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect how each image is arranged and composed automatically in the case of panorama-composing a plurality of images having overlapped parts. SOLUTION: An area is set to a plurality of images read and expanded from a recording medium 110 in which cross reference points are to be extracted, and a cross reference point detection section 114 detects the cross reference points between the images as to the area, and an evaluation data arithmetic section 115 obtains evaluation data to obtain arrangement information based on the detected cross reference points, and an arrangement information extract section 116 obtains the arrangement information based on the evaluation data and an image composing section 117 arranges each image according to the arrangement information to compose the images into one image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing apparatus suitable for use when a plurality of images taken by a camera are stitched together to create a panoramic image.

[0002]

2. Description of the Related Art Conventionally, an image synthesizing apparatus for obtaining a single panoramic image by joining a plurality of images captured by an electronic camera and partially overlapping each other according to arrangement information indicating their arrangement, For example, JP-A-4-5263
As disclosed in Japanese Patent No. 5 publication, an electronic camera provided with means for detecting a three-dimensional position and azimuth is used to detect the position and azimuth at the time of shooting, and a panoramic image is reproduced based on the position and azimuth information during reproduction. There are things I tried to get. This obtains the arrangement information between images by using a special means for detecting the three-dimensional position and orientation.

[0003]

However, in the above-mentioned conventional apparatus, it is necessary to install special detecting means in the electronic camera in order to obtain the arrangement information between the images, so that the apparatus becomes large in size. Become. Further, it is necessary to record the detected three-dimensional position and orientation in association with each image signal, which complicates the apparatus and newly requires a special image format, which is a problem in terms of cost and the like. .

An object of the present invention is to solve the above problems, and an object thereof is to obtain an image synthesizing apparatus which can easily and accurately extract arrangement information between a plurality of images.

[0005]

According to a first aspect of the invention, corresponding point extracting means for extracting corresponding points between a plurality of input images, and the plurality of images based on the extracted corresponding points. Arrangement information extracting means for extracting arrangement information for arrangement and image combining means for combining the plurality of images based on the extracted arrangement information and corresponding points are provided.

According to a seventh aspect of the present invention, reduced image generating means for generating reduced images of a plurality of input images, and first corresponding point extracting means for extracting corresponding points between the plurality of reduced images are provided. Arrangement information extracting means for extracting arrangement information for arranging the plurality of images based on the extracted corresponding points, and a plurality of the plurality of arrangement information based on corresponding points between the extracted arrangement information and the extracted reduced images. Second corresponding point extracting means for extracting corresponding points between the images, and image synthesizing means for synthesizing the plurality of images based on the extracted arrangement information and the corresponding points extracted between the plurality of images. It is provided.

[0007]

According to the first aspect of the present invention, first, the points corresponding to each other in the plurality of images are extracted, and the overlapping portion of each image is obtained from the corresponding points. Next, the arrangement of the images is determined based on the corresponding points, and the images are combined.

According to the invention of claim 7, a plurality of images are first reduced, corresponding points and arrangement information are obtained for each reduced image, and corresponding points between the plurality of images are obtained again based on these.
The respective images are combined based on the obtained corresponding points and the arrangement information.

[0009]

BEST MODE FOR CARRYING OUT THE INVENTION

(First Embodiment) The configuration and function of an image composition system as an image composition apparatus according to the present invention will be described below with reference to FIG. In FIG. 1, reference numeral 110 is a recording medium, and a magnetic disk, a tape or the like can be used. 1
An image reading unit 11 reads image information recorded in the recording medium 110. An image expansion unit 112 expands the read image information in the memory.
An area setting unit 113 sets an area to be used for image layout information extraction in two images used for composition.

Reference numeral 114 is a corresponding point extraction unit, which obtains a corresponding relationship between the set areas. An evaluation data calculation unit 115 calculates data for extracting arrangement information between images from the obtained information on the correspondence relationship.
Reference numeral 116 is a placement information extraction unit that compares evaluation data,
The placement information is determined. An image combining unit 117 combines the respective images using the information on the correspondence relationship used when determining the arrangement information. An image writing unit 118 writes the combined image information on a recording medium or the like.

Next, the operation of the image synthesizing system having the above configuration will be described. The image information read from the recording medium 110 by the image reading unit 111 is expanded on the memory by the image expansion unit 112. FIG. 2 is an outline of the image 210 and the image 220 developed on the memory. For the sake of explanation, the image size is width W × height H. The area setting unit 113 sets areas L1 to L4 and R1 to R4 for obtaining the correspondence relationship in the overlapping portions of the images 210 and 220 developed on the memory. In this way, the regions L1 to L4 or R1 to detect corresponding points in each of the images 210 and 220 are
By setting R4 and executing the corresponding point extraction processing described later only in this limited area, the processing can be sped up.

In the present embodiment, the correspondence is extracted by the template matching method. An outline of the template matching method is shown in FIG. In FIG. 3, 410
Indicates the region L1 in FIG. 2, and 420 indicates the region R2. In the area 410, each area surrounded by a dotted line indicates a template. The template is composed of n × m pixels with an arbitrary pixel as the center.

FIG. 4 is an enlarged view of the template T11 in FIG. 3, which is composed of 5 × 5 pixels centered on the pixel L (x11, y11). Corresponding points are extracted by moving the template T11 on the area 420 of FIG. 3 while moving the template T11 and the area 42 at each position.
The residual with each pixel R at 0 is calculated by the following formula.

[0014]

[Equation 1]

The minimum residual E obtained by the equation (1)
The position where (x, y) is given is the position of the corresponding point with respect to the pixel L (x11, y11).

The area setting unit 113 sets a pair of a template cutout area and a search area for extracting corresponding points as shown in FIG. Here, FIG.
L1 to L4 and R1 to R4 correspond to the areas in FIG.

Next, the corresponding point extraction unit 114 extracts the corresponding points by the above template matching method. FIG. 6 shows corresponding points extracted between the regions L1 and R2 of FIG. 2 and the results are represented as flows F11, F12, ..., In which the coordinates and templates of the regions L1 and R2 are shown simultaneously with the upper left corner as the origin. It is a thing. In FIG. 6, each region at the right end shows templates T11, T12, ..., T27. Further, the flow F11 includes the coordinates (x11, y11) of the center of the template T11 and the coordinates (x'11,
y'11) and the difference (Δx11, Δy11). The same applies to other flows.

The corresponding point extraction unit 114 creates a flow based on the coordinates of the detected corresponding points and creates a histogram for the flows in the x and y directions. FIG.
It is the outline of the histogram created based on the flow of the corresponding points obtained by FIG. Here, for the sake of explanation, it is assumed that FIG. 7 is created based on the flow in the x direction. It is possible to set an arbitrary value as the width of the histogram.
In FIG. 7, the width is set to 5.

After creating the histogram, the flow that gives the maximum frequency is extracted. In FIG. 7, Δx _{min + 10 to} Δx
_{min + 15} is the flow that gives the maximum frequency. Similarly, for the y direction, the flow that gives the maximum frequency is extracted. The corresponding point extraction unit 114 removes corresponding point pairs other than the flow that gives the maximum frequency extracted by the histogram processing as noise.

FIG. 8 shows a pair of corresponding points remaining after the noise removal is executed by the histogram.
Compared to FIG. 6, the noise component is removed.

Next, the evaluation data calculation unit 115 generates evaluation data for determining the arrangement of the two images. As the evaluation data, the number of templates cut out in the template cutting area shown in FIG. 6 is TN, and the number of templates remaining after noise removal by histogram processing is TK, which is generated by the following formula. D1 = TK / TN × 100 (%) (2) In the examples of FIGS. 6 and 8, TN = 14 and TK = 8,
D1 = 57 (%).

Regarding the four arrangements of to shown in FIG.
Evaluation data D1 to D4 are generated by performing the above corresponding point extraction, histogram processing, and calculation processing of evaluation data.

Next, the arrangement information extracting unit 116 compares the generated evaluation data D1 to D4 and extracts the direction giving the maximum evaluation data. For example, when D1 is detected to be the maximum, the two images are arranged as shown in FIG.
It shall be shown in (a). The arrangements of (a) to (d) of FIG. 9 correspond to the pairs of to of FIG.

Next, the image synthesizing unit 117 extracts parameters for synthesizing based on the arrangement information obtained by the arrangement information extracting unit 116 and the coordinate data of the corresponding points detected in the extracted arrangement. The image synthesizing unit 117 synthesizes two images using the following formula by affine transformation.

[0025]

[Equation 2]

Next, the composite image generated according to the equation (3) is written in the memory or the recording medium by the image writing unit 118. Further, the combined image may be displayed on a display unit (not shown).

(Second Embodiment) FIG. 10 shows the second embodiment, and the characteristic parts are an edge image generating section 119 and a corresponding point extracting section 120. The other parts have the same functions and operations as those in FIG. 1 according to the first embodiment, and the description thereof will be omitted. The edge image generation unit 119 generates an edge image based on the input image. For example, the following masks are used for the edge image generation.

[0028]

(Equation 3)

The edge images L'and R'are generated by the following equation.

[0030]

(Equation 4)

As in the first embodiment, the area setting unit 113 sets an area for obtaining the layout information for the generated edge image.

In the corresponding point extraction unit 120, a template is set in the edge image as shown in FIG. 3, but at this time, the following processing is newly added. That is, the pixel value in each template, that is, the edge level is compared with the threshold value S. For example, for a 5 × 5 template, 25
And the threshold value S are compared to obtain the number k of edges equal to or larger than the threshold value.

Next, it is detected whether or not the number k of edges is _{equal to} or more than the threshold value S _n , and if the number is _{equal to} or less than the threshold value S _n , the template is not used for extracting the corresponding points. Therefore, the number of edges k is the threshold value S in the number of cut-out templates TN used in the evaluation data calculation unit 115.
Do not include templates below _n . The following processing is the first
It is performed in the same manner as the embodiment of.

As a result, it is possible to prevent erroneous correspondence caused by using an image with few feature points and an image with a large density difference, and it is possible to improve the accuracy of layout information extraction.

(Third Embodiment) FIG. 11 shows the third embodiment. The characteristic part is the rough image generation unit 1.
21 and two corresponding point extraction units 122. Other parts are shown in FIGS. 1 and 1 according to the first and second embodiments.
Since it has the same function and operation as 0, description thereof will be omitted. In the present embodiment, the placement information is extracted from the coarse image obtained by reducing the read image, and based on the flow of the corresponding points obtained from the coarse image, the overlapping area between the images in the original image is predicted to detect the corresponding points. By executing the processing, the processing time is shortened and the accuracy at the time of composition is improved.

Next, the operation of FIG. 11 will be described. The coarse image generation unit 121 converts the read image W × H into a coarse image reduced to W / 2 × H / 2. Conversion is performed by the following equation. P ′ _{i, j} = (P _{2i, 2j} + P _{2i + 1,2j} + P _{2i, 2j + 1} + P _{2i + 1,2j + 1} ) / 4 i = 0 ... W / 2, j = 0 ... H / 2 ... (6) Here, P'is the pixel value of the coarse image, and P is the pixel value of the original image. Note that here, the vertical and horizontal directions of the original image are 1 /
Although the image reduced to 2 is used as the raw image, it is needless to say that the image reduced to 1/4 or 1/8 may be used as the raw image.

Next, the rough images of L and R are set as L _s and R _s, and the arrangement information is extracted based on the created rough images L _s and R _s by the same method as in the second embodiment. After extracting the layout information, the maximum flow F of the corresponding points used in the layout information extraction
The overlapping area in the edge images L'and R'of the original image is predicted based on _max .

FIG. 12 shows an outline of the overlap area prediction. FIG.
In the edge images L'and R'of the original image, reference numeral 2 is an outline of the overlapping area (hatched portion) and the search area predicted based on the flow of the corresponding points. Maximum flow F _{ma x} (Δ
x _max , Δy _max ) is obtained in the coarse image reduced to ½ in the vertical and horizontal directions, the overlapping area is (W−Δx _max).
It becomes a region of × 2, H−Δy _max × 2).

The search area is the pixel L'of FIG.
It is possible to predict the point corresponding to (x, y) based on the maximum flow obtained from the rough image and set the point as the center. As described above, the corresponding points are extracted again by the overlapping area and the search area predicted based on the corresponding point extraction result of the rough image, and the synthesized image is generated as in the second embodiment.

(Fourth Embodiment) In each of the embodiments described so far, the synthesis of two images is shown, but the fourth embodiment shows the synthesis of three or more images. FIG. 13 shows an outline of a case where the arrangement is automatically detected and synthesized from three or more images. here,
The composition of the four images 900 to 903 shown in FIG.

FIG. 13B shows the first two images 9
This shows the synthesis of 00 and 901. In this case, the regions L1 to L4 and R1 to R4 for placement detection are set and the placement is extracted as in the above-described embodiments. As the arrangement extraction method, as in the first embodiment, the template matching method according to the equation (1) and the removal of the miscorresponding point component by the histogram processing can be used.

The minimum residual E _min (x, y) obtained for each template by the equation (1) as the evaluation data.
It is also possible to use the average value of. That is, in FIG.
Mean value D of residuals for each of the four arrangements
The arrangement between images is extracted using 1'-D4 'as evaluation data.

If there are four images to be combined, ₄ C
Execute the above process for _two combinations of images,
The average value of the residuals used as the evaluation data is calculated, two images giving the minimum value are selected, and the images are combined according to the direction giving the minimum average value of the residuals. FIG. 13 (b)
In the above, the images 900 and 901 are selected in accordance with the above-mentioned processing, and the synthesis is executed according to the directions of L1 and R1 which are the directions that give the minimum average value of the residuals.

The image 910 in FIG. 13C is an image 900.
And 901 are the combined images. Composite image 91
For 0, the area L1 for extracting the arrangement information is the same as described above.
~ L4 is set, and this and the remaining image 90 of FIG.
Evaluation data for extracting the arrangement information is generated between 2 and 903. In FIG. 13C, the image 910 and the image 902 are selected as a pair that gives the minimum residual average value, and L is the direction that gives the minimum residual average value.
Synthesis is performed according to the directions of 4 and R3.

In this case, since the layout information has already been extracted from the corresponding points between the regions L1 and R2, it is not necessary to search the corresponding points for the regions corresponding to L1 and R2 surrounded by the dotted line in the image 910. Absent.

The image 911 in FIG. 13D is the image 910.
And a result of synthesizing the image 902. Similarly, for the image 911, areas L1 to L4, L for extracting the layout information
1'and L4 'are set, and evaluation data for extracting the layout information is generated between this and the image 903. Thereafter, by the same processing as described above, the respective images are combined in the direction in which the average value of the minimum residuals is given. Here, the synthesis of four images is shown, but it goes without saying that more images can be handled by expanding the above process.

[0047]

As described above, according to the invention of claim 1, it is possible to input a plurality of images and automatically extract the arrangement information between the plurality of input images. A troublesome operation for setting is unnecessary.

According to the second aspect of the present invention, since the miscorresponding component is removed at the time of extracting the corresponding points, it is possible to prevent a situation in which incorrect placement information is extracted due to the miscorresponding component.
Detection accuracy can be improved. Further, according to the invention of claim 3, since the evaluation information is generated based on the number of candidate points at the time of extracting the corresponding points and the number of the corresponding points after the extraction, a complicated operation is not required and a short time is required. The placement information can be extracted.

Further, according to the invention of claim 4, since the corresponding points are extracted based on the edge image, it is possible to prevent erroneous correspondence due to the density difference between the images, and to perform highly accurate corresponding point extraction and image synthesis. It can be carried out. Further, according to the invention of claim 5, since the candidate points are set based on the edge information,
Corresponding points can be extracted in a characteristic part, and the cause of miscorrespondence can be removed before executing the corresponding point extraction processing. Further, according to the invention of claim 6, since the arrangement information is extracted based on the reduced image, the processing time can be shortened.

Further, according to the invention of claim 7, the arrangement information is extracted from the reduced image, and the corresponding point extracting process is executed again using the original image, so that a high-precision composite image can be obtained in a short time. Can be generated.

Further, according to the inventions of claims 8 and 9, it is possible to speed up the processing because it is not necessary to perform unnecessary corresponding point extraction processing.

[Brief description of drawings]

FIG. 1 is a block diagram showing a first embodiment of the present invention.

FIG. 2 is a configuration diagram for explaining an image area used for extracting layout information.

FIG. 3 is a configuration diagram for explaining a template matching method.

FIG. 4 is a configuration diagram for explaining a template matching method.

FIG. 5 is a configuration diagram for explaining correspondence of areas used for arrangement information extraction.

FIG. 6 is a configuration diagram showing an outline of corresponding point extraction processing.

FIG. 7 is a characteristic diagram showing histogram processing of corresponding point extraction processing.

FIG. 8 is a configuration diagram showing an outline of erroneous correspondence removal of corresponding point extraction.

FIG. 9 is a configuration diagram showing an arrangement of images.

FIG. 10 is a block diagram showing a second embodiment of the present invention.

FIG. 11 is a block diagram showing a third embodiment of the present invention.

FIG. 12 is a configuration diagram showing an outline of corresponding point extraction.

FIG. 13 is a configuration diagram showing an outline of arrangement information extraction.

[Explanation of symbols]

 110 recording medium 111 image reading unit 112 image expansion unit 113 region setting unit 114 corresponding point extraction unit 115 evaluation data generation unit 116 placement information extraction unit 117 image synthesis unit 118 image writing unit 119 edge image generation unit 121 rough image generation unit

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Kenji Hatori 3-30-2 Shimomaruko, Ota-ku, Tokyo Inside Canon Inc.

Claims (9)

[Claims] 1. Corresponding point extracting means for extracting corresponding points between a plurality of input images, and arrangement information extracting means for extracting arrangement information for arranging the plurality of images based on the extracted corresponding points. An image synthesizing apparatus comprising: an image synthesizing unit that synthesizes the plurality of images based on the extracted arrangement information and the corresponding points. 2. The corresponding point extracting means sets a plurality of candidate points for extracting the corresponding points in the plurality of images,
The image synthesizing apparatus according to claim 1, wherein the corresponding points are obtained by performing corresponding point extraction processing on the plurality of candidate points and removing erroneous corresponding components of the corresponding point information obtained by this processing. 3. The arrangement information extraction means generates evaluation information on the number of the plurality of candidate points and the number of corresponding points after the removal of the erroneous corresponding components, and extracts the arrangement information based on the evaluation information. The image synthesizing device according to claim 2. 4. The image synthesizing apparatus according to claim 1, wherein the corresponding point extracting means generates edge images of the plurality of images and performs corresponding point extraction based on the generated edge images. 5. The corresponding point extracting means sets a plurality of candidate points for extracting the corresponding points in the plurality of images, and sets the candidate points based on edge information around the candidate points. The image synthesizing device according to claim 4. 6. A reduced image generating means for generating reduced images of the plurality of images is provided, the corresponding point extracting means extracts corresponding points between the plurality of reduced images, and the arrangement information extracting means extracts the extracted points. The image synthesizing apparatus according to claim 1, wherein the arrangement information of the plurality of images is extracted based on the determined corresponding points. 7. A reduced image generating means for generating reduced images of a plurality of input images, a first corresponding point extracting means for extracting corresponding points between the plurality of reduced images, and the extracted corresponding points. Arrangement information extracting means for extracting arrangement information for arranging the plurality of images based on, and corresponding points between the plurality of images based on corresponding points between the extracted arrangement information and the extracted reduced image. An image synthesizing device comprising: second corresponding point extracting means for extracting; and image synthesizing means for synthesizing the plurality of images based on the extracted arrangement information and the corresponding points extracted between the plurality of images. 8. The image synthesizing apparatus according to claim 1, wherein the corresponding point extracting means sets an area for extracting the corresponding points in the plurality of images. 9. The image synthesizing means is configured such that, after synthesizing two images, the corresponding point extracting means sets the area and extracts corresponding points for the synthesized image and another image. 9. At this time, the corresponding point extraction processing is not performed for the areas set in the two images, and the image composition apparatus according to claim 8.