JP6099447B2 - Stereo pair determination device and stereo pair determination program - Google Patents

Description

  The present invention corresponds to scene data (image shooting data, image shooting position, image shooting direction, image shooting time, and each pixel on the image used for calculating a digital elevation model (DEM) which is a digital elevation model. The present invention relates to a stereo pair determination device and a stereo pair determination program for determining a set of data composed of line-of-sight vectors.

In recent years, it has become possible to take high-resolution images due to the development of sensors for image acquisition.
As an image acquisition sensor, an image sensor is arranged in a two-dimensional manner, an area sensor that obtains a two-dimensional image at a time, and a sensor in which the image sensor is arranged in a one-dimensional manner. There is a line sensor that acquires a one-dimensional image while moving the sensor and constructs a two-dimensional image from a plurality of one-dimensional images.
Generally, a line sensor can acquire a higher resolution image than an area sensor.

There is a process for generating new information by using an image acquired by an image acquisition sensor and parameters (shooting position and shooting direction) at the time of image shooting.
For example, as shown in FIG. 8, by matching two images, a corresponding point (hereinafter referred to as “corresponding point”) on the two images is obtained, and the coordinates of the corresponding point are obtained. There is a process of calculating the three-dimensional coordinates of the intersection point p by obtaining the intersection point p from the line-of-sight vectors v ₁ and v ₂ and the shooting positions x ₁ and x _{2 of the two} images.
Advanced _{h p} of intersection p can be obtained as the intersection of the straight line _{x 1} + t · _v 1, the straight line _{x 2} + s · _v 2. However, t and s are variables. However, in three dimensions, an intersection of straight lines may not be obtained due to an error, and in this case, the nearest point (intersection point) of both straight lines is regarded as p.
Of the three-dimensional coordinates, a group of altitude values is a DEM, and the DEM represents a ground surface, a building, or the like (for example, see Non-Patent Document 1).

According to the survey procedure of aviation laser disclosed in Non-Patent Document 2 below, DEM can be calculated if there are at least two images (stereo pairs). When there are a large number, a plurality of stereo pairs are generated, and the DEM is calculated from each of the plurality of stereo pairs, so that the calculation accuracy of the DEM can be improved. Therefore, it is important to use many images in order to calculate the DEM with high accuracy.
However, in order to calculate one DEM, it is costly to repeat the imaging at the same place many times.
Therefore, as much as possible, it is necessary to calculate the DEM using images taken in the past without newly photographing the same place.

When calculating the DEM, as described above, it is necessary to find corresponding points on the two images.
Generally, the degree of coincidence between images (an index indicating how much the surrounding images including corresponding points match) is calculated by image processing, and the corresponding points are obtained based on the degree of coincidence.
Further, in the DEM calculation, as described above, the intersection point p of the line-of-sight vectors is calculated (see FIG. 8), but the angle formed by the _two line-of-sight vectors v ₁ and v ₂ (corresponding to the shooting parallax) is small. In this case, a small shift of corresponding points on the two images appears as a large difference in altitude. In general, since the corresponding points are easily displaced, when the line-of-sight vectors v ₁ and v ₂ with small parallax are used, the calculation error of the DEM tends to increase.

On the other hand, when the parallax is large, the shooting direction of the image is greatly different, so even if the same place is shot, the appearance may be greatly different. In such a case, image processing (matching processing) becomes difficult, and as a result, there is a high possibility that the corresponding points will be mistaken.
Therefore, in the DEM calculation, it is not necessary that the parallax is too large or too small, so it is desirable to use an image with an appropriate parallax.

In addition to the parallax, it is necessary to consider whether or not the range in which the DEM calculation is performed varies.
That is, it is necessary to consider whether or not “the altitudes of the locations to be subjected to DEM calculation are really coincided” at the time when each image is taken.
For example, if the interval between the time when two images were taken is about several seconds, there is no need to consider the terrain change or the construction of a new building, but if the interval is several months, etc. There is a good possibility that topographic changes and construction of new buildings are underway.
If the scenery including the surroundings has changed significantly, there is a high possibility that the matching process will not match (there is a high possibility that no corresponding point will be found), and it will not be possible to calculate the DEM. Even if the corresponding point is found by the matching process, the position error of the intersection obtained from such a corresponding point becomes large, so it is desirable that the time difference between the photographing times of the two images is short.
Note that in the DEM calculation, since it is usually assumed that “the altitude of the current state (newest state) is calculated”, it is considered that the degree of coincidence between the calculation result and the current state is higher when new data is used.

In the matching process performed when calculating the DEM, in general, a partial image that is a partial region in one image is superimposed on the other image, and the degree of coincidence is calculated at the overlapping position (for example, matching) Calculate a normalized correlation value that represents degrees).
The degree of coincidence is calculated while changing the overlapping position, and the overlapping position with the highest degree of coincidence is assumed to be a corresponding point.
However, if the degree of coincidence does not increase at any position, it is desirable that no corresponding point exists and these images are not handled as a stereo pair.

`` Digital elevation model '' http://www.nnk.co.jp/research/product/dem.html "Aerial Laser Survey" http://www1.gsi.go.jp/geowww/Laser_HP/senmon.html

Since the conventional stereo pair determination device is configured as described above, a plurality of stereo pairs can be generated from many images taken at the same place. The stereo pair is not generated by confirming whether or not the parallax is appropriate. For this reason, there has been a problem that two images with an inappropriate photographing parallax are generated as a stereo pair and the calculation accuracy of the DEM may deteriorate.
In addition, it is desirable not to treat two images that do not have a high degree of coincidence as a stereo pair. However, it is not intended to generate a stereo pair by checking whether or not the degree of coincidence between the two images is high. . For this reason, there is a problem that two images that do not have a high degree of coincidence are generated as a stereo pair and the calculation accuracy of the DEM may deteriorate.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide a stereo pair determination device and a stereo pair determination program capable of selecting a stereo pair capable of increasing the DEM calculation accuracy. And

The stereo pair determining device according to the present invention stores scene data including a plurality of scene data including image capturing data, image capturing position, image capturing direction, image capturing time, and line-of-sight vector corresponding to each pixel on the image. Using the shooting direction included in the plurality of scene data stored by the storage means and the scene data storage means, the shooting parallax between the images related to each scene data is calculated, and from among the plurality of scene data , a set of scene data captured parallax between images is within a predetermined range and a stereo pair retrieving means for retrieving as a stereo pair, the stereo each pair retrieved by scan Tereopea search means, constitute the stereo pair The degree of coincidence between the images indicated by the shooting data included in the two scene data is calculated, and the degree of coincidence is less than a predetermined value. Excludes au pair, grouping the matching degree that based on the stereo pair selection means to leave only a stereo pair of predetermined value or more, the degree of matching of a stereo pair left by stereo pair selection means, grouping stereopair For each group to which the means and one or more stereo pairs belong, the shooting time of the image included in the scene data belonging to the group is compared to identify the latest shooting time, and in each group Compare the most recent shooting time, arrange each group in the order of shooting time, and output the sorting order of each group, and the sorting order output from the group sorting means in order from the previous group For each stereo pair belonging to the group, included in the two scene data that make up the stereo pair By calculating the time difference of the photographing time of the image it is, by arranging each stereo pair time difference ascending order is obtained by so and a stereo pair sorting means for outputting the order of the stereo pair.

  According to the present invention, the shooting parallax between the images related to each scene data is calculated using the shooting directions included in the plurality of scene data stored by the scene data storage means, and the plurality of scene data Stereo pair search means for searching as a stereo pair a set of scene data whose shooting parallax between images is included within a predetermined range is provided, and the stereo pair selection means is searched for by the stereo pair search means. Each time, the degree of coincidence between images indicated by the shooting data included in the two scene data constituting the stereo pair is calculated, and the stereo pair whose degree of coincidence is less than a predetermined value is excluded, and the coincidence is calculated. Since the configuration is such that only stereo pairs having a degree equal to or higher than a predetermined value are left, it is possible to select stereo pairs that can improve the calculation accuracy of the DEM. There is.

It is a block diagram which shows the stereo pair determination apparatus by Embodiment 1 of this invention. FIG. 5 is an explanatory diagram showing a state in which an image indicated by shooting data R (n) is moved by ± e pixels in the X and Y directions to search for a position where the normalized correlation value cor (a (b, l)) is maximum. is there. It is a block diagram which shows the stereo pair determination apparatus by Embodiment 2 of this invention. It is a block diagram which shows the stereo pair determination apparatus by Embodiment 3 of this invention. It is a block diagram which shows the stereo pair determination apparatus by Embodiment 4 of this invention. It is a block diagram which shows the stereo pair determination apparatus by Embodiment 5 of this invention. It is a block diagram which shows the stereo pair determination apparatus by Embodiment 5 of this invention. It is explanatory drawing which shows the calculation process of a three-dimensional coordinate.

Embodiment 1 FIG.
1 is a block diagram showing a stereo pair determining apparatus according to Embodiment 1 of the present invention.
In FIG. 1, a scene data storage unit 1 is composed of a storage device such as a RAM or a hard disk, and stores a plurality of scene data S (i).
The scene data includes image shooting data G (i) (for example, image data in JPEG or TIFF format), image shooting position l (i), image shooting direction d (i), and image shooting time t (i ) And the line-of-sight vector v _{x, y} (i) corresponding to each pixel on the image. Here, i is a variable for identifying scene data, i = 1, 2,... I is the number of scene data stored in the scene data storage unit 1.
The scene data storage unit 1 constitutes scene data storage means.

The information input unit 2 performs a process of inputting a plurality of scene data S (i) stored in the scene data storage unit 1 and calculation target range information indicating a calculation target range (designated region).
The parallax calculation unit 3 uses the imaging direction d (i) of the images included in the plurality of scene data S (i) input by the information input unit 2, and uses the image capturing direction d (i) between the images related to each scene data S (i). And a pair of scene data S (i) in which the shooting parallax P between images is included in a predetermined range from a plurality of scene data S (i). As a result, search processing is performed. Here, a is a variable for identifying a stereo pair, and a = 1, 2,. A is the number of stereo pairs searched by the parallax calculation unit 3.
Note that the parallax calculator 3 constitutes a stereo pair search means.

For each stereo pair SP (a) searched by the parallax calculation unit 3, the image search unit 4 captures shooting data G included in the two scene data S (i) constituting the stereo pair SP (a). From (i), a process of taking out the shooting data R (i) of the portion shooting the calculation target range indicated by the calculation target range information input by the information input unit 2 is performed.
In addition, the image search unit 4 uses, for each stereo pair SP (a) searched by the parallax calculation unit 3, a normalized correlation value cor (a) between images indicated by the two shooting data G (i) (match between images). Index) indicating that the normalized correlation value cor (a) is less than the predetermined threshold value β and the stereo pair SP (a) is excluded, and the normalized correlation value cor (a) is equal to or greater than the threshold value β. A process of leaving only the stereo pair SP (a) is performed. The image search unit 4 constitutes a stereo pair selection unit.

The grouping unit 5 performs a process of grouping the stereo pairs SP (a) on the basis of the normalized correlation value cor (a) of the stereo pairs SP (a) left by the image search unit 4. The grouping unit 5 constitutes grouping means.
For each group, the time comparison unit 6 captures images t (i) of images included in the two scene data S (i) constituting one or more stereo pairs SP (a) belonging to the group. by comparing, to implement a process to identify the most new shooting time t _new.
In addition, the time comparison unit 6 compares the latest shooting time t _new in each group, arranges the groups in the order of the shooting time t _new , and outputs the arrangement order of the groups. The time comparison unit 6 constitutes a group sorting unit.

  The time difference calculation unit 7 configures the stereo pair SP (a) for each stereo pair SP (a) belonging to the group in order from the previous group in the arrangement order output from the time comparison unit 6. A time difference Δt between the shooting times t (m) and t (n) of the images related to the scene data S (m) and S (n), which are two scene data S (i), is calculated, and each time difference Δt is increased in order. A process of arranging the stereo pairs SP (a) and outputting the arrangement order of the stereo pairs SP (a) is performed. The time difference calculator 7 constitutes a stereo pair sorter.

In the example of FIG. 1, the scene data storage unit 1, the information input unit 2, the parallax calculation unit 3, the image search unit 4, the grouping unit 5, the time comparison unit 6, and the time difference calculation unit 7 that are components of the stereo pair determination device. Each component is composed of dedicated hardware (each component other than the scene data storage unit 1 is composed of, for example, a semiconductor integrated circuit on which a CPU is mounted or a one-chip microcomputer) Assuming, the stereo pair determining device may be configured by a computer.
When the stereo pair determination device is configured by a computer, the scene data storage unit 1 is configured on the memory of the computer, and the information input unit 2, the parallax calculation unit 3, the image search unit 4, the grouping unit 5, and the time comparison unit 6 and the time difference calculation unit 7 may be stored in a computer memory so that the CPU of the computer executes the stereo pair determination program stored in the memory. .

Next, the operation will be described.
First, the information input unit 2 inputs a plurality of scene data S (i) stored in the scene data storage unit 1 and outputs the plurality of scene data S (i) to the parallax calculation unit 3.
Further, the information input unit 2 inputs calculation target range information indicating a calculation target range (designated region) set externally, and inputs the calculation target range information (four corner coordinates, DEM resolution, etc.) to the image search unit 4. Output.

Upon receiving a plurality of scene data S (i) from the information input unit 2, the parallax calculation unit 3 selects any two scene data S (i) from the plurality of scene data S (i).
Here, for convenience of explanation, it is assumed that scene data S (1) and scene data S (2) are selected.
When the parallax calculation unit 3 selects the scene data S (1) and the scene data S (2), the shooting direction d (1) of the image included in the scene data S (1) and the scene data S (2) The angle (0 to 180 degrees) formed with the shooting direction d (2) of the image included in the image is the shooting parallax P (1,1) between the images related to the scene data S (1) and S (2). 2) is calculated.

式（１）において、ｄ（１）・ｄ（２）は、ｄ（１）とｄ（２）の内積を表している。 That is, the parallax calculation unit 3 sets the image capturing direction d (1) to d (1) = (dx (1), dy (1), dz (1)) and the image capturing direction d (2) to d (1). 2) = (dx (2), dy (2), dz (2)) Assuming that the shooting parallax P (1,2) between images is calculated as the following equation (1).

In equation (1), d (1) · d (2) represents the inner product of d (1) and d (2).

When the parallax calculation unit 3 calculates the shooting parallax P (1,2) between the images, the parallax calculation unit 3 determines whether or not the shooting parallax P (1,2) is included in the predetermined range.
That is, the parallax calculation unit 3 determines whether or not the following formula (2) is satisfied, and if the formula (2) is satisfied, the shooting parallax P (1,2) is included in the predetermined range. Judge.
α1 ≦ P (1,2) ≦ α2 (2)
In equation (2), α1 and α2 are preset angle thresholds, and α1 ≦ α2.

If the parallax calculation unit 3 determines that the shooting parallax P (1,2) is included within the predetermined range, the pair of the scene data S (1) and the scene data S (2) is converted into the stereo pair SP (1). = {S (1), S (2)} is output to the image search unit 4.
Here, the parallax calculation unit 3 selects the scene data S (1) and the scene data S (2) from the plurality of scene data S (i), but the scene data S (1) and the scene data S are selected. All combinations other than (2) are also sequentially selected and the same processing as described above is performed.
That is, if the two scene data selected from the plurality of scene data S (i) are S (m) and S (n), the images related to the two scene data S (m) and S (n). If the shooting parallax P (m, n) satisfies the expression (2), the pair of the scene data S (m) and the scene data S (n) is represented as a stereo pair SP (a) = {S (m), S (n)} is output to the image search unit 4.
Therefore, a set of two scene data in which the shooting parallax P (m, n) between images is not included in the predetermined range and the shooting parallax P (m, n) between images does not satisfy the equation (2) is The stereo pair SP (a) is not output to the image search unit 4.
As a result, the shooting parallax P (m, n) is narrowed down to an appropriate stereo pair SP (a).
For example, if the number of sets of scene data S (m) and scene data S (n) satisfying Expression (2) is A, a = 1, 2,...

When the image search unit 4 receives one or more stereo pairs SP (a) from the parallax calculation unit 3, the image search unit 4 stores the scene data S (m) and the scene data S (n) in each stereo pair SP (a). Since the shooting ranges are not necessarily the same, the calculation indicated by the calculation target range information output from the information input unit 2 from the shooting data G (m) included in the scene data S (m). While taking out the shooting data R (m) of the part shooting the target range, the calculation target output from the information input unit 2 from the shooting data G (n) included in the scene data S (n) Processing is performed to extract the shooting data R (n) of the portion shooting the calculation target range indicated by the range information.
The shooting data R (m) and the shooting data R (n) extracted by the image search unit 4 are data of the same shooting range.

When the image search unit 4 extracts the shooting data R (m) and the shooting data R (n) for each stereo pair SP (a), each point (four corner coordinates and coordinates determined by the DEM resolution (b , L)) and the normalized correlation value cor (a (b, l)) indicating the degree of coincidence between the image indicated by the photographic data R (m) and the image indicated by the photographic data R (n), respectively. calculate. The normalized correlation value cor (a (b, l)) is a value in the range of 0-1.
Since the calculation process itself of the normalized correlation value cor (a (b, l)) is a known technique, a detailed description thereof is omitted, but the image indicated by the shooting data R (m) and the shooting data R (n) are omitted. In addition to the normalized correlation value cor (a (b, l)) with the image indicated by, the image indicated by the shooting data R (m) and the image indicated by the shooting data R (n) The normalized correlation value cor (a (b, l)) with the image moved by ± e pixels (for example, e = 2) in the direction) is calculated.
Then, among the normalized correlation values cor (a (b, l)), the image with the largest normalized correlation value cor (a (b, l)) indicated by the imaging data R (m) is captured. The normalized correlation value cor (a (b, l)) with the image indicated by the data R (n) is assumed.
FIG. 2 shows a state where the image indicated by the photographic data R (n) is moved ± e pixels in the X and Y directions to search for a position where the normalized correlation value cor (a (b, l)) is the largest. It is explanatory drawing.

When the image search unit 4 calculates the normalized correlation value cor (a (b, l)) for each stereo pair SP (a) and each point (b, l) of the calculation target range, the normalized correlation value cor (A (b, l)) is compared with a predetermined threshold value β, and the normalization correlation value cor (a (b, l)) is excluded from the stereo pair SP (a) whose threshold value β is less than the normal value β. Only the stereo pair SP (a) whose correlation value cor (a (b, l)) is equal to or greater than the threshold value β is left, and the left stereo pair SP (a) is output to the grouping unit 5. Further, the largest normalized correlation value cor (a (b, l)) relating to the remaining stereo pair SP (a) is output to the grouping unit 5.
This narrows down the stereo pair SP (a) having a high degree of coincidence of images for each point (b, l) in the calculation target range.
When the image search unit 4 outputs a stereo pair SP (a) whose normalized correlation value cor (a (b, l)) is greater than or equal to the threshold value β, a DEM calculation processing unit (not shown) has a high degree of coincidence. In order to enable calculation of DEM using an image, information indicating the position of the imaged data R (n) at which the normalized correlation value cor (a (b, l)) is largest is not shown. You may make it output to a part.

The grouping unit 5 sends one or more stereo pairs SP (a) from the image search unit 4 (stereo pairs SP (a) whose normalized correlation value cor (a (b, l)) is greater than or equal to the threshold value β) and their stereos. When the normalized correlation value cor (a (b, l)) of the pair SP (a) is received, the stereo pair SP (a) is referred to based on the normalized correlation value cor (a (b, l)). ).
For example, when the normalized correlation value is 0.05, the normalized correlation value is 0.95 or more, the normalized correlation value is 0.9 or more and less than 0.95, and the normalized correlation value is 0.9. Each stereo pair SP (a) is classified into a group of less than.

When the grouping unit 5 divides each stereo pair SP (a) into groups, the time comparison unit 6 sets two scene data constituting one or more stereo pairs SP (a) belonging to the group. By comparing the shooting times t (i) of the images included in S (i), the newest shooting time t _new is specified.
Time comparison section 6 in each group and to identify the newest shooting time t _{new new,} by comparing the most recent photographing time t _{new new} in each group, the photographing time t _{new new} side by side each group in reverse chronological order, each group Output the sort order.

However, in more than one group, when most new shooting time t _new is the same, in favor of the group of people is large number of stereo pairs SP (a) that belong to the order of the group ahead.
If the number of stereo pairs SP (a) to which they belong is the same, the arrangement order at the time of output from the grouping unit 5 is retained.
The time comparison unit 6 outputs a group list GL indicating the arrangement order of the groups to the time difference calculation unit 7.

When the time difference calculation unit 7 receives the group list GL from the time comparison unit 6, the time difference calculation unit 7 refers to the group list GL and recognizes the first group in the arrangement order (the first group in the arrangement order). For each stereo pair SP (a) to which it belongs, the shooting times t (m) and t of images included in the scene data S (m) and S (n) constituting the stereo pair SP (a) The time difference Δt of (n) is calculated.
Δt = | t (m) −t (n) | (3)
The time difference calculation unit 7 calculates the time difference Δt between the shooting times t (m) and t (n) for each stereo pair SP (a) belonging to the first group in the arrangement order. Stereo pairs SP (a) are arranged, and a stereo pair list SPL indicating the arrangement order of the stereo pairs SP (a) is generated.

Next, the time difference calculation unit 7 refers to the group list GL, recognizes the second group in the order of arrangement, and also sets the second group to the second group as in the case of the first group. The stereo pairs SP (a) to which they belong are arranged in ascending order of the time difference Δt, and the stereo group SP (a) generated by the previously generated stereo pair list SPL is arranged in the second group at the back of the arrangement order. The arrangement order of each stereo pair SP (a) to which it belongs is added.
For the third and subsequent groups indicated by the group list GL, the stereo pair SP (a) belonging to the third and subsequent groups is performed by performing the same processing as described above in order from the previous group. Are added after the stereo pair list SPL generated earlier.

As a result, the arrangement order of the stereo pair SP (a) belonging to the first group (the group to which the stereo pair SP (a) having a high degree of coincidence of images belongs) comes first, and the same group For the stereo pair SP (a) belonging to the stereo pair SP (a), there is a stereo pair list SPL indicating that the stereo pair SP (a) having the smaller time difference Δt between the photographing times t (m) and t (n) is arranged first. Finally generated, the stereo pair list SPL is output from the time difference calculator 7 to a DEM calculation processor (not shown).
The DEM calculation processing unit can increase the calculation accuracy of the DEM by calculating the DEM by preferentially using the stereo pair SP (a) whose arrangement order indicated by the stereo pair list SPL is first.

As apparent from the above, according to the first embodiment, each of the shooting directions d (i) included in the plurality of scene data S (i) stored in the scene data storage unit 1 is used. The shooting parallax P (m, n) between the images related to the scene data S (i) is calculated, and the shooting parallax P (m, n) between the images of the plurality of scene data S (i) is within a predetermined range. A parallax calculation unit 3 for searching for a pair of scene data S (m) and S (n) included therein as a stereo pair SP (a), and the image search unit 4 is searched by the parallax calculation unit 3 For each stereo pair SP (a), shooting data G (m) and G (n) included in the scene data S (m) and S (n) constituting the stereo pair SP (a) are shown. The normalized correlation value cor (a (b, l)) between images is determined for each point within the DEM calculation target range. b, l), and the stereo correlation SP cor (a (b, l)) whose normalized correlation value cor (a (b, l)) is less than the threshold β is excluded. Since l)) is configured to leave only the stereo pair SP (a) having a threshold value β or more, there is an effect that it is possible to select a stereo pair capable of improving the DEM calculation accuracy.
That is, there is an effect that it is possible to select a stereo pair having an appropriate shooting parallax between two images and having a high degree of coincidence between the two images for each point in the calculation target range.

Further, according to the first embodiment, the stereo pair SP (a) is based on the normalized correlation value cor (a (b, l)) of the stereo pair SP (a) left by the image search unit 4. Grouping unit 5 for grouping images, and for each group, the shooting times of images included in two scene data S (i) constituting one or more stereo pairs SP (a) belonging to the group by comparing the t (i), as well as identify the most new shooting time t _new, as compared to the most recent shooting time t _new in each group, the imaging time t _new side by side with each group to the new order, of each group A time comparison unit 6 that outputs an arrangement order, and the time difference calculation unit 7 outputs a stereo pair SP belonging to the group in the order in which the arrangement order output from the time comparison unit 6 is from the previous group. For each a), the time difference Δt between the photographing times t (m) and t (n) of the images included in the scene data S (m) and S (n) constituting the stereo pair SP (a) is obtained. Since the stereo pairs SP (a) are arranged in order of increasing time difference Δt and the arrangement order of the stereo pairs SP (a) is output, the stereo with a short time difference between the shooting times of the two images is calculated. The effect which can select a pair is produced.

Embodiment 2. FIG.
3 is a block diagram showing a stereo pair determining apparatus according to Embodiment 2 of the present invention. In the figure, the same reference numerals as those in FIG.
When the region dividing unit 11 inputs the calculation target range information indicating the calculation target range (designated region), the region dividing unit 11 divides the calculation target range indicated by the calculation target range information, and calculates the calculation target range information indicating the calculation target range after division. The process given to the information input unit 2 is performed. The area dividing unit 11 constitutes a designated area dividing means.

In the example of FIG. 3, the scene data storage unit 1, the information input unit 2, the parallax calculation unit 3, the image search unit 4, the grouping unit 5, the time comparison unit 6, the time difference calculation unit 7, Each of the region dividing units 11 is configured by dedicated hardware (each component other than the scene data storage unit 1 is configured by, for example, a semiconductor integrated circuit on which a CPU is mounted, a one-chip microcomputer, or the like. However, the stereo pair determination device may be configured by a computer.
When the stereo pair determination device is configured by a computer, the scene data storage unit 1 is configured on the memory of the computer, and the information input unit 2, the parallax calculation unit 3, the image search unit 4, the grouping unit 5, and the time comparison unit 6. A stereo pair determination program describing the processing contents of the time difference calculation unit 7 and the area division unit 11 is stored in the memory of the computer, and the CPU of the computer executes the stereo pair determination program stored in the memory What should I do?

  In the first embodiment, the image search unit 4 is input by the information input unit 2 from the shooting data G (m) and G (n) included in the scene data S (m) and S (n). In the above example, the shooting data R (m) and R (n) of the part shooting the calculation target range indicated by the calculated calculation target range information is shown. When the range is too wide, since the matching accuracy between the image indicated by the photographic data R (m) and the image indicated by the photographic data R (n) is deteriorated, the normalized correlation value cor (a (b, l)) is small. Therefore, there is a possibility that an appropriate stereo pair SP (a) cannot be selected.

Therefore, in the second embodiment, when the region dividing unit 11 has a calculation target range indicated by the calculation target range information input from the outside wider than the preset range, the calculation target range is larger than the preset range. The calculation target range is divided so as to be smaller, and calculation target range information indicating the calculation target range after division is given to the information input unit 2.
The processing contents from the information input unit 2 to the time difference calculation unit 7 are the same as those in the first embodiment, and the region division unit 11 outputs the stereo pair list SPL related to the calculation target range after the division from the time difference calculation unit 7. Then, the stereo pair list SPL is output to a DEM calculation processing unit (not shown).
The area dividing unit 11 may output the stereo pair list SPL to a DEM calculation processing unit (not shown) every time the stereo pair list SPL related to the calculation target range after the division is received from the time difference calculating unit 7. However, after the stereo pair lists SPL for all the calculation target range information are gathered, the stereo pair list SPL related to the divided calculation target ranges may be output in parallel to a DEM calculation processing unit (not shown).

  As apparent from the above, according to the second embodiment, the region dividing unit 11 divides the calculation target range indicated by the calculation target range information input from the outside, and shows the calculation target range after the division. Since the target range information is provided to the information input unit 2, an appropriate stereo pair SP (a) can be selected even when the calculation target range indicated by the calculation target range information input from the outside is too wide. There is an effect.

Embodiment 3 FIG.
FIG. 4 is a block diagram showing a stereo pair determining apparatus according to Embodiment 3 of the present invention. In the figure, the same reference numerals as those in FIGS.
The calculation target range setting unit 12 includes scene data S _new (including the _latest photographed data G _new (i) among the plurality of scene data S (i) stored in the scene data storage unit 1. i) is specified, the range of the image indicated by the shooting data G _new (i) included in the scene data S _new (i) is set as the calculation target range (designated region), and the calculation target range is indicated. The process which gives calculation target range information to the information input part 2 is implemented. The calculation target range setting unit 12 constitutes a designated area setting unit.

In the example of FIG. 4, the scene data storage unit 1, the information input unit 2, the parallax calculation unit 3, the image search unit 4, the grouping unit 5, the time comparison unit 6, the time difference calculation unit 7, Each calculation target range setting unit 12 is configured by dedicated hardware (each component other than the scene data storage unit 1 is, for example, a semiconductor integrated circuit on which a CPU is mounted or a one-chip microcomputer, etc. However, the stereo pair determination device may be configured by a computer.
When the stereo pair determination device is configured by a computer, the scene data storage unit 1 is configured on the memory of the computer, and the information input unit 2, the parallax calculation unit 3, the image search unit 4, the grouping unit 5, and the time comparison unit 6. A stereo pair determination program that describes the processing contents of the time difference calculation unit 7 and the calculation target range setting unit 12 is stored in the memory of a computer, and the CPU of the computer stores the stereo pair determination program stored in the memory. It should be executed.

  In the second embodiment, when the region dividing unit 11 inputs the calculation target range information indicating the calculation target range, the calculation target range indicated by the calculation target range information is divided, and the calculation target range after the division is calculated. Although the target range information is given to the information input unit 2, the calculation target range setting unit 12 may automatically set the calculation target range without inputting the calculation target range information from the outside.

That is, the calculation target range setting unit 12 compares the shooting times t (i) of the images included in the plurality of scene data S (i) stored by the scene data storage unit 1, and the latest shooting is performed. The scene data S _new (i) including the captured data G _new (i) is specified.
When the calculation target range setting unit 12 specifies the scene data S _new (i), the calculation target range setting unit 12 sets the range of the image indicated by the shooting data G _new (i) included in the scene data S _new (i) as the calculation target range. Then, the calculation target range information indicating the calculation target range is given to the information input unit 2.
Processing contents other than the calculation target range setting unit 12 are the same as those in the second embodiment.
Thereby, the calculation target range can be automatically set to an appropriate range, and the appropriate stereo pair SP (a) can be selected.

Embodiment 4 FIG.
5 is a block diagram showing a stereo pair determining apparatus according to Embodiment 4 of the present invention. In the figure, the same reference numerals as those in FIG.
The image search unit 20 performs the same processing as the image search unit 4 of FIG. 1, but compares the normalized correlation value cor (a (b, l)) with the threshold value β, and normalizes the correlation value cor (a (a ( b, l)) is included in the two scene data S (m) and S (n) constituting the stereo pair SP (a) when the stereo pair SP (a) having a threshold β is excluded. Of the images included in the plurality of scene data (i) stored in the scene data storage unit 1 and the older one of the shooting times t _old of the shooting times t (m) and t (n). As the time difference Δt from the latest shooting time t _new among the shooting times t (i) is larger, the larger threshold value β is set. The image search unit 20 constitutes a stereo pair selection unit.

In the example of FIG. 5, the scene data storage unit 1, the information input unit 2, the parallax calculation unit 3, the image search unit 20, the grouping unit 5, the time comparison unit 6, and the time difference calculation unit 7 that are components of the stereo pair determination device. Each component is composed of dedicated hardware (each component other than the scene data storage unit 1 is composed of, for example, a semiconductor integrated circuit on which a CPU is mounted or a one-chip microcomputer) Assuming, the stereo pair determining device may be configured by a computer.
When the stereo pair determination device is configured by a computer, the scene data storage unit 1 is configured on the memory of the computer, and the information input unit 2, the parallax calculation unit 3, the image search unit 20, the grouping unit 5, the time comparison unit 6 and the time difference calculation unit 7 may be stored in a computer memory so that the CPU of the computer executes the stereo pair determination program stored in the memory. .

  In the first to third embodiments, the image search unit 4 compares the preset threshold value β with the normalized correlation value cor (a (b, l)), but the image search unit 20 The threshold value β may be set as shown below and compared with the normalized correlation value cor (a (b, l)).

The image search unit 20 includes an image capturing time t (m) included in one scene data S (m) constituting the stereo pair SP (a) and the other scene data S (n). The image capturing time t _old of the older image is specified by comparing the image capturing time t (n) of the captured image.
Further, the image searching unit 20 compares the capture time of image included in the plurality of stored by the scene data storing unit 1 scene data S (i) t (i) , the newest shooting time t _{new new} Is identified.

When specifying the oldest shooting time t _old and the latest shooting time t _new , the image search unit 20 calculates a time difference Δt between these times as shown in the following equation (4).
Δt = t _new −t _old (4)
When the time difference Δt is calculated, the image search unit 20 uses the threshold function β (Δt) that monotonously increases with respect to the time difference Δt, and sets a larger threshold β as the time difference Δt increases.

The comparison processing of the normalized correlation value cor (a (b, l)) and the threshold value β is the same as that of the image search unit 4 in FIG. 1, but the oldest shooting time t _old and the latest shooting time t _new are the same. The larger the time difference Δt is, the larger the threshold value β is set. Therefore, the probability that the normalized correlation value cor (a (b, l)) is excluded from the stereo pair SP (a) is assumed to be less than the threshold value β. Only the stereo pair SP (a) that is likely to be more appropriate remains.

Embodiment 5. FIG.
In the first to fourth embodiments, the image search unit 4 (or the image search unit 20) is installed behind the parallax calculation unit 3, but as shown in FIG. (Or the image search part 20) may be installed before the parallax calculation part 3, and there can exist an effect similar to the said Embodiment 1-4.

  In the first to fourth embodiments, the time difference calculation unit 7 is installed behind the time comparison unit 6. However, as shown in FIG. 7, the time difference calculation unit 7 includes the time comparison unit 6. It may be installed before and can produce the same effects as those of the first to fourth embodiments.

  In the present invention, within the scope of the invention, any combination of the embodiments, or any modification of any component in each embodiment, or omission of any component in each embodiment is possible. .

  DESCRIPTION OF SYMBOLS 1 Scene data storage part (scene data storage means), 2 Information input part, 3 Parallax calculation part (stereo pair search means), 4,20 Image search part (stereo pair selection means), 5 Grouping part (grouping means), 6 Time comparison unit (group sorting unit), 7 time difference calculation unit (stereo pair sorting unit), 11 area dividing unit (designated area dividing unit), 12 calculation target range setting unit (designated area setting unit).

Claims (8)

Scene data storage means for storing a plurality of scene data composed of image data, image capturing position, image capturing direction, image capturing time, and line-of-sight vector corresponding to each pixel on the image;
The shooting parallax between the images related to each scene data is calculated using the shooting directions included in the plurality of scene data stored by the scene data storage means, and between the images is calculated from the plurality of scene data. Stereo pair search means for searching as a stereo pair a set of scene data whose shooting parallax is included in a predetermined range;
For each stereo pair searched by the stereo pair searching means, the degree of coincidence between images indicated by the shooting data included in the two scene data constituting the stereo pair is calculated, and the degree of coincidence is a predetermined value. Stereo pair selection means that excludes less than stereo pairs and leaves only stereo pairs with a matching value equal to or higher than a predetermined value ;
Grouping means for grouping the stereo pairs on the basis of the matching degree of the stereo pairs left by the stereo pair selection means;
For each group to which one or more stereo pairs belong, the shooting time of the image included in the scene data belonging to the group is compared to identify the latest shooting time, and the newest shooting in each group A group sorting means for comparing the times, arranging the groups in the order from the latest shooting time, and outputting the order of the groups;
Shooting times of images included in the two scene data constituting the stereo pair for each stereo pair belonging to the group in the order in which the arrangement order output from the group sorting unit is from the previous group A stereo pair determining device comprising: stereo pair sorting means for calculating a time difference of the stereo pairs, arranging the stereo pairs in order of increasing time difference, and outputting the arrangement order of the stereo pairs . Scene data storage means for storing a plurality of scene data composed of image data, image capturing position, image capturing direction, image capturing time, and line-of-sight vector corresponding to each pixel on the image;
The shooting parallax between the images related to each scene data is calculated using the shooting directions included in the plurality of scene data stored by the scene data storage means, and between the images is calculated from the plurality of scene data. Stereo pair search means for searching as a stereo pair a set of scene data whose shooting parallax is included in a predetermined range;
For each stereo pair searched by the stereo pair search means, the degree of coincidence between images indicated by the shooting data included in the two scene data constituting the stereo pair is calculated and the stereo pair is configured. Of the images included in the two scene data, and the shooting time of the image included in the plurality of scene data stored by the scene data storage means. The larger the time difference from the latest shooting time, the larger the predetermined value is set, excluding stereo pairs whose coincidence is less than the predetermined value, and only stereo pairs whose coincidence is greater than or equal to the predetermined value. Stereo pair selection means to leave
Stereo pair determination device equipped with. For each stereo pair left by the stereo pair selection means, the time difference between the photographing times of the images included in the two scene data constituting the stereo pair is calculated, and each stereo pair is selected in ascending order of the time difference. The stereo pair determination device according to claim 1 or 2, further comprising stereo pair sorting means for arranging and outputting the arrangement order of each stereo pair. The stereo pair selecting means is a part of photographing a set designated area from among photographing data included in two scene data constituting the stereo pair searched by the stereo pair searching means. Taking out each photographic data, calculating the degree of coincidence between images indicated by the taken photographic data, excluding stereo pairs with the degree of coincidence being less than a predetermined value, and removing only stereo pairs with the degree of coincidence being a predetermined value or more. The stereo pair determination device according to any one of claims 1 to 3, wherein the stereo pair determination device is left. A designated area dividing means for dividing the set designated area;
The stereo pair selection means includes the designated area divided by the designated area dividing means from the photographing data included in the two scene data constituting the stereo pair searched by the stereo pair search means. Each piece of shooting data of the portion being shot is taken out, the degree of coincidence between images indicated by the taken shot data is calculated, stereo pairs with the degree of coincidence being less than a predetermined value are excluded, and the degree of coincidence is a predetermined value The stereo pair determination device according to any one of claims 1 to 3, wherein only the above stereo pair is left. Comprising designated area setting means for setting the designated area as the range of the image indicated by the latest photographed image data;
The stereo pair selecting means selects the designated area set by the designated area setting means from the shooting data included in the two scene data constituting the stereo pair searched by the stereo pair searching means. Each piece of shooting data of the portion being shot is taken out, the degree of coincidence between images indicated by the taken shot data is calculated, stereo pairs with the degree of coincidence being less than a predetermined value are excluded, and the degree of coincidence is a predetermined value The stereo pair determination device according to any one of claims 1 to 3, wherein only the above stereo pair is left. A scene data storage processing procedure for storing a plurality of scene data composed of image data, image capturing position, image capturing direction, image capturing time, and line-of-sight vector corresponding to each pixel on the image;
The shooting parallax between the images related to each scene data is calculated using the shooting directions included in the plurality of scene data stored by the above-described scene data storage processing procedure, and images are selected from the plurality of scene data. Stereo pair search processing procedure for searching for a pair of scene data whose shooting parallax is within a predetermined range as a stereo pair;
For each stereo pair searched by the stereo pair search processing procedure, the degree of coincidence between images indicated by the shooting data included in the two scene data constituting the stereo pair is calculated, and the degree of coincidence is predetermined. Stereo pair selection processing procedure that excludes stereo pairs that are less than the value and leaves only stereo pairs with the matching degree equal to or higher than a predetermined value,
A grouping process procedure for grouping the stereo pairs based on the degree of matching of the stereo pairs left by the stereo pair selection process procedure;
For each group to which one or more stereo pairs belong, the shooting time of the image included in the scene data belonging to the group is compared to identify the latest shooting time, and the newest shooting in each group A group sort processing procedure for comparing the times, arranging the groups in the order from the latest shooting time, and outputting the order of each group;
Photographing images included in the two scene data constituting the stereo pair for each stereo pair belonging to the group in the order from the previous group in the order of output output from the group sort processing procedure. A stereo pair determination program that calculates a time difference of time, arranges each stereo pair in ascending order of time difference, and outputs a stereo pair sorting process procedure for outputting the arrangement order of each stereo pair . A scene data storage processing procedure for storing a plurality of scene data composed of image data, image capturing position, image capturing direction, image capturing time, and line-of-sight vector corresponding to each pixel on the image;
The shooting parallax between the images related to each scene data is calculated using the shooting directions included in the plurality of scene data stored by the above-described scene data storage processing procedure, and images are selected from the plurality of scene data. Stereo pair search processing procedure for searching for a pair of scene data whose shooting parallax is within a predetermined range as a stereo pair;
For each stereo pair searched by the stereo pair search processing procedure, the degree of coincidence between images indicated by the shooting data included in the two scene data constituting the stereo pair is calculated, and the degree of coincidence is predetermined. Stereo pair selection processing procedure that excludes stereo pairs that are less than the value and leaves only stereo pairs with the matching degree equal to or higher than a predetermined value ,
For each stereo pair searched by the stereo pair search processing procedure, the degree of coincidence between images indicated by the shooting data included in the two scene data constituting the stereo pair is calculated, and the stereo pair is Shooting of an image included in a plurality of scene data stored by the above-described scene data storage processing procedure and the shooting time of the older one among the shooting times of images included in the two configured scene data A larger predetermined value is set as the time difference with the latest shooting time is larger, and stereo pairs with the matching degree less than the predetermined value are excluded, and the stereo pair with the matching degree equal to or higher than the predetermined value is excluded. A stereo pair determination program that causes a computer to execute a stereo pair selection process procedure that only leaves .

Images