JP4506501B2 - Image composition device and imaging system - Google Patents

Description

  The present invention relates to an imaging system that obtains a wide viewing angle or high-resolution video using a plurality of imaging devices, and an image composition device used in such an imaging system.

  There are a technique known as a video mosaicing technique and a technique described in Patent Document 1 as a photographing system that obtains a wide viewing angle or high-resolution video using a plurality of photographing apparatuses. The technique described in Patent Document 1 includes one wide-angle photographing device and a plurality of normal angle-of-view photographing devices, and an image obtained by synthesizing images of the respective normal angle-of-view photographing devices based on one wide-angle photographing device. Is generated. All of these technologies estimate the internal parameters (focal length, etc.) and external parameters (position, orientation, etc.) of multiple photographic devices, and synthesize each image after conversion based on the parameters. By performing the process, a wide viewing angle or high-resolution video that is connected smoothly is generated.

JP 2004-135209 A

  In the prior art, inconsistency in visibility due to differences in the projection center positions of a plurality of photographing apparatuses occurs in principle, and thus image quality deterioration due to this is unavoidable. Similarly, inconsistencies with respect to optical characteristics having directivity, such as the state of specular reflection, occur in principle, and image quality deterioration due to this is unavoidable.

  In order to solve the above-described problems, the present invention includes one whole photographing device that photographs image data serving as a reference, and a detailed photographing device that photographs a range including a part of the photographing range of the whole photographing device, Further, means for performing geometric transformation on the image data photographed by the whole photographing apparatus to generate whole image data, and detailed image data by performing geometric transformation on the image data photographed by the detailed photographing apparatus Generating means, means for determining the presence or absence of consistency for each pixel by comparing the whole image data and the detailed image data, and the determination result of the pixel value of the image data to be output is “no consistency” A pixel that is determined using the entire image data and a pixel whose previous determination result is “consistent” is determined using the detailed image data. It was. As for the presence / absence of consistency, “consistency” is obtained when the difference between the pixel value of the whole image data and the pixel value of the detailed image data is within a predetermined allowable range, and otherwise. Judged to be “no consistency”.

  As the predetermined allowable range, for example, an appropriate value determined according to various evaluation methods for the difference in pixel values may be used as exemplified below.

  For example, in the case of a monochrome image, when each pixel is handled with an 8-bit value and the absolute value of the difference between the pixel values (0 to 255) is evaluated as a difference, for example, “5 or less”. Etc. may be used as a predetermined tolerance.

  In the case of a color image, when each image is converted into a monochrome image based on a predetermined conversion formula and the absolute value of each pixel value difference is evaluated as a difference, a certain threshold value T is used. “T or less” or the like may be used as a predetermined allowable range, or the absolute value of the difference between the chromaticity points when converted to the UCS color system and the Y value when converted to the XYZ color system When a difference is evaluated based on the above, using a certain threshold value T1 and another threshold value T2, “the distance between chromaticity points is equal to or less than T1 and the absolute value of the difference between Y values is equal to or less than T2” or the like It may be used as an allowable range.

  According to the present invention, it is possible to generate high-resolution image data using a plurality of photographing devices while ensuring that image data consistent with image data photographed by one photographing device is captured. It becomes.

  Hereinafter, embodiments of the present invention will be described using examples.

  Hereinafter, the first embodiment will be described with reference to FIGS. 1 to 22 and 60.

  FIG. 1 is a conceptual diagram for explaining the effect.

  In FIG. 1, image data 110 is image data photographed by the whole photographing device 201, image data 120 is image data photographed by the left surface photographing device 202, and image data 130 is image data photographed by the right surface photographing device 203. The image data 111 is image data that is output when the image data 110, the image data 120, and the image data 130 are input to the image composition apparatus 200 of the present embodiment. The whole image capturing device 201, the left surface image capturing device 202, the right surface image capturing device 203, and the image composition device 200 will be described with reference to FIG.

  The image synthesizing apparatus basically generates image data 111 from the image data 110. At this time, the image data 121 generated based on the image data 120 is combined with the image data 111 within a range that can be matched with the image data 110. Then, by combining the image data 131 generated based on the image data 130 with the image data 111, it is possible to increase the resolution of a part or the whole of the image data 111.

  Here, for example, in the range that can be matched with the image data 110, the overlap between the person 140 and the person 141 looks different from each other in the image data 110 and the image data 130. If there is such inconsistency, the image data 111 is generated with priority given to the appearance in the image data 110. A specific method for generating the image data 111 will be described with reference to FIG.

  FIG. 1 is a conceptual diagram for explaining the effect, and the actual shapes of the high-definition region 121 and the high-definition region 131 may be significantly different from the shape shown in FIG.

  FIG. 2 is a system configuration diagram of the system according to the present embodiment.

  In FIG. 2, image data captured by the whole image capturing device 201 is input to the image composition device 200 via the video input unit 210 and stored in the input data storage unit 220. Similarly, image data captured by the left side imaging device 202 and the right side imaging device 203 is input to the image composition device 200 via the video input unit 210 and stored in the input data storage unit 220. The imaging field angle of the overall imaging device 201 is configured to be wider than the imaging angle of view of the left imaging device 202 and the imaging imaging angle of the right imaging device 203, and the left imaging device 202 and the right imaging device 203 are the same as those of the overall imaging device 201. Shoot within the shooting area. In addition, the imaging device density of the entire imaging device 201, the imaging device density of the left surface imaging device 202, and the imaging device density of the right surface imaging device 203 are configured to be approximately the same. With this configuration, the image data captured by the left image capturing device 202 and the image data captured by the right image capturing device 203 are images that are close to those obtained by capturing a part of the image data captured by the overall image capturing device 201 in more detail. It becomes data.

  The image composition device 200 includes a video input unit 210 that acquires image data captured by the whole image capturing device 201, the left surface image capturing device 202, and the right surface image capturing device 203, an input data storage unit 220 that stores the acquired image data, and an input An image processing unit 230 that performs image processing on the data stored in the data storage unit 220 and the intermediate data storage unit 260 while reading parameters stored in the parameter storage unit 240 as necessary, and stores the data in the parameter storage unit 240 A parameter setting unit 250 that acquires parameters to be performed, an intermediate data storage unit 260 that stores intermediate data output by the processing of the image processing unit 230, and a video output unit 270 that outputs generated image data or default image data. , The video input unit 210, the image processing unit 230, and the video output unit 270. Configured to include an overall controller 280 that performs timing control.

  Note that FIG. 3 is used for data stored in the input data storage unit 220, FIG. 4 is used for data stored in the overall control unit 280, and FIG. 5 is used for data stored in the parameter storage unit 240. The data stored in the intermediate data storage unit 260 will be described with reference to FIG.

  FIG. 3 is an explanatory diagram of data stored in the input data storage unit 220 in the system of this embodiment.

  In accordance with a control signal from the overall control unit 280, the input data storage unit 220 sets image data captured by the overall imaging apparatus 201 as overall input image data 221 and image data captured by the left side imaging apparatus 202 as left side input image data 222. In addition, the image data captured by the right side photographing device 203 is stored in the right side input image data 223, respectively. When time-series image data is input, for example, a frame buffer is prepared for two screens, one is a processing frame buffer for the image processing unit 230 to process, and the other is a video input unit 210 for image data. It may be configured so that it can be used by switching every frame so that it becomes an input frame buffer for writing.

  FIG. 4 is an explanatory diagram of data stored in the overall control unit 280 in the system of this embodiment.

  The overall control unit 280 is configured to store an overall input state flag 281, a left side input state flag 282, a right side input state flag 283, a state mode flag 284, and default image data 285.

  The whole input state flag 281 is a flag that holds a state of whether or not image data is input from the whole photographing apparatus 201, and whether or not image data is inputted from the whole photographing apparatus 201 via the video input unit 210. It detects and updates sequentially. Similarly, the left side input state flag 282 is a flag for holding a state of whether or not image data is input from the left side photographing apparatus 202, and whether or not the image data is input from the left side photographing apparatus 202 is displayed in the video input unit 210. And sequentially updating them. Similarly, the right side input state flag 283 is a flag for holding a state of whether or not image data is input from the right side photographing apparatus 203, and whether or not the image data is input from the right side photographing apparatus 203 is displayed in the video input unit 210. And sequentially updating them. The state mode flag 284 is a flag that holds a state indicating whether or not the initialization of the image composition apparatus 200 has been completed, and the initial value is “uninitialized”.

  The default image data 285 is image data output from the image composition device 200 when no image data is input from the overall photographing device 201. In this embodiment, the default image data 285 is stored in the overall control unit 280. However, the default image data 285 may be stored in the intermediate data storage unit 260 or the video output unit 270.

  FIG. 5 is an explanatory diagram of data stored in the parameter storage unit 240 in the system of the present embodiment.

  The parameter storage unit 240 is configured to store internal parameter data 241, external parameter data 242, color tone correction data 243, and output parameter data 244.

  The internal parameter data 241 is data that retains internal parameters of the whole image capturing device 201, the left surface image capturing device 202, and the right surface image capturing device 203. Specifically, it relates to a single photographing apparatus such as a projection method, focal length, lens distortion parameters, non-orthogonal distortion of the axis, coordinate values on the image of the intersection of the imaging surface and the optical axis, and arrangement of the imaging element. Stores necessary parameters among parameters. Here, the projection method is information on whether an equidistant fisheye lens or the like is used, or whether a lens represented by general perspective projection is used.

  The external parameter data 242 is a parameter that represents the relative relationship between the whole photographing device 201 and the left surface photographing device 202 and the relative relationship between the whole photographing device 201 and the right surface photographing device 203. Specifically, for example, the three-dimensional position coordinates of the projection center position of the left surface photographing device 202 and the three dimensional position coordinates of the projection center position of the right surface photographing device 203 when the projection center position of the whole photographing device 201 is taken as the coordinate origin. And the third-order rotation matrix that represents the photographing posture of the left-side photographing device 202 when the photographing posture of the whole photographing device 201 is used as a reference, and the photographing posture of the right-side photographing device 203 when the photographing posture of the whole photographing device 201 is used as a reference. And a third-order rotation matrix representing.

  The color tone correction data 243 is data for matching the color tones of the whole image capturing device 201, the left surface image capturing device 202, and the right surface image capturing device 203. Necessary among data for correcting decrease in peripheral brightness, data for correcting difference in color space, data for converting image data shot with different exposure to image data shot with reference exposure, etc. Store things.

  The output parameter data 244 is data that defines a method for generating output image data of the image composition device 200. The projection center position is set equal to the projection center position of the whole image capturing apparatus 201, and parameters such as the projection method and the angle of view when the coordinate system is also equal to the coordinate system of the entire image capturing apparatus 201 are stored.

  FIG. 6 is an explanatory diagram of data stored in the intermediate data storage unit 260 in the system of the present embodiment.

  The intermediate data storage unit 260 is configured to store map image data 610, converted image data 620, reference image data 630, change image data 640, and output image data 650.

  The map image data 610 is image data including information for sampling input image data to generate new image data. The entire map image data 611, the left side map image data 612, and the right side map image data. 613, left side map corrected image data 614, right side map corrected image data 615, and composite map image data 616. Here, the resolution of the entire map image data 611, the left map image data 612, the right map image data 613, the left map correction image data 614, the right map correction image data 615, and the composite map image data 616 is the resolution of the output image data 650. To be equal to The data format of the pixel values of various map image data included in the map image data 610 will be described with reference to FIG.

  The converted image data 620 is data obtained by converting the entire input image data 221 based on the entire map image data 611 and data obtained by converting the left input image data 222 based on the left map image data 612. It consists of certain left side converted image data 622 and right side converted image data 623 which is data obtained by converting the right side input image data 213 based on the right side map image data 613. Here, the resolutions of the whole converted image data 621, the left side converted image data 622, and the right side converted image data 623 are configured to be equal to the resolution of the output image data 650.

  The reference image data 630 is data representing a state at the end of the processing at time (T-1) in order to detect a portion changed from the state at time (T-1) when performing processing at time T. And consists of general reference image data 631, left side reference image data 632, and right side reference image data 633. Here, the overall reference image data 631, the left side reference image data 632, and the right side reference image data 633 are configured to have the same resolution as that of the output image data 650.

  The change image data 640 is binary image data representing a portion changed from the state at the time (T-1) in the process at the time T. The change image data 640 includes the entire change image data 641, the left side change image data 642, and the right side. It consists of change image data 643. Here, the overall change image data 641, the left side change image data 642, and the right side change image data 643 are configured to have the same resolution as the output image data 650.

  FIG. 7 is a diagram illustrating a data format of pixel values of various map image data in the system of the present embodiment.

  The data format 710 is a data format of pixel values used in the entire map image data 611, the left side map image data 612, the right side map image data 613, the left side map corrected image data 614, and the right side map corrected image data 615. A reference source X coordinate value 711 and a reference source Y coordinate value 712 for performing sampling processing are provided.

  The data format 720 is a data format of pixel values used in the composite map image data 616. An input data identifier 721 that determines which image data of the entire input image data 221, left side input image data 222, and right side input image data 223 is used as a reference source image, and a reference source X coordinate value for performing sampling processing 722 and a reference source Y coordinate value 723.

  Note that, in both the data format 710 and the data format 720, if there is no appropriate reference source coordinate value, an exception value is stored as the reference source coordinate value. In addition, regarding the pixel value of the pixel sampled based on the reference source coordinate value which is an exceptional value, the processing is also performed as an exceptional value, and it is determined whether or not the difference between the two pixel values is within an allowable range. Is always treated as “out of tolerance”.

  FIG. 8 is a situation explanatory diagram for explaining the operation of the image composition processing in the system of the present embodiment.

  When the entire photographing apparatus 201 photographs the entire conference room 810 and the left photographing apparatus 202 and the right photographing apparatus 203 are each configured to photograph a part thereof, the right photographing apparatus 203 at time (T-1). The operation when the person who is at the position 820 within the shooting range moves to the position 821 outside the shooting range of the right side photographing apparatus 203 at time T will be described with reference to FIG.

  FIG. 9 is a conceptual diagram for explaining the operation of the image composition process in the system of the present embodiment. The conceptual diagram of FIG. 9 represents processing in the situation of FIG.

  The image data 910 is the entire input image data 221 at time (T-1), and the image data 920 is the right side input image data 223 at time (T-1). The image data 911 is the entire input image data 221 at time T, and the image data 921 is the right side input image data 223 at time T. It can be seen that the person who was at position 820 at time (T-1) moved to position 821 at time T.

  The image data 912 is the entire reference image data 631 before update, and the image data 922 is the right side reference image data 633 before update.

  The image data 913 is whole converted image data 621, and the image data 923 is right side converted image data 623. Even though the person at the position 820 has disappeared, since the sampling based on the state at the time (T-1) is performed, it can be seen that the corresponding portion of the image data 923 has been whitened out.

  The image data 914 is overall change image data 641 and the image data 924 is right side change image data 643. The image data 914 is generated by detecting a difference between the image data 912 and the image data 913, and the image data 924 is generated by detecting a difference between the image data 922 and the image data 923. Is.

  The image data 915 is the updated overall reference image data 631, and the image data 925 is the updated right side reference image data 633. It can be seen that the expected image data is generated by updating the sampling data of the portion where the state has changed based on the state at time T.

  FIG. 10 is a flowchart of image composition processing in the system of this embodiment. The image composition process in FIG. 10 is a process that is performed at predetermined time intervals in the overall control unit 280.

  When the image composition process is started in step 1000, first, in step 1001, it is determined whether or not there is an entire video input. This determination is made by referring to the entire input state flag 281. If it is determined in step 1001 that there is no entire video input, the process proceeds to step 1002 where the status mode flag 284 is set to “uninitialized”, and then the process proceeds to step 1003 to receive a default from the video output unit 270. After outputting the image data 285, the process proceeds to step 1004 and the image composition process is terminated.

  If it is determined in step 1001 that there is an entire video input, input image data update processing is performed in step 1005. In the input image data update process in step 1005, the input data storage unit 220 is controlled so that the image data photographed by the whole photographing device 201 is the whole input image data 221 and the image data photographed by the left photographing device 202 is the left surface input image. This is a process of storing, in the data 222, image data taken by the right side photographing device 203 in the right side input image data 223, respectively.

  Since there is no input from the left side photographing device 202 can be detected by referring to the left side input state flag 282 and no input from the right side photographing device 203 can be detected by the right side input state flag 283, respectively, actually FIG. In the case of performing the above process, the process regarding the left side input image data 222 or the right side input image data 223 may be appropriately skipped based on the left side input state flag 282 or the right side input state flag 283. In order to avoid complication of the flowchart, in this specification, for convenience, image data in which all pixel values are “black” when there is no video input is stored.

  Next, in step 1006, it is determined whether or not the initialization completion mode. This determination is made by referring to the state mode flag 284. If the state mode flag 284 is “initialization complete”, the process proceeds to an intermediate data update process in step 1007. If the state mode flag 284 is “initialization not completed”, the process proceeds to an intermediate data generation process in step 1008. If the process proceeds to the intermediate data generation process in step 1008, the status mode flag 284 is set to “initialization complete” in step 1009. The intermediate data update process in step 1007 will be described with reference to FIG. 20, and the intermediate data generation process in step 1008 will be described with reference to FIG.

After step 1007 or step 1009 is completed, a composite image data output process is performed in step 1010. This is a process of outputting the output image data 650 stored in the intermediate data storage unit 260 from the video output unit 270.
After completion of the composite image data output process in step 1010, the process proceeds to step 1004 to end the image composition process.

FIG. 11 is a flowchart of intermediate data generation processing in the system of this embodiment.
When the intermediate data generation processing is started in step 1100, first, map image data initialization processing is performed in step 1101. Details of the map image data initialization process will be described with reference to FIG.

  Next, in step 1102, color tone correction processing is performed. In the color tone correction process, based on the color tone correction data 243, image correction is performed on each of the total input image data 221, the left side input image data 222, and the right side input image data 223, and then the total input image data 221, This process is to store the left side input image data 222 and the right side input image data 223 again.

  Next, in step 1103, a converted image data generation process is performed. The converted image data generation process includes a whole converted image data generation process, a left side converted image data generation process, and a right side converted image data generation process. Of these, details of the entire converted image data generation processing will be described with reference to FIG. In the left side conversion image data generation process, the left side map image data 612 is used instead of the whole map image data 611 used in the whole conversion image data generation process, and the left side input image data 222 is used instead of the whole input image data 221. Instead of the image data 621, the left side conversion image data 622 may be used. The same applies to the right side conversion image data generation processing.

Next, in step 1104, each surface map image data generation process is performed. Each surface map image data generation process includes a left surface map image data generation process and a right surface map image data generation process. Of these, details of the left map image data generation processing will be described with reference to FIG. The right map image data generation process can also be performed by the same process.
Next, in step 1105, reference image data generation processing is performed. The reference image data generation process includes an overall reference image data generation process, a left side reference image data generation process, and a right side reference image data generation process. Of these, details of the overall reference image data generation processing will be described with reference to FIG. The left side reference image data generation process refers to the left side input image data 222 instead of the whole input image data 221 used in the whole reference image data generation process, and the left side map image data 612 instead of the whole map image data 611. The left side reference image data 632 may be used instead of the image data 631. The same applies to the right side reference image data generation processing.

  Next, in step 1106, composite map image data generation processing is performed. Details of the composite map image data generation processing will be described with reference to FIGS. 17 and 18.

  Next, in step 1107, output image data generation processing is performed. Details of the output image data generation processing will be described with reference to FIG.

After completing the above processing, the intermediate data generation processing is ended in step 1108.
FIG. 12 is a flowchart of map image data initialization processing in the system of this embodiment.

  When the map image data initialization process is started in step 1200, first, the internal parameter data 241 is read in step 1201, and then the output parameter data 244 is read in step 1202.

  Next, in step 1203, an overall initial value generation process is performed. In the overall initial value generation process, in order to generate the output image data 650, each pixel of the output image data 650 calculates which pixel value of the coordinates of the overall input image data 221 should be sampled. , A process of storing the coordinate value of the reference source in the entire map image data 611. The calculation process of the reference source coordinate value can be easily performed by a known technique using the internal parameter data 241 and the output parameter data 244.

  Next, in step 1204, external parameter data 242 is read.

  Next, in step 1205, left side initial value generation processing is performed. In the initial value generation process for the left side, in order to generate the output image data 650, each pixel of the output image data 650 calculates which coordinate value of the left side input image data 222 should be sampled. , A process of storing the coordinate value of the reference source in the left map image data 612. However, since the three-dimensional shape of the subject is not known at that time, the reference source coordinate value is calculated on the assumption that all subjects are on a predetermined curved surface. The calculation of the reference source coordinate value can be easily performed from the internal parameter data 241, the external parameter data 242, and the output parameter data 244 when a predetermined curved surface is given.

  Next, in step 1206, right side initial value generation processing is performed. The right side initial value generation process can be performed by the same process as the left side initial value generation process.

  After the above process is completed, the map image data initialization process is ended in step 1207.

  FIG. 13 is a flowchart of the entire converted image data generation process in the system of this embodiment.

When the entire converted image data generation process is started in step 1300, first, in step 1301, n, which is the serial number of the pixel to be processed, is initialized to zero.
Next, in step 1302, the pixel value of the nth pixel of the entire map image data 611 is read.

Next, in step 1303, the pixel value at the position corresponding to the reference source coordinate value read in step 1302 on the entire input image data 221 is calculated and set as the pixel value of the nth pixel in the entire converted image data 621. To do. Here, the reference source coordinate value is not generally an integer value, but the pixel value may be calculated by a known method such as bilinear interpolation.
Next, in step 1304, n, which is the serial number of the pixel to be processed, is incremented by 1, and in step 1305, it is determined whether or not the processing has been completed for all the pixels in the entire converted image data 621. If the process has been completed for all the pixels in the entire converted image data 621, the process proceeds to step 1306 to end the entire converted image data generation process. If there is a pixel that has not been processed yet, the process proceeds to step 1306. Proceed to 1302 and continue processing.

  FIG. 14 is a flowchart of left-side map image data generation processing in the system of this embodiment.

  When the left side map image data generation process is started in step 1400, first, the entire converted image data 621 is read in step 1401, and then the left side converted image data 622 is read in step 1402.

  Next, in step 1403, left-side map corrected image data generation processing is performed. The left side map corrected image data generation process is a process of generating left side map corrected image data 614 for correcting an error in the left side map image data 612 based on the whole converted image data 621 and the left side converted image data 622. . For each pixel position of the entire converted image data 621, it is only necessary to calculate from which coordinate position of the left plane converted image data 622 the sampling should be performed, and this can be performed using a known technique such as pattern matching. It should be noted that for pixels for which correspondence cannot be obtained, an exceptional value may be stored so that the subsequent processing can be skipped as appropriate.

  Next, in step 1404, left side map image data composition processing is performed. Details of the left map image data composition processing will be described with reference to FIG.

  After the above process is finished, the left map image data generation process is finished in step 1405.

FIG. 15 is an explanatory diagram of a map image data synthesis method in the system of this embodiment.
The reference source coordinate value of the pixel 1511 on the left side map corrected image data 1510 is the coordinate value 1521 on the left side map image data 1520, and the reference source coordinate value of the coordinate value 1521 is the coordinate value 1531 of the left side input data 1530. At this time, the reference source coordinate value of the pixel 1511 on the left side map corrected image data 1510 is updated to the coordinate value 1531. After this update is performed on all the pixels of the left-side map corrected image data 1510, the left-side map corrected image data 1510 is stored again as the left-side map image data 1520.

  Here, since the coordinate value 1521 generally does not coincide with the position of the pixel center, it is necessary to calculate the coordinate value 1531 by bilinear interpolation or the like. That is, in FIG. 15, instead of obtaining the reference source coordinate value of the coordinate value 1521, the left map image data 1520 for each of the coordinate value 1522, the coordinate value 1523, the coordinate value 1524, and the coordinate value 1525 at the pixel center position. The coordinate value 1531 corresponding to the coordinate value 1521 is calculated by obtaining a reference source coordinate value with reference to the pixel values of the pixel values and performing bilinear interpolation on the reference source coordinate values.

  FIG. 16 is a flowchart of overall reference image data generation processing in the system of the present embodiment.

When the entire reference image data generation process is started in step 1600, first, in step 1601, n, which is the serial number of the pixel to be processed, is initialized to zero.
Next, in step 1602, the pixel value of the nth pixel of the entire map image data 611 is read.

  Next, in step 1603, the pixel value at the position corresponding to the reference source coordinate value read in step 1602 on the entire input image data 221 is calculated and set as the pixel value of the nth pixel of the entire reference image data 631. To do. Here, the reference source coordinate value is not generally an integer value, but the pixel value may be calculated by a known method such as bilinear interpolation.

  In step 1604, n, which is the serial number of the pixel to be processed, is incremented by 1. Subsequently, in step 1605, it is determined whether or not the processing has been completed for all the pixels in the overall reference image data 631. If the process has been completed for all the pixels in the overall reference image data 631, the process proceeds to step 1606 to end the overall reference image data generation process, and if there is a pixel that has not been completed yet, the process proceeds to step 1606. Proceeding to 1602, processing continues.

  FIG. 17 is a flowchart of the composite map image data generation process in the system of this embodiment.

When the composite map image data generation process is started in step 1700, first, in step 1701, n, which is the serial number of the pixel to be processed, is initialized to zero.
Next, in step 1702, the pixel value VT of the nth pixel of the overall reference image data 631, the pixel value VL of the nth pixel of the left surface reference image data 632, and the nth pixel of the right surface reference image data 633. The pixel value VR is read.

  Next, in step 1703, adopted pixel determination processing is performed. Details of the adopted pixel determination process will be described with reference to FIG.

  In step 1704, n, which is the serial number of the pixel to be processed, is incremented by 1. Subsequently, in step 1705, it is determined whether or not processing has been completed for all the pixels in the composite map image data 616. If the process has been completed for all the pixels in the composite map image data 616, the process proceeds to step 1706 to end the composite map image data generation process, and if there is a pixel that has not been processed yet, the process proceeds to step 1706. Proceed to 1702 and continue processing.

  FIG. 18 is a flowchart of adopted pixel determination processing in the system of this embodiment.

  When the adopted pixel determination process is started in step 1800, it is first determined in step 1801 whether or not the difference between VT and VL is within an allowable range based on a predetermined threshold value. If it is determined in step 1801 that the value is within the allowable range, the process proceeds to step 1802 where “left-side input image data” is set in the input data identifier 721 as the pixel value of the nth pixel in the composite map image data 616. The reference source X coordinate value 722 is set to the reference source X coordinate value 711 which is the pixel value of the nth pixel of the left side map image data 612, and the reference source Y coordinate value 723 is set to n of the left side map image data 612. A reference source Y coordinate value 712 that is the pixel value of the th pixel is set.

  If it is determined in step 1801 that the value is outside the allowable range, the process proceeds to step 1803. In step 1803, it is determined whether or not the difference between VT and VR is within an allowable range based on a predetermined threshold. If it is determined in step 1803 that the value is within the allowable range, the process proceeds to step 1804 where “right-side input image data” is set in the input data identifier 721 as the pixel value of the nth pixel of the composite map image data. The reference source X coordinate value 722 is set with the reference source X coordinate value 711 that is the pixel value of the n th pixel of the right side map image data 613, and the reference source Y coordinate value 723 is set to the n th number of the right side map image data 613. A reference source Y-coordinate value 712 that is the pixel value of this pixel is set.

  If it is determined in step 1803 that the value is outside the allowable range, the process proceeds to step 1805, where “total input image data” is set in the input data identifier 721 as the pixel value of the nth pixel of the composite map image data. The reference source X coordinate value 722 is set to the reference source X coordinate value 711 that is the pixel value of the nth pixel of the entire map image data 611, and the reference source Y coordinate value 723 is set to the nth pixel of the entire map image data 611. A reference source Y-coordinate value 712 that is the pixel value of this pixel is set.

  As described above, after completion of any one of the processing in step 1802, step 1804, and step 1805, the adopted pixel determination processing is terminated in step 1806.

  FIG. 19 is a flowchart of output image data generation processing in the system of this embodiment.

When the output image data generation process is started in step 1900, first, n, which is the serial number of the pixel to be processed, is initialized to 0 in step 1901, and then in step 1902, the nth of the composite map image data 616 is initialized. The pixel value of the pixel is read.
Next, in step 1903, the pixel value of the coordinates represented by the reference source X coordinate value 722 and the reference source Y coordinate value 723 is obtained on the input data represented by the input data identifier 721 of the pixel value read in step 1902. Calculate and set as the pixel value of the nth pixel of the output image data 650. Here, the reference source X coordinate value 722 and the reference source Y coordinate value 723 are generally not integer values, but the pixel value may be calculated by a known method such as bilinear interpolation.

  Next, n, which is the serial number of the pixel to be processed, is incremented by 1 in step 1904, and then in step 1905, it is determined whether or not the processing has been completed for all the pixels in the output image data 650. If all pixels in the output image data 650 have been processed, the process proceeds to step 1906 to end the output image data generation process. If there are pixels that have not been processed yet, the process proceeds to step 1902. Proceed and continue processing.

  FIG. 20 is a flowchart of intermediate data update processing in the system of this embodiment.

  When the intermediate data update process is started in step 2000, first, a color tone correction process is performed in step 2001. The color tone correction process in step 2001 is the same process as the color tone correction process in step 1102.

  Next, in step 2002, a converted image data generation process is performed. The converted image data generation process in step 2002 is the same process as the converted image data generation process in step 1103.

  Next, in step 2003, change image data generation processing is performed. The change image data generation process includes an entire change image data generation process, a left face change image data generation process, and a right face change image data generation process. Of these, details of the entire change image data generation processing will be described with reference to FIG. In the left side change image data generation processing, the left side reference image data 632 is used instead of the whole reference image data 631 used in the whole change image data generation processing, and the left side conversion image data 622 is used instead of the whole conversion image data 621. The left side change image data 642 may be used instead of the image data 641. The same applies to the right-side change image data generation processing.

  Next, in step 2004, each surface map image data update process is performed. Each surface map image data update process includes a left surface map image data update process and a right surface map image data update process. Of these, details of the left map image data update processing will be described with reference to FIG. The right map image data update process can also be performed by the same process.

  Next, in step 2005, reference image data generation processing is performed. The reference image data generation process in step 2005 is the same process as the reference image data generation process in step 1105.

  Next, in step 2006, composite map image data generation processing is performed. The composite map image data generation process in step 2006 is the same process as the composite map image data generation process in step 1106.

  Next, in step 2007, output image data generation processing is performed. The output image data generation process in step 2007 is the same process as the output image data generation process in step 1107.

After completing the above processing, in step 2008, the intermediate data update processing is ended.
FIG. 21 is a flowchart of overall change image data generation processing in the system of this embodiment.

When the whole change image data generation process is started in step 2100, first, in step 2101, n, which is the serial number of the pixel to be processed, is initialized to 0.
Next, in step 2102, the pixel value V0 of the nth pixel of the entire reference image data 631 is read. Subsequently, in step 2103, the pixel value V1 of the nth pixel of the entire converted image data 621 is read.

  Next, in step 2104, it is determined whether or not the difference between V0 and V1 is within an allowable range based on a predetermined threshold. If it is determined in step 2104 that the value is within the allowable range, the process proceeds to step 2105 to set the pixel value of the nth pixel of the entire change image data 641 to 0. If it is determined in step 2104 that the value is outside the allowable range, the process proceeds to step 2106 to set the pixel value of the nth pixel of the entire change image data 641 to 1.

  After step 2105 or step 2106, n, which is the serial number of the pixel to be processed in step 2107, is incremented by 1. Subsequently, in step 2108, whether or not the processing has been completed for all the pixels in the whole change image data 641. Determine. If the process has been completed for all the pixels in the whole change image data 641, the process proceeds to step 2109 to end the whole change image data generation process, and if there is a pixel that has not been finished, step is performed. Go to 2102 and continue processing.

  FIG. 22 is a flowchart of left map image data update processing in the system of the present embodiment.

  When the left side map image data update process is started in step 2200, first, the whole converted image data 621 and the left side converted image data 622 are read in step 2201, and then in step 2202, the whole changed image data 641 and the left side changed image are read. Data 642 is read.

  Next, in step 2203, a left map corrected image data update process is performed. In the left-side map corrected image data update processing, the left-side map corrected image data generation described in step 1403 of FIG. 14 is performed for pixels whose pixel value is 1 in either the entire changed image data 641 or the left-side changed image data 642. The pixel value of the left-side map modified image data 614 is determined in the same manner as the processing, and for the pixel whose pixel value is 0 in both the entire change image data 641 and the left-side change image data 642, the coordinate value of the pixel is set. This is a process of setting the reference source coordinate value of the pixel as it is. By using the change image data, the region where the state has changed can be known in advance, so that the processing such as pattern matching can be expected to be faster and more robust. In addition, since the amount of calculation related to a portion where there is no change in the state can be reduced, the processing speed can be increased.

Next, in step 2204, left side map image data composition processing is performed. The left side map image data composition process in step 2204 is the same process as the left side map image data composition process described in step 1404 of FIG.
After finishing the above processing, the left map image data update processing is finished in step 2205.
FIG. 60 is a hardware configuration diagram.

The image composition device 6000 includes a USB interface 6011, a USB interface 6012, a USB interface 6013, a CPU 6020, a memory 6021, a hard disk 6022, an image board 6023, and a network card 6024.
The hard disk 6022 stores various programs corresponding to the various processes described so far, various data to be stored in the overall control unit 280, and various data to be stored in the parameter storage unit 240. .

The system of this embodiment is realized by the CPU 6020 executing various programs loaded from the hard disk 6022 to the memory 6021 with the hardware configuration shown in FIG.
The functions of the input data storage unit 220 and the intermediate data storage unit 260 are realized by the memory 6021, and the function of the video input unit 210 is realized by the USB interface 6011, the USB interface 6012, and the USB interface 6013, and the video output unit 270. Is realized by the image board 6023, and the function of the parameter setting unit 250 is realized by the network card 6024.

  Further, the functions of the whole image capturing device 201, the left surface image capturing device 202, and the right surface image capturing device 203 are realized by the USB camera 6001, the USB camera 6002, and the USB camera 6003, respectively.

  It should be noted that the hard disk 6022 should have a function as long as the stored information is not lost when the power is turned off, and can be replaced by a non-volatile memory, or a CD-ROM or floppy (registered trademark) disk. A combination of such media and an interface for reading the media can be used instead.

  A function that the USB interface 6011, the USB interface 6012, and the USB interface 6013 should have is that an input signal to the image composition device 200 is written in the memory 6021 as digital image data. Instead, use the appropriate one. For example, when the input signal is an analog signal, a capture card having an A / D conversion function may be used, and when the input signal is digital data sent via a network, a network card is used. do it.

  The function that the image board 6023 should have is to output the combined image data to the outside, and the presence / absence of the D / A conversion function may be changed according to the output destination. Furthermore, when storing the synthesized data in a storage medium such as a hard disk, an interface corresponding to the medium may be used, and when transmitting the data via a network, a network card may be used. Further, a network card having a data compression function such as an MPEG2 encoding function may be used.

  A function that the network card 6024 should have is that various data to be stored in the parameter storage unit 240 can be written in the hard disk 6022, and a medium such as a CD-ROM or a floppy (registered trademark) disk and the medium are read. For example, a pair with an interface may be substituted, or an external input interface such as a keyboard may be substituted. This hardware configuration can also be applied to other embodiments.

  As described above, according to the present embodiment, high-resolution image data is generated using a plurality of imaging devices while ensuring that image data consistent with image data captured by one imaging device is captured. It becomes possible.

  In this embodiment, the change image data is used from the viewpoint of processing speed and robustness. However, the embodiment of the present invention is not limited to this, and processing is always performed for all pixels. Obviously, it may be possible to do this.

  Hereinafter, the second embodiment will be described with reference to FIGS. 23 to 33.

The present embodiment is an example in which intermediate data can be updated by a simpler combination of processes with respect to the example described in the first embodiment.
FIG. 23 is a system configuration diagram of the system of this embodiment. 23, the same reference numerals as those in FIG. 2 denote the same elements as those in FIG. The same reference numerals as those in FIG. 6 denote the same elements as those in FIG.
In the image composition device 2300 of this embodiment, the intermediate data storage unit 2360 is configured to store the depth image data 2360 and the search depth set data 2370 with respect to the image composition device 200, and further, the image processing unit The intermediate data generation process and the intermediate data update process executed in 2330 are modified.

  Here, the depth image data 2360 is composed of entire depth image data 2361 and background depth image data 2362. Here, the resolution of the entire depth image data 2361 and the background depth image data 2362 is configured to be equal to the resolution of the output image data 650.

  Further, the search depth set data 2370 is composed of prescribed search depth set data 2371 that is a set of predetermined depth values, temporary search distance set data 2372, and maximum search distance set data 2373. The configuration method of the prescribed search depth set data 2371 will be described with reference to FIG. 26, and the configuration method of the temporary search distance set data 2372 and the maximum search distance set data 2373 will be described with reference to FIG.

FIG. 24 is a flowchart of intermediate data generation processing in the system of this embodiment. 24, the same reference numerals as those in FIG. 11 represent the same processes as those in FIG.
Details of the depth image data generation processing in step 2401 will be described with reference to FIG.

  Each surface map image data generation process in step 2402 includes a left map image data generation process and a right map image data generation process. Of these, details of the left map image data generation processing will be described with reference to FIG. The right map image data generation process can also be performed by the same process.

  FIG. 25 is a flowchart of depth image data generation processing in the system of this embodiment.

  When the depth image data generation process is started in step 2500, first, in step 2501, n, which is the serial number of the pixel to be processed, is initialized to zero.

  Next, in step 2502, a predetermined precision ratio with respect to a three-dimensional direction that is defined by the output parameter data 244 and the nth pixel position of the output image data 650 and that faces the pixel position direction from the projection center position as a base point. Based on the evaluation function, for each depth value included in each specified depth set data, a precision ratio when the depth value is the depth value is calculated.

  Next, in step 2503, the depth value with the highest relevance ratio calculated in step 2502 is selected and set as the nth pixel value of the entire depth image data 2361.

  Next, in step 2504, n, which is the serial number of the pixel to be processed, is incremented by 1, and in step 2505, it is determined whether or not the processing has been completed for all the pixels in the entire depth image data 2361. If all pixels in the entire depth image data 2361 have been processed, the process proceeds to step 2506. If there is a pixel that has not been processed yet, the process proceeds to step 2502 to continue the process.

  In step 2506, the pixel value of each pixel of the background depth image data 2362 is set to the same value as the pixel value of the pixel at the same position in the entire depth image data 2361. Then, in step 2507, the depth image data generation process is terminated.

  FIG. 26 is a diagram for explaining a method of configuring the prescribed search depth set data 2371 in the system of the present embodiment.

  In FIG. 26, each white circle represents one depth value. In this embodiment, the depth values may be arranged at equal intervals as in the depth set data 2620 with respect to the predetermined minimum depth value 2610 and the maximum depth value 2611, or “the amount of deviation on the image data” From the viewpoint of “so that they are substantially equal”, the smaller the depth value, such as the depth set data 2621, may be densely arranged.

  FIG. 27 is a flowchart of left-side map image data generation processing in the system of the present embodiment.

When the left map image data generation process is started in step 2700, first, in step 2701, n, which is the serial number of the pixel to be processed, is initialized to 0.
Next, in step 2702, the entire depth image data with respect to a three-dimensional direction that is defined by the output parameter data 244 and the nth pixel position of the output image data 650 and that faces the pixel position direction from the projection center position as a base point. A position P in the three-dimensional space is determined from the depth value on 2361.

  Next, in step 2703, a corresponding position on the left side input image data 222 is determined based on the position P determined in step 2702, the internal parameter data 241 and the external parameter data 242, and the nth of the left side map image data 612 is determined. Is set as the pixel value of the pixel.

  Next, in step 2704, n, which is the serial number of the pixel to be processed, is incremented by 1, and then, in step 2705, it is determined whether or not the processing has been completed for all the pixels in the left map image data 612. If the process has been completed for all the pixels in the left-side map image data 612, the process proceeds to step 2706 to end the left-side map image data generation process, and if there are pixels that have not been processed yet, the step is completed. Proceed to 2702 and continue processing.

FIG. 28 is a flowchart of the intermediate data update process in the system of this embodiment. 28, the same reference numerals as those in FIG. 20 represent the same processes as those in FIG.
Details of the entire depth image data update processing in step 2801 will be described with reference to FIG.

  Details of the background depth image data update processing in step 2802 will be described with reference to FIG.

  Each area map image data update process in step 2803 includes a left side map image data update process and a right side map image data update process. Of these, details of the left map image data update processing will be described with reference to FIG. The right map image data update process can also be performed by the same process.

  FIG. 29 is a flowchart of the entire depth image data update process in the system of the present embodiment.

  When the entire depth image data update process is started in step 2900, first, in step 2901, n, which is the serial number of the pixel to be processed, is initialized to zero.

  Next, in step 2902, pixel value priority depth determination processing is performed for the nth pixel of the entire depth image data 2361. Details of the pixel value priority depth determination processing will be described with reference to FIGS. 30 and 31. FIG.

  Next, in step 2903, n, which is the serial number of the pixel to be processed, is incremented by 1, and in step 2904, it is determined whether or not the processing has been completed for all the pixels in the entire depth image data 2361. If processing has been completed for all the pixels in the entire depth image data 2361, the process proceeds to step 2905 to end the entire depth image data update process, and if there are pixels that have not been processed yet, step is performed. Proceed to 2902 and continue processing.

  FIG. 30 is a flowchart of pixel value priority depth determination processing in the system of the present embodiment, and FIG. 31 illustrates a configuration method of the temporary search distance set data 2372 and the maximum search distance set data 2373 in the system of the present embodiment. FIG. 31, the same reference numerals as those in FIG. 26 denote the same elements as those in FIG.

  When the pixel value priority depth determination process is started in step 3000, first, an overall depth value neighborhood search process is performed in step 3001. This is to read an overall depth value that is the pixel value of the nth pixel of the overall depth image data 2361, and based on a predetermined interval and a predetermined precision evaluation function, the depth value near the overall depth value, and the depth This is a process of calculating a set of the precision ratio in value. For example, in FIG. 31, each matching rate is calculated in step 3001 for the overall depth value 3101 and the depth value 3102 and depth value 3103 in the vicinity thereof.

  Next, in step 3002, background depth value neighborhood search processing is performed. This reads a background depth value which is the pixel value of the nth pixel of the background depth image data 2362, and based on a predetermined interval and a predetermined precision evaluation function, the depth value near the background depth value and the depth This is a process of calculating a set of the precision ratio in value. For example, in FIG. 31, for each of the background depth value 3111 and the depth value 3112 and depth value 3113 in the vicinity thereof, calculation of the relevance ratio is performed in step 3002.

  Next, in step 3003, a prescribed depth value search process is performed. This is a process for obtaining the matching rate based on a predetermined matching rate evaluation function with respect to the depth value included in the prescribed search depth set data 2371.

  Next, in step 3004, an optimum depth value selection process is performed. This is a process of selecting the depth value having the highest relevance ratio from the depth values processed in Step 3001, Step 3002, and Step 3003, and setting it as the pixel value of the nth pixel of the entire depth image data 2361. is there. As a result, a process equivalent to selecting the depth value having the highest precision after obtaining the precision based on a predetermined precision evaluation function for each of the depth values included in the depth set data 3121. It becomes.

  After the processing of step 3004 ends, the pixel value priority depth determination processing ends in step 3005.

  FIG. 32 is a flowchart of background depth image data update processing in the system of the present embodiment.

  When the background depth image data update process is started in step 3200, first, in step 3201, n, which is the serial number of the pixel to be processed, is initialized to zero.

  Next, in step 3202, the pixel value V0 of the nth pixel of the entire depth image data 2361 is read. Subsequently, in step 3203, the pixel value V1 of the nth pixel of the background depth image data 2362 is read.

  Next, in step 3204, the values of V0 and V1 are compared. If it is determined that V0 is larger, the process proceeds to step 3205, and the pixel value of the nth pixel of the background depth image data 2362 is set to V0. Set to.

  After the above processing is completed, n, which is the serial number of the pixel to be processed, is incremented by 1 in step 3206. Subsequently, in step 3207, whether or not the processing has been completed for all the pixels in the background depth image data 2362 is determined. judge. If the processing has been completed for all the pixels in the background depth image data 2362, the process proceeds to step 3208 to end the background depth image data update processing, and if there are pixels that have not been processed yet, step Proceed to 3202 and continue processing.

  By updating the background depth image data 2362 in this way, each pixel of the background depth image data 2362 can hold the largest value among the depth values selected in the past with respect to the pixel.

  FIG. 33 is a flowchart of left map image data update processing in the system of the present embodiment.

  When the left map image data update process is started in step 3300, first, in step 3301, n, which is the serial number of the pixel to be processed, is initialized to zero.

  Next, in Step 3302, it is determined whether or not the subject is a pixel determined to have changed. This may be performed by referring to the entire change image data 641 and the left surface change image data 642. If it is determined in step 3302 that there is no change, the process proceeds to step 3305.

  If it is determined in step 3302 that there is “change”, in step 3303, the pixel position with the projection center position defined as the output parameter data 244 and the nth pixel position of the output image data 650 as the base point. A position P in the three-dimensional space is determined from the depth value on the entire depth image data 2361 with respect to the three-dimensional direction facing the direction.

Next, in step 3304, the corresponding coordinate value on the left side input image data 222 is determined from the position P determined in step 3303, the internal parameter data 241 and the external parameter 242, and the nth of the left side map image data 612 is determined. Is set as the pixel value of the pixel.
After the above processing is completed, n, which is the serial number of the pixel to be processed, is incremented by 1 in step 3305. Subsequently, in step 3306, it is determined whether or not processing has been completed for all the pixels in the left-side map image data 612. judge. If all the pixels in the left side map image data 612 have been processed, the process proceeds to step 3307 to end the left side map image data update process, and if there are pixels that have not yet been processed, step Go to 3302 and continue processing.

  As described above, according to the present embodiment, it is possible to update the intermediate data by a simpler combination of processes with respect to the system of the first embodiment.

  Hereinafter, the third embodiment will be described with reference to FIGS. 34 and 35. FIG.

  The present embodiment is an example in which a depth value search can be efficiently and robustly performed by detecting a moving object, compared to the example described in the second embodiment.

  Note that the system according to the present embodiment includes an intermediate data storage unit 3460 that can further store data described with reference to FIGS. 34 and 35 in place of the intermediate data storage unit 2360 in the system configuration of the second embodiment. Constitute.

  FIG. 34 is a flowchart of the entire depth image data update process in the system of this embodiment. 34, the same reference numerals as those in FIG. 29 represent the same processes as those in FIG.

  The differences between the entire depth image data update process of FIG. 34 and the entire depth image data update process of FIG. 29 are step 3401, step 3402, step 3403, and step 3404.

  The entire depth image data duplicating process in step 3401 is a process of duplicating the entire depth image data 2361 to generate duplicate entire depth image data 3410. The generated entire copy depth image data 3410 is stored in the intermediate data storage unit 3460. In the overall depth image data duplication processing, since the pixel value in the overall depth image data at a position other than the pixel position being processed is required in the flow priority depth determination processing described with reference to FIG. This is a process for preliminarily storing data at the previous time before being updated by the process for each pixel.

  The optical flow estimation processing in step 3402 estimates from which image data captured at time (T-1) and the image data captured at time T the subject that was captured in which pixel is now captured in which pixel. And can be performed using a known technique. As specific input data when applied to the present embodiment, overall reference image data 631 is used as image data taken at time (T-1), and overall converted image data 621 is used as image data taken at time T. . The result of the optical flow estimation process is stored in the intermediate data storage unit 3460 as optical flow image data 3420 having a pair of the reliability of the flow and the coordinate value of the movement source as a pixel value.

  In step 3403, the pixel value of the corresponding pixel in the optical flow image data 3420 is read, and it is determined whether or not the flow of the pixel is reliable based on a predetermined threshold value. If it is determined to be reliable, a flow priority depth determination process is performed in step 3404. If it is determined to be unreliable, the process proceeds to step 2902 and the pixel value priority depth determination process is performed as in the second embodiment. . Details of the flow priority depth determination process will be described with reference to FIG.

  FIG. 35 is a flowchart of flow priority depth determination processing in the system of the present embodiment.

  When the flow priority depth determination process is started in step 3500, first, in step 3501, a movement source depth value neighborhood search process is performed. The movement source depth value neighborhood search process is the following process.

  First, the movement source coordinate value of the pixel currently being processed is read from the optical flow image data 3420. Next, the pixel value corresponding to the movement source coordinate value of the entire copy depth image data 3410 is read and set as the movement source depth value. Then, based on a predetermined interval and a predetermined precision evaluation function, the precision is calculated for the movement source depth value and the depth values before and after the movement source depth value, and the combination of the depth value and the precision ratio is an intermediate data storage unit 3460. To store.

  Subsequently, at step 3502, an optimum depth value selection process is performed. This is a process of selecting a pair having the highest precision from the combinations of the depth value and precision that are stored in step 3501. The depth value of the set selected at this time is set as the flow optimum depth value 3510, and the matching rate of the set selected at this time is set as the flow optimum matching rate 3511.

  In step 3503, it is determined based on a predetermined threshold whether or not the flow optimum matching rate 3511 is sufficient as the matching rate.

  If it is determined in step 3503 that the matching rate is sufficient, then in step 3504, the flow optimum depth value 3510 is written as the pixel value of the pixel currently being processed in the entire depth image data 2361.

  If it is determined in step 3503 that the matching rate is not sufficient, candidate pixel value priority depth determination processing is performed in step 3505. The candidate-attached pixel value priority depth determination process is the adaptation that becomes the reference for selection by adding the flow optimal depth value 3510 as an option of the optimum depth value in Step 3004 of the pixel value priority depth determination process described in FIG. As the rate, the flow optimum matching rate 3511 is used.

  After the above processing is completed, the flow priority depth determination processing is ended in step 3506.

  As described above, according to the present embodiment, the depth value search can be performed more efficiently and robustly with respect to the example described in the second embodiment.

  Hereinafter, the fourth embodiment will be described with reference to FIGS. 36 to 39.

  In the present embodiment, output image data that is close to the image data photographed from the projection center designated by the user is generated in addition to the example described in the second embodiment. This is an example suitable for a case where, for example, in a teleconference system or the like, image data captured in accordance with the angle of view of the display device is distorted and displayed.

  FIG. 36 is a system configuration diagram of the system of this embodiment. 36, the same reference numerals as those in FIG. 23 denote the same elements as those in FIG.

  In the image composition device 3600 of this embodiment, the image storage device 3300 is configured such that the parameter storage unit 3640 can store predetermined video generation viewpoint data 3645, and the intermediate data storage unit 3660 includes environment reconstruction data. 3680 can be stored, and an image processing unit 3630 in which the processing content of the image composition processing is changed is provided.

  The video generation viewpoint data is data that defines a method for generating output image data of the image composition device 3600. The projection center position is set equal to the projection center position of the whole image capturing apparatus 201, and parameters such as the projection method and the angle of view when the coordinate system is also equal to the coordinate system of the entire image capturing apparatus 201 are stored.

  The environment reconstruction data 3680 is data that expresses where and what color of the space area photographed by this system exists. Specifically, each vertex has three-dimensional coordinates. Such data is composed of triangular mesh data and color information of each vertex.

  The method for generating the environment reconstruction data 3680 will be described with reference to FIG. 39, and the contents of the image composition processing after the change will be described with reference to FIGS.

  FIG. 37 is a flowchart of image composition processing in the system of this embodiment. 37, the same reference numerals as those in FIG. 10 represent the same processes as those in FIG.

  The image composition process of this embodiment is obtained by adding an output image data update process 3701 before the composite image data output process 1010 to the image composition process described with reference to FIG. Details of the output image data update processing 3701 will be described with reference to FIG.

  FIG. 38 is a flowchart of output image data update processing in the system of this embodiment.

  When the output image data update process is started in step 3800, the entire depth image data 2361 is read in step 3801, and then the output image data 650 is read in step 3802.

  Next, in Step 3803, environment reconstruction data 3680 is generated based on the entire depth image data 2361 and the output image data 650. Specific processing for generating the environment reconstruction data 3680 will be described with reference to FIG.

  Next, in step 3804, the video generation viewpoint data 3645 is read.

  In step 3805, the output image data 650 is redrawn. The output image data 650 may be generated again from the video generation viewpoint data 3645 and the environment reconstruction data 3680 by the principle of perspective transformation, and can be implemented by a known technique.

  After finishing the above processing, in step 3806, the output image data update processing is finished.

  FIG. 39 is a diagram for explaining a method for generating environment reconstruction data in the system according to the present embodiment.

  Each pixel of the output image data 650 is assigned to each vertex of the environment reconstruction data 3680. Regarding the three-dimensional coordinate value of each vertex, the overall depth is defined with respect to the three-dimensional direction that is defined by the output parameter data 244 and the respective pixel positions of the output image data 650 and that faces the pixel position direction from the projection center position. What is necessary is just to determine based on the depth value stored in the corresponding pixel on the image data 2361. Further, as shown in FIG. 39, with respect to how to connect the vertices for obtaining triangular mesh data, the direction in which the diagonal line is inserted is determined in advance for the unit cell formed by each pixel, and the direction is determined. This can be done by connecting them regularly.

  Specifically, after connecting the vertices in a square lattice shape, the vertex 3900, the vertex 3901, the vertex 3902, and the vertex 3903 are connected to either the vertex 3900 and the vertex 3903 or the vertex 3901 and the vertex 3902. These rules are determined and connected in accordance with the rules. FIG. 39 shows an example in which a vertex 3900 and a vertex 3903 are connected by a diagonal line 3910. For the color information of each vertex, the pixel value of the corresponding pixel in the output image data 650 may be used as it is.

  As described above, according to the present embodiment, although there are differences in visibility and optical characteristics having directivity, the image data captured from the projection center designated by the user is further added to the example described in the second embodiment. It is possible to generate output image data that is close to pseudo.

  Note that the difference between the present embodiment and the publicly known technique is that the image quality is not deteriorated due to different projection centers of a plurality of photographing apparatuses.

  Hereinafter, the fifth embodiment will be described with reference to FIGS. 40 to 44.

  The present embodiment is an example in which the shooting ranges of the left and right photographing apparatuses can be overlapped with the example described in the first embodiment, and a wide area in the output image data is increased. This is an example suitable for connecting and expressing fine image data.

  FIG. 40 is a diagram illustrating the relationship between the imaging range of each imaging apparatus and the output image generation range in the system of the present embodiment.

  In FIG. 40, an imaging range 4000 is the imaging range of the overall imaging device 201, an imaging range 4010 is the imaging range of the left side imaging device 202, and an imaging range 4011 is the imaging range of the right side imaging device 203. An output image generation range 4020 is an output image generation range. FIG. 40 is for explaining the conditions when applying the present embodiment, and the actual shape of each range may be greatly different from that shown in FIG.

  The present embodiment is an example suitable for application to a case where the shooting range 4000 includes an output image generation range 4020. In this example, it is possible to cope with the case where there is a range included in both the shooting range 4010 and the shooting range 4011. However, there is a range included in both the shooting range 4010 and the shooting range 4011. This is not necessarily a necessary condition for the present embodiment.

  FIG. 41 is a diagram for explaining the data format of the pixel value of the composite map image data 616 in the system of the present embodiment.

  In this embodiment, a data format 4120 is used instead of the data format 720. The data format 4120 is composed of whole sampling data 4100, left side sampling data 4101, and right side sampling data 4012.

  The entire sampling data 4100 is composed of a weight value WT, a reference source X coordinate location XT, and a reference source Y coordinate value YT. The weight value WT, the reference source X coordinate value XT, and the reference source Y coordinate value YT are all real values, and the reference source X coordinate location XT and the reference source Y coordinate value YT represent coordinate values on the entire input image data 221. .

  Similarly, the left side sampling data 4101 includes a weight value WL, a reference source X coordinate value XL, and a reference source Y coordinate value YL. The weight value WL, the reference source X coordinate value XL, and the reference source Y coordinate value YL are all real values, and the reference source X coordinate value XL and the reference source Y coordinate value YL represent the coordinate values on the left side input image data 222. .

  Similarly, the right side sampling data 4102 includes a weight value WR, a reference source X coordinate value XR, and a reference source Y coordinate value YR. The weight value WR, the reference source X coordinate value XR, and the reference source Y coordinate value YR are all real values, and the reference source X coordinate value XR and the reference source Y coordinate value YR represent coordinate values on the right side input image data 223. .

  FIG. 42 is a diagram for explaining a weight value calculation method in the system according to the present embodiment.

  A weight calculation method when the reference source coordinate value is the coordinate value 4220 of the input image data 4210 will be described with reference to FIG. First, a calculation reference distance DX in the X direction is calculated. This is the smaller value of the X-direction distance DX1 and the X-direction distance DX2 in FIG. Next, similarly, a calculation reference distance DY in the Y direction is calculated. That is, the smaller one of the Y-direction distance DY1 and the Y-direction distance DY2 in FIG. After calculating the calculation reference distance DX in the X direction and the calculation reference distance DY in the Y direction, the weight value W is determined as the product of DX and DY.

FIG. 43 is a flowchart of adopted pixel determination processing in the system of this embodiment.
When the adopted pixel determination process is started in step 4300, first, in step 4301, the values of WT, WL, and WR are initialized. Specifically, “WT = 1” is set, and the values of WL and WR are described with reference to FIG. 42 based on the reference source X coordinate value and the reference source Y coordinate value stored in the composite map image data 616. Initialize by the method you did.

  Next, in Step 4302, it is determined whether or not the difference between VT and VL is within the allowable range based on a predetermined threshold. If it is determined that the difference is outside the allowable range, the value of WL is set to 0 in Step 4303. To do.

  Next, in step 4304, based on a predetermined threshold value, it is determined whether or not the difference between VT and VR is within the allowable range. If it is determined that the difference is outside the allowable range, the value of WR is set to 0 in step 4305. To do.

  Next, in step 4306, it is determined whether or not “WL + WR> 0”, that is, at least one of the values of WL and WR is not zero. If at least one of the values of WL and WR is not 0, in step 4307, the value of WT is set to 0.

  Subsequently, in step 4308, processing for normalizing the values of WT, WL, and WR is performed. Specifically, the sum of the values of WT, WL, and WR is set to 1 by dividing each of WT, WL, and WR by the sum of their values (WT + WL + WR).

  After the above processing is completed, the adopted pixel determination processing is ended in step 4309.

  FIG. 44 is a flowchart of output image data generation processing in the system of this embodiment.

  When the output image data generation process is started in step 4400, first, in step 4401, n, which is the serial number of the pixel to be processed, is initialized to zero.

  Next, in step 4402, the pixel value of the nth pixel is read from the composite map image data 616.

  Next, in step 4403, the pixel value of the nth pixel of the output image data 650 is determined by taking a weighted sum of the pixel values of each input image data based on the data read in step 4402.

  In step 4404, n, which is the serial number of the pixel to be processed, is incremented by 1. Subsequently, in step 4405, it is determined whether or not the processing has been completed for all the pixels in the output image data 650. If the process has been completed for all the pixels in the output image data 650, the process proceeds to step 4406 to end the output image data generation process. If there is a pixel that has not been processed yet, the process proceeds to step 4402. Proceed and continue processing.

  As described above, according to the present embodiment, the present invention can be applied to the example described in the first embodiment even when the shooting ranges of the left and right imaging apparatuses overlap.

  Hereinafter, the sixth embodiment will be described with reference to FIGS. 45 to 49.

  In this embodiment, in addition to the example described in the fifth embodiment, the conditions imposed on the shooting range of the whole shooting apparatus include only the area where the left shooting apparatus and the right shooting apparatus are shooting in an overlapping manner. This is an example that is relaxed under the condition that it is only necessary to have a wide viewing angle in a remote conference system or the like.

  FIG. 45 is a situation explanatory diagram for explaining a photographing range of each photographing apparatus in the system of the present embodiment. 45, the same reference numerals as those in FIG. 8 represent the same elements as those in FIG.

  Unlike the case of FIG. 8, the present embodiment differs from the case of FIG. 8 in that the angle of view of the whole image taking device 201 is not sufficiently wide, and the left side image taking device 202 is used alone to display the entire area to be displayed. This is an example in which an area that is imaged in the area or an area that is photographed independently by the right-side imaging apparatus 203 is required.

  FIG. 46 is a diagram illustrating the relationship between the imaging range of each imaging apparatus and the output image generation range in the system of the present embodiment.

  In FIG. 46, an imaging range 4600 is an imaging range of the entire imaging device 201, an imaging range 4610 is an imaging range of the left side imaging device 202, and an imaging range 4611 is an imaging range of the right side imaging device 203. An output image generation range 4620 is an output image generation range. FIG. 46 is for explaining the conditions when this embodiment is applied, and the actual shape of each range may be greatly different from that shown in FIG.

  This embodiment is an example suitable for application to a case where a part of the output image generation range 4620 protrudes outside the imaging range 4600. In this example, it is possible to cope with the case where there is a range included in both the shooting range 4610 and the shooting range 4611. However, there is a range included in both the shooting range 4610 and the shooting range 4611. This is not necessarily a necessary condition for the present embodiment.

  Regarding the application of this embodiment, it is a necessary condition that the entire output image generation range 4620 is included in the union of the shooting range 4600, the shooting range 4610, and the shooting range 4611. In addition, when there is a range that is commonly included in the shooting range 4610, the shooting range 4611, and the output image generation range 4620, the shooting range 4600 preferably includes the range.

  FIG. 47 is a diagram for explaining a difference in how inconsistencies occur depending on the subject position in the system of the present embodiment.

  In FIG. 47, a viewpoint position 4700 is a position serving as a projection center when the entire image capturing apparatus 201 generates image data, that is, a position serving as a projection center when generating output image data, and a viewpoint position 4710 is a left-side image capturing apparatus 202. Is a position serving as a projection center when image data is generated, and a viewpoint position 4711 is a position serving as a projection center when the right side photographing apparatus 203 generates image data.

  A subject 4730 and a subject 4731 are subjects on a circle 4720 centered on the viewpoint position 4700, that is, subjects that are equidistant with respect to the viewpoint position 4700. For these two subjects, the difference between the directions seen from the viewpoint position 4710 and the viewpoint position 4711 is an angle 4740 for the subject 4730 and an angle 4741 for the subject 4731. Here, it is clear that the angle 4740 is larger than the angle 4741.

  From this example, it can be said that the inconsistency is less noticeable in the peripheral portion than in the central portion of the photographing region of the photographing system. The present embodiment utilizes this property, generates image data that is consistent with the image data captured by one photographing apparatus for the central portion, and separates the peripheral portion so as to be smoothly connected to the central portion. This is an attempt to connect image data photographed by a photographing apparatus. In the present embodiment, it is essentially impossible to capture image data that matches image data captured by a single imaging device, but an output image that is visually inferior to such image data. Generate data.

  FIG. 48 is a flowchart of adopted pixel determination processing in the system of this embodiment. 48, the same reference numerals as those in FIG. 43 denote the same elements as those in FIG.

  The adopted pixel determination process in FIG. 48 differs from the adopted pixel determination process in FIG. 43 in Step 4801, Step 4802, Step 4803, and Step 4804. Details of these steps will be described below.

  The difference in the initialization process in step 4801 is the WT initialization process. In the initialization process described in step 4301 of FIG. 43, initialization is performed so that “WT = 1”. In step 4801 of the present embodiment, as shown in FIG. Performs initialization depending on. For initialization of WL and WR, the same processing as step 4301 in FIG. 43 is performed.

  Step 4802 is processing corresponding to step 4303 in FIG. In step 4303 of FIG. 43, the value of WL is set to 0. However, in step 4802 of this embodiment, the value of WL is set to W0. Here, W0 is a very small number that does not affect the result when WT is 1. Specifically, when the possible range of the pixel value of the output image data 650 is 0 to 255, for example, 1/512 may be adopted as the value of W0.

  This is based on the fact that the pixel value of the output image data 650 is a discrete value. Such processing is performed because the reference source coordinate value in the entire sampling data 4100 is the entire input image data 221. This is because the output image data 650 is generated with priority given to the pixel values of the left-side input image data 222 or the right-side input image data 223 even when a mismatch occurs as it approaches the end. As a result, the pixel value of the output image data 650 is determined from only the pixel value of the left side input image data 222 or the right side input image data 223 from the region shown in the whole input image data 221 and not in the whole input image data 221. It is necessary to make a smooth connection to the area where it is necessary to determine.

  Similarly, Step 4803 is processing corresponding to Step 4305 in FIG. In step 4305 in FIG. 43, the value of WR is set to 0. However, in step 4803 of this embodiment, the value of WR is set to W0.

  Step 4804 is processing corresponding to step 4306 in FIG. In step 4306 of FIG. 43, it is determined whether or not “WL + WR> 0”, but in step 4804 of this embodiment, it is determined whether or not “WL + WR ≧ 1”. This is because there is consistency between the pixel value of the overall reference image data 631 and the pixel value of the left reference image data 632, or between the pixel value of the overall reference image data 631 and the pixel value of the right reference image data 633. In other cases, the pixel value of the entire input image data 221, the pixel value of the left surface input image data 222, and the pixel value of the right surface input image data 223 are blended in consideration of the weight. This is because the pixel value of the output image data 650 is determined.

  FIG. 49 is a diagram for explaining a method of determining the weight value of the entire sampling data 4100 in the system of the present embodiment. 49, the same reference numerals as those in FIG. 46 denote the same elements as those in FIG.

  The initial value of WT is “WT = 1” at least in a range included in common with the shooting range 4610, the shooting range 4611, and the output image generation range 4620, and from that point toward the boundary of the shooting range 4600, “WT” = 0 ”continuously.

  Specifically, a function 4910 is used as a function for determining the initial value of WT in the cross section 4900 of FIG. The function 4910 is “WT = 1” in the region 4921 including the range included in common with the shooting range 4610, the shooting range 4611, and the output image generation range 4620, and the region 4920 and the region 4022 which are other regions. The continuous function is such that “WT = 0” as it approaches the boundary of the imaging range 4600.

  As described above, according to the present embodiment, in addition to the example described in the fifth embodiment, an area in which the left side photographing device and the right side photographing device are photographing the conditions imposed on the photographing range of the whole photographing device. It can be applied even when relaxed to the condition that it only has to be included.

  Hereinafter, the seventh embodiment will be described with reference to FIGS. 50 to 55.

  The present embodiment is an example in which the focal length and posture of an imaging apparatus other than the whole imaging apparatus can be changed with respect to the example described in the second embodiment. For example, in a remote conference system, when the position of the speaker can be specified by voice recognition or the like, the posture is controlled so that the speaker is reflected in the center, and then the speaker is zoomed in and photographed. This is an example suitable for application to an application that expresses the expression of the speaker in detail.

  FIG. 50 is a system configuration diagram of the system of this embodiment. 50, the same reference numerals as those in FIG. 23 denote the same elements as those in FIG.

  In the present embodiment, the left surface photographing device 5020 includes an attitude control unit 5021 and an angle of view control unit 5022. In addition to the function of the left surface photographing device 202, a photographing posture and a photographing image are controlled by a control signal from the photographing device control device 5010. It has a function to change the corner (or focal length). Similarly, the right side photographing device 5030 includes an attitude control unit 5031 and an angle of view control unit 5032, and in addition to the functions of the left side photographing device 203, the photographing posture and the photographing angle of view are changed by a control signal from the photographing device control device 5010. It has a function to do.

  The photographing device control device 5010 controls the photographing posture and the photographing angle of view of the left surface photographing device 5020 and the right surface photographing device 5030 according to an external control signal, and the photographing posture and photographing of the left surface photographing device 5020 and the right surface photographing device 5030 after the change. It has a function of transmitting information on the angle of view to the image composition device 5000.

  The image composition device 5000 according to the present embodiment adds the overall control unit 5080 to the dynamic internal state flag 281, the left side input state flag 282, the right side input state flag 283, the state mode flag 284, and the default image data 285. The parameter data 5081 and the dynamic external parameter data 5082 can be stored, and the information on the shooting angle of view of the left side imaging device 5020 and the right side imaging device 5030 transmitted from the imaging device control unit 501 is obtained as the dynamic internal parameter data 5081. In addition, it is configured to store the information of the photographing posture in the dynamic external parameter data 5082, respectively.

  Further, the image processing unit 5040 is configured to be able to further process the left side intermediate data update process described with reference to FIG. 53 and the corresponding right side intermediate data update process with respect to the image processing unit 2330. A part of the processing content of the intermediate data update processing 5101 is changed. Specifically, the adopted pixel determination process is changed to that described with reference to FIG. 54, and the output image data update process described with reference to FIG. 55 is executed after the output image data generation process 2007 is completed. I made it.

  FIG. 51 is a flowchart of image composition processing in the system of this embodiment. 51, the same reference numerals as those in FIG. 10 represent the same processes as those in FIG.

  In the intermediate data update process in step 5101, the adopted pixel determination process is changed to that described with reference to FIG. 54 with respect to the intermediate data update process described in FIG. 28, and after the output image data generation process 2007 is completed. , Output image data update processing described with reference to FIG. 55 is executed.

  Details of the photographing parameter change related processing in step 5102 will be described with reference to FIG.

  FIG. 52 is a flowchart of the photographing parameter change related process in the system of the present embodiment.

  When the shooting parameter change-related process is started in step 5200, first, in step 5201, it is determined whether or not the shooting parameter change of the left side shooting device 5020 has been completed. At this time, if the photographing parameter of the left side photographing device 5020 has not changed, it is determined as “No”.

  If it is determined in step 5201 that the change of the imaging parameter of the left side imaging device 5020 has been completed, the left side intermediate data update process is performed in step 5202. Details of the left side intermediate data update processing will be described with reference to FIG.

  Subsequently, in step 5203, it is determined whether or not the change of the shooting parameter of the right surface shooting device 5030 is completed. At this time, if the photographing parameter of the right side photographing device 5030 has not changed, it is determined as “No”.

  If it is determined in step 5203 that the change of the shooting parameters of the right side photographing device 5030 has been completed, right side intermediate data update processing is performed in step 5204. The right side intermediate data update process can be performed by the same process as the left side intermediate data update process in step 5202.

  After finishing the above processing, in step 5205, the photographing parameter change related processing is finished.

  FIG. 53 is a flowchart of the left side intermediate data update process in the system of the present embodiment.

  When the left side intermediate data update process is started in step 5300, first, in step 5301, the internal parameter data 241 and the external parameter data 242 are read.

  Next, in step 5302, dynamic internal parameter data 5081 and dynamic external parameter data 5082 are read.

  Next, in step 5303, a remappable area extraction process is performed. This is a process of extracting a region that is commonly photographed before and after the photographing parameter change on the coordinates of the output image data 650. The internal parameter data 241, the external parameter data 242, the output parameter data 244, and the dynamic internal It is calculated based on the parameter data 5081 and the dynamic external parameter data 5082.

  Next, in step 5304, intermediate data remapping processing is performed on the pixels in the remappable area. This uses internal parameter data 241, external parameter data 242, output parameter data 244, dynamic internal parameter data 5081, and dynamic external parameter data 5082, and depth image data 2360 based on the value before the change of the shooting parameter. This is a process for updating the map image data 610 and the reference image data 630.

  Next, in step 5305, the internal parameter data 241 is updated with the dynamic internal parameter data 5081. Subsequently, in step 5306, the external parameter data 242 is updated with the dynamic external parameter data 5082.

  Next, in step 5307, intermediate data generation processing is performed for pixels existing in an area other than the remappable area. This is the same process as the intermediate data generation process described with reference to FIG. 24 except that the pixels to be processed are limited.

  After finishing the above processing, the left side intermediate data update processing is ended in step 5308.

FIG. 54 is a flowchart of adopted pixel determination processing in the system of this embodiment.
When the adopted pixel determination process is started in step 5400, first, in step 5401, it is determined whether or not the shooting parameter of the left surface shooting device 5020 is being changed. If it is determined that the imaging parameter of the left side imaging device 5020 is not being changed, it is determined in step 5402 whether or not the difference between VT and VL is within the allowable range, and it is determined that the imaging is within the allowable range. In step 5403, “left side input image data” is set in the input data identifier 721 as the pixel value of the nth pixel in the composite map image data 616, and the left side map image is set in the reference source X coordinate value 722. The reference source X coordinate value 711 that is the pixel value of the nth pixel of the data 612 is set, and the reference source Y coordinate value that is the pixel value of the nth pixel of the left map image data 612 is set as the reference source Y coordinate value 723. After setting 712, the process proceeds to step 5404 to end the adopted pixel determination process.

  If it is determined in step 5401 that the imaging parameter of the left side imaging device 5020 is being changed, or if it is determined in step 5402 that the difference between VT and VL is not within the allowable range, the process proceeds to step 5405.

  In step 5405, it is determined whether or not the shooting parameter of the right side shooting device 5030 is being changed. If it is determined that the shooting parameter of the right-side imaging device 5030 is not being changed, it is determined in step 5406 whether or not the difference between VT and VR is within the allowable range. In step 5407, “right side input image data” is set as the input data identifier 721 as the pixel value of the n th pixel of the composite map image data 616, and the right side map image is set as the reference source X coordinate value 722. The reference source X coordinate value 711 that is the pixel value of the nth pixel of the data 613 is set, and the reference source Y coordinate value that is the pixel value of the nth pixel of the right surface map image data 613 is set as the reference source Y coordinate value 723. After setting 712, the process proceeds to step 5404 to end the adopted pixel determination process.

  If it is determined in step 5405 that the imaging parameter of the right side imaging device 5030 is being changed, or if it is determined in step 5406 that the difference between VT and VR is not within the allowable range, the process proceeds to step 5408. As the pixel value of the nth pixel of the composite map image data 616, “entire input image data” is set in the input data identifier 721, and the nth pixel of the entire map image data 611 is set in the reference source X coordinate value 722. A reference source X coordinate value 711 that is a pixel value is set, and a reference source Y coordinate value 712 that is a pixel value of the nth pixel of the entire map image data 611 is set as the reference source Y coordinate value 723, and then step 5404. Then, the adopted pixel determination process is terminated.

  FIG. 55 is a flowchart of output image data update processing in the system of this embodiment. In the output image data update processing, when the shooting parameters of the left side photographing device 202 and the right side photographing device 203 are changed, for example, it is possible to know how much of the portion of the output image data 650 is zoomed up. It is a process to make.

  When the output image data update process is started in step 5500, first, in step 5501, the shooting area extraction data during state change is read. Specifically, the state-changing shooting area extraction data includes the entire depth image data 2361, internal parameter data 241, external parameter data 242, output parameter data 244, dynamic internal parameter data 5081, and dynamic external parameter data 5082. It is.

  Next, in step 5502, output image data 650 is read.

Next, in step 5503, the boundary line of the imaging region during the state change is drawn in the output image data 650 and stored again as output image data 650.
After finishing the above processing, in step 5504, the output image data update processing is finished.

  As described above, according to the present embodiment, the present invention can also be applied to cases where the focal length and posture of an imaging apparatus other than the whole imaging apparatus can be changed.

  Hereinafter, the eighth embodiment will be described with reference to FIGS. 56 to 59.

  The present embodiment is an example in which, in addition to the example described in the fifth embodiment, an area with less movement among the portions corresponding to the background area in the video can be displayed with high definition.

  FIG. 56 is a system configuration diagram of the system of this embodiment. 56, the same reference numerals as those in FIG. 2 denote the same elements as those in FIG. 56, the same reference numerals as those in FIG. 6 denote the same elements as those in FIG.

  In the image composition device 5600 of the present embodiment, first, an image processing unit 5630 in which the processing contents of the adopted pixel determination processing and the output image data generation processing are changed from the image composition device of the fifth embodiment is provided. The processing contents of the adopted pixel determination process after the change will be described with reference to FIG. 58, and the processing contents of the output image data generation process after the change will be described with reference to FIG.

  In the image composition device 5600 of this embodiment, the intermediate data storage unit 5660 further includes the pixel value of the composite map image data in addition to the converted image data 620, the reference image data 630, the changed image data 640, and the output image data 650. The map image data 5610 and the high-definition background image data 5690 obtained by further changing the data format from the format described in FIG. 41 can be stored. The newly changed pixel value format of the composite map image data will be described with reference to FIG.

  The high-definition background image data 5690 is taken using the whole photographing apparatus 201 so that the image data is higher quality than the whole input image data 221.

  In the case where the entire photographing apparatus 201 does not have a special function, for example, photographing may be performed a plurality of times, and the averaged image data may be generated and stored as high-definition background image data 5690. In this case, a high-quality image with less noise than the entire input image data 221 can be obtained due to the effect of time averaging.

  In addition, when the entire input image data 221 has a function of switching between the moving image mode and the still image mode, in general, it is structured such that high quality image data can be captured when the image is captured in the still image mode. Therefore, the image data captured in the still image mode may be stored as the high-definition background image data 5690 as it is.

  In addition, when the focal length control or the shooting posture control can be performed on the whole photographing apparatus 201, high-definition background image data 5690 is generated and stored by image processing using a technique called super-resolution. May be.

  FIG. 57 is a diagram for explaining the data format of the pixel value of the composite map image data 616 in the system of the present embodiment. 57, the same reference numerals as those in FIG. 41 denote the same elements as those in FIG.

  In this embodiment, instead of the data format 4120, a data format 5720 that is a format obtained by adding the high-definition background sampling data 5700 to the data format 4120 is used. The high-definition background sampling data 5700 includes a weight value WH, an X-coordinate value XH of the high-definition background image data, and a Y-coordinate value YH of the high-definition background image data. The weight value WH, the X coordinate value XH, and the Y coordinate value YH are all real values.

  Here, the X coordinate value XH and the Y coordinate value YH are the pixel values of all the pixels of the composite map image data when it is assumed that the subject is stationary over the entire imaging region of the imaging system of the present embodiment. In FIG. 59, the output image data generation process described in FIG. 59 is executed with “WT = 1, WL = WR = WH = 0”, and the pixel values of all the pixels of the composite map image data are “WH = 1, WT = The generation method of the high-definition background image data 5690 and the output parameter data 244 are set so that the output image data substantially matches the output image data generation processing described in FIG. 59 when “WL = WR = 0”. Based on the above, the value is calculated and stored in advance.

FIG. 58 is a flowchart of adopted pixel determination processing in the system of this embodiment.
When the adopted pixel determination process is started in step 5800, first, in step 5801, the values of WT, WL, WR, and WH are initialized. Specifically, WT, WL, and WR are initialized with the same values as in step 4301 in FIG. 43, and WH is set to 1.

  Next, in the VL determination processing in step 5802, based on a predetermined threshold value, it is determined whether or not the difference between the values of VT and VL is within the allowable range. If the difference is outside the allowable range, the value of WL is set to 0. I do.

  Subsequently, in the VR determination process in step 5803, it is determined whether or not the difference between the values of VT and VR is within the allowable range based on a predetermined threshold value. If the difference is outside the allowable range, the WR value is set to 0. I do.

  Subsequently, in Step 5804, it is determined whether or not “WL + WR> 0”, that is, at least one of the values of WL and WR is not 0. If at least one of the values of WL and WR is not 0, the value of WT and WH is set to 0 in step 5805, and then the process proceeds to step 5809. If both the values of WL and WR are 0, they are left as they are. Proceed to step 5806.

  In step 5806, based on a predetermined threshold, it is determined whether or not the difference between the values of VT and VH is within an allowable range. Here, VH is a value calculated from the high-definition background sampling data 5700 and the high-definition background image data 5690 at the pixel value of the nth pixel of the composite map image data 616. If the determination result is within the allowable range, the process proceeds to step 5807 to set the WT value to 0, and if it is outside the allowable range, the process proceeds to step 5808 to set the WH value to 0.

  Subsequently, in step 5809, processing for normalizing the values of WT, WL, WR, and WH is performed. Specifically, the sum of the values of WT, WL, WR, and WH is set to 1 by dividing each of WT, WL, WR, and WH by the sum of their values (WT + WL + WR + WH).

  After the above processing ends, the adopted pixel determination processing ends at step 5810.

  FIG. 59 is a flowchart of output image data generation processing in the system of this embodiment.

  When the output image data generation process is started in step 5900, first, in step 5901, n, which is the serial number of the pixel to be processed, is initialized to 0.

  Next, in step 5902, the pixel value of the nth pixel is read from the composite map image data 616.

  Next, in step 5903, based on the data read in step 5902, the output image data 650 is obtained by taking a weighted sum of the pixel values of each input image data including the pixel values of the high-definition background image data 5690. The pixel value of the nth pixel is determined.

  In step 5904, n, which is the serial number of the pixel to be processed, is incremented by 1. Subsequently, in step 5905, it is determined whether or not processing has been completed for all the pixels in the output image data 650. If all pixels in the output image data 650 have been processed, the process proceeds to step 5906 to end the output image data generation process. If there are pixels that have not been processed yet, the process proceeds to step 5902. Proceed and continue processing.

  As described above, according to the present embodiment, in addition to the example described in the fifth embodiment, it is possible to display a region with less movement among portions corresponding to the background region in the video with high definition. Further, since the high-definition background image data 5690 is data that can be generated in advance, the quality of the output image data can be improved without increasing the network load in an application such as a remote conference system. .

  In the above-described embodiment, the high-definition background image data 5690 is taken using the whole photographing apparatus 201. However, the pixels of the high-definition background image data 5690 are consistent with the whole reference image data 631. Since the determination is performed, the embodiment is not limited to this. It may be generated by performing geometric transformation processing in consideration of internal parameter data and external parameter data on the image data photographed using the left surface photographing device 202 and the right surface photographing device 203, or another newly provided photographing device It may be generated by applying the same geometric transformation process to the image data photographed using.

  In the above-described examples from Example 1 to Example 8, the embodiment has been described by way of an example in which one overall photographing apparatus and two other photographing apparatuses are provided. However, the embodiment is not limited thereto. The present invention is not limited, and it is obvious that the present invention can be applied to a general configuration including a plurality of other photographing apparatuses in addition to one whole photographing apparatus.

It is a conceptual diagram for demonstrating an effect. 1 is a system configuration diagram of a system of Example 1. FIG. It is explanatory drawing of the data stored in an input data storage part in the system of Example 1. FIG. It is explanatory drawing of the data stored in a whole control part in the system of Example 1. FIG. It is explanatory drawing of the data stored in a parameter memory | storage part in the system of Example 1. FIG. It is explanatory drawing of the data stored in an intermediate data storage part in the system of Example 1. FIG. It is a figure explaining the data format of the pixel value of the various map image data in the system of Example 1. FIG. FIG. 6 is a situation explanatory diagram for explaining the operation of image composition processing in the system according to the first embodiment. FIG. 3 is a conceptual diagram for explaining an operation of image composition processing in the system of Embodiment 1. 3 is a flowchart of image composition processing in the system according to the first exemplary embodiment. 3 is a flowchart of intermediate data generation processing in the system according to the first embodiment. 3 is a flowchart of map image data initialization processing in the system of Embodiment 1; 3 is a flowchart of overall conversion image data generation processing in the system of Embodiment 1; It is a flowchart of the left surface map image data generation process in the system of Example 1. It is explanatory drawing of the synthetic | combination method of the map image data in the system of Example 1. FIG. 6 is a flowchart of overall reference image data generation processing in the system of Embodiment 1; 6 is a flowchart of a composite map image data generation process in the system according to the first embodiment. 3 is a flowchart of adopted pixel determination processing in the system of Embodiment 1; 3 is a flowchart of output image data generation processing in the system of Embodiment 1. 3 is a flowchart of intermediate data update processing in the system according to the first embodiment. 4 is a flowchart of overall change image data generation processing in the system of Embodiment 1; It is a flowchart of the left surface map image data update process in the system of Example 1. FIG. 6 is a system configuration diagram of a system according to a second embodiment. 10 is a flowchart of intermediate data generation processing in the system according to the second embodiment. 10 is a flowchart of depth image data generation processing in the system according to the second embodiment. It is a figure for demonstrating the structure method of the regulation search depth set data in the system of Example 2. FIG. It is a flowchart of the left surface map image data generation process in the system of Example 2. 10 is a flowchart of intermediate data update processing in the system according to the second embodiment. 10 is a flowchart of an entire depth image data update process in the system according to the second embodiment. 10 is a flowchart of pixel value priority depth determination processing in the system according to the second embodiment. It is a figure for demonstrating the construction method of the temporary search distance set data and the maximum search distance set data in the system of Example 2. 10 is a flowchart of a background depth image data update process in the system according to the second embodiment. It is a flowchart of the left surface map image data update process in the system of Example 2. 10 is a flowchart of an entire depth image data update process in the system according to the third embodiment. 12 is a flowchart of flow priority depth determination processing in the system according to the third embodiment. FIG. 10 is a system configuration diagram of a system according to a fourth embodiment. 10 is a flowchart of image composition processing in the system of Embodiment 4. 14 is a flowchart of output image data update processing in the system of Embodiment 4. FIG. 10 is a diagram for explaining a method for generating environment reconstruction data in the system according to the fourth embodiment. FIG. 10 is a diagram illustrating a relationship between a shooting range of each shooting apparatus and a generation range of an output image in the system according to the fifth embodiment. It is a figure explaining the data format of the pixel value of the composite map image data in the system of Example 5. FIG. 10 is a diagram for explaining a weight value calculation method in the system according to the fifth embodiment. 14 is a flowchart of adopted pixel determination processing in the system of Example 5. 10 is a flowchart of output image data generation processing in the system of Embodiment 5. FIG. 10 is a situation explanatory diagram for explaining a photographing range of each photographing apparatus in the system of Example 6; FIG. 10 is a diagram illustrating a relationship between a shooting range of each shooting apparatus and a generation range of an output image in the system according to the sixth embodiment. FIG. 10 is a diagram for explaining a difference in how inconsistency occurs depending on a subject position in the system according to the sixth embodiment. 18 is a flowchart of adopted pixel determination processing in the system of Example 6. FIG. 10 is a diagram for explaining a method of determining weight values of entire sampling data in the system of the sixth embodiment. FIG. 10 is a system configuration diagram of a system according to a seventh embodiment. 18 is a flowchart of image composition processing in the system according to the seventh embodiment. 18 is a flowchart of imaging parameter change related processing in the system according to the seventh embodiment. It is a flowchart of the left surface intermediate data update process in the system of Example 7. 18 is a flowchart of adopted pixel determination processing in the system of Example 7. 18 is a flowchart of output image data update processing in the system of Embodiment 7. FIG. 10 is a system configuration diagram of a system according to an eighth embodiment. FIG. 20 is a diagram illustrating a data format of pixel values of composite map image data in the system of Example 8. 18 is a flowchart of adopted pixel determination processing in the system of Example 8. 16 is a flowchart of output image data generation processing in the system of Example 8. It is a hardware block diagram.

Explanation of symbols

110, 120, 130 ... input image data, 111 ... output image data, 910 ... total input image data at time (T-1), 920 ... right input image at time (T-1) Data, 911 ... Overall input image data at time T, 912 ... Overall reference image data before update, 913 ... Overall converted image data, 914 ... Overall changed image data, 915 ... After update ,...,... Right side input image data at time T, 922... Right side reference image data before update, 923... Right side converted image data, 924. .. right-side reference image data after update, 2620, 2621, 3121 ... depth set data, 4000, 4010, 4011 ... shooting range, 4020 Output range 4600, 4610, 4611 ... Shooting range, 4612 ... Overlapping shooting range, 4620 ... Output range, 4700, 4710, 4711 ... Projection center position, 4730, 4731 ... Subject position , 4910... Function for determining the initial value of WT

Claims (7)

An image synthesis device that synthesizes first input image data and second input image data and outputs output image data of a predetermined resolution,
A first geometric transformation parameter defining which position in the first input image data each pixel position of the output image data corresponds to; and each pixel position of the output image data is the second input image. A second geometric transformation parameter defining which position in the data corresponds to; a geometric transformation parameter storage unit for storing;
A first image conversion unit that converts the first input image data based on the first geometric transformation parameter to generate first output candidate image data;
A second image conversion unit that converts the second input image data based on the second geometric conversion parameter to generate second output candidate image data;
The first output candidate image data and the second output candidate image data are compared, and whether or not there is consistency is determined by determining whether or not the difference between the two is within a predetermined allowable range for each pixel. A consistency determination unit to perform,
For each pixel of the output image data, the pixel value of the pixel determined to be inconsistent by the consistency determination unit is determined based on the pixel value of the first output candidate image data, and the consistency determination An output image data generation unit that generates the output image data by determining the pixel value of the pixel determined to be consistent by the unit based on the pixel value of the second output candidate image data;
An image composition device comprising: An image synthesis device that synthesizes first input image data, second input image data, and third input image data and outputs output image data of a predetermined resolution,
A first geometric transformation parameter defining which position in the first input image data each pixel position of the output image data corresponds to; and each pixel position of the output image data is the second input image. A second geometric transformation parameter defining which position in the data corresponds to, and a third geometric transformation defining which position in the third input image data each pixel position of the output image data corresponds to A parameter storage unit for storing geometric transformation parameters;
A first image conversion unit that converts the first input image data based on the first geometric transformation parameter to generate first output candidate image data;
A second image conversion unit that converts the second input image data based on the second geometric conversion parameter to generate second output candidate image data;
A third image conversion unit that converts the third input image data based on the third geometric conversion parameter to generate third output candidate image data;
A first consistency determination unit that compares the first output candidate image data with the second output candidate image data and determines whether or not there is consistency between the two for each pixel;
A second consistency determination unit that compares the first output candidate image data and the third output candidate image data and determines whether or not there is consistency between the two for each pixel;
In order to determine the pixel value of the pixel determined to be consistent by both the first consistency determination unit and the second consistency determination unit, the pixel value of the second output candidate image data A contribution rate determination unit that determines a contribution rate of pixel values of the third output candidate image data;
For each pixel of the output image data
The pixel values of the pixels determined to be inconsistent by the first consistency determining unit and determined to be inconsistent by the second consistency determining unit are the values of the first output candidate image data. Based on the pixel value,
The pixel values of the pixels that are determined to be consistent by the first consistency determination unit and are determined to be non-consistent by the second consistency determination unit are included in the second output candidate image data. Based on the pixel value,
The pixel values of the pixels determined to be non-consistent by the first consistency determining unit and determined to be consistent by the second consistency determining unit are included in the third output candidate image data. Based on the pixel value,
The pixel value of the pixel determined to be consistent by the first consistency determination unit and determined to be consistent by the second consistency determination unit is determined by the contribution rate determination unit. By determining based on the pixel value of the second output candidate image data and the pixel value of the third output candidate image data based on the contribution rate,
An output image data generation unit for generating output image data;
An image composition device comprising: The image composition device according to claim 1,
further,
A high-quality image data storage unit that stores high-quality image data; and high-quality consistency that compares the first output candidate image data with the high-quality image data and determines whether or not there is consistency between the two for each pixel. A determination unit,
The output image data generation unit matches each pixel of the output image data with respect to the pixel value of the pixel determined to be inconsistent by the consistency determination unit, by the high quality consistency determination unit. If the pixel is determined to be compatible, it is determined based on the pixel value of the high quality image data, and if the pixel is determined to be non-consistent by the high quality consistency determination unit The pixel value of the pixel determined based on the pixel value of the first output candidate image data and determined to be consistent by the consistency determination unit is based on the pixel value of the second output candidate image data. An image synthesizing apparatus characterized in that output image data is generated by determination. An imaging system comprising: a first imaging device; a second imaging device that performs imaging so as to have an overlapping range with an imaging range of the first imaging device; and the image composition device according to claim 1,
The image composition device includes:
Image data captured by the first image capturing device is input as the first input image data, and image data captured by the second image capturing device is input as the second input image data. An imaging system, wherein image data having a predetermined resolution is output. The imaging system according to claim 4 ,
The imaging system, wherein the second imaging device is configured to take an image of the inside of the imaging range of the first imaging device. An imaging system comprising a first imaging device, a second imaging device, a third imaging device, and the image composition device according to claim 2,
The photographing range of the second photographing device and the photographing range of the third photographing device are configured to overlap each other, and the photographing range of the first photographing device is the second photographing device and the second photographing device. 3 is configured so as to include a range where the photographing device overlaps with the photographing device,
The image composition device includes:
Image data captured by the first image capturing device is input as the first input image data, image data captured by the second image capturing device is input as the second input image data, and the third input image data is input. An image pickup system configured to output image data having a predetermined resolution by inputting image data taken by an image pickup apparatus as the third input image data. A first photographing apparatus having a high quality photographing mode that prioritizes the image quality of the photographed image data and a normal photographing mode that prioritizes other characteristics of the photographed image data, and overlaps with the photographing range of the first photographing apparatus. An imaging system comprising a second imaging device for imaging such a range, and the image composition device according to claim 3 ,
further,
A high-quality image initial setting unit that stores image data captured by setting the first imaging device in the high-quality imaging mode as the high-quality image data;
The image composition device includes:
Image data captured by setting the first imaging device to the normal imaging mode is input as the first input image data, and image data captured by the second imaging device is input to the second input image data. By configuring to enter as
An imaging system, wherein image data having a predetermined resolution is output.

Images