JP2019032600A - Three-dimensional image generation device, three-dimensional image generation method, and three-dimensional image generation program - Google Patents

Abstract

To superpose far-infrared images on respective visible images in position.SOLUTION: A first image acquisition unit acquires a visible image group composed of a plurality of visible images of a subject captured by a visible camera and a second image acquisition unit acquires a far-infrared image group composed of a plurality of far-infrared images of the subject captured by a far-infrared camera. A first reconstitution unit reconstitutes a three-dimensional model from the visible image group to obtain a visible distance image and a second reconstitution unit reconstitutes a three-dimensional model from the far-infrared image group to obtain a far-infrared distance image. A storage device preliminarily stores a between-camera position attitude relationship between the visible camera and the far-infrared camera. A one-dimensional search unit compares the visible distance image with the far-infrared distance image by one-dimensional search to determine scale information representing a correspondence between a size of the between-camera position attitude relationship and a size of a three-dimensional structure. A superposition unit superposes, on the visible images, the respective far-infrared images using the between-camera position attitude relationship on the basis of the determined scale information to obtain a three-dimensional image of the subject.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a three-dimensional image generation apparatus, a three-dimensional image generation method, and a three-dimensional image generation program.

  A temperature image showing temperature information that cannot be captured by a visible camera can be captured by a temperature camera. In this specification, the “temperature camera” refers to an invisible camera (far infrared camera) capable of capturing an infrared image (far infrared image) having a wavelength in the range of 8 μm to 1000 μm. Infrared rays in this wavelength range include thermal infrared rays and far infrared rays. Therefore, by using a temperature camera (invisible camera; far-infrared camera), it is possible to obtain an abnormality such as a plant or facility from a temperature image (far-infrared image).

  However, there is a problem in that it is difficult to understand the geometric shape only by the temperature image (far infrared image). Therefore, studies have been made to compensate for this drawback by using a visible camera and a temperature camera (far infrared camera) in combination.

  For example, Patent Document 1 discloses a three-dimensional temperature image conversion device that converts a two-dimensional temperature image of an object into a three-dimensional three-dimensional image. The temperature image three-dimensional device disclosed in Patent Document 1 is a device that three-dimensionalizes a temperature image of an object based on the three-dimensional information and temperature image information of the object. The three-dimensional apparatus includes thermography (temperature image acquisition means), three-dimensional information acquisition means, and coordinate correspondence means. The thermography (temperature image acquisition means) acquires temperature image information based on heat generated from the object and the reference object. The three-dimensional information acquisition unit acquires three-dimensional information of the target object. The coordinate association means associates specific temperature image coordinates in the temperature image information obtained by thermography with a specific actual image in the three-dimensional information obtained by the three-dimensional information acquisition means. On the monitor, the temperature image and three-dimensional image of the object can be displayed individually or simultaneously on the same screen. The three-dimensional device includes a reference object that generates a heat source (eg, a plurality of miniature bulbs) having known coordinates as a reference point in order to determine a reference point of the object.

  In Patent Document 1, the three-dimensional device acquires temperature image information (information having temperature information and coordinates) of an object and a miniature light bulb at the same time, and acquires three-dimensional information of the object. The three-dimensional apparatus determines the conversion formula from the actual coordinates to the temperature image coordinates by calibrating the actual coordinates of the miniature light bulb and the temperature image coordinates. Then, the three-dimensional device converts all the real images of the three-dimensional information into temperature image coordinates based on the determined conversion formula, extracts the temperature information based on the converted temperature image information, and converts the temperature image into 3D. In Patent Document 1, the three-dimensional information acquisition unit images a transmitter that forms a magnetic vector in a predetermined area, a light projecting unit that irradiates light on an object, and light irradiated on the object surface by the light projecting unit. And a three-dimensional magnetic sensor for detecting the three-dimensional position and orientation of the three-dimensional device.

  Patent Document 2 discloses a method and apparatus for visualizing the periphery of a vehicle by fusing an infrared image and a visible image. The device disclosed in Patent Document 2 for visualizing the periphery of a vehicle in the dark includes one visible camera that provides a visible image including the surrounding digital data and the surrounding digital data. And a camera device having one infrared camera for providing an infrared image. A visible visible object is displayed on the visible image, and an infrared image generated by the visually visible object and / or other object is displayed on the infrared image. The image processing unit performs image fusion between the visible image and the infrared image. The image display unit displays the peripheral image generated by the image processing unit.

  In Patent Document 2, the image processing unit includes a first normalization device, a second normalization device, an alignment device, and an overlay / fusion device. The distortion or optical aberration of the visible image is removed by the first normalization device. The distortion or optical aberration of the infrared image is removed by the second normalization device. The normalized images are aligned with each other in the registration process using the registration parameters by the alignment apparatus. That is, both images after normalization are aligned with each other so that the same object appears in the fused image at substantially the same point and substantially the same size. The images aligned with each other are superimposed or fused by digital data processing in a superposition / fusion apparatus.

  Patent Document 3 discloses a portable imaging device that is excellent in portability and can easily detect the presence or absence of a defect such as a crack in a structure or a cavity. The imaging device disclosed in Patent Literature 3 includes a digital camera that captures visible image data of a structure that is an imaging target and an infrared camera that captures thermal image data of the structure inside the casing. The digital camera and the infrared camera are adjusted so that their optical axes are parallel to each other. This makes it possible to adjust the center of the image data captured by each of the digital camera and the infrared camera to indicate a point on the subject that is separated by the same distance as the distance between the digital camera and the infrared camera. Taking into account the ratio of field size calculated based on the angle of view of each optical system for digital cameras and infrared cameras, the angle of view of the image with the smaller angle of view from the vicinity of the center of the image with the larger angle of view. Are moved and overlapped by the same distance as the distance between the digital camera and the infrared camera. This makes it possible to superimpose visible image data and thermal image data regardless of the distance to the subject, and highly accurate thermal-visible fusion image data obtained by superimposing and fusing thermal image data and visible image data. Can be generated.

  In Patent Document 3, when visible image data and thermal image data are supplied, the image processing unit corrects the visible image data and the thermal image data to an equal scale, and the corrected isometric image data is converted to SDRAM (Synchronous Dynamic Data). Random Access Memory). In the imaging apparatus, lens aberration correction processing and parallax correction processing are executed by the image processing unit. When the camera lens angle of view of the digital camera and infrared camera is different, the distance between the center of the optical system of each of the digital camera and the infrared camera, and the distance to the structure acquired using the autofocus function by the digital camera Can be used to correct parallax due to a difference in viewpoint between visible image data and thermal image data. When such a parallax correction process is performed, thermo-visible fusion image data is generated by superimposing and fusing the visible image data and the thermal image data having the same field of view and angle of view.

  Patent Document 4 discloses a distance image generation method and apparatus for generating a distance image using a stereo matching method or the like. The “distance image” is an image in which the distance to each object point of the subject is the pixel value of each pixel. In Patent Document 4, the camera body is provided with an imaging unit (main imaging unit) PA and an imaging unit (sub-imaging unit) PB. The imaging unit PA captures an image of visible light. On the other hand, the imaging unit PB is used to generate a distance image, and captures an image of visible light from a different viewing direction than the imaging unit PA. Accordingly, the two images captured by the imaging unit PA and the imaging unit PB are used as a stereo image for generating a distance image by the stereo matching method.

  In Patent Document 4, an infrared light irradiation unit that emits infrared light is provided above and below the photographing lens LB of the imaging unit PB. An image formed by infrared light (light including infrared light) irradiated and reflected by the subject from these infrared light irradiation units is captured by the imaging unit PB. The image (infrared image) imaged by the infrared light is used to generate a distance image by the TOF (Time of Flight) method using the infrared light. The image signal indicating the infrared image is output from the image sensor, converted into a digital signal by the A / D converter, and then taken into the TOF distance image generation unit. In the TOF distance image generation unit, a distance image with the distance to each object point of the subject as the pixel value of each pixel is generated by the TOF method.

  Patent Document 5 discloses a calibration board for calibrating a plurality of different types of cameras.

Japanese Patent No. 5522630 Special table 2008-530667 gazette JP 2005-338359 A JP 2008-116308 A International Publication No. 2017/056473

  However, the above-described prior art documents have the following problems.

  In Patent Document 1, the three-dimensional information acquisition unit is realized as a large number of physical hardware including a transmitter, a light projecting unit, an imaging unit, a three-dimensional magnetic sensor, and the like. Therefore, there is a problem that the apparatus becomes large and it is difficult to reduce the size.

  Patent Document 2 merely discloses a technique for fusing a visible image and an infrared image to generate a two-dimensional fused image. That is, in Patent Document 2, a three-dimensional fusion image cannot be obtained.

  Patent Document 3 simply discloses a technique for generating two-dimensional thermo-visible fusion image data by superimposing and fusing visible image data and thermal image data. Therefore, even in Patent Document 3, three-dimensional thermo-visible fusion image data cannot be obtained.

  Patent Document 4 merely discloses a method for generating a distance image using a stereo matching method or a TOF method.

  Patent Document 5 merely discloses a calibration board that calibrates a plurality of different types of cameras.

  An object of the present invention is to provide a 3D image generation apparatus, a 3D image generation method, and a 3D image generation program that can solve the above-described problems.

  A three-dimensional image generation apparatus according to an aspect of the present invention includes a visible camera that can capture a visible image, and a far-infrared camera that can capture a far-infrared image in a wavelength range of 8 μm to 1000 μm. A three-dimensional image generation device for generating a three-dimensional image of the subject by moving an imaging device having a fixed relative position with respect to the far-infrared camera relative to the subject, and imaging with the visible camera A first image acquisition unit configured to acquire a visible image group including a plurality of visible images of the subject; and far infrared including a plurality of far infrared images of the subject captured by the far infrared camera. Second image acquisition means for acquiring an image group; first reconstruction means for reconstructing a three-dimensional model from the visible image group to obtain a visible distance image; and a three-dimensional model from the far-infrared image group Reconstruct the far-infrared distance image Second reconstructing means for obtaining an image; storage means for preliminarily storing a position and orientation relationship between the visible camera and the far-infrared camera; and primary processing of the visible distance image and the far-infrared distance image One-dimensional search means for determining scale information representing a correspondence relationship between the size of the position and orientation relationship between the cameras and the size of the three-dimensional structure in comparison with the original search; based on the determined scale information, Superimposing means for superimposing corresponding far-infrared images on each visible image using a position and orientation relationship between cameras to obtain a three-dimensional image of the subject.

  A three-dimensional image generation method according to another aspect of the present invention includes a visible camera that can capture a visible image, and a far-infrared camera that can capture a far-infrared image in a wavelength range of 8 μm to 1000 μm. Three-dimensional image generation in a three-dimensional image generation device that generates a three-dimensional image of the subject by continuously moving an imaging device having a fixed relative position to the far-infrared camera with respect to the subject A first image acquisition step in which a first image acquisition unit acquires a visible image group made up of a plurality of visible images of the subject imaged by the visible camera; and a second image acquisition unit A second image acquisition step of acquiring a far-infrared image group composed of a plurality of far-infrared images of the subject imaged by the far-infrared camera; and a first reconstruction unit includes the visible image Reconstruct 3D model from group A first reconstruction step for obtaining a viewing distance image; a second reconstruction step for obtaining a far-infrared distance image by a second reconstruction unit reconstructing a three-dimensional model from the far-infrared image group; Storing a positional orientation relationship between the visible camera and the far-infrared camera in a storage device in advance; and a one-dimensional search unit firstly processing the visible distance image and the far-infrared distance image. A one-dimensional search step for determining scale information representing a correspondence relationship between the size of the position and orientation relationship between the cameras and the size of the three-dimensional structure in comparison with an original search; And a superimposing step of superimposing corresponding far-infrared images on each visible image using a position and orientation relationship between the cameras to obtain a three-dimensional image of the subject.

  A three-dimensional image generation program according to another aspect of the present invention includes a visible camera that can capture a visible image, and a far-infrared camera that can capture a far-infrared image in a wavelength range of 8 μm to 1000 μm. A tertiary that causes the computer of the three-dimensional image generation device to generate a three-dimensional image of the subject by continuously moving an imaging device having a fixed relative position with the far-infrared camera relative to the subject. A first image acquisition procedure, which is an original image generation program, for acquiring a visible image group made up of a plurality of visible images of the subject captured by the visible camera; A second image acquisition procedure for acquiring a far-infrared image group composed of a plurality of far-infrared images of a subject; and a first reconstruction for reconstructing a three-dimensional model from the visible image group to obtain a visible distance image Procedure and A second reconstruction procedure for reconstructing a three-dimensional model from the far-infrared image group to obtain a far-infrared distance image; and a positional posture relationship between the visible camera and the far-infrared camera. A procedure for storing in advance in a storage device; the visible distance image and the far-infrared distance image are compared by a one-dimensional search, and the correspondence relationship between the size of the position and orientation relationship between the cameras and the size of the three-dimensional structure A one-dimensional search procedure for determining scale information representing the image; superimposing each corresponding far-infrared image on each visible image using a position and orientation relationship between the cameras based on the determined scale information And a superimposing procedure for obtaining a three-dimensional image of the subject.

  According to the present invention, each far-infrared image can be superimposed on each corresponding visible image at a correct position.

1 is a block diagram illustrating a schematic configuration of a three-dimensional image generation system including a three-dimensional image generation apparatus according to an embodiment of the present invention. It is a block diagram which shows the internal structure of the three-dimensional image generation apparatus shown by FIG. It is a block diagram for demonstrating the method of calculating the position-and-orientation relationship between cameras used for the three-dimensional image generation system shown by FIG. It is a flowchart for demonstrating operation | movement of the one-dimensional search part used for the three-dimensional image generation system shown by FIG. 2 is a flowchart for explaining the overall operation of the three-dimensional image generation system shown in FIG. 1. It is a figure which shows the example of an image for demonstrating the whole operation | movement of the three-dimensional image generation system shown in FIG.

  Next, embodiments for carrying out the invention will be described in detail with reference to the drawings.

[Description of configuration]
FIG. 1 is a block diagram showing a schematic configuration of a 3D image generation system 10 including a 3D image generation apparatus 100 according to an embodiment of the present invention.

  Referring to FIG. 1, a 3D image generation system 10 includes an imaging device including a visible camera 12 and a far infrared camera 14, a 3D image generation device 100 according to an embodiment of the present invention, and a monitor 16. .

  The visible camera 12 is a camera that can capture a visible image. The far infrared camera 14 is a camera that can capture a far infrared image (temperature image) in a wavelength range of 8 μm to 1000 μm. The imaging device is a device in which the relative position between the visible camera 12 and the far infrared camera 14 is fixed. That is, the imaging apparatus is a handheld device in which the visible camera 12 and the far-infrared camera 14 are fixed in parallel on the same axis (not shown). The imaging apparatus can photograph the subject (environment) 15 by moving itself.

  The visible camera 12 is also called an RGB (red, green, blue) camera, and the far-infrared camera 14 is also called a temperature camera.

  The three-dimensional image generation apparatus 100 generates a three-dimensional image of the subject 15 using such an imaging apparatus, as will be described later. The generated three-dimensional image of the subject 15 is also called an RGB-T (temperature) three-dimensional image.

  The monitor 16 is a device that displays an RGB-T three-dimensional image. The monitor 16 includes a display device such as an LCD (Liquid Crystal Display) or a PDP (Plasma Display Panel).

  The three-dimensional image generation apparatus 100 can be realized by a personal computer (PC) that operates by program control.

  FIG. 2 is a block diagram illustrating an internal configuration of the 3D image generation apparatus 100. The three-dimensional image generation apparatus 100 includes a storage device 102 that stores programs and data to be described later, and a data processing device 104 that processes data.

  The storage device 102 includes a memory such as a hard disk, a read only memory (ROM), and a random access memory (RAM). The storage device 102 has a function of storing processing information (described later) and programs 201 necessary for various processing in the data processing device 104.

  The data processing device 104 includes a microprocessor such as an MPU (micro processing unit) or a central processing unit (CPU). The data processing device 104 has a function of realizing various processing units that read the program 201 from the storage device 102 and process data according to the program 201.

  The main processing units realized by the data processing apparatus 104 are a first image acquisition unit 301, a second image acquisition unit 302, a first reconstruction unit 303, a second reconstruction unit 304, and a one-dimensional search unit. 305 and a superimposing unit 306.

  The first image acquisition unit 301 acquires a visible image group 202 composed of a plurality of visible images of the subject 15 captured by the visible camera 12. The acquired visible image group 202 is stored in the storage device 102.

  Similarly, the second image acquisition unit 302 acquires a far infrared image group 203 including a plurality of far infrared images of the subject 15 captured by the far infrared camera 14. The acquired far-infrared image group 203 is stored in the storage device 102.

  The first reconstruction unit 303 reconstructs a three-dimensional model from the visible image group 202 by DSO (Direct Sparse Odometry) to obtain a visible distance image 204. The obtained visible distance image 204 is stored in the storage device 102. Here, DSO is a visual mileage measurement method based on a novel high-precision sparse direct configuration motion equation, and is a well-known technique in this technical field.

  Similarly, the second reconstruction unit 304 reconstructs a three-dimensional model from the far-infrared image group 203 by DSO, and obtains a far-infrared distance image 205. The obtained far-infrared distance image 205 is stored in the storage device 102.

  The storage device 102 stores in advance a positional orientation relationship 206 between the visible camera 12 and the far-infrared camera 14. A method of generating the position / orientation relationship 206 between cameras will be described later.

  The one-dimensional search unit 305 compares the visible distance image 204 and the far-infrared distance image 205 by a one-dimensional search, and calculates the size of the position / posture relationship 206 between the cameras and the size of the three-dimensional structure, as will be described later. Scale information 207 representing the correspondence is determined. The determined scale information 207 is stored in the storage device 102.

  Based on the determined scale information 207, the superimposing unit 306 superimposes each corresponding far-infrared image on each visible image using the positional orientation relationship 206 between the cameras, and the three-dimensional image 208 of the subject 15. Get. The obtained three-dimensional image 208 is stored in the storage device 102.

  The feature of this embodiment is that the scale information 207 is determined. More specifically, it is assumed that a three-dimensional image is to be generated without using hardware as disclosed in Patent Document 1. In this case, a three-dimensional image is generated without considering the movement amount of the imaging device (in other words, without acquiring three-dimensional information). As a result, in the obtained three-dimensional image, the scale information becomes indefinite, the far-infrared image (temperature image) captured by the far-infrared camera 14 is aligned, and the visible image captured by the visible camera 12 is displayed. Cannot be superimposed on the image.

  Therefore, in the present embodiment, the one-dimensional search unit 305 determines the scale information 207 by comparing the visible distance image 204 and the far-infrared distance image 205 by a one-dimensional search. As a result, the superimposing unit 306 superimposes each corresponding far-infrared image on each visible image using the position / orientation relationship 206 between the cameras based on the determined scale information 207, so that the tertiary of the subject 15 is obtained. The original image 208 can be obtained.

  With reference to FIG. 3, a method of calculating the position / orientation relationship 206 between the cameras will be described. In this method, the calibration board 17 is used. As this calibration board 17, the calibration board currently disclosed by patent document 5 mentioned above can be used, for example. Specifically, the calibration board 17 provides a temperature difference by, for example, placing a heat-insulating material on a hot carpet to create a temperature-prone portion and a hard-to-rise portion. It is a board that can be calibrated even from the camera 14.

  First, the first image acquisition unit 301 acquires a visible image 202-1 obtained by capturing the calibration board 17 with the visible camera 12, and stores it in the storage device 102. Similarly, the second image acquisition unit 302 acquires a far-infrared image 203-1 obtained by capturing the calibration board 17 with the far-infrared camera 14, and stores it in the storage device 102.

  The data processing device 104 includes a parameter calculation unit 307 and an attitude relationship calculation unit 308.

  The parameter calculation unit 307 calculates the internal parameters of the visible camera 12 and the far infrared camera 14 and the external parameters between the cameras using the visible image 202-1 and the far infrared image 203-1. The posture relationship calculation unit 308 calculates a position / posture relationship 206 between the cameras using the calculated internal parameter and the calculated external parameter. The calculated position / posture relationship 206 between the cameras is stored in the storage device 102.

  Next, a method for calculating the scale information 207 will be described with reference to FIG. 4 in addition to FIG. FIG. 4 is a flowchart for explaining the operation of the one-dimensional search unit 305.

  First, referring to FIG. 2, the first reconstruction unit 303 generates a visible distance image 204 from the position and orientation of the visible camera 12 and the visible image group 202 using a multi-baseline stereo method. The generated visible distance image 204 is stored in the storage device 102.

  Similarly, the second reconstruction unit 304 generates a far-infrared distance image 205 from the position and orientation of the far-infrared camera 14 and the far-infrared image group 203 using a multi-baseline stereo method. The generated far-infrared distance image 205 is stored in the storage device 102.

  The “multi-baseline stereo method” is a well-known method in this technical field, and thus the description thereof is omitted.

  4, first, the one-dimensional search unit 305 converts the visible distance image 204 into a distance image on the image coordinates of the far infrared camera 14 (step S101).

  Next, the one-dimensional search unit 305 obtains a score by comparing the converted distance image and the far-infrared distance image 205 using the mutual information amount (step S102).

  Finally, the one-dimensional search unit 305 determines the scale information 207 that gives the highest score (step S103). The determined scale information 207 is stored in the storage device 102.

[Description of operation]
Next, the overall operation of the 3D image generation system 10 according to the present embodiment will be described in detail with reference to FIGS.

  First, the subject 15 is imaged by the visible camera 12 and the far-infrared camera 14 while moving the imaging device (steps S201A and S201B).

  Thereby, the first image acquisition unit 301 acquires a visible image group 202 composed of a plurality of visible images of the subject 15 captured by the visible camera 12 (step S202A), and the second image acquisition unit 302 A far-infrared image group 203 composed of a plurality of 15 far-infrared images of the subject imaged by the far-infrared camera 14 is acquired (step S202B). The acquired visible image group 202 and far-infrared image group 203 are stored in the storage device 102.

  Subsequently, the first reconstruction unit 303 reconstructs a three-dimensional model from the visible image group 202 by DSO to obtain a visible distance image 204 (step S203A). Similarly, the second reconstruction unit 304 reconstructs a three-dimensional model from the far-infrared image group 203 by DSO, and obtains a far-infrared distance image 205 (step S203B). The obtained visible distance image 204 and far infrared distance image 205 are stored in the storage device 102.

  Next, the one-dimensional search unit 305 determines the scale information 207 by comparing the visible distance image 204 and the far-infrared image 205 by a one-dimensional search (step S204). The determined scale information 207 is stored in the storage device 102.

  Based on the determined scale information 207, the superimposing unit 306 superimposes each corresponding far-infrared image on each visible image using the position / orientation relationship 206 between the cameras, and performs RGB-T tertiary of the subject 15. An original image 208 is obtained (step S205).

  The RGB-T three-dimensional image 208 obtained in this way is output to the monitor 16 and displayed on the screen of the monitor 16 (step S206).

[Description of effects]
Next, the effect of this embodiment will be described.

  By using the embodiment of the present invention, each far-infrared image can be superimposed on each corresponding visible image at a correct position. This is because the scale information of the three-dimensional structure is determined by comparing the distance images by a one-dimensional search.

  In addition, what is necessary is just to implement | achieve each part of a three-dimensional image generation apparatus using the combination of hardware and software. In a form in which hardware and software are combined, a three-dimensional image generation program is expanded in a random access memory (RAM), and hardware such as a control unit (CPU (central processing unit)) is based on the three-dimensional image generation program. Each part is realized as various means. Further, the three-dimensional image generation program may be recorded and distributed on a recording medium. The image composition program recorded on the recording medium is read into the memory via the wired, wireless, or recording medium itself, and operates the control unit and the like. Examples of the recording medium include an optical disk, a magnetic disk, a semiconductor memory device, and a hard disk.

  To describe the above-described embodiment in another expression, a computer that operates as a three-dimensional image generation apparatus is based on a three-dimensional image generation program developed in a RAM. The first image acquisition unit 301 and the second image acquisition It can be realized by operating as the unit 302, the first reconstruction unit 303, the second reconstruction unit 304, the one-dimensional search unit 305, and the superimposition unit 306.

  Moreover, the imaging device including the visible camera 12 and the far-infrared camera 14 and the three-dimensional image generation device 100 may be configured integrally or may be configured separately. When configured separately, the imaging device temporarily records a visible image group and a far infrared image group captured by the visible camera 12 and the far infrared camera 14 on a small memory card such as a memory stick, A signal indicating the visible image group and the far-infrared image group may be wirelessly transmitted from the transmission unit to the outside. In this case, the 3D image generation apparatus 100 includes a reading unit that reads information recorded on a small memory card and a receiving unit that receives information transmitted from a transmission unit.

  It should be noted that the specific configuration of the present invention is not limited to the above-described embodiment, and changes within a range not departing from the gist of the present invention are included in the present invention.

  Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

  INDUSTRIAL APPLICABILITY The present invention can be used for detection of water leaks in distribution pipes, prevention of secondary disasters at fire sites, and detection of damage to large-scale plants.

  Specifically, in the detection of water leaks in the water distribution pipes, it is assumed that there are tears in the water pipes in the walls of the old house and water leaks have occurred. In this case, the heating speed varies depending on the amount of moisture contained in the wall. For this reason, even when water does not soak to the surface of the outer wall, the possibility of water leakage can be visualized by using the three-dimensional image generation system according to the present invention. As a result, it is possible to detect an abnormality in the internal water distribution pipe without breaking the outer wall.

  Further, in a fire scene, it is possible to visualize how hot a door knob made of metal is by using the three-dimensional image generation system according to the present invention. Therefore, even if the firefighter does not actually open the door, the situation inside can be predicted. As a result, secondary disasters such as backfire that may occur when the firefighter opens the door can be prevented.

  Furthermore, in the large-scale plant, by using the three-dimensional image generation system according to the present invention, it is possible to visualize a temperature distribution that is clearly different from the normal temperature state due to the slight damage of the facility outer wall. As a result, it is possible to make a repair plan before an accident due to an abnormality occurs.

10 3D image generation system 12 Visible camera (RGB camera)
14 Far-infrared camera (temperature camera)
DESCRIPTION OF SYMBOLS 15 Subject 16 Monitor 17 Calibration board 100 Three-dimensional image generation apparatus 102 Storage apparatus 104 Data processing apparatus 201 Program 202 Visible image group 202-1 Visible image 203 Far infrared image group 203-1 Far infrared image 204 Visible distance image 205 Far-infrared distance image 206 Position and orientation relationship between cameras 207 Scale information 208 Three-dimensional image (RGB-T three-dimensional image)
301 first image acquisition unit 302 second image acquisition unit 303 first reconstruction unit 304 second reconstruction unit 305 one-dimensional search unit 306 superimposition unit 307 parameter calculation unit 308 posture relation calculation unit

Claims (12)

A visible camera capable of capturing a visible image and a far infrared camera capable of capturing a far-infrared image in a wavelength range of 8 μm to 1000 μm, and a relative position between the visible camera and the far-infrared camera is fixed A three-dimensional image generation device that continuously moves an imaging device with respect to a subject to generate a three-dimensional image of the subject,
A first image acquisition unit that acquires a visible image group composed of a plurality of visible images of the subject imaged by the visible camera;
A second image acquisition unit that acquires a far-infrared image group composed of a plurality of far-infrared images of the subject imaged by the far-infrared camera;
A first reconstruction unit that reconstructs a three-dimensional model from the visible image group to obtain a visible distance image;
A second reconstruction unit for reconstructing a three-dimensional model from the far-infrared image group to obtain a far-infrared distance image;
A storage device that stores in advance a positional orientation relationship between the visible camera and the far-infrared camera;
A one-dimensional search that compares the visible distance image and the far-infrared distance image by a one-dimensional search, and determines scale information representing a correspondence relationship between the size of the position and orientation relationship between the cameras and the size of the three-dimensional structure. And
Based on the determined scale information, using the position and orientation relationship between the cameras, superimposing each corresponding far infrared image on each visible image, to obtain a three-dimensional image of the subject;
A three-dimensional image generation apparatus comprising:   The position and orientation relationship between the cameras is obtained by using internal images of the visible camera and the far-infrared camera and external parameters between the cameras using an image obtained by capturing the calibration board with the visible camera and the far-infrared camera. The three-dimensional image generation apparatus according to claim 1, wherein the three-dimensional image generation device is calculated in advance using the calculated internal parameter and the calculated external parameter. The first reconstruction unit generates the visible distance image from a position and orientation of the visible camera and the visible image group using a multi-baseline stereo method,
The second reconstruction unit generates the far-infrared distance image from the position and orientation of the far-infrared camera and the far-infrared image group using the multi-baseline stereo method.
The three-dimensional image generation apparatus according to claim 1 or 2.   The one-dimensional search unit converts the visible distance image into a distance image on the image coordinates of the far infrared camera, and compares the converted distance image and the far infrared distance image using mutual information. The three-dimensional image generation apparatus according to any one of claims 1 to 3, wherein a score is obtained by determining the scale information that provides the highest score. A visible camera capable of capturing a visible image and a far infrared camera capable of capturing a far-infrared image in a wavelength range of 8 μm to 1000 μm, and a relative position between the visible camera and the far-infrared camera is fixed A three-dimensional image generation method in a three-dimensional image generation apparatus for continuously moving an imaging device with respect to a subject to generate a three-dimensional image of the subject,
A first image acquisition step in which a first image acquisition unit acquires a visible image group composed of a plurality of visible images of the subject imaged by the visible camera;
A second image acquisition step of acquiring a far-infrared image group composed of a plurality of far-infrared images of the subject imaged by the far-infrared camera;
A first reconstruction unit that reconstructs a three-dimensional model from the visible image group to obtain a visible distance image; and
A second reconstruction step of reconstructing a three-dimensional model from the far-infrared image group to obtain a far-infrared distance image;
Storing in advance a positional orientation relationship between the visible camera and the far-infrared camera in a storage device;
A one-dimensional search unit compares the visible distance image and the far-infrared distance image by a one-dimensional search, and represents scale information representing the correspondence between the size of the position and orientation relationship between the cameras and the size of the three-dimensional structure A one-dimensional search process for determining
A superimposing unit superimposes each corresponding far-infrared image on each visible image using the position and orientation relationship between the cameras based on the determined scale to obtain a three-dimensional image of the subject. Process,
A three-dimensional image generation method including:   The position and orientation relationship between the cameras is obtained by using internal images of the visible camera and the far-infrared camera and external parameters between the cameras using an image obtained by capturing the calibration board with the visible camera and the far-infrared camera. The three-dimensional image generation method according to claim 5, wherein the three-dimensional image is calculated in advance using the calculated internal parameter and the calculated external parameter. In the first reconstruction step, the first reconstruction unit generates the visible distance image from the position and orientation of the visible camera and the visible image group using a multi-baseline stereo method,
In the second reconstruction step, the second reconstruction unit uses the multi-baseline stereo method to calculate the far-infrared distance from the position and orientation of the far-infrared camera and the far-infrared image group. Generate images,
The three-dimensional image generation method according to claim 5 or 6.   In the one-dimensional search step, the one-dimensional search unit converts the visible distance image into a distance image on the image coordinates of the far-infrared camera, and converts the converted distance image and the far-infrared distance image to each other. The three-dimensional image generation method according to any one of claims 5 to 7, wherein a score is obtained by comparison using an amount of information, and the scale information that determines the highest score is determined. A visible camera capable of capturing a visible image and a far infrared camera capable of capturing a far-infrared image in a wavelength range of 8 μm to 1000 μm, and a relative position between the visible camera and the far-infrared camera is fixed A three-dimensional image generation program for causing a computer of a three-dimensional image generation device to generate a three-dimensional image of the subject by continuously moving the imaging device being moved with respect to the subject,
A first image acquisition procedure for acquiring a visible image group composed of a plurality of visible images of the subject imaged by the visible camera;
A second image acquisition procedure for acquiring a far-infrared image group composed of a plurality of far-infrared images of the subject imaged by the far-infrared camera;
A first reconstruction procedure for reconstructing a three-dimensional model from the visible image group to obtain a visible distance image;
A second reconstruction procedure for reconstructing a three-dimensional model from the far-infrared image group to obtain a far-infrared distance image;
A procedure for storing a positional orientation relationship between the visible camera and the far-infrared camera in a storage device in advance,
A one-dimensional search that compares the visible distance image and the far-infrared distance image by a one-dimensional search, and determines scale information representing a correspondence relationship between the size of the position and orientation relationship between the cameras and the size of the three-dimensional structure. Procedure and
Based on the determined scale information, the position and orientation relationship between the cameras is used to superimpose each corresponding far-infrared image on each visible image to obtain a three-dimensional image of the subject;
A three-dimensional image generation program for causing the computer to execute.   The position and orientation relationship between the cameras is obtained by using internal images of the visible camera and the far-infrared camera and external parameters between the cameras using an image obtained by capturing the calibration board with the visible camera and the far-infrared camera. The three-dimensional image generation program according to claim 9, which is calculated in advance using the calculated internal parameter and the calculated external parameter. The first reconstruction procedure causes the computer to generate the visible distance image from the position and orientation of the visible camera and the visible image group using a multi-baseline stereo method,
The second reconstruction procedure causes the computer to generate the far-infrared distance image from the position and orientation of the far-infrared camera and the far-infrared image group using the multi-baseline stereo method.
The three-dimensional image generation program according to claim 9 or 10.   In the one-dimensional search procedure, the visible distance image is converted into a distance image on the image coordinates of the far-infrared camera, and a mutual information amount is converted between the converted distance image and the far-infrared distance image. The three-dimensional image generation program according to any one of claims 9 to 11, wherein the three-dimensional image generation program is configured to obtain a score by using and comparing and to determine the scale information with the highest score.

Images