JP5691630B2 - Image composition method - Google Patents

Description

  The present invention relates to an image composition method for compositing a photographed real image and a virtual image prepared in advance.

  As an image composition method for compositing a captured real image and an additional image such as a virtual image prepared in advance, for example, the subject to be photographed has a marker, and the virtual image is placed on the actual surface where the marker is arranged. An image composition method is known in which a composite image is generated by combining so as to be substantially in the same plane (see, for example, Patent Document 1).

JP 2010-49690 A

  In the above conventional image composition method, the virtual image is arranged on the real surface so as to be on the same plane, but the real image and the virtual image are to be synthesized by matching the position in the same plane more accurately. There is. For this reason, when arranging the data of each pixel of a virtual image prepared in advance at an accurate position on a captured real image, there is a problem that work is complicated and a great deal of time and cost must be spent.

  The present invention has been made in view of such problems, and an object of the present invention is to provide an image composition method capable of accurately and easily arranging and compositing a virtual image with a captured real image. There is to do.

In order to achieve such an object, the image composition method of the present invention shoots the first area by including a first position information subject whose position can be specified within the shooting range when shooting the first area. A photographing step of generating image data of the first region image;
A storage step of storing the first region image and the first position specified by the first position information subject in association with each other;
A second position specified by the second position information subject in image data of a second region image obtained by photographing the second area by including the second position information subject in the second area including the first area; An image combining step of combining the image data of the first region image by positioning based on the first position;
Have
During shooting of the first region and shooting of the second region, the distance between the shooting viewpoint and the first position information subject and the second position information subject is measured and the shooting direction is specified,
Based on the identified distance from the viewpoint, the shooting direction, and the marker position information of the first position information subject and the second position information subject, the first region to be combined with the image data of the second region image An image composition method characterized by processing image data of a region image and combining it with the first region image .

  According to such an image composition method, the first area image whose position is specified by the first position information subject and the second area where the position is specified by the second position information subject and the second area including the first area are photographed. Since the two-region image is positioned and synthesized based on the second position and the first position, the first region image can be synthesized at a more accurate position of the second region image. At this time, the first region image includes the first position information subject, and the second region image includes the second position information subject. Therefore, the first region image and the second region image are captured. It is possible to easily specify the positions of the first area image and the second area image, and to associate the first area image and the second area image with each other. For this reason, it is possible to arrange | position and synthesize | combine a 1st area image on a 2nd area image more correctly and easily.

The image data of the first region image to be combined with the image data of the second area image was measured at the time of photographing the photographing and the second region of the first region, viewpoint and first position information object and the second position of the photographing Since processing is performed based on the distance to the information subject, the shooting direction, and the marker position information of the first position information subject and the second position information subject, the second region image and the first region image are used when shooting. Even if the distance and the shooting direction are different from each other, it is possible to process the first area image in correspondence with the second area image to obtain an image shot from the same distance and the same direction. Such a composite image can be generated.

  In this image composition method, the first region image is based on marker position information associated with the first position information subject, a distance from the viewpoint measured at the time of photographing, and a direction of the photographing. The object to be photographed is preferably stored in association with a three-dimensional model obtained by modeling a three-dimensional model.

  According to such an image composition device, the first region image is based on the marker position information associated with the first position information subject, the distance from the viewpoint measured at the time of photographing, and the direction of photographing. Since it is stored in association with the model, the two-dimensional first area image that has been photographed is associated with a three-dimensional model in which the subject to be photographed is three-dimensionally modeled. Information can be added and stored.

  ADVANTAGE OF THE INVENTION According to this invention, it is possible to provide the image composition method which can arrange | position easily and synthesize | combine a virtual image correctly with the image | photographed real image.

It is a block diagram which shows an example of the apparatus for implement | achieving the image composition method which concerns on this invention. It is a flowchart for demonstrating the process at the time of construction of the image composition method of this embodiment. It is a flowchart for demonstrating the process after completion of the image composition method of this embodiment. It is a figure which shows the real image at the time of construction, and its imaging | photography range. It is a figure which shows the image which matches image data of the real image at the time of construction, the positional information on A marker, and imaging | photography information with 3D model. FIG. 6A is a diagram showing a photographing range of a real image after completion, and FIG. 6B is a diagram for explaining a display when an area is designated by the user. (C) is a figure which shows the synthesized image displayed on the display part. It is a figure which shows the virtual image which match | combined the imaging direction of the real image before construction with the imaging direction after completion.

  The image composition method according to the present embodiment is used, for example, when photographing a wall of a concrete structure after completion and displaying the state of bar arrangement during construction of the photographed wall as a composite image.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram showing an example of an apparatus for realizing an image composition method according to the present invention.

  An image composition apparatus 1 for realizing the image composition method of the present embodiment includes an image composition unit 10 that photographs a subject, and a process that combines a real image of the photographed subject and a pre-formed virtual image. A control unit 12 that controls the apparatus 1, a storage unit 14 that stores various information and data necessary for combining images, a display unit 16 that displays the combined image, and the present image combining apparatus 1. And an operation unit 18 for operation.

  The image composition device 1 of the present embodiment has, for example, an imaging unit 10 and a display unit 16 that displays a captured image, and is portable, for example, a digital camera, a digital video camera, a mobile phone, a personal digital assistant (PDA). ), A program capable of executing image composition processing is stored in a mobile computer or the like.

  Then, when the user of the image composition device 1 photographs the subject, the captured actual image is displayed on the display unit 16. At this time, a marker 21 as an image-corresponding subject in which predetermined composite information such as a QR code (registered trademark) or a bar code set in advance is associated with the photographed real image 20 (FIG. 4). Is included, information is acquired by detecting the marker 21, and based on the acquired information, the synthesized real image 20 and a pre-stored virtual image 23 (FIG. 6) are combined. An image 24 (FIG. 6) is generated and the composite image 24 is displayed on the display unit 16.

  The imaging unit 10 converts optical information into electrical information by an imaging element such as a CCD sensor that generates image data of a plurality of pixels constituting an image by photographing, such as a digital camera or a digital video camera. It is configured as follows.

  The display unit 16 is, for example, a liquid crystal panel 16a that can display image data of a plurality of pixels constituting an image in correspondence with each element, and is transparently provided on the surface of the liquid crystal panel 16a. And a touch panel 18a having optical properties. Here, the image data of each pixel is, for example, data indicating each component obtained by decomposing each pixel into an R (red) component, a G (green) component, and a B (blue) component.

  The operation unit 18 is a part where a user who uses the image composition device 1 performs an operation such as photographing or changing settings on the image composition device 1, and photographs at least the actual image 20. So-called shutters are provided. Here, the touch panel 18 a is also included in the operation unit 18.

  The storage unit 14 stores various types of information for generating a composite image in the image composition apparatus 1, and also stores an information storage area 14a in which information is stored and an image in the process of generating the composite image. It has an image virtual development area 14b which is a work area in which data is rewritten.

  In the information storage area 14a, information for specifying the virtual image 23 associated with the position information, image data of the virtual image 23, position information indicating a position of the virtual image 23 to be arranged with respect to the actual image 20, and the like. It is remembered. The information storage area 14a also stores calculated information such as information indicating the distance from the viewpoint of the imaging unit 10 to the subject measured at the time of shooting. The information storage area 14a also stores 3D model data as a three-dimensional model obtained by three-dimensionally modeling a concrete structure as a subject.

  The image virtual development area 14b is a storage area in which photographed image data is stored for each pixel. As described above, since the image data of each pixel has data indicating three components of R, G, and B, the image virtual development area 14b has a storage area that is three times the number of pixels constituting the composite image. ing.

  The image data stored in the image virtual development area 14b is rewritten with the corresponding pixel image data as necessary when generating a composite image. The three data stored for each pixel in the image virtual development area 14b are rewritten respectively for the R component, the G component, and the B component when the image data of each pixel is rewritten. At this time, in the image virtual development area 14b, the storage area does not need to be arranged in the same manner as the elements of the display unit 16 or the imaging unit 10, and the image data of each storage area is the element of the display unit 16 or the imaging unit 10. As long as they are stored in association with each other.

  The control unit 12 controls the imaging unit 10 and the display unit 16 by operating the operation unit 18 by the user, or detects the marker 21 from the image data of the captured real image 20. Various processes such as an analysis process for analyzing the displayed information and an image synthesis process for synthesizing the actual image 20 and the virtual image 23 stored in the storage unit 14 are executed based on the analyzed information.

Next, an image composition method by the control unit 12 will be described.
The image composition method according to the present embodiment is provided in the image composition apparatus 1, and a process is started by operating a shutter which is an operation unit 18 for photographing with the imaging unit 10. At this time, the operation unit 18 may be provided with a switch for switching processing between when the image composition device 1 simply captures a real image and when a composite image is generated, such as a digital camera. Further, it is set to always execute the process of generating the composite image, and the process of generating the composite image may be executed only when the marker 21 is detected in the captured real image.

  In the present embodiment, as described above, when shooting is performed by the imaging unit 10 of the image composition device 1, the position information in which position information such as a QR code (registered trademark) or a barcode is associated with the shooting range. A composite image 24 is generated based on information associated with the marker 21 as a subject.

  In the present embodiment, description will be made using two markers 21 in which different positional information is associated with each other. In order to distinguish between the two markers 21, one marker 21 will be described as an A marker 21a, and the other marker 21 as a B marker 21b. Here, in the A marker 21a and the B marker 21b, information indicating the actual size of the A marker 21a and the B marker 21b and the x coordinate, the y coordinate, and the z coordinate of the provided position are provided as marker position information. It is associated.

  FIG. 2 is a flowchart for explaining processing at the time of construction of the image composition method of the present embodiment. FIG. 3 is a flowchart for explaining processing after completion of the image composition method of the present embodiment. FIG. 4 is a diagram showing a construction actual image and its photographing range. FIG. 5 is a diagram showing an image in which image data of a construction actual image, position information of the A marker, and shooting information are associated with the 3D model.

  In this embodiment, first, for example, a state before the concrete on the wall 30 is placed is photographed when the concrete structure is constructed. At this time, photographing is performed so that the A marker 21a as the first position information subject is included in the photographing range to be photographed. The position on the earth where the A marker 21a is provided or the position on the concrete structure is associated with the A marker 21a as marker position information indicated by the x coordinate, the y coordinate, and the z coordinate. In this A marker 21a, the associated x coordinate, y coordinate, and z coordinate correspond to the first position. Based on the marker position information of the x coordinate, y coordinate, and z coordinate, the GPS, position sensor, etc. The position is accurately measured and arranged. Here, the region including the reinforcing bar 31 before the concrete is placed at the time of construction and the region including the A marker 21a corresponds to the first region.

  When the shutter is operated, the control unit 12 captures a first area as a subject in the image capturing unit 10 and generates image data of each pixel constituting the actual construction image 20a as the captured first region image. Then, the generated image data of the actual construction image 20a is stored in the information storage area 14a of the storage unit 14 (storage step S2). Here, it is assumed that the image is captured in the imaging range F1 as shown in FIG. As shown in FIG. 4, reinforcing bars 31 are arranged vertically and horizontally at the site where the wall 30 is formed.

  At this time, the control unit 12 uses, for example, the upper half of the lens and the lower half of the lens based on the information acquired from the imaging unit 10 at the time of capturing a real image, using the principle of an autofocus mechanism mounted on a general camera. The distance between the subject and the viewpoint position of the imaging unit 10 is measured using the difference in the center of gravity position that occurs in half (S3). Further, the photographing direction by the image composition device 1 is specified using a digital compass or the like (S3). The measured distance and the specified shooting direction are stored in the information storage area 14a of the storage unit 14 as shooting position information (S4).

  Thus, marker position information indicating the x-coordinate, y-coordinate, and z-coordinate of the marker 21 each time the image data of the construction actual image 20a photographed and generated at a desired position of the concrete structure is photographed, and photographing. Along with the shooting position information indicating the distance and direction of the subject at the time, the information is stored in association with the 3D model 33 of the concrete structure stored in the information storage area 14a as shown in FIG. 5 (S5). Here, the subject to be imaged at the time of constructing the concrete structure is not limited to the one showing the bar arrangement state, and may be a pipe state, a wiring state, or the like.

  FIG. 6A is a diagram showing a photographing range of a real image after completion, and FIG. 6B is a diagram for explaining a display when an area is designated by the user. (C) is a figure which shows the synthesized image displayed on the display part. FIG. 7 is a diagram illustrating a virtual image in which the shooting direction of the actual image before construction is matched with the shooting direction after completion.

  Next, as shown in FIG. 6, for example, after the completion of the concrete structure, a B marker 21 b as a second position information subject is pasted on the wall 30 where the bar arrangement state is photographed together with the A marker 21 a at the time of construction. Photographing is performed so that the marker 21b is included in the photographing range. The position on the earth where the B marker 21b is provided or the position on the concrete structure is associated with the B marker 21b as marker position information indicated by the x coordinate, the y coordinate, and the z coordinate. In this B marker 21b, the x coordinate, y coordinate, and z coordinate associated with each other correspond to the second position. Based on the marker position information of the x coordinate, the y coordinate, and the z coordinate, the B marker 21b is accurately detected by a GPS, a position sensor, or the like. Measure and place in Here, the area including the wall 30 and the B marker 21b after the concrete is placed after completion corresponds to the second area.

  When the shutter is operated, the control unit 12 captures the subject with the imaging unit 10, generates image data of each pixel constituting the captured actual image 20b (S11), and generates the generated actual image after completion. The image data 20b is stored in the image virtual development area 14b of the storage unit 14 (S12). Here, it is assumed that the image is captured in the imaging range F2 as shown in FIG. As shown in FIG. 6A, the wall 30 to which the B marker 21b is attached is taken from an angle different from the actual image 20a at the time of construction, for example, from the front.

  At this time, the control unit 12 acquires the marker position information associated with the B marker 21b, measures the distance between the wall 30 and the viewpoint position of the imaging unit 10, and uses a digital compass or the like to measure the image. The shooting direction by the synthesizer 1 is specified, and the specified shooting direction and the measured distance are stored in the information storage area 14a of the storage unit 14 as shooting position information (S13).

  Next, the control unit 12 sets the position included in the shooting direction on the 3D model 33 of the concrete structure stored in the information storage area 14a based on the shooting position and shooting direction at the time of shooting after completion. The presence / absence of the associated construction actual image 20a is determined (S14). At this time, the wall 30 to which the B marker 21b was pasted in the actual image 20b after completion was taken with the A marker 21a because it is a wall 30 in which the reinforcing bar provided with the A marker 21a was embedded at the time of construction. The actual image 20a at the time of construction is detected. At this time, when the construction actual image 20a associated with the position included in the photographing direction is not stored in the information storage area 14a, the processing is terminated, and the actual image after completion is displayed on the display unit 16. Yes.

  When the construction actual image 20a associated with the A marker 21a is detected, the control unit 12 completes the construction on the display unit 16 based on the image data of the completed actual image 20b stored in the image virtual development area 14b. While displaying the after actual image 20b, the message which prompts to input the area | region which wants to see the state of the bar arrangement at the time of construction from a touch panel is displayed (S15).

  In accordance with this message, the user traces from the touch panel so as to draw a frame 32 in the area where the wall 30 is displayed, for example, as shown in FIG. At this time, a frame traced by the user is drawn on the image of the display unit 16.

  Next, the marker position information and the shooting position information stored in association with the construction actual image 20a photographed together with the A marker 21a are acquired from the information storage area 14a (S16), and the image data of the construction actual image 20a is acquired. And the inclination of the wall 30 with respect to the shooting direction is calculated (S17). For example, as shown in FIG. 2, the actual construction image 20 a is displayed on the display unit 16 so that the length indicating the height on the right end side of the wall 30 is shorter than the length indicating the height on the left end side of the wall 30. ing. For this reason, the length of the right end side of the A marker 21a, the length of the left end side of the A marker 21a, and the width of the A marker 21a are measured from the image data of the actual image 20a during construction. The inclination of the wall 30 with respect to the shooting direction is calculated from the measured value and the actual size of the A marker 21a stored in the information storage area 14a.

  Next, the marker position information and shooting position information stored in association with the actual image 20b after completion taken with the B marker 21b are acquired from the information storage area 14a, and the image data of the actual image 20b after completion is analyzed. Then, the inclination of the wall 30 with respect to the photographing direction is calculated (S18). At this time, the post-completion actual image 20b has the same height as the right end of the wall 30 and the left end of the wall 30 as shown in FIG. Thus, it is detected that the image is taken from the front perpendicular to each other. Further, based on the calculated shooting direction and inclination of the actual image 20a and the actual image 20b after completion and the marker position information of the A marker 21a and the B marker 21b, the actual image 20a and the actual image 20b after completion are captured. Calculate the positional deviation.

  The image data of the construction actual image 20a is based on the shooting direction and inclination of the construction actual image 20a and the completed actual image 20b, and the deviation of the photographing position between the construction actual image 20a and the construction actual image 20b. As shown in FIG. 7, the actual image 20a at the time of construction is processed into a virtual image 23 positioned so as to be photographed from the same direction as the photographing direction of the actual image 20b after completion (S19).

  Next, in order to make the virtual image 23 obtained by processing the actual image 20a at the time of construction the size of the area designated from the touch panel by the user, the image outside the designated area is deleted (S20). The image data of the virtual image 23 processed into the image of the area is rewritten with the image data of the completed actual image 20b stored in the image virtual development area 14b (S21). As a result, as shown in FIG. 6C, a composite image 24 is generated in which a virtual image in which the state of the bar arrangement is shown in the designated area of the actual image after completion is combined. Here, the processing of S19 to S21 in FIG. 3 corresponds to an image composition step.

  Finally, the composite image 24 is displayed on the display unit 16 based on the image data of the composite image 24 stored in the image virtual development area 14b (S22).

  According to the image synthesizing method of the present embodiment, the image includes the actual image 20a at the time of construction whose position is specified by the A marker 21a and the region whose position is specified by the B marker 21b and photographed together with the A marker 21a at the time of construction. An actual image 20b after completion of photographing an area including the B marker 21b later, and an x coordinate, y coordinate, z coordinate associated with the A marker 21a, and an x coordinate, y coordinate, z associated with the B marker 21b Positioning based on the coordinates, and forming and synthesizing the virtual image 23 according to the area specified by the user, the virtual image 23 processed from the construction actual image 20a is more accurate than the actual image 20b after completion. It is possible to synthesize at any position. At this time, the construction actual image 20a includes the A marker 21a, and the post-completion actual image 20b includes the B marker 21b. Therefore, the construction actual image 20a and the post-completion actual image 20b are photographed. By simply identifying the positions of the virtual image 23 processed from the actual image 20a at the time of construction and the actual image 20b after completion, the virtual image 23 processed from the actual image 20a at the time of construction and the actual image 20b after completion are associated with each other It is possible to attach. For this reason, it is possible to arrange the virtual image 23 processed from the actual image 20a at the time of construction on the actual image 20b after completion more accurately and easily synthesize it.

  In addition, even if the image data of the construction actual image 20a to be combined with the image data of the actual image 20b after completion is different from each other in the photographing position, the distance from the viewpoint, and the photographing direction at the time of construction and photographing after completion, Since the actual image 20a at the time of construction is processed according to the shooting position, distance from the viewpoint, and the shooting direction of the actual image 20b after completion, the actual image 20a at the time of construction was shot from the same distance and the same direction as the actual image 20b after completion. Such a virtual image 23 can be generated. And since the produced | generated virtual image 23 is synthesize | combined with the real image 20 after completion, it is possible to produce | generate the synthesized image superimposed as if it image | photographed at once.

  The actual construction image 20a is stored in association with the 3D model 33 based on the marker position information associated with the A marker 21a, the distance from the viewpoint measured during photographing, and the photographing direction. Therefore, by associating the photographed two-dimensional actual construction image 20a with the 3D model 33 in which the subject to be photographed is three-dimensionally modeled, three-dimensional coordinate information is attached to the construction actual image 20a. It is possible to memorize it.

  The above embodiment is for facilitating the understanding of the present invention, and is not intended to limit the present invention. The present invention can be changed and improved without departing from the gist thereof, and it is needless to say that the present invention includes equivalents thereof.

  In the above embodiment, the example in which the A marker 21a is arranged at an accurate position has been described. However, the A marker 21a does not necessarily have to be provided at the accurately measured position. For example, unlike the A marker 21a, An accurate measurement of a reference marker or the like is provided, and an actual image captured so that the reference marker and the A marker 21a are included in the imaging range is analyzed to calculate an accurate position of the A marker 21a. I do not care.

  In the above embodiment, an example of combining still images has been described. However, when combining moving images, a combined image may be generated and displayed in accordance with the frame rate of the moving images. For example, if the frame rate is 15 fps, it can be realized by rewriting the image on the display unit by repeating the image composition processing as described above 15 times per second.

DESCRIPTION OF SYMBOLS 1 Image composition apparatus 10 Image pick-up part 12 Control part 14 Storage part 14a Information storage area 14b Image virtual expansion | deployment area | region 16 Display part 16a Liquid crystal panel 18 Operation part 18a Touch panel 20 Real image 20a Construction actual image 20b Completion actual image 21 Marker 21a A Marker 21b B marker 23 Virtual image 24 Composite image 30 Wall 31 Reinforcing bar 32 Frame 33 3D model F1 Shooting range at construction F2 Shooting range after completion

Claims (2)

A shooting step of shooting the first area by including a first position information subject whose position can be specified within a shooting range when shooting the first area, and generating image data of the first area image;
A storage step of storing the first region image and the first position specified by the first position information subject in association with each other;
A second position specified by the second position information subject in image data of a second region image obtained by photographing the second area by including the second position information subject in the second area including the first area; An image combining step of combining the image data of the first region image by positioning based on the first position;
Have
During shooting of the first region and shooting of the second region, the distance between the shooting viewpoint and the first position information subject and the second position information subject is measured and the shooting direction is specified,
Based on the identified distance from the viewpoint, the shooting direction, and the marker position information of the first position information subject and the second position information subject, the first region to be combined with the image data of the second region image An image composition method comprising processing image data of a region image and combining it with the first region image . The image composition method according to claim 1 ,
The first area image is a three-dimensional image of a subject to be photographed based on marker position information associated with the first position information subject, a distance from the viewpoint measured at the time of photographing, and the direction of photographing. An image composition method characterized by being stored in association with a modeled stereo model.

Images