JP2012174218A - Image composition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image composition method in which the photographed actual image and the generated virtual image can be composited into the image in line with more reality.SOLUTION: The image composition method includes: a photographing step of photographing a subject; a real image storage step of storing image data constituting the photographed actual image; an information acquisition step in which composition information for compositing the actual image and the virtual image is acquired which includes information for specifying the virtual image composited with the actual image based on the information acquired by photographing, and the information for specifying a setting region where arrangement of the image data of the actual image is set in advance; and an image composition step in which, when the actual image and the virtual image are composited with each other based on the composition information, the image data of the actual image is arranged in the specified setting region, and the composition image is generated.

Description

  The present invention relates to an image composition method for compositing a captured real image and a generated virtual image.

  As an image composition method for synthesizing the captured real image and the generated virtual image, for example, an additional image corresponding to a specific subject included in the captured captured image is obtained in advance by capturing the additional image corresponding to the specific subject. An image composition method for generating a composite image by arranging it at a position on the image is known (for example, see Patent Document 1). Such a composite image is synthesized by placing the additional image at a predetermined position of the captured image, and therefore, in a region where the captured image and the additional image overlap, the additional image is placed in the foreground and the captured image is added. It is synthesized so as to be hidden by the image.

JP 2010-187052 A

In the conventional image composition method, since the additional image is arranged in the foreground and the captured image is combined so as to be hidden by the additional image, for example, the additional image is a window frame, When the additional image of the window frame is combined, a landscape appears outside the window frame.
Further, in the case where the captured image has two or more objects whose positions are different in the depth direction, even when it is desired to synthesize the captured image and an additional image arranged between the two objects, There is a problem that an image cannot be synthesized so that a part of the additional image is hidden by a part of the captured image.

  The present invention has been made in view of such problems, and an object of the present invention is to provide an image composition method capable of compositing a captured real image and a generated virtual image into a more realistic image. It is to provide.

  In order to achieve such an object, an image composition method of the present invention is based on a photographing process for photographing a subject, a real image storing process for storing image data constituting a photographed real image, and information acquired by photographing. The real image and the virtual image including the information specifying the virtual image to be combined with the real image and the information specifying the setting area in which the image data of the real image is set in advance are combined An information acquisition step for acquiring composite information for performing, and when combining the real image and the virtual image based on the composite information, the real image set in advance in the specified setting area And an image composition step of generating image data by arranging image data.

  According to such an image composition method, it is possible to synthesize a real image and a virtual image based on information acquired by photographing a subject and to arrange a real image in a setting area. In addition to synthesizing the real image and the virtual image, the real image can be arranged and synthesized in the setting area according to the setting. For this reason, it is possible to generate a composite image that is more realistic.

  In this image composition method, the composition information includes distance information indicating a distance from the viewpoint of shooting set in the virtual image and a distance from the viewpoint set in the previous setting area. In the image compositing step, of the real image in order from the image data of the region in which the distance from the viewpoint is set longer as the distance information among the region of the virtual image and the setting region. It is desirable that the image data is rewritten.

  According to such an image composition method, the virtual image area and the setting area are rewritten with the image data of the captured real image in order from the area where the distance from the viewpoint is set longer in the composition information. The one with the shortest distance from the viewpoint, that is, the one located closer to the viewpoint appears on the composite image. For this reason, it is possible to arrange the real image and the virtual image as if they are visible from the viewpoint.

  In this image composition method, the image data constituting the photographed real image is stored in two image storage areas, and in the image composition step, based on the combination information, the two image storage areas When one of the image storage areas is used as a work area, the image data of the stored real image is rewritten to generate a composite image, and when the image data of the real image is arranged in the setting area, It is desirable to rewrite the image data of the actual image stored in one image storage area with the image data stored in the other image storage area.

  According to such an image composition method, image data of a real image generated by photographing a subject is stored in two image storage areas, and one of the two image storage areas is used as a work area. Since the image data of the stored real image is rewritten to generate a composite image, image data of the composite image obtained by combining the real image and the virtual image is generated in one image storage area, and the other image storage area It is possible to hold image data of a real image generated by shooting. When the image data of the real image is arranged in the setting area, the image data of the real image stored in the one image storage area is stored in the other image storage area that is held without being rewritten. Therefore, it is possible to reliably arrange the photographed actual image in the setting area.

  Also included in a photographing step for photographing a subject, a real image storing step for storing image data constituting the photographed real image, a virtual image to be combined with the real image based on information acquired by photographing, and a real image Information for identifying the image element to be imaged, distance information indicating the distance of the actual image, the virtual image, and the image element from the shooting viewpoint, and image data of the actual image are arranged in advance. An information acquisition step for acquiring composite information for combining the real image and the virtual image, and based on the composite information, the virtual image is included in the virtual image. An image synthesis step of synthesizing the image and the image element to generate a synthesized image, and the image synthesis step adds image data of the real image to the captured image data based on the synthesis information. Of the image data and the image data of the image element, the image data with the shorter distance from the viewpoint is written, and the actual image when the shorter distance from the viewpoint is set in the setting area The image composition method is characterized in that the image data is written.

  According to such an image composition method, based on the composite information acquired by photographing, image data with a shorter distance from the viewpoint of the virtual image and the image element is added to the image data of the photographed real image. In addition to writing, if the shorter distance from the viewpoint is set in the setting area, the image data of the actual image is written, so the actual image and the virtual image are simply synthesized based only on the distance from the viewpoint. In addition, when the area having a short distance from the viewpoint is the setting area, it is possible to always place the actual image in the foreground.

  The composite information for generating the composite image includes information specifying the virtual image to be combined with the real image and the image elements included in the real image, the real image, the virtual image, and the photographing viewpoint of the image element. Distance information indicating the distance from the camera and information for specifying a setting area in which image data of the actual image is set in advance are included, so that information according to the situation at the time of shooting can be acquired. In addition, the composite information can be acquired by photographing. For this reason, it is possible to easily generate a composite image that is more realistic.

  In this image composition method, it is preferable that the image element is set in the setting area.

  According to such an image composition method, the image elements included in the actual image are set in the setting area. Therefore, when the actual image is to be overwritten after the virtual image is overlaid on the actual image, the actual image can be easily obtained. It is possible to place the image in front.

  In this image composition method, the image data constituting the photographed real image is stored in two image storage areas, and in the image composition step, based on the combination information, the two image storage areas When one of the image storage areas is used as a work area, the image data of the stored real image is rewritten to generate a composite image, and when the image data of the real image is arranged in the setting area, It is desirable to rewrite the image data stored in one image storage area to the image data stored in the other image storage area.

  According to such an image composition method, image data of a real image generated by photographing a subject is stored in two image storage areas, and one of the two image storage areas is used as a work area. Since the image data of the stored real image is rewritten to generate a composite image, image data of the composite image obtained by combining the real image and the virtual image is generated in one image storage area, and the other image storage area It is possible to hold image data of a real image generated by shooting. When the image data of the real image is arranged in the setting area, the image data of the real image stored in the one image storage area is stored in the other image storage area that is held without being rewritten. Therefore, it is possible to reliably arrange the photographed actual image in the setting area.

  In this image composition method, the image data constituting the photographed real image is stored in two image storage areas, and in the image composition step, based on the combination information, the two image storage areas When one of the image storage areas is used as a work area, the image data of the stored real image is rewritten to generate a composite image, and when the image data of the real image is arranged in the setting area, It is desirable to rewrite the image data of the actual image stored in one image storage area with the image data stored in the other image storage area.

  According to such an image composition method, image data of a real image generated by photographing a subject is stored in two image storage areas, and one of the two image storage areas is used as a work area. Since the image data of the stored real image is rewritten to generate a composite image, image data of the composite image obtained by combining the real image and the virtual image is generated in one image storage area, and the other image storage area It is possible to hold image data of a real image generated by shooting. When the image data of the real image is arranged in the setting area, the image data of the real image stored in the one image storage area is stored in the other image storage area that is held without being rewritten. Therefore, it is possible to reliably arrange the photographed actual image in the setting area.

  In this image composition method, the composite image, the real image, and the virtual image are formed of a plurality of pixels, and the setting area is set for each of the pixels. A distance information storage area for storing the distance information of each pixel is provided corresponding to each pixel in the image storage area, and the image data stored in the one image storage area is rewritten in the image composition step. If it is, it is preferable that the distance information in the distance information storage area is rewritten to the distance information of the rewritten image data.

  According to such an image composition method, the distance information corresponding to each pixel of the image data stored in one image storage area is stored in the distance information storage area, and the image data in one image storage area is stored. Every time it is rewritten, the distance information in the distance information storage area is rewritten to the distance information of the rewritten image data. Therefore, the distance information stored in the distance information storage area is always associated with the image data in one image storage area. It is possible to update to the latest state.

  In this image composition method, it is preferable that each pixel constituting the image data of the real image stored in the one image storage area is set to have an infinite distance from the viewpoint.

  According to such an image composition device, each pixel constituting the image data of the actual image stored in the one image storage area is set to an infinite distance from the viewpoint, so the captured actual image It is possible to arrange a virtual image to be synthesized and an image associated with the image element. For this reason, it is possible to synthesize an image while rewriting image data using a captured real image as an original image.

  In this image composition method, in the image composition step, the distance information in the distance information storage area and the distance information set in the virtual image or the setting area are set for each pixel. It is desirable to compare the distance information.

 According to such an image composition method, when compositing an image, distance information in the distance information storage area for each pixel, distance information set in the virtual image, and distance information set in the image element, Therefore, the distance information of the virtual image to be combined and the distance information of the setting area with respect to the distance information of each pixel of the distance information storage area that is always updated to the latest state corresponding to one image storage area. As a result of comparison, for each pixel, the image data in one image storage area can be rewritten to image data closer to the viewpoint more accurately. For this reason, it is possible to generate a composite image that is more realistic.

  According to the present invention, it is possible to combine a captured real image and a generated virtual image into a more realistic image.

It is a block diagram which shows the apparatus for implement | achieving the image-synthesis method based on this invention. It is a flowchart for demonstrating the image composition method of 1st Embodiment. It is a figure which shows the image of the process in which an image is synthesize | combined by the image synthesis method of 1st Embodiment. It is a flowchart for demonstrating the image composition method of 2nd Embodiment. FIG. 5A is a diagram illustrating a real image, a virtual image, an image set in a setting area, a real image, a virtual image, a depth buffer of an image element, and a depth image in the image composition method of the second embodiment, and FIG. FIG. 5B is a diagram illustrating an image of a captured real image and a depth buffer. FIG. 5B is a diagram illustrating an image element of the front desk and an image of a depth set in the image element. FIG. FIG. 5D shows an image of a desk and an image of the depth set in the image element. FIG. 5D shows an image of the near side image and an image of the depth set in the near side image. FIG. These are figures which show the image of the depth set to the back person image and this back person image. FIG. 6A is a diagram showing an image synthesis process and an image of a depth buffer by the image synthesis method of the second embodiment, and FIG. 6A shows an image for synthesizing a captured real image and a front person image. FIG. 6B is a diagram illustrating an image obtained by combining an image obtained by combining a real image and a front person image and an image element of a front desk, and FIG. 6C illustrates an actual image, a front person image, FIG. 6D is a diagram illustrating an image obtained by combining an image obtained by combining the image elements of the front desk and the image elements of the back desk. FIG. 6D is a real image, a front person image, an image element of the front desk, and an image element of the back desk. The figure which shows the image which synthesize | combines the image and back person image which were synthesize | combined, FIG.6 (e) is a figure which shows the completed synthesized image.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram showing an example of an apparatus for realizing an image composition method according to the present invention.

  The image composition method according to the present embodiment, for example, combines a real image and a virtual image when combining a photographed real image and a pre-formed virtual image based on information acquired by photographing a subject. This is a method of generating a composite image that matches reality.

  An image composition apparatus 1 for realizing such an image composition method includes an image capturing unit 10 that captures a subject, and a process that combines a real image of the captured subject and a virtual image that is formed in advance. The control unit 12 for controlling the image, the storage unit 14 for storing various information and data necessary for synthesizing the image, the display unit 16 for displaying the synthesized image, and the image synthesizing apparatus 1 And an operation unit 18 for this purpose.

  The image composition device 1 of the present embodiment has, for example, an imaging unit 10 and a display unit 16 that displays a captured image, and is portable, for example, a digital camera, a digital video camera, a mobile phone, a personal digital assistant (PDA). ), A program capable of executing image composition processing is stored in a mobile computer or the like.

  Then, when the user of the image composition device 1 photographs the subject, the captured actual image is displayed on the display unit 16. At this time, the photographed real image 20 includes a preset marker 21 indicating predetermined composite information such as a QR code (registered trademark) or a barcode, or a GPS or a digital compass. When information such as a shooting position (location) and a shooting direction is measured, the marker 21 is detected and analyzed to obtain combined information, or the combined information is acquired based on the shooting position and direction information. Based on the acquired composite information, a composite image 24 (FIG. 3) is generated by combining the captured real image 20 (FIG. 3) and the virtual image 22 (FIG. 3) stored in advance. (FIG. 3) is configured to be displayed on the display unit 16.

  The imaging unit 10 converts optical information into electrical information by an imaging element such as a CCD sensor that generates image data of a plurality of pixels constituting an image by photographing, such as a digital camera or a digital video camera. The display unit 16 is configured to display an image based on the converted image data.

  The display unit 16 is configured by, for example, a liquid crystal panel that can display image data of a plurality of pixels constituting an image corresponding to each element. Here, the image data of each pixel is, for example, data indicating each component obtained by decomposing each pixel into an R (red) component, a G (green) component, and a B (blue) component.

  The operation unit 18 is a part where a user who uses the image composition device 1 performs an operation such as photographing or changing settings on the image composition device 1, and photographs at least the actual image 20. So-called shutters are provided.

  The storage unit 14 includes a composite information storage area 14a in which various types of information for generating a composite image in the image composition apparatus 1 are stored, and two images that store image data of a captured real image for each pixel. It has storage areas 14b and 14c and a pixel information storage area 14d for storing information for each pixel necessary for synthesizing an image.

  In the composite information storage area 14a, the virtual image 22 associated with information indicated by the marker 21 included in the subject, information such as the shooting position and direction, and the actual image 20 of the virtual image 22 are arranged. The position information indicating the position where the marker 21 is provided or the distance information indicating the distance from the viewpoint at the time of shooting of the image composition device 1 and the real image 20 and the virtual image 22 are superimposed. In some cases, setting area information indicating a relative position between a setting area 26 in which placement of either a real image or a virtual image is set in advance and a viewpoint position at the time of shooting of the setting area 26 is stored.

  Each of the two image storage areas 14b and 14c is a storage area in which captured image data is stored for each pixel, and one image storage area is rewritten in the process of generating a composite image. The other image storage area is an image virtual development area 14b to be a working area, and the other image storage area associates image data that has already been shot until the next shooting is performed with position information and distance information of each pixel in the actual image. This is an actual image storage area 14c that is held as a texture. As described above, since the image data of each pixel has data indicating three components of R, G, and B, the image virtual development area 14b and the actual image storage area 14c each have the number of pixels constituting the composite image. It has three times the storage area.

  The image data stored in the image virtual development area 14b is rewritten with the corresponding pixel image data as necessary when generating a composite image. The three data stored for each pixel in the image virtual development area 14b are rewritten respectively for the R component, the G component, and the B component when the image data of each pixel is rewritten. At this time, the real image storage area 14c and the image virtual development area 14b do not have to be arranged in the same manner as the elements of the display unit 16 or the imaging unit 10, and the image data of the image storage areas 14b and 14c May be stored in association with the elements of the display unit 16 or the imaging unit 10.

  The pixel information storage area 14d is a storage area for storing, for each pixel, information indicating the position of the image associated with each pixel with respect to the viewpoint at which the subject is photographed. For this reason, the pixel information storage area 14d has storage areas for the number of pixels constituting the composite image.

  The control unit 12 controls the imaging unit 10 and the display unit 16 by operating the operation unit 18 by the user, or extracts the marker 21 from the image data of the captured real image 20. Image synthesis for synthesizing the real image 20 and the virtual image 22 stored in the storage unit 14 based on the analysis processing for analyzing the information shown, the analyzed information, and the information indicating the distance and the direction measured in photographing. Various processes such as processes are executed.

  Next, an image composition method executed by the control unit 12 will be described.

<First Embodiment>
The image synthesizing method of the first embodiment is provided in the image synthesizing apparatus 1, and a process is started by operating a shutter which is an operation unit 18 for photographing with the imaging unit 10. At this time, the operation unit 18 may be provided with a switch for switching processing between when the image composition device 1 simply captures a real image and when a composite image is generated, such as a digital camera. Further, it is set so that a process for generating a composite image is always executed, and the composite image is displayed only when the marker 21 is detected in the photographed real image or when the photographing position and the photographing direction are specified. It is good also as performing the process to produce | generate.

  In the case of the first embodiment, it is assumed that the subject is a wall that partitions the outdoor and indoors, and the marker 21 is provided on the wall. The marker 21 includes information for specifying the virtual image 22 to be combined with the captured real image.

  FIG. 2 is a flowchart for explaining the image composition method of the first embodiment, and FIG. 3 is a diagram showing an image in the process of image synthesis by the image composition method of the first embodiment.

  When the shutter is operated, the control unit 12 captures a subject with the imaging unit 10 (imaging step S1), and stores the data of each pixel constituting the captured actual image in the actual image storage storage area 14c of the storage unit 14. And the image virtual development area 14b (actual image storage step S2). In this state, the image virtual development area 14b stores image data forming a real image 20 of the wall provided with the markers 21 as shown in FIG.

  At this time, the control unit 12 uses, for example, the upper half of the lens and the lower half of the lens based on the information acquired from the imaging unit 10 at the time of capturing a real image, using the principle of an autofocus mechanism mounted on a general camera. The distance between the subject and the viewpoint position of the imaging unit 10 is measured using the difference in the center of gravity position generated in half, and the measured distance information is stored in the storage unit 14. This distance information is included in information acquired by photographing.

  Next, the control unit 12 analyzes whether or not the marker 21 is captured in the image data of the actual image 20 (S3). At this time, if the marker 21 is not photographed in the image data of the real image 20, it is determined that there is no image to be combined, and the photographed real image 20 is displayed on the display unit 16 (S11). Further, when the marker 21 is detected, the marker 21 is further analyzed, and the composite information associated with the information indicated by the marker 21 and the information of the marker 21 is acquired from the composite information storage area 14a (information acquisition) Step S4).

  The information acquired at this time includes information for specifying the virtual image 22 to be superimposed on the captured real image 20, the position on the plane on which the specified virtual image 22 should be superimposed on the real image 20, and the specification. The distance information indicating the distance to the photographed viewpoint where the virtual image should be placed, and the region in which the real image 20 or the virtual image 22 is to be placed in the generated composite image are set in advance. Information including setting area information.

  In the case of the first embodiment, as the virtual image 22, a window frame virtual image 22a as shown in FIG. 3B and a landscape virtual image 22b as shown in FIG. 3C are specified. . Further, as the composite information, the marker 21 is provided with distance information arranged so that the virtual image 22a of the window frame is provided on the wall of the actual image 20, and the virtual image 22a of the window frame and the virtual image 22b of the scenery. Position information that is arranged with the existing position as a center, and information that indicates a region that is superimposed on the actual image 20 as shown in FIG. Here, as the setting area 26, area information indicating that an area located outside the window frame is set in the scenery virtual image 22b among the scenery and window frame virtual images 22a and 22b, and an image of the setting area includes The distance information that the distance from the photographing viewpoint to be arranged is the same distance as the wall on which the marker 21 is provided, and that the actual image 20 is arranged in the setting area 26 is set. Is obtained.

  Next, the control unit 12 associates the virtual image 22a of the window frame and the virtual image 22b of the scenery with the size of the actual image based on the acquired composite information and the distance information measured and stored at the time of shooting. Convert to size as appropriate. At this time, the size of the virtual image 22 is set as the size of the area displayed on the display unit 16 based on the distance information, and the pixel on which the virtual image 22 is displayed on the display unit 16 is specified.

  Then, the control unit 12 identifies a region where the virtual image 22 and the real image 20 that have been appropriately converted to a size overlap, and a setting region on the composite image 24 based on the acquired position information. Specifically, in the image virtual development area 14b, a pixel in which the window frame virtual image 22a overlaps, a pixel in which the landscape virtual image 22b overlaps, and a pixel in the setting area are specified (specification step S5).

  In the present embodiment, the setting area 26 is obtained when the virtual image 22a of the window frame whose size is converted based on the distance information and the virtual image 22b of the scenery are superimposed with the position of the marker 21 as the center. It is set as an area corresponding to the scenery virtual image 22b arranged outside the virtual image 22a of the window frame, and is set so that image data of the actual image is arranged in each pixel included in the setting area 26. ing.

  Next, among the identified window frame virtual image 22a, landscape virtual image 22b, and setting area 26, the control unit 12 selects the one having the longest distance from the shooting viewpoint (S7). Here, the virtual image 22b having the longest distance from the shooting viewpoint is selected.

  Next, the control unit 12 determines whether or not the selected object is the setting area 26 (S8). In this case, since it is not the setting area 26, the control unit 12 uses the image data of the pixel in which the scenery virtual image 22b is arranged among the pixels of the image virtual development area 14b as the image data of the scenery virtual image 22b. Rewrite to In this state, image data forming an image in which the entire virtual image 22b of the scenery is arranged on the real image 20 of the photographed wall as shown in FIG. Is remembered.

  Next, the control unit 12 determines whether there is an image to be further synthesized (S6). In this case, since the virtual image 22a of the window frame and the setting area 26 remain, the process of combining the images is continued.

  That is, the control unit 12 selects an object having a long distance from the shooting viewpoint from the virtual image 22a of the window frame and the setting area 26 (S7). In this case, the distance from the shooting viewpoint in the virtual image 22a of the window frame is a position protruding forward from the wall on which the marker 21 is provided, and the setting area 26 is the surface of the wall on which the marker 21 is provided. Therefore, the setting area 26 is selected.

  Next, the control unit 12 determines whether or not the selected object is the setting area 26 (S8). In this case, since it is the setting area 26, the control unit 12 stores the image data of the pixel in which the setting area 26 is arranged among the pixels of the image virtual development area 14b in the actual image storage area 14c. The actual image 20 is rewritten with the image data (S9). In this state, in the image virtual development area 14b, a landscape virtual image 22b is displayed only in an area corresponding to the inside of the window frame on the photographed wall real image 20, as shown in FIG. Image data forming the arranged image is stored. In the case of the first embodiment, it is determined whether or not the area to be combined is the setting area 26 (S8). In the case of the setting area 26, the image data in the image virtual development area 14b is stored in the real image storage area 14c. The image data of the real image 20 stored is rewritten (S9), and if it is not the setting area 26, the image data of the image virtual development area 14b is rewritten to the image data of the virtual image 22 (S10). include.

  Next, the control unit 12 determines whether there is an image to be further synthesized (S6). In this case, since the virtual image 22a of the window frame remains, the process of combining the images is continued. In this case, the virtual image 22a of the window frame is selected regardless of the distance from the shooting viewpoint (S7).

  Next, the control unit 12 determines whether or not the selected object is the setting area 26 (S8). In this case, since it is not the setting area 26, the control unit 12 converts the image data of the pixel in which the window frame virtual image 22a is arranged among the pixels of the image virtual development area 14b to the window frame virtual image 22a. (S10). In this state, in the image virtual development area 14b, as shown in FIG. 3G, a composite image 24 having a window frame on the photographed wall and a landscape image arranged inside the window frame is formed. The image data to be stored is stored.

  Next, the control unit 12 determines whether there is an image to be further synthesized (S6). At this time, since there are no remaining objects to be synthesized, the process of synthesizing the image is terminated, and the image is displayed on the display unit 16 based on the image data stored in the image virtual development area 14b (S11).

  According to the image composition method of the first embodiment, when the wall provided with the marker 21 is photographed, the virtual image 22a of the window frame or the virtual image of the landscape is displayed on the real image 20 based on the composite information indicated by the marker 21. Since the real image 20 is arranged in the setting area, the real image 20 is simply combined with the virtual image 22a of the window frame and the virtual image 22b of the scenery, as well as the setting area 26. It is possible to arrange the set real image 20. For this reason, by setting the outside of the window frame where the scenery should not be originally visible as the setting area 26, it is possible to place the actual image 20 in the setting area 26 and use the area outside the window frame as a wall. is there. For this reason, it is possible to generate a composite image that is more realistic.

  Further, the virtual image 22a of the window frame, the virtual image 22b of the scenery, the setting area 26, and the like are sequentially rewritten with the image data of the photographed real image 20 from the composite information in which the distance from the viewpoint is set to be long. Therefore, the one having the shortest distance from the viewpoint, that is, the one located closer to the viewpoint appears on the composite image 24. For this reason, it is possible to arrange the real image 20, the virtual image 22a of the window frame, and the virtual image 22b of the scenery so as to be seen from the viewpoint.

  In addition, since the image data of the actual image 20 is arranged in the setting area 26 set in the area of the virtual image 22b of the scenery, for example, where a distance longer than the distance from the viewpoint of the photographed subject is set, The image data of the scenery virtual image 22b to be hidden after the image 20 is rewritten to the image data of the real image 20. Therefore, the synthesized image 24 can be synthesized with the real image 20, the window frame virtual image 22 a, and the landscape virtual image 22 b as if viewed from the viewpoint. Therefore, it is possible to generate a composite image 24 that is more realistic.

Second Embodiment
The image synthesis method of the second embodiment is the same as that of the first embodiment in that processing is started by operating a shutter for photographing by the imaging unit 10. Hereinafter, members similar to those in the first embodiment are denoted by the same reference numerals, and a part of the description is omitted.

  In the case of the second embodiment, it is assumed that a room in which two desks are arranged and a room in which a marker 21 is provided is taken as a subject. At this time, the two desks are arranged in the front-rear direction in the depth direction, and the front desk (hereinafter referred to as the front desk) is arranged slightly to the left of the back desk (hereinafter referred to as the back desk). . In the photographed real image, the two desks are set as image elements 23a and 23b different from the background (floor or wall), and the image data of the real image, that is, the image element of the front desk is based on a predetermined condition. 23a and a setting area set so that image data of the back image element 23b is arranged.

  In the second embodiment, a numerical value (hereinafter referred to as a depth) as distance information corresponding to the distance of the image indicated by each pixel with respect to the position of the viewpoint of the imaging unit 10 is stored in association with each pixel. A depth buffer for storing the depth of each pixel is provided in a pixel information storage area 14d as a distance information storage area. This distance information is given as a depth corresponding to the distance to the viewpoint position of the imaging unit 10, and a small depth is given from an image arranged at a position close to the viewpoint position of the imaging unit 10.

 FIG. 4 is a flowchart for explaining an image composition method according to the second embodiment. FIG. 5 is a diagram illustrating a real image, a virtual image, an image set in a setting region, a real image, a virtual image, and an image element depth buffer image in the image composition method according to the second embodiment. FIG. 6 is a diagram illustrating a process in which images are combined by the image combining method according to the second embodiment and an image of a depth buffer. In FIG. 6, for convenience of explanation, the cells indicating the image are overlapped with the depth buffer and the depth.

  When the shutter is operated, the control unit 12 captures a subject with the imaging unit 10 (imaging step S101), and stores data of each pixel constituting the captured actual image in the actual image storage area 14c of the storage unit 14. And the image virtual development area 14b (actual image storage step S102). Further, at the time of shooting, the distance from the subject and the viewpoint of the imaging unit 10 is measured, and the measured distance information is stored in the storage unit 14. In the second embodiment, the distance from the viewpoint of the imaging unit 10 of the two desks provided with the markers 21 and the wall arranged behind the desk is measured and stored in the storage unit 14.

  The image data of the real image 20 stored in the image virtual development area 14b is given a depth indicating a position farthest from the viewpoint position of the imaging unit 10 (infinity), and in the real image storage and storage step S102. When the image data of the real image 20 is stored, the depth buffer of the pixel information storage area 14d is initialized (S103). In the initialization of the depth buffer, for example, the depth “9” given to each pixel of the real image 20 stored in the image virtual development area 14b is stored as the depth of all the pixels in the depth buffer. In this state, image data forming a real image 20 in a room where two desks are arranged as shown in the left diagrams of FIGS. 5A and 6A are stored in the image virtual development area 14b. In the depth buffer, depth “9” is stored in all pixels.

  In addition, two desks existing on the real image 20 are set as image elements included in the real image, and the area of the two desk image elements is set in the setting area 26. It is stored as information. For example, as shown in FIGS. 5B and 5C, the front desk and the back desk are set as image elements 23a and 23b, and the depth “1” is set as position information for each pixel forming the image element 23a of the front desk. However, the depth “3” is set as position information for each pixel forming the back desk image element 23b based on the information of the marker 21 acquired at the time of photographing. The front desk image element 23a and the back desk image element 23b are provided with the same number of pixel data so as to have the same size as the size of the image data of the captured real image.

  Next, the control unit 12 analyzes whether or not the marker 21 is captured in the image data of the actual image 20 (S104). At this time, if the marker 21 is not captured in the image data of the actual image 20, it is determined that there is no image to be combined, and the captured actual image 20 is displayed on the display unit 16 (S115). Further, when the marker 21 is detected, the marker 21 is further analyzed, and the composite information associated with the information indicated by the marker 21 and the information of the marker 21 is acquired from the composite information storage area 14a (information acquisition) Step S105).

  In the case of the second embodiment, the virtual images 22c and 22d representing the whole body of two persons are specified as the virtual image 22 by the marker 21, and the image elements of the two desks are included as the image elements 23 included in the actual image. 23a and 23b are specified, and information in which the image elements 23a and 23b of the two desks are set in the setting area 26 is acquired.

  Further, as composite information, a virtual image 22c of one person (hereinafter referred to as a front person image) 22c of two persons is located between two desks. Distance information indicating that a virtual image (hereinafter referred to as a back person image) 22d of the other person is located between the back desk and the wall is acquired. Since this distance information is given as a depth corresponding to the distance to the position of the viewpoint of the imaging unit 10, in this example, for example, a depth “2” is added as distance information to the front person image 22c, and a depth person image 22d is displayed. Depth “4” is given as position information corresponding to each pixel.

  Further, in the image elements 23a and 23b indicating the two desks, composite information such as distance information indicating that the distance from the photographing viewpoint is the same distance as each desk on which the marker 21 is provided is acquired. Here, the distance information between the image elements 23a and 23b of the two desks is the depth “1” as the position information for each pixel of the image element 23a of the front desk, and the position information for each pixel of the image element 23b of the back desk. The depth “3” is acquired. In the following description, the two virtual images 22c and 22d and the two image elements 23a and 23b are also referred to as objects.

  Next, the control unit 12 selects one object from the virtual images 22c and 22d of the two persons and the two image elements 23a and 23b (S106). The objects selected at this time are selected in a preset order such as the specified order. Here, for example, it is assumed that the front person image 22c is first selected.

  Next, the image data of the selected foreground person image 22c is appropriately converted into a size by the control unit 12 based on the distance information measured and stored at the time of shooting. Based on the converted image data of the foreground person image 22c, a pixel in which the foreground person image 22c is arranged in the image virtual development area 14b is specified (S108).

  Next, the control unit 12 specifies the depth stored in the depth buffer and the foreground human image 22c for each pixel of the image virtual development area 14b, which is specified as the pixel data of the foreground human image 22c being arranged. The depth set in the pixel data is sequentially compared for each pixel (S109). At this time, since the depth buffer is initialized and “9” is stored as the depth of all the pixels, the depth “2” set for all the pixels of the foreground person image 22c is smaller and the shooting is performed. Determined to be close to the viewpoint.

  When the depth “2” set in the front person image 22c is smaller than the depth of the corresponding pixel in the depth buffer, the pixel is in the setting area 26, that is, the image element 23a and the back of the front desk. It is determined whether or not the image element 23b is included in the desk (S110). In the case of the second embodiment, the depth set to the front person image 22c is “2”, the depth set to the front desk image element 23a is “1”, and the back desk image element 23b is set. Since the set depth is “3”, it is not determined whether or not the pixel of the front person image 22c is included in the setting area 26 set in the image element 23b of the back desk.

  If the target pixel of the front human image 22c is not included in the front desk image element 23a set in the setting area, the image data of the target pixel of the image virtual development area 14b is The image data of the front person image 22c, which is a virtual image, is rewritten (S112), and the depth buffer depth is updated to the depth “2” set in the front person image 22c (S113).

  On the other hand, when the target pixel of the front person image 22c is included in the front desk image element 23a set in the setting area, the image data of the target pixel of the image virtual development area 14b is The image data of the real image 20 stored in the real image storage area 14c is rewritten (S111), and the depth buffer depth is updated to the depth “1” set in the image element 23a of the front desk (S113). .

  In this way, for each pixel included in the foreground person image 22c, the depth is compared and it is determined whether or not it is the setting area 26. If necessary, the image data in the real image storage area 14c is rewritten, and the depth buffer The depth is updated, and the processing is completed for all the pixels of the front person image 22c.

  In this state, the image virtual development area 14b stores image data forming an image in which the front person image 22c is drawn between the front desk and the back desk, as shown in the left diagram of FIG. 6B. In the depth buffer, the depth “1” is present in the area of the front person image 22 c that overlaps the image element 23 a of the front desk, and the depth “2” is present in the area of the front person image 22 c that does not overlap the image element 23 a of the front desk. Is written. Here, the step (S113) of updating the depth buffer from the step (S109) of comparing the depth of each pixel constituting the virtual image with the depth of the depth buffer is included in the image synthesis step.

  Next, the control unit 12 determines whether all the target objects have been processed (S104). In this case, since the process of the back person image 22d and the two image elements 23a and 23b has not been performed, for example, the image element 23a of the front desk is selected from the unprocessed objects (S106). The process is executed. Specifically, first, the image data of the image element 23 a of the front desk is appropriately converted into a size based on the distance data by the marker 21 by the control unit 12. Based on the converted image data of the front desk image element 23a, the pixel of the image virtual development area 14b where the front desk 23a is arranged is specified (S108).

  Next, for each pixel in the image virtual development area 14b in which the pixel data of the front desk image element 23a is arranged, the control unit 12 determines the depth stored in the depth buffer and the front desk image element 23a. The depth set in the pixel data is sequentially compared for each pixel (S109). At this time, the depth buffer stores the depth “9” in the pixels other than the area of the front human image 22c as shown in the left diagram of FIG. The depth “2” is stored. For this reason, each pixel of the front desk image element 23a has a depth smaller than that of the pixel rewritten by the process of the front person image 22c. Therefore, all the pixels of the front desk image element 23a are in the setting area 26. Is determined (S110).

  In the case of the second embodiment, since all the pixels of the image element 23a of the front desk are set in the setting area, the image data of the pixel that is the target of the image virtual development area 14b is converted into the real image storage area 14c. (S111) and the depth buffer depth is updated to the depth “1” set in the image element 23a of the front desk (S113).

  In this way, for each pixel of the image element 23a on the front desk, the depth is compared and it is determined whether or not it is the setting area 26. If necessary, the image data in the actual image storage area 14c is rewritten, and the depth buffer Is updated, and the processing is completed for all the pixels of the image element 23a on the front desk (S107).

  In this state, in the image virtual development area 14b, as shown in the left diagram of FIG. 6C, image data forming an image in which the front person image 22c is drawn between the front desk and the back desk is stored. The depth buffer stores the depth “1” in the area of the front desk image element 23 a and the depth “2” in the area of the front human image 22 c that does not overlap the front desk image element 23 a. . Here, the step (S113) of updating the depth buffer from the step (S109) of comparing the depth of each pixel of the image element with the depth of the depth buffer is included in the image composition step.

  Next, the control unit 12 determines whether all the target objects have been processed (S104). In this case, since the back person image 22d and the back desk image element 23d have not been processed, for example, the back desk image element 23b is selected from the unprocessed objects (S106). Processing is executed. Specifically, first, the image data of the back desk image element 22 d is converted into an appropriate size by the control unit 12 based on the distance data by the marker 21. Based on the converted image data of the back image element 22d, the pixel of the image virtual development area 14b where the back 22d is arranged is specified (S108).

  Next, for each pixel in the image virtual development area 14b in which the pixel data of the back image element 23b is arranged, the control unit 12 determines the depth stored in the depth buffer and the back image element 23b. The depth set in the pixel data is sequentially compared for each pixel (S109). At this time, as shown in FIG. 6C, the depth buffer stores the depth “9” in the pixels other than the area of the front person image 22c and the front desk image element 23a by the processing up to the front desk image element 23a. In the area of the front person image 22c, the depth “2” is stored in the pixel that does not overlap the front desk image element 23a, and the depth “1” is stored in the pixel of the front desk image element 23a. ing. For this reason, all the pixels of the back image element 23b in which the depth “3” is stored are all rewritten in the process up to the image element 23a of the front desk. Among the pixels of the element 23b, it is determined whether or not the pixel that does not overlap with the front person image 22c and the front desk image element 23a is the setting region 26 (S110).

  In the case of the second embodiment, since all the pixels of the back image element 23b are set in the setting area, the image data of the pixel that is the target of the image virtual development area 14b is converted into the real image storage area 14c. (S111) and the depth buffer depth is updated to the depth “3” set in the back image element 23b (S113).

  In this way, for each pixel of the back image element 23b, the depth is compared and it is determined whether or not it is the setting area 26. If necessary, the image data in the actual image storage area 14c is rewritten, and the depth buffer The depth is updated, and the processing is completed for all the pixels of the back image element 23b (S107).

  In this state, in the image virtual development area 14b, as shown in FIG. 6D, image data forming an image in which a front person image 22c is drawn between the front desk and the back desk is stored, The depth buffer includes a depth “1” in the area of the front desk image element 23 a, a depth “2” in the area of the front human image 22 c that does not overlap the front desk image element 23 a, and the front desk image element 23 a. The depth “3” is written in the area of the image element 23b of the back desk that does not overlap with any of the front person image 22c. Here, the step (S113) of updating the depth buffer from the step (S109) of comparing the depth of each pixel of the image element with the depth of the depth buffer is included in the image composition step.

  Next, the control unit 12 determines whether all the target objects have been processed (S104). In this case, since the back person image 22d has not been processed, the back person image 22d that has not been processed is selected (S106), and the process is executed.

  Specifically, first, the image data of the back person image 22d is converted into an appropriate size by the control unit 12 based on the distance data measured and stored at the time of shooting. Based on the converted image data of the back person image 22d, the pixel of the image virtual development area 14b where the back person image 22d is arranged is specified (S108).

  Next, the control unit 12 specifies the depth stored in the depth buffer and the back person image 22d for each pixel in the image virtual development area 14b, which is specified when the pixel data of the back person image 22d is arranged. The depth set in the pixel data is sequentially compared for each pixel (S109). At this time, the depth buffer performs processing up to the back desk image element 23b, and pixels other than the front person image 22c, the front desk image element 23a, and the back desk image element 23b are processed as shown in FIG. “9” is stored, the depth “2” is stored in the pixel of the front desk image 22c in the pixel that does not overlap the image element 23a of the front desk, and the depth “1” is stored in the pixel of the image element 23a of the front desk. ”Is stored, and the depth“ 3 ”is stored in the area of the image element 23 b of the back desk that does not overlap either the image element 23 a of the front desk or the front person image 22 c. For this reason, all the pixels of the back desk image element 23b in which the depth “3” is stored are all rewritten in the processing up to the back desk image element 23b, and therefore the depth of the back desk image element 23b is small. Among the pixels of the element 23b, it is determined whether or not the pixel that does not overlap any of the front person image 22c, the front desk image element 23a, and the back desk image element 23b is the setting area 26 (S110).

  At this time, since the back person image 22d is not the setting area 26, pixels that do not overlap any of the front person image 22c, the front desk image element 23a, and the back desk image element 23b are targets of the image virtual development area 14b. Is rewritten with the image data of the real image 20 stored in the real image storage area 14c (S111), and the depth “4” set in the depth person image 22d is set in the depth buffer. (S113). Here, the step (S113) of updating the depth buffer from the step (S109) of comparing the depth of each pixel of the image element with the depth of the depth buffer is included in the image composition step.

  Next, the control unit 12 determines whether all the target objects have been processed (S104). At this time, since the processing has been completed for the four objects 22c, 22d, 23a, and 23b, the image composition processing is completed, and display is performed based on the pixel data of the image stored in the image virtual development area 14b. An image is displayed on the unit 16 (S115). At this time, the display unit 16 has two desks in the room viewed from the viewpoint position of the image synthesizing apparatus 1 as shown in FIG. 6E, and the front person image 22c is located between the front desk and the back desk. Is displayed, and an image in which the back person image 22d is placed between the back desk and the wall of the real image is displayed.

  According to the image composition method of the second embodiment, the image data of the photographed real image 20 is obtained from the viewpoint of the virtual images 22c and 22d and the image elements 23a and 23b based on the composition information acquired by photographing. Is rewritten to the image data with the shorter distance, and if the shorter distance from the viewpoint is set in the setting area 26, the image data of the actual image 20 is rewritten and combined, so the distance from the viewpoint is simply The real image 20 and the virtual images 22c and 22d are not only synthesized based on the image, but the real image 20 can always be placed in the front when the setting region 26 has a short distance from the viewpoint. is there.

  The composite information for generating the composite image 24 includes virtual images 22c and 22d to be combined with the real image 20, information specifying the image elements 23a and 23b included in the real image 20, the real image 20, and the virtual image. The distance information indicating the distance from the shooting viewpoint of the images 22c and 22d and the image elements 23a and 23b, and information for specifying the setting area 26 in which the image data of the actual image 20 is set in advance are included. Therefore, it is possible to acquire information according to the situation at the time of shooting, and it is possible to acquire composite information by shooting. For this reason, it is possible to easily generate the composite image 24 that is more realistic.

  In addition, since the image elements 23a and 23b included in the real image 20 are set in the setting area 26, it is easy to overwrite the real image 20 after the virtual images 22c and 22d are overlaid on the real image 20. In addition, the actual image 20 can be arranged on the front surface.

  Further, the depth as distance information corresponding to each pixel of the image data stored in the image virtual development area 14b is stored in the pixel information storage area 14d, and each time the image data in the image virtual development area 14b is rewritten. Since the depth of the pixel information storage area 14d is rewritten to the depth of the rewritten image data, the depth stored in the pixel information storage area 14d is always updated to correspond to the image data of the image virtual development area 14b. It is possible to keep it.

  In addition, each pixel constituting the image data of the actual image 20 stored in the image virtual development area 14b is set to an infinite distance from the viewpoint, and therefore should be synthesized on the captured actual image 20. Image elements 23a and 23b associated with the virtual images 22c and 22d and the setting area 26 can be arranged. For this reason, it is possible to synthesize an image while rewriting image data using the captured real image 20 as an original image.

 Further, when compositing images, the depth of the pixel information storage area 14d is compared for each pixel with the depth set in the virtual images 22c and 22d and the depth set in the image elements 23a and 23b. Therefore, the depth of each of the virtual images 22c and 22d to be combined and the depth of the image elements 23a and 23b with respect to the depth of each pixel of the image virtual development region 14b that is always updated to the latest state corresponding to the image virtual development region 14b. Are compared, and the image data of the image virtual development area 14b can be more accurately rewritten to image data closer to the viewpoint for each pixel. Therefore, it is possible to generate a composite image 24 that is more realistic.

  In the second embodiment, the depth of one pixel constituting the virtual image 22 is compared with the depth of the depth buffer, and when the depth of the virtual image 22 is smaller (close to the viewpoint), the target pixel However, the present invention is not limited to this. For example, a depth buffer that is initialized after shooting is stored in advance with depths set in pixels included in the setting area, and a depth buffer in which the depth of pixels corresponding to the setting area 26 in the depth buffer is rewritten, and the virtual image 22 Images may be synthesized by comparing the depths of the two.

  In the above two embodiments, the example in which the marker 21 is provided on the subject has been described. However, the present invention is not limited to this. For example, the shooting position is specified by GPS or the like, the shooting direction is specified by a digital compass or the like, and the virtual image and the setting area stored in advance in the storage area are specified based on the specified shooting position and shooting direction. Alternatively, composite information associated with the virtual image and the setting area may be acquired to generate a composite image.

  Also, the absolute coordinates of the virtual image are stored in the storage area in advance, and the distance between the imaging unit and an object such as a wall included in the range of the screen shot by a 3D laser scanner or the like at the same time as specifying the shooting position and shooting direction (Distance between each point for each pixel and the image pickup unit) is measured, and the absolute coordinates of the virtual image are included in the range of coordinates that the image pickup unit of the image composition device should have taken, If the distance between the virtual image and the imaging unit is closer than the distance between the object and the imaging unit, the virtual image may be superimposed on the captured image and displayed for the corresponding pixel portion.

  Specifically, for example, it is assumed that a virtual “window” is synthesized on the wall of a real building. At this time, as the window frame, virtual images of a plurality of types of designs are stored in the image composition device, and the user operates the image composition device to switch the design of the window frame. Displays a composite image that combines the scenery. When such an image composition device is used, it is possible to check the window harmony design from various angles in the room, and the user can select the design of the window frame with the best appearance. At this time, the wall around the window frame is set as a setting (masking) region so that the scenery is correctly cut out in accordance with the window frame. When using the marker as described above, the marker specifies the window frame design to be synthesized and acquires information indicating the relative positional relationship between the window frame mounting position and the imaging unit. It becomes possible.

  On the other hand, when the marker is not used, the window frame design to be combined is selected and switched using, for example, a user interface provided in the image combining apparatus. The information indicating the relative positional relationship between the window frame attachment position and the imaging unit is obtained, for example, when the latitude / longitude of the attachment position is known in advance. Thus, it is possible to obtain the relative position of the imaging unit. As another method, for example, the orientation of the imaging unit is acquired with a digital compass, the distance to the mounting position is measured with a laser distance meter, and the relative position is obtained even if coordinate conversion is performed. It is possible.

  The above embodiment is for facilitating the understanding of the present invention, and is not intended to limit the present invention. The present invention can be changed and improved without departing from the gist thereof, and it is needless to say that the present invention includes equivalents thereof.

  In the embodiment described above, the viewpoint of the image composition device 1 is arranged in front of the wall on which the window frame of the first embodiment is provided and the indoor wall located at the innermost position of the second embodiment. As described above, if the virtual image has data that can be expressed in three dimensions, the viewpoint of the image composition device 1 may be set at a position inclined with respect to the wall. When the viewpoint of the image composition device 1 is set at a position inclined with respect to the wall, the virtual image and the shape of the setting area are changed based on the position information in correspondence with the inclination.

  In the above embodiment, an example of combining still images has been described. However, when combining moving images, a combined image may be generated and displayed in accordance with the frame rate of the moving images. For example, if the frame rate is 15 fps, it can be realized by rewriting the image on the display unit by repeating the image composition processing as described above 15 times per second.

DESCRIPTION OF SYMBOLS 1 Image composition apparatus, 10 imaging part, 12 control part, 14 memory | storage part,
14a composite information storage area, 14b image virtual development area,
14c real image storage area, 14d pixel information storage area, 16 display section,
18 operation unit, 20 real image, 21 marker, 22 virtual image,
22a Virtual image of window frame, 22b Virtual image of scenery, 22c Front person image,
22d Back person image, 23a Front desk image element, 23b Back desk image element,
24 composite image, 26 setting area

Claims (9)

A shooting process for shooting a subject;
A real image storage step of storing image data constituting the photographed real image;
Including information for specifying a virtual image to be combined with the real image based on information acquired by shooting, and information for specifying a setting area in which image data of the real image is set in advance. An information acquisition step of acquiring combined information for combining the real image and the virtual image;
An image compositing step for generating a composite image by arranging the image data of the real image in the specified setting area when compositing the real image and the virtual image based on the composite information;
An image synthesizing method characterized by comprising: The image composition method according to claim 1,
The composite information includes distance information indicating a distance from the viewpoint of shooting set in the virtual image and a distance from the viewpoint set in the setting area,
In the image compositing step, the image data of the real image is sequentially from the image data of the region in which the distance from the viewpoint is set longer as the distance information in the virtual image region and the setting region. An image composition method characterized by being rewritten. The image composition method according to claim 1 or 2,
The image data constituting the photographed real image is stored in two image storage areas,
In the image composition step, based on the composition information, one image storage area of the two image storage areas is used as a work area, and the image data of the stored real image is rewritten to generate a composite image,
When the image data of the real image is arranged in the setting area, the image data of the real image stored in the one image storage area is stored in the other image storage area. An image composition method characterized by rewriting data. A shooting process for shooting a subject,
A real image storage step for storing image data constituting the photographed real image;
Based on information acquired by photographing, information for specifying a virtual image to be combined with the real image and an image element included in the real image, and the photographing of the real image, the virtual image, and the image element For synthesizing the actual image and the virtual image, including distance information indicating a distance from the viewpoint of the image and information specifying a setting area in which image data of the actual image is set in advance. An information acquisition step of acquiring composite information of, and
An image synthesis step of generating a synthesized image by synthesizing the virtual image and the image element on the real image based on the synthesis information;
In the image synthesis step, based on the synthesis information, image data of the captured real image is image data having a shorter distance from the viewpoint of the image data of the virtual image and the image data of the image element. , And if the shorter distance from the viewpoint is set in the setting area, the image data of the actual image is written. The image composition method according to claim 4,
An image composition method, wherein the image element is set in the setting area. The image composition method according to claim 4 or 5, wherein:
The image data constituting the photographed real image is stored in two image storage areas,
In the image composition step, based on the composition information, one image storage area of the two image storage areas is used as a work area, and the image data of the stored real image is rewritten to generate a composite image,
When the image data of the real image is arranged in the setting area, the image data of the real image stored in the one image storage area is stored in the other image storage area. An image composition method characterized by rewriting data. The image composition method according to claim 6, comprising:
The composite image, the real image, and the virtual image are formed of a plurality of pixels,
The setting area is set for each pixel,
A distance information storage area for storing the distance information of each pixel in correspondence with each pixel of the one image storage area;
When the image data stored in the one image storage area is rewritten in the image composition step, the distance information in the distance information storage area is changed to the distance information of the rewritten image data. An image composition method characterized by being rewritten. The image composition method according to claim 7, wherein
An image synthesizing apparatus, wherein each pixel constituting the image data of the real image stored in the one image storage area has a distance from the viewpoint set to infinity. The image composition method according to claim 7 or 8,
In the image composition step, the distance information in the distance information storage area is compared for each pixel with the distance information set in the virtual image or the distance information set in the image element. An image composition method characterized by the above.