JP2010199898A - Image processor, imaging apparatus, image generation method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate three-dimensional images from captured images. <P>SOLUTION: A detection part 400 detects an imaging position and an imaging direction during the imaging operation of an imaging part 100. A control part 210 acquires numerical map data corresponding to the imaging position from numerical map data stored in a storage part 250, and generates three-dimensional modeling data to be the same angle as the captured image on the basis of the detected imaging direction. The control part 210 extracts an object image from the captured image on the basis of the generated three-dimensional modeling data, maps it on the three-dimensional modeling data and thus generates the three-dimensional image using the captured image. The control part 210 specifies a light source direction or the like during imaging, records it as a light illumination condition, and when changing a view point in the generated three-dimensional image, calculates a light illumination state on the basis of the light illumination condition and performs rendering. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing device, an imaging device, an image generation method, and a program, and more particularly, to an image processing device, an imaging device, an image generation method, and a program suitable for generating a three-dimensional image.

  Digital cameras that generate captured images using an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) are widely used.

  In a digital camera, since a captured image is obtained as digital data, various image processing can be performed during imaging (for example, Patent Document 1).

JP 11-355705 A

  However, in a conventional digital camera, a captured image is generated by forming a subject image obtained through a lens on a flat image sensor, and thus there is no distance information with respect to the subject. For this reason, it was not possible to generate a stereoscopic image (three-dimensional image) from an image captured by a general monocular digital camera.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus, an imaging apparatus, an image generation method, and a program that can generate a three-dimensional image from a captured image.

In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention provides:
Captured image acquisition means for acquiring a captured image generated by the imaging operation;
Image capturing time information acquisition means for acquiring position information and direction information indicating the position and direction during the image capturing operation, and recording the information in association with the captured image acquired by the captured image acquisition means;
Map information acquisition means for acquiring numerical map data of an imaging location based on the position information associated with the captured image;
Modeling that generates three-dimensional modeling data corresponding to the captured image based on the position information and the direction information associated with the captured image and the numerical map data acquired by the map information acquisition unit. Means,
3D image generation means for mapping the captured image to 3D modeling data generated by the modeling means to generate a 3D image;
It is characterized by providing.

In the image processing apparatus,
The three-dimensional image generation means includes
Object detection means for detecting an object constituting the captured image;
Object image extraction means for extracting an object image indicating the object detected by the object detection means from the captured image;
In this case,
It is preferable that the three-dimensional image is generated by mapping the object image extracted by the object image extraction unit to the three-dimensional modeling data.

In the image processing apparatus,
The three-dimensional image generation means includes
An illumination condition specifying means for specifying an illumination condition during the imaging operation;
It is desirable to further include image processing means for processing an image to be mapped to the three-dimensional modeling data based on the illumination condition specified by the illumination condition specifying means.

In order to achieve the above object, an imaging apparatus according to a second aspect of the present invention provides:
Position detecting means for detecting a position at the time of imaging;
Direction detection means for detecting the direction during imaging;
Storage means for storing position information indicating the position detected by the position detection means and direction information indicating the direction detected by the direction detection means in association with a captured image obtained by the imaging;
Map information acquisition means for acquiring numerical map data of an imaging location based on the position information associated with the captured image;
Based on the position information and the direction information associated with the captured image, and the numerical map data acquired by the map information acquisition unit, three-dimensional modeling data corresponding to the captured image is generated, and the 3 3D image generation means for mapping the captured image to 3D modeling data to generate a 3D image;
It is characterized by providing.

The imaging apparatus is
It is desirable to further comprise map information storage means for storing the numerical map data, in this case,
The map information acquisition means preferably acquires numerical map data from the map information storage means.

In the imaging apparatus,
Preferably, the three-dimensional image generation means detects and extracts an object constituting the captured image, maps the extracted object image to the three-dimensional modeling data, and generates the three-dimensional image.

In the imaging apparatus,
The three-dimensional image generation means preferably processes an image to be mapped to the three-dimensional modeling data based on illumination conditions at the time of imaging.

In order to achieve the above object, an image generation method according to a third aspect of the present invention includes:
An image generation method for an image processing apparatus to generate a three-dimensional image,
The image processing apparatus is
A captured image acquisition step of acquiring a captured image generated by the imaging operation;
An imaging time information acquisition step of acquiring position information indicating a position at the time of the imaging operation and direction information indicating a direction, and recording in association with the captured image acquired in the captured image acquisition step;
Based on the position information associated with the captured image, a map information acquisition step of acquiring numerical map data of the imaging location;
Modeling that generates three-dimensional modeling data corresponding to the captured image based on the position information and the direction information associated with the captured image and the numerical map data acquired in the map information acquisition step. Steps,
A 3D image generation step of generating a 3D image by mapping the captured image onto the 3D modeling data generated in the modeling step;
It is characterized by including.

In order to achieve the above object, a program according to the fourth aspect of the present invention is:
A computer that controls an image processing apparatus that performs image processing on a captured image obtained by the imaging apparatus;
A function of acquiring a captured image generated by the imaging operation;
A function of acquiring position information indicating a position at the time of the imaging operation and direction information indicating a direction, and recording the information in association with the acquired captured image;
Based on the position information associated with the captured image, a function of acquiring numerical map data of the imaging location;
A function of generating three-dimensional modeling data corresponding to the captured image based on the position information and the direction information associated with the captured image and the acquired numerical map data;
A function of mapping the captured image to the generated three-dimensional modeling data to generate a three-dimensional image;
Is executed.

In order to achieve the above object, a program according to the fifth aspect of the present invention is:
To a computer that controls an imaging apparatus provided with detection means for detecting the position and direction at the time of imaging,
A function of storing position information and direction information indicating the position and direction detected by the detection unit in a storage device in association with a captured image obtained by the imaging;
Based on the position information associated with the captured image, a function of acquiring numerical map data of the imaging location;
A function of generating three-dimensional modeling data corresponding to the captured image based on the position information and the direction information associated with the captured image, and the acquired numerical map data;
A function of mapping the captured image to the generated three-dimensional modeling data to generate a three-dimensional image;
It is characterized by realizing.

  According to the present invention, a three-dimensional image can be generated from a captured image.

It is a block diagram which shows the structure of the digital camera concerning embodiment of this invention. It is a functional block diagram which shows the function implement | achieved by the control part shown in FIG. It is a flowchart for demonstrating "3D image generation process at the time of imaging" concerning embodiment of this invention. It is a figure which shows the example of the image processed by the "3D image generation process at the time of imaging" shown in FIG. 3, (a) shows the example of a captured image, (b) shows the example of the generated modeling data. FIG. 4 is a diagram for explaining an object image extracted by the “3D image generation process during imaging” illustrated in FIG. 3, where (a) illustrates an example of a captured image, and (b) to (d) are extracted from the captured image. An example of the obtained object image is shown.

  Embodiments according to the present invention will be described below with reference to the drawings. In the present embodiment, a case where the present invention is realized by a digital still camera (hereinafter referred to as a digital camera) is illustrated.

  FIG. 1 is a block diagram showing a configuration of a digital camera 1 according to an embodiment of the present invention. The schematic configuration of the digital camera 1 according to the present embodiment includes an imaging unit 100, a data processing unit 200, an interface (I / F) unit 300, a detection unit 400, and the like as illustrated.

  The imaging unit 100 is a part that performs an imaging operation of the digital camera 1, and includes an optical device 110, an image sensor unit 120, and the like as shown in FIG.

  The optical device 110 includes, for example, a lens, a diaphragm mechanism, a shutter mechanism, and the like, and performs an optical operation related to imaging. In other words, the operation of the optical device 110 collects incident light and adjusts optical elements related to the angle of view, focus, exposure, and the like, such as focal length, aperture, and shutter speed. Note that the shutter mechanism included in the optical device 110 is a so-called mechanical shutter, and when the shutter operation is performed only by the operation of the image sensor, the optical device 110 may not include the shutter mechanism. The optical device 110 operates under the control of the control unit 210 described later.

  The image sensor unit 120 generates an electrical signal corresponding to the incident light collected by the optical device 110, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). (Semiconductor) image sensor. The image sensor unit 120 performs photoelectric conversion to generate an electrical signal corresponding to the received light and output it to the data processing unit 200.

  The data processing unit 200 processes the electrical signal generated by the imaging operation by the imaging unit 100 to generate digital data indicating the captured image, and performs image processing on the captured image. As shown in FIG. 1, the data processing unit 200 includes a control unit 210, an image processing unit 220, an image memory 230, an image output unit 240, a storage unit 250, an external storage unit 260, and the like.

  The control unit 210 includes, for example, a processor such as a CPU (Central Processing Unit), a main storage device (memory) such as a RAM (Random Access Memory), and the like. Each part of the digital camera 1 is controlled by executing the stored program. Further, in the present embodiment, by executing a predetermined program, a function related to each process described later is realized by the control unit 210.

  The image processing unit 220 includes, for example, an ADC (Analog-Digital Converter), a buffer memory, an image processing processor (so-called image processing engine), and the like, and is generated by the image sensor unit 120. Based on the electrical signal, digital data indicating the captured image is generated.

  That is, when the analog electric signal output from the image sensor unit 120 is converted into a digital signal by the ADC and sequentially stored in the buffer memory, the image processing engine performs so-called development processing on the buffered digital data. Adjust image quality and compress data.

  The image memory 230 includes, for example, a storage device such as a RAM or a flash memory, and temporarily stores captured image data generated by the image processing unit 220, image data processed by the control unit 210, and the like.

  The image output unit 240 includes, for example, an RGB signal generation circuit and the like, converts the image data expanded in the image memory 230 into an RGB signal and the like, and outputs the RGB signal to a display screen (a display unit 310 to be described later).

  The storage unit 250 includes a storage device such as a ROM (Read Only Memory) or a flash memory, and stores programs and data necessary for the operation of the digital camera 1. In the present embodiment, it is assumed that the operation program executed by the control unit 210 and the like, parameters and arithmetic expressions necessary for the processing are stored in the storage unit 250.

  The external storage unit 260 includes a storage device that can be attached to and detached from the digital camera 1 such as a memory card, and stores image data captured by the digital camera 1.

  The interface unit 300 has a configuration related to an interface between the digital camera 1 and its user or an external device. As shown in FIG. 1, the display unit 310, an external interface (I / F) unit 320, an operation unit 330, etc. Consists of

  The display unit 310 includes, for example, a liquid crystal display device, and displays and outputs various screens necessary for operating the digital camera 1, live view images at the time of shooting, captured images, and the like. In the present embodiment, display output of a captured image or the like is performed based on an image signal (RGB signal) from the image output unit 240.

  The external interface unit 320 includes, for example, a USB (Universal Serial Bus) connector, a video output terminal, and the like, and performs output of image data to an external computer device, display output of a captured image to an external monitor device, and the like. . In addition, the external interface unit 320 may include an interface for communicating with an external computer device or the like via a communication network, for example. In this case, for example, a wireless LAN module or the like may be included in the external interface unit 320 and may be connected to the Internet or the like by wireless communication with a wireless LAN access point.

  The operation unit 330 includes various buttons configured on the outer surface of the digital camera 1, generates an input signal corresponding to an operation by the user of the digital camera 1, and inputs the input signal to the control unit 210. As buttons constituting the operation unit 330, for example, a shutter button for instructing a shutter operation, a mode button for designating an operation mode of the digital camera 1, a cross key and a function button for performing various settings, and the like Is included.

  The detection unit 400 is for detecting the position and direction of the digital camera 1 at the time of imaging, and includes a position detection unit 410 and a direction detection unit 420 as shown in FIG.

  The position detection unit 410 is for detecting the position of the digital camera 1 at the time of an imaging operation, and includes, for example, a GPS (Global Positioning System) module. In this case, the position detection unit 410 acquires coordinate information indicating the position of the digital camera 1 by the GPS function.

  The direction detection unit 420 is for detecting the imaging direction at the time of the imaging operation, and is composed of, for example, a three-axis azimuth sensor including a geomagnetic sensor and an angular velocity sensor, and detects the imaging direction of the digital camera 1. In this case, for example, the azimuth (azimuth angle) in the imaging direction is detected by a geomagnetic sensor or the like, and the vertical angle (elevation angle) in the imaging direction is detected by an angular velocity sensor or the like. That is, the direction detection unit 420 detects the imaging direction of the digital camera 1 in a three-dimensional direction. In this case, it is assumed that the optical axis direction of the lens constituting the optical device 110 is set to be detected.

  In the digital camera 1 having such a configuration, the control unit 210 executes the operation program stored in the storage unit 250 to realize each process described later. In this case, the control unit 210 implements each process. The function will be described with reference to FIG.

  FIG. 2 is a functional block diagram illustrating functions realized by the control unit 210. Here, a functional configuration necessary for realizing an operation of generating a three-dimensional image from a captured image obtained by the imaging operation of the digital camera 1 is shown. In this case, as illustrated, the control unit 210 functions as an operation mode processing unit 211, an imaging control unit 212, a detection information acquisition unit 213, a map information acquisition unit 214, a three-dimensional image generation unit 215, and the like.

  The operation mode processing unit 211 cooperates with the display unit 310 to display screens necessary for the user of the digital camera 1 to specify various operation modes of the digital camera 1 and setting screen displays for each specified operation mode. In addition, the operation mode designated by the user is recognized in cooperation with the operation unit 330, and a program or an arithmetic expression necessary for executing the operation mode is read from the storage unit 250 to store the main memory of the control unit 210. Load to device (memory). In the present embodiment, it is assumed that an operation mode (hereinafter referred to as “3D mode”) for generating a three-dimensional image (3D image) from a captured image at the time of imaging with the digital camera 1 is prepared, and the selection of the 3D mode is performed. When recognized by the operation mode processing unit 211, the following functions are realized by the control unit 210.

  The imaging control unit 212 executes the imaging operation in the digital camera 1 by controlling the imaging unit 100 according to an input signal from the operation unit 330 when the shutter button is operated. Here, a captured image obtained by the imaging operation of the imaging unit 100 is developed in the image memory 230 after being processed by the image processing unit 220. The file format of the captured image is, for example, Exif (Exchangeable Image File Format). It is assumed that the parameters at the time of the imaging operation are recorded together with the image data of the captured image. In the present embodiment, it is assumed that at least a parameter indicating the angle of view of the captured image (for example, a lens focal length) is recorded.

  The detection information acquisition unit 213 controls the detection unit 400, operates the position detection unit 410 and the direction detection unit 420 during the imaging operation under the 3D mode, acquires detection information indicating the respective detection results, and acquires the captured image. Record in association with image data. In this case, for example, it is recorded as a part of parameters recorded in the Exif format. The detection information includes position information indicating the position (latitude / longitude) of the digital camera 1 detected by the position detection unit 410 at the time of imaging, and direction information indicating the imaging direction detected by the direction detection unit 420.

  The map information acquisition unit 214 acquires numerical map data corresponding to the specified imaging position based on the position information associated with the captured image. In the present embodiment, it is assumed that the numerical map data is stored in the storage unit 250 in advance, and the map information acquisition unit 214 acquires the numerical map data corresponding to the imaging position from the storage unit 250. The numerical map data used in the present embodiment includes numerical map data indicating a map image (for example, “numerical map 25000 (map image)” (Geographical Survey Institute)) or mesh data including altitude data ( For example, it is assumed that “numerical map 5 m mesh (elevation)” (Geographical Survey Institute) is included.

  The three-dimensional image generation unit 215 generates a three-dimensional image using the numerical map data acquired by the map information acquisition unit 214 and the captured image. In this case, the three-dimensional image generation unit 215 performs modeling data (three-dimensional) that has the same angle as the captured image based on the imaging position and imaging direction detected when the captured image is captured, the lens focal length indicated by the imaging parameter, and the like. Modeling data) is generated from the numerical map data, and the captured image is mapped to the modeling data as a mapping texture, thereby generating a three-dimensional image corresponding to the captured image. The three-dimensional image generation unit 215 sets illumination conditions in the generated three-dimensional image, and performs rendering according to the set illumination conditions.

  The above is the function realized by the control unit 210. In the present embodiment, each function described above is realized by a logical process performed by the control unit 210 executing a program. These functions are, for example, ASIC (Application Specific Integrated Circuit). Or an integrated circuit). In this case, among the functions shown in FIG. 2, the functions related to image processing may be realized by the image processing unit 220.

  The configuration of the digital camera 1 described above is a configuration necessary for realizing the present invention, and a configuration used for a basic function and various additional functions as a digital camera is provided as necessary. .

  The operation of the digital camera 1 having such a configuration will be described below. Here, “3D image generation processing during imaging” for generating a three-dimensional image from a captured image obtained by the imaging operation when the 3D mode is selected will be described with reference to the flowchart shown in FIG. This “3D image generation process during imaging” is started when the user of the digital camera 1 operates the operation unit 330 to set the operation mode of the digital camera 1 to the 3D mode. Note that the 3D image generation according to the present embodiment is preferably applied to a captured image obtained by outdoor shooting such as a landscape photograph. Therefore, in the present embodiment, 3D Assume that the mode is set.

  When the process is started, the operation mode processing unit 211 notifies the detection information acquisition unit 213 that the 3D mode has been set. When the input signal generated by the user operating the shutter button by the user of the digital camera 1 is input to the control unit 210 (step S101: Yes), the imaging control unit 212 controls the imaging unit 100 to perform the imaging operation. Let it run. A captured image is generated by the electrical signal generated by the imaging operation of the imaging unit 100 being processed by the image processing unit 220, and is developed in the image memory 230 (step S102).

  Here, at the same time when the shutter operation is detected, the imaging control unit 212 notifies the detection information acquisition unit 213 to that effect. In response to the notification from the imaging control unit 212, the detection information acquisition unit 213 controls the position detection unit 410 and the direction detection unit 420 of the detection unit 400, and the position and direction of the digital camera 1 at the time when the shutter operation is performed. To detect position information and direction information indicating the detection result (step S103).

  When the detection information acquisition unit 213 detects the position information and the direction information from the detection unit 400, the detection information acquisition unit 213 associates these with the captured image developed in the image memory 230 (step S104). Here, for example, position information and direction information are added as Exif data of a captured image.

  When the position information and the direction information indicating the position and direction at the time of imaging are acquired, the detection information acquisition unit 213 notifies the map information acquisition unit 214 to that effect. Here, for example, the map information acquisition unit 214 is notified of information (such as a file name) specifying the captured image.

  The map information acquisition unit 214 accesses the captured image developed in the image memory 230 in response to the notification from the detection information acquisition unit 213, and refers to the position information associated with the captured image. In the present embodiment, since position detection is performed by GPS, the recorded position information indicates the latitude and longitude of the digital camera 1 at the time of imaging. The map information acquisition unit 214 accesses the numerical map data stored in the storage unit 250, acquires the numerical map data corresponding to the referenced position information (latitude / longitude), and develops it in the memory of the control unit 210, for example. (Step S105).

  Here, since the altitude data is included in the numerical map data, the altitude at the point indicated by the specified latitude / longitude can also be specified, so the map information acquisition unit 214 determines the altitude specified in this way. The information is added to the position information associated with the captured image. That is, an imaging position (point) is indicated by position information indicating three-dimensional space coordinates.

  When the numerical map data of the imaging position is acquired, the map information acquisition unit 214 notifies the three-dimensional image generation unit 215 to that effect. In this case, the map information acquisition unit 214 notifies the three-dimensional image generation unit 215 of, for example, the acquired numerical map data and an address on the memory for specifying.

  The 3D image generation unit 215 performs modeling using the numerical map data notified from the map information acquisition unit 214. In this case, the three-dimensional image generation unit 215 generates modeling data (three-dimensional modeling data) by, for example, polygon modeling by performing modeling using a known technique for generating a three-dimensional image.

  In performing modeling, the three-dimensional image generation unit 215 uses the direction information associated with the captured image and the information indicating the angle of view recorded as the imaging parameter (such as the lens focal length) as a three-dimensional parameter. Modeling data is generated (step S106).

  In other words, the three-dimensional coordinates indicating the imaging position (point) are applied to the pseudo three-dimensional space reproduced by the numerical map data, and the viewpoint is set so as to obtain the angle of view at the time of imaging in the direction indicated by the direction information. Thus, modeling data having the same angle as the captured image is generated. For example, in the case of a captured image as shown in FIG. 4A, the modeling data generated from the position information, direction information, and angle-of-view parameters obtained at the time of imaging is shown in FIG. 4B. It becomes a thing of the same angle as such a captured image.

  Since modeling data generated in this way is generated from numerical map data, various objects in the modeling data (for example, natural objects, buildings, etc. that form terrain such as mountains, rivers, lakes, etc.) Artifacts that can be included as map information such as roads) can be identified. Here, as shown in FIG. 4, since modeling data having the same angle as the captured image is generated, the image size and resolution between the image obtained by projecting the modeling data on the two-dimensional plane and the captured image are determined. Can be processed in the same coordinate system by making them the same or performing coordinate transformation between images. For this reason, the part corresponding to the object recognized on modeling data can be specified on a captured image.

  When the three-dimensional image generation unit 215 specifies the object part on the captured image by such a method, the three-dimensional image generation unit 215 extracts an image region (object image) indicating each object (step S107). An example of the extracted object image is shown in FIG. As illustrated, object images as illustrated in FIGS. 5B to 5D are extracted from the captured image illustrated in FIG. Here, in the extraction of the object image, a known method (for example, flood fill or the like) that performs range selection based on luminance information or the like is used to select a range at the coordinate position of each object specified from the modeling data.

  When the object image is extracted from the captured image in this way, the three-dimensional image generation unit 215 performs mapping using the extracted object image as a mapping texture. That is, the object image extracted from the captured image in step S107 is mapped to the modeling data generated in step S106 (FIG. 4B) (step S108). Thereby, a 3D image (three-dimensional image) using the captured image is generated.

  Since the 3D image generated in this way is obtained by mapping the captured image to modeling data having the same angle as the captured image, the mapped image reflects the illumination state (light source direction, attenuation, etc.) at the time of imaging. It has been done.

  On the other hand, since it is generated by a general three-dimensional computer graphics technique of mapping a mapping texture onto modeling data, the viewpoint of the generated 3D image can be changed by changing various parameters.

  When the viewpoint is changed, the light source direction and the like also change, so a reference illumination condition is set. Here, the three-dimensional image generation unit 215 sets illumination conditions at the time of imaging, and records the illumination conditions in association with the generated 3D image (step S109).

  In the present embodiment, since a captured image obtained by photographing outdoors such as a landscape photograph is assumed, the light source is the sun. Therefore, the sun direction (azimuth angle and elevation angle) at the time of imaging is set as the illumination condition. The sun's direction can be obtained from the position on the earth (latitude / longitude), date, and time, so the position information (latitude / longitude) acquired in this process and the image that is normally recorded in Exif data, etc. Based on the date and time information, the sun direction at the time of imaging can be obtained.

  In the present embodiment, it is assumed that an arithmetic expression for calculating the sun direction from latitude / longitude, date, time, and time is stored in the storage unit 250 in advance, and the three-dimensional image generation unit 215 has a latitude indicated by the position information. By calculating the longitude and imaging date and time by applying this formula, the sun direction at the time of imaging is specified and set as an illumination condition.

  Here, in order for the user of the digital camera 1 to visually recognize the 3D image generated in step S <b> 108, the three-dimensional image generation unit 215 displays the generated 3D image on the display unit 310. In this case, the display unit 310 displays a 3D image and, for example, a slider bar that can be operated with the cross key of the operation unit 330. This slider bar is for instructing to change the viewpoint of the displayed 3D image. When the user desires to change the viewpoint, the slider bar is moved by operating the operation unit 330 to change the direction of the viewpoint change. Enter the amount of movement.

  When an input signal related to such an operation is input from the operation unit 330 to the control unit 210, the three-dimensional image generation unit 215 determines that a viewpoint change operation has been performed (step S110: Yes). In this case, the three-dimensional image generation unit 215 changes the parameters of the modeling data in accordance with the input viewpoint change direction and movement amount, and changes the modeling data from the specified viewpoint. A three-dimensional image is generated by mapping the object image extracted in step S107 to the changed modeling data. However, since the viewpoint position that was the imaging position at the beginning has changed, the illumination state has also changed. .

  Therefore, the three-dimensional image generation unit 215 performs rendering by changing the illumination parameter in the 3D image after changing the viewpoint based on the illumination condition set in step S109 (step S111). In this case, since the sun direction at the time of imaging is recorded as the illumination condition, the three-dimensional image generation unit 215 calculates the sun direction in the 3D image based on the imaging direction by the viewpoint change.

  In the extraction of the object image, since the object is recognized from the modeling data generated from the numerical map data, the distance from the viewpoint to the object in the three-dimensional coordinate space of the modeling data can be calculated. Therefore, the three-dimensional image generation unit 215 calculates the distance from the changed viewpoint to each object from the three-dimensional space coordinates of the modeling data.

  Thus, since the light source direction in the 3D image after the viewpoint change and the distance to each object are calculated, the three-dimensional image generation unit 215 determines the changed light source direction on the modeling data after the viewpoint change. By performing rendering using the distance to each object as a parameter, the illumination state for the image to be mapped is calculated.

  The three-dimensional image generation unit 215 processes the object image to be mapped according to the calculated illumination state. In this case, a mapping texture reflecting the illumination state after changing the viewpoint is obtained by changing the luminance of each object image.

  The three-dimensional image generation unit 215 generates a 3D image from the viewpoint designated by the user by mapping the object image processed in this way. Also in this case, since the object image extracted from the captured image is used as the mapping texture, a 3D image from an angle different from that at the time of imaging is generated while using the captured image.

  In this way, by changing the parameter according to an arbitrary viewpoint change instruction, a 3D image from any angle can be obtained. Therefore, the user can change the viewpoint as many times as necessary by operating a slider bar or the like displayed together with the 3D image.

  Here, the viewpoint can be changed until the user gives an instruction to save the generated 3D image (step S112: No). When the user instructs to save the 3D image by operating the operation unit 330 (step S112: Yes), the 3D image generation unit 215 stores the generated 3D image in the external storage unit 260 or the like. (Step S113). In this case, the illumination conditions set in step S109 are stored in association with the 3D image. The captured image developed in the image memory 230 is also stored in the external storage unit 260 in association with the 3D image together with the position information and the direction information associated with each other.

  In this way, for the captured image obtained by the imaging operation, the 3D image generated using the captured image and the captured image are stored in the external storage unit 260, and then a predetermined end event (for example, in the 3D mode). If there is no cancellation, etc. (step S114: No), the processing after step S101 is repeated, and a 3D image related to the next imaging operation is generated.

  Then, the process ends when an end event occurs (step S114: Yes).

  As described above, by applying the present invention as in the above embodiment, generation of a three-dimensional image using a captured image is realized.

  In this case, by obtaining information indicating the imaging position and the imaging direction together with the captured image, modeling with the same angle as the captured image can be performed using the numerical map data, and the captured image is mapped to the modeling data. As a result, it is possible to generate a three-dimensional image as if the captured image was stereoscopically displayed.

  Here, when mapping the captured image, it is desirable to map the object image extracted from the captured image. As a result, the shooting distance can be calculated for each object, so that the illumination state can be set for each object when the viewpoint is changed, and appropriate rendering is performed even when the angle is different from the actual captured image. Can do.

  In addition, when the sun is used as the light source, the light source direction in the captured image can be specified based on the position information and direction information acquired for the captured image, the imaging date and time, etc., so the illumination conditions at the time of imaging are specified. Thus, even when the viewpoint of the generated three-dimensional image is changed, it is possible to obtain a three-dimensional image appropriately illuminated.

  And it can produce | generate on the spot which imaged the three-dimensional image using the captured image by setting it as the imaging device carrying GPS and the sensor which detects the imaging position and imaging direction at the time of imaging.

  In this case, it is possible to generate a three-dimensional image in a shorter time by storing numerical map data used for generating a three-dimensional image in the imaging apparatus in advance.

  The said embodiment is an example and the application range of this invention is not restricted to this. That is, various applications are possible, and all embodiments are included in the scope of the present invention.

  For example, in the above embodiment, the numerical map data corresponding to the imaging position is acquired from the numerical map data stored in advance in the storage unit 250. However, the imaging apparatus can be connected to a communication network such as the Internet. In this case, the numerical map data may be acquired from an external device by communication via a communication network.

  In the case where the present invention is realized by an imaging apparatus such as the digital camera 1 exemplified in the above embodiment, the configuration and functions according to the present invention can be provided as an imaging apparatus, and each function of the control unit 210 can be provided. By applying a program that realizes the same function as the above, it is possible to make an existing imaging device function as the imaging device according to the present invention. In this case, if the imaging apparatus has a function of detecting the imaging position and imaging direction at the time of imaging, the application of the present invention can generate a three-dimensional image using the captured image.

  Moreover, although the case where this invention was implement | achieved as an imaging device was illustrated in the said embodiment, you may implement | achieve the image processing apparatus of this invention as an apparatus (for example, personal computer etc.) without an imaging function.

  That is, in the above-described embodiment, the case where the present invention is realized as an imaging device is illustrated, but any device other than the imaging device can be used as long as information indicating the imaging position and imaging direction at the time of imaging can be acquired together with the captured image. The present invention can be applied to.

  Also in this case, by applying the program, an existing apparatus can be made to function as the image processing apparatus according to the present invention.

  The application method of such a program is arbitrary. For example, the program can be applied by being stored in a storage medium such as a CD-ROM or a memory card, or can be applied via a communication medium such as the Internet.

DESCRIPTION OF SYMBOLS 1 ... Digital camera, 100 ... Imaging part, 110 ... Optical apparatus, 120 ... Image sensor part, 200 ... Data processing part, 210 ... Control part, 211 ... Operation mode processing part, 212 ... Imaging control part, 213 ... Acquisition of detection information , 214 ... Map information acquisition unit, 215 ... 3D image generation unit, 220 ... Image processing unit, 230 ... Image memory, 240 ... Image output unit, 250 ... Storage unit, 260 ... External storage unit, 300 ... Interface unit, DESCRIPTION OF SYMBOLS 310 ... Display part, 320 ... External interface part, 330 ... Operation part, 400 ... Detection part, 410 ... Position detection part, 420 ... Direction detection part

Claims (10)

Captured image acquisition means for acquiring a captured image generated by the imaging operation;
Image capturing time information acquisition means for acquiring position information and direction information indicating the position and direction during the image capturing operation, and recording the information in association with the captured image acquired by the captured image acquisition means;
Map information acquisition means for acquiring numerical map data of an imaging location based on the position information associated with the captured image;
Modeling that generates three-dimensional modeling data corresponding to the captured image based on the position information and the direction information associated with the captured image and the numerical map data acquired by the map information acquisition unit. Means,
3D image generation means for mapping the captured image to 3D modeling data generated by the modeling means to generate a 3D image;
An image processing apparatus comprising: The three-dimensional image generation means includes
Object detection means for detecting an object constituting the captured image;
Object image extraction means for extracting an object image indicating the object detected by the object detection means from the captured image;
Further comprising
Mapping the object image extracted by the object image extraction means to the three-dimensional modeling data to generate the three-dimensional image;
The image processing apparatus according to claim 1. The three-dimensional image generation means includes
An illumination condition specifying means for specifying an illumination condition during the imaging operation;
Image processing means for processing an image to be mapped to the three-dimensional modeling data based on the illumination condition specified by the illumination condition specifying means;
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Position detecting means for detecting a position at the time of imaging;
Direction detection means for detecting the direction during imaging;
Storage means for storing position information indicating the position detected by the position detection means and direction information indicating the direction detected by the direction detection means in association with a captured image obtained by the imaging;
Map information acquisition means for acquiring numerical map data of an imaging location based on the position information associated with the captured image;
Based on the position information and the direction information associated with the captured image, and the numerical map data acquired by the map information acquisition unit, three-dimensional modeling data corresponding to the captured image is generated, and the 3 3D image generation means for mapping the captured image to 3D modeling data to generate a 3D image;
An imaging apparatus comprising: Map information storage means for storing the numerical map data;
The map information acquisition means acquires numerical map data from the map information storage means;
The imaging apparatus according to claim 4. The three-dimensional image generation means detects and extracts an object constituting the captured image, maps the extracted object image to the three-dimensional modeling data, and generates the three-dimensional image.
The imaging apparatus according to claim 4 or 5, wherein The three-dimensional image generation means processes an image to be mapped to the three-dimensional modeling data based on illumination conditions at the time of imaging.
The image pickup apparatus according to claim 4, wherein the image pickup apparatus is an image pickup apparatus. An image generation method for an image processing apparatus to generate a three-dimensional image,
The image processing apparatus is
A captured image acquisition step of acquiring a captured image generated by the imaging operation;
An imaging time information acquisition step of acquiring position information indicating a position at the time of the imaging operation and direction information indicating a direction, and recording in association with the captured image acquired in the captured image acquisition step;
Based on the position information associated with the captured image, a map information acquisition step of acquiring numerical map data of the imaging location;
Modeling that generates three-dimensional modeling data corresponding to the captured image based on the position information and the direction information associated with the captured image and the numerical map data acquired in the map information acquisition step. Steps,
A 3D image generation step of generating a 3D image by mapping the captured image onto the 3D modeling data generated in the modeling step;
An image generation method comprising: A computer that controls an image processing apparatus that performs image processing on a captured image obtained by the imaging apparatus;
A function of acquiring a captured image generated by the imaging operation;
A function of acquiring position information indicating a position at the time of the imaging operation and direction information indicating a direction, and recording the information in association with the acquired captured image;
Based on the position information associated with the captured image, a function of acquiring numerical map data of the imaging location;
A function of generating three-dimensional modeling data corresponding to the captured image based on the position information and the direction information associated with the captured image and the acquired numerical map data;
A function of mapping the captured image to the generated three-dimensional modeling data to generate a three-dimensional image;
A program characterized by having executed. In a computer that controls an imaging apparatus provided with a detecting means for detecting a position and a direction at the time of imaging,
A function of storing position information and direction information indicating the position and direction detected by the detection unit in a storage device in association with a captured image obtained by the imaging;
Based on the position information associated with the captured image, a function of acquiring numerical map data of the imaging location;
A function of generating three-dimensional modeling data corresponding to the captured image based on the position information and the direction information associated with the captured image, and the acquired numerical map data;
A function of mapping the captured image to the generated three-dimensional modeling data to generate a three-dimensional image;
A program characterized by realizing.

Images