JPWO2019026919A1 - Image processing system, image processing method, and program - Google Patents

Abstract

Image processing that can generate data that causes an event according to a user's operation or action when displaying an image using an image obtained by shooting a subject as a material without requiring more time and effort than before Provide the system etc. The image processing system includes a photographing unit that generates image data of a moving image or a still image by photographing a subject, and, in response to a predetermined trigger, provides identification information to a frame or a still image included in the moving image generated by the photographing unit. Based on the marking processing unit to give, the image data of the moving image or the still image generated by the photographing unit, the image processing unit that generates the image data of the moving image or the still image, and the image processing unit generated. An image data storage unit that stores image data.

Description

  The present invention relates to an image processing system, an image processing method, and a program for displaying an image obtained by shooting a subject.

  2. Description of the Related Art In recent years, in business fields such as tourism and real estate, a service is provided in which an image obtained by photographing a tourist spot or a real estate property is processed using various techniques and is browsed by a customer. For example, an image (also referred to as a celestial sphere image) showing a subject at 360 ° around the shooting point is displayed on a glasses-type or goggles-type display device called a head mounted display (hereinafter, also referred to as “HMD”), By changing the display range of the image in accordance with the movement of the head of the user wearing the HMD, the user can feel as if he / she entered the space of the displayed image. As described above, the technique of making a virtual but real-like feeling experience is also called virtual reality (VR), and a virtual space recognized in a three-dimensional (3D) manner by VR is VR. Also called space.

  As a technique related to VR, for example, Patent Document 1 includes a city VR display unit that displays three-dimensional city VR information and a map display unit that displays two-dimensional map information, and the viewpoint position in the city VR space. There is disclosed a display device that displays information related to a city VR space in real time by moving information on the map and a viewpoint on the displayed map.

  Further, Patent Document 2 discloses an image reproducing method for reproducing a spherical omnidirectional moving image captured by a spherical omnidirectional camera by an image reproducing device having a gyro sensor, and a viewer's line of sight during the reproduction. Accordingly, an image reproduction method for activating an event corresponding to a video at the gaze point is disclosed.

JP 2002-351309 A JP, 2016-134742, A

  By the way, when an image for expressing a 3D space is to be created by a CG (computer graphics) technique, image processing such as rendering, which requires a high calculation cost, must be performed. Therefore, a high-specification dedicated device capable of performing such image processing is required.

  On the other hand, there is also known a technology that expresses a 3D space in a pseudo manner by processing an image using a still image or a moving image obtained by shooting a subject as a material and stereoscopically utilizing the parallax of both eyes of a person. Has been. Alternatively, there is also known a technique in which a two-dimensional image is displayed (panoramic display) as if it is arranged on a spherical surface so that the user feels as if he or she were surrounded by a 3D space. When such a technique is used, the calculation cost required for processing an image can be significantly suppressed.

  However, in order to display the 3D space or the panorama display on the HMD or the like and generate some event in accordance with the operation performed or the user's operation during that, specific information must be embedded in the image data in advance. Therefore, when a still image or a moving image obtained by shooting a subject is used as a material, an editing work for embedding necessary information in the image data is required after shooting, which requires a lot of time and work cost. Will end up.

  The present invention has been made in view of the above, and requires less time and effort than before, and an image obtained by shooting a subject is used as a material for operation and operation by a user when displaying an image. An object is to provide an image processing system, an image processing method, and a program capable of creating data that causes a corresponding event.

  In order to solve the above-mentioned problems, an image processing system according to an aspect of the present invention includes a photographing unit that generates image data of a moving image or a still image by photographing a subject, and the photographing unit according to a predetermined trigger. A marking processing unit that adds identification information to a frame or a still image included in the moving image generated by the above, and a moving image or a still image for display based on the image data of the moving image or the still image generated by the photographing unit. And an image data storage unit for storing the image data generated by the image processing unit.

  The image processing system includes a marking list storage unit that stores one or more types of identification information, and an operation input unit that selects one identification information from the one or more types of identification information according to an operation performed from the outside. Further, the marking processing unit may add the identification information selected by the operation input unit to the frame or the still image generated by the imaging unit at a timing according to the trigger.

  The image processing system further includes a position detection unit that detects the position of the imaging unit and outputs position information, and the marking processing unit is the position information output from the position detection unit at a timing according to the trigger. May be added as the identification information to the frame or the still image generated by the imaging unit at a timing according to the trigger.

  In the image processing system described above, the trigger may be an operation on a predetermined operation unit performed while shooting a moving image.

  In the image processing system described above, the trigger may be an operation on an operation unit that is operated when capturing a still image.

  The image processing system includes an external display unit capable of displaying a moving image or a still image based on the image data stored in the image data storage unit, a display control unit controlling image display on the display unit, and an external display unit. The display control unit further includes an operation detection unit configured to detect an operation to be performed, and the identification information is added to the display control unit when the identification information is added to a frame or a still image of a moving image being displayed on the display unit. Information indicating that the information is displayed on the display unit, and when the operation detection unit detects an operation of selecting the information, the image display on the display unit is added to the frame or still image of the moving image being displayed. You may switch to the frame of a moving image or the still image to which the same identification information as the identification information currently provided is added.

  In the image processing system, the image data storage unit stores image data of a plurality of moving images, each of the plurality of moving images includes a frame to which the same identification information is added, and the display control unit includes a plurality of the plurality of moving images. The image display on the display unit may be switched between the moving images.

  In the image processing system, the moving image includes a plurality of frames that are generated at different times and are provided with the same identification information, and the display control unit displays an image on the display unit between the plurality of frames. You may switch.

  In the image processing system, the moving image includes a frame having the same identification information as the identification information given to the still image, and the display control unit has the same identification information as the still image. The image display on the display unit may be switched between frames.

  In the above image processing system, the image processing unit may further construct image data of a three-dimensional model based on the image data of the moving image or still image generated by the photographing unit, and perform the marking process. Based on the identification information given to the moving image or the still image by the unit, the information indicating the position is given to each place of the model, the display control unit, the information given to each place of the model, The image display on the display unit may be switched between the model and the moving image or the still image based on the identification information given to the moving image or the still image.

  The image processing system includes a second image processing unit that generates image data of a moving image or a still image displayed three-dimensionally based on the image data of the moving image or the still image generated by the photographing unit. You may prepare further.

  An image processing method according to another aspect of the present invention includes a step (a) of generating image data of a moving image or a still image by photographing a subject, and a moving image generated by the photographing unit according to a predetermined trigger. Image data of a moving image or a still image for display based on the image data of the moving image or the still image generated in step (a) And a step (d) of storing the image data generated in step (c).

  A program that is still another aspect of the present invention includes a photographing unit that generates image data of a moving image or a still image by photographing a subject, a storage unit, and an operation input unit that receives an operation performed from the outside. A program to be executed by an imaging device, wherein the storage section stores in advance one or more types of identification information, and one identification information is selected from the one or more types of identification information according to an operation on the operation input section. And a step of assigning the one piece of identification information to the frame or still image generated by the photographing unit at a timing according to a predetermined trigger.

  A program according to yet another aspect of the present invention includes a photographing unit that generates image data of a moving image or a still image by photographing a subject, and a position detection unit that detects the position of the photographing unit and outputs position information. And a step of causing the position detection unit to output position information at a timing according to a predetermined trigger, and the program generated by the imaging unit at a timing according to the trigger. The step of adding the position information as identification information to a frame or a still image is executed.

  A program that is still another aspect of the present invention includes a display unit that can display a moving image or a still image, a display control unit that controls image display on the display unit, and an operation detection unit that detects an operation performed from the outside. A program to be executed by a display device including, when predetermined identification information is added to a frame or still image of a moving image being displayed on the display unit, the predetermined identification information is added Is displayed on the display unit, and when the operation detection unit detects an operation of selecting the information, the image display on the display unit is added to the frame or still image of the moving image being displayed. And a step of switching to a frame of a moving image or a still image to which the same identification information as the predetermined identification information is added.

  According to the present invention, identification information is added to a moving image or a still image while shooting a subject, and image data of the moving image or the still image for display is generated based on the image data of the moving image or the still image. Therefore, it is not necessary to separately perform the editing work for embedding necessary information in the image data. Therefore, it is possible to create data for generating an event corresponding to a user's operation or action at the time of displaying an image, using an image obtained by shooting a subject as a material without requiring more time and effort than before. It will be possible.

It is a block diagram which shows schematic structure of the image processing system which concerns on the 1st Embodiment of this invention. It is a schematic diagram which shows the state which mounted the display apparatus shown in FIG. 1 on the user. It is a schematic diagram which shows an example of the screen displayed on the display part of the display device shown in FIG. 3 is a flowchart showing an operation of the image capturing apparatus according to the first embodiment of the present invention. It is a schematic diagram for demonstrating an example of the photography procedure of a real estate property. It is a schematic diagram which illustrates the screen displayed on the display part of an imaging device. It is a schematic diagram which illustrates the screen displayed on the display part of an imaging device. It is a schematic diagram which illustrates the screen displayed on the display part of an imaging device. It is a schematic diagram which illustrates the screen displayed on the display part of an imaging device. It is a schematic diagram for demonstrating another example of the photography procedure of a real estate property. It is a schematic diagram for demonstrating another example of the photography procedure of a real estate property. FIG. 9 is a schematic diagram for explaining an example of an image switching procedure in the display device. FIG. 9 is a schematic diagram for explaining an example of an image switching procedure in the display device. FIG. 9 is a schematic diagram for explaining an example of an image switching procedure in the display device. 3 is a flowchart showing the operation of the display device shown in FIG. 1. It is a schematic diagram which shows the example of a display of the image on a display apparatus. It is a schematic diagram which shows the example of a display of the image on a display apparatus. It is a schematic diagram which shows the example of a display of the image on a display apparatus. 7 is a timing chart showing display switching timing of images in the display device. 7 is a timing chart showing display switching timing of images in the display device. 7 is a timing chart showing display switching timing of images in the display device. 7 is a timing chart showing display switching timing of images in the display device. 7 is a flowchart showing the operation of the display device in the modification of the first embodiment of the present invention. It is a schematic diagram which shows the example of a display in the display device in the modification of the 1st Embodiment of this invention. 8 is a flowchart showing an operation of the image capturing apparatus according to the second embodiment of the present invention. It is a schematic diagram which illustrates the screen displayed on the display part of an imaging device. It is a flowchart which shows operation | movement of the imaging device in the 3rd Embodiment of this invention. It is a schematic diagram which shows the example of a display of a 3D model. It is a flowchart which shows operation | movement of the display apparatus in the 3rd Embodiment of this invention. It is a block diagram which shows schematic structure of the image processing apparatus which concerns on the 4th Embodiment of this invention.

  Hereinafter, an image processing system, an image processing method, and a program according to embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to these embodiments. In the description of each drawing, the same parts are denoted by the same reference numerals.

(First embodiment)
FIG. 1 is a block diagram showing a schematic configuration of an image processing system according to the first embodiment of the present invention. The image processing system 1 according to the present embodiment is a system that generates an image by shooting a subject and virtually displays a three-dimensional space using these images.

  In the following, as an object (subject) to be imaged by the image processing system 1, the inside of a real estate such as an individual house or office building is exemplified, but the example of the subject is not limited to this. For example, a subject that can be photographed by a general digital optical camera, such as a tourist spot, an unexplored area, a large furniture store, or a model of a building, can be a subject in the image processing system 1.

  As shown in FIG. 1, the image processing system 1 includes a photographing device 10 that photographs a subject to generate image data, a server 20 that stores the image data, and a display device 30 that displays an image of the subject. The imaging device 10, the server 20, and the display device 30 are connected via a communication network N.

  As the communication network N, for example, an Internet line, a telephone line, a LAN, a dedicated line, a mobile communication network, a communication line such as WiFi (Wireless Fidelity), Bluetooth (registered trademark), or a combination thereof is used. The communication network N may be wired, wireless, or a combination thereof.

  The image capturing apparatus 10 is a device that captures an image of a subject and generates image data, and can be realized by a digital camera, a smartphone in which a dedicated application is installed, or the like. In this embodiment, a smartphone is used as the image capturing device 10. Specifically, the image capturing device 10 includes a communication unit 11, an image capturing unit 12, an operation input unit 13, a display unit 14, a position detecting unit 15, a storage unit 16, and a processor 17.

  The communication unit 11 connects the imaging device 10 to the communication network N and transmits / receives information to / from other devices connected to the communication network N. The communication unit 11 is configured by using, for example, a soft modem, a cable modem, a wireless modem, an ADSL modem or the like.

  Here, when a digital camera is used as the photographing device 10, a communication function may not be provided depending on the model. In such a case, the image data generated in the image capturing apparatus 10 may be temporarily transferred to a personal computer (PC) via a cable, and the image data may be uploaded from the PC to the server 20 or the like via the communication network N. Good. Alternatively, the image data may be directly transferred from the photographing device 10 to the display device 30 via a cable.

  The image capturing unit 12 is a camera built in a smartphone and capable of capturing still images and moving images. By attaching a commercially available wide-angle lens (or an ultra-wide-angle lens) or a fish-eye lens to the imaging unit 12, it is possible to acquire an image with a wide angle of view (for example, 180 ° or more).

  Alternatively, as the image capturing unit 12, a camera that is externally attached to the smartphone by wire or wireless may be used. For example, by using a so-called omnidirectional camera equipped with two ultra-wide-angle lenses with an angle of view of 180 ° or more as the photographing unit 12, a celestial sphere image in which a subject of 360 ° around the photographing device 10 is captured is obtained. can do.

The operation input unit 13 is an input device such as an operation button or a touch sensor provided on the display unit 14, receives an operation performed from the outside, and inputs a signal corresponding to the operation to the processor 17.
The display unit 14 is a display including a display panel and a drive unit formed of liquid crystal or organic EL (electroluminescence).
The position detection unit 15 includes a GPS module, detects the position of the imaging device 10, and outputs position information.

  The storage unit 16 is a computer-readable storage medium such as a semiconductor memory such as a ROM or a RAM. The storage unit 16 stores, in addition to the operating system program and the driver program, application programs that execute various functions, various parameters used during the execution of these programs, and the like. Specifically, the storage unit 16 includes a program storage unit 161, which stores various programs, an image data storage unit 162, which stores image data, and a marking list storage unit 163, which stores a marking list.

  Here, the marking list is a list of the names of tags added to the image data as identification information for identifying the content of the image and recorded. For example, when photographing a real estate property, "entrance", "kitchen", "bedroom", etc. are used as the tag names.

  The processor 17 is configured by using, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and reads various programs stored in the program storage unit 161 to comprehensively control each unit of the imaging apparatus 10. At the same time, various processes for displaying the image captured by the image capturing unit 12 on the display device 30 are executed. Specifically, the functional units executed by the processor 17 include a photographing control unit 171, an image processing unit 172, a marking list registration unit 173, and a marking processing unit 174.

The photographing control unit 171 controls the photographing operation in the photographing unit 12 according to the signal input from the operation input unit 13.
The image processing unit 172 generates image data for display by performing image processing such as demosaicing, white balance processing, and gamma correction on the image data generated by the photographing unit 12.

  The marking list registration unit 173 registers the name of the tag added as identification information to the image data generated by the photographing unit 12 in the marking list according to the signal input from the operation input unit 13.

  The marking processing unit 174 executes a process (hereinafter referred to as a marking process) of adding a tag selected from tags registered in the marking list to the image data by using a signal input from the operation input unit 13 as a trigger. .

  Such an image capturing apparatus 10 captures a subject to generate image data of a still image or a moving image, adds a tag to the image data according to a predetermined trigger, and stores the image data in a predetermined file format. 162. The image files stored in the image data storage unit 162 are transmitted to the server 20 by a user operation.

  The server 20 is composed of a host computer having a high calculation processing capacity, and a server function is exhibited by a predetermined server program operating in the host computer. More specifically, the server 20 connects the server 20 to the communication network N and controls the operation of the server 20 and a communication unit 21 that transmits and receives information to and from other devices connected to the communication network N. Control unit 22, an image data storage unit 23 that stores image data (image files of moving images and still images) uploaded from the image capturing apparatus 10, and information about a real estate property that is a subject (such as real estate property name and location). Property information database 24 for storing information).

  The processor 22 may include an image processing unit 221 that performs predetermined image processing on the image data stored in the image data storage unit 23. Specifically, the image processing unit 221 performs a process of generating image data of a moving image or a still image that is displayed three-dimensionally based on the image data of the moving image or the still image uploaded from the photographing device 10. . Specifically, the image data for stereoscopic viewing using parallax is generated. However, such image processing does not necessarily have to be performed in the server 20, and may be performed in the image capturing device 10, the display device 30 described later, or a separately provided image processing computer.

  The server 20 performs image processing on the image uploaded from the imaging device 10 as necessary, stores the image in association with the property information, and transmits the image file to the display device 30 according to a request from the display device 30. .

  The display device 30 is a device that causes a user to recognize a three-dimensional virtual space by displaying a two-dimensional still image or moving image. As the display device 30, a goggle-type dedicated device (so-called head mounted display (HMD)) that is directly worn on the user's head may be used, or a dedicated smartphone installed with a dedicated application may be used. It may be attached to a goggle type holder for use.

  The display device 30 includes a communication unit 31, an operation input unit 32, a display unit 33, an orientation / posture detection unit 34, a storage unit 35, and a processor 36. The hardware configurations of the communication unit 31, the operation input unit 32, the display unit 33, the storage unit 35, and the processor 36 are the communication unit 11, the operation input unit 13, the display unit 14, the storage unit 16, and the processor of the imaging device 10. The same as 17 respectively.

  The orientation / posture detection unit 34 includes various sensors such as a gyro sensor and an acceleration sensor, and detects the orientation and attitude (inclination with respect to the vertical axis) of the display device 30.

  The storage unit 35 stores a program storage unit 351 that stores various programs and various parameters used during execution of the programs, and image data storage that stores image data (image files of moving images and still images) downloaded from the server 20. And part 352.

  The processor 36 comprehensively controls each unit of the display device 30 by reading various programs stored in the program storage unit 351, and causes the display unit 33 to display the image data stored in the image data storage unit 352. Performs various processes for The function unit executed by the processor 36 includes a display control unit 361 and an operation detection unit 362.

  The display control unit 361 causes the display unit 33 to display a moving image or a still image based on the image data stored in the image data storage unit 352. Further, the display control unit 361 performs control such as changing the range of the image displayed on the display unit 33 and control for switching the image displayed on the display unit 33 based on the detection result of the operation detection unit 362.

  The operation detection unit 362 detects an operation performed by the user wearing the display device 30 according to a signal input from the operation input unit 32 and a detection signal output from the orientation / posture detection unit 34. For example, the operation detection unit 362 detects the orientation of the head of the user who wears the display device 30 based on the detection signal output from the orientation and orientation detection unit 34, so that the user can view the image within the image being displayed. It is determined which of the objects the eyes are directed to, or whether the objects are directed toward the object for a predetermined time or more.

  FIG. 2 is a schematic diagram showing a state in which the user is wearing the display device 30. FIG. 3 is a schematic diagram showing an example of a screen displayed on the display unit 33. When a smartphone is used as the display device 30, the user can view the display unit 33 hands-free by mounting the display unit 33 on the holder 37 with the side provided with the display unit 33 facing the user. Inside the holder 37, two lenses are attached at positions corresponding to the left and right eyes of the user. When allowing the user to stereoscopically view an image, as shown in FIG. 3, two images with parallax are displayed in two areas 33a and 33b in the display unit 33, respectively. By viewing the areas 33a and 33b through the two lenses in the holder 37, the user can three-dimensionally recognize the images reflected in the areas 33a and 33b.

Next, the operation of the image processing system 1 will be described.
FIG. 4 is a flowchart showing the operation of the image capturing apparatus 10 according to the first embodiment of the present invention. FIG. 5 is a schematic diagram for explaining an example of a shooting procedure according to the first embodiment of the present invention. 6 to 9 are schematic views illustrating screens displayed on the display unit 14 of the image capturing apparatus 10. Of these, FIG. 6 shows a marking list registration screen, FIG. 7 shows a shooting mode selection screen, FIG. 8 shows a moving image shooting screen, and FIG. 9 shows a still image shooting screen. 10A and 10B are schematic diagrams for explaining another example of the photographing procedure by the photographing device 10.

  In the following, as an example, it is assumed that the property (one room in the apartment) shown in FIG. 5 is photographed. The order of shooting is not particularly limited, but here, the user shoots a moving image four times while moving along the moving image shooting routes R1 to R4, and shoots a still image 11 times at the still image shooting points P1 to P11. And

  By activating the image capturing unit 12 in the image capturing apparatus 10, the marking list registration screen shown in FIG. 6 is displayed on the display unit 14, and a series of image capturing operations described below is started.

  First, in step S100, the imaging device 10 registers a marking list according to a signal input from the operation input unit 13. In detail, for example, when photographing the real estate property shown in FIG. 5, the user taking the photograph refers to the floor plan, and enters the name of each room (for example, the entrance, the kitchen, the bedroom) in the input field 14a shown in FIG. Etc.) in the text and tap the OK button 14b. When detecting the operation on the OK button 14b, the imaging apparatus 10 registers the text input in the input field 14a in the marking list 14c as the tag name. The text input as the tag name is not limited to the room name. In short, any information that allows the user himself / herself to photograph can identify which position (portion) of the subject (real estate property) is photographed.

  In subsequent step S101, the image capturing apparatus 10 determines whether or not the shooting mode is set to moving image. Specifically, on the shooting mode selection screen shown in FIG. 7, it is determined which of “photograph (still image)” and “moving image” is selected from the icons 14e indicating the shooting mode. When the "moving image" icon is selected, the shooting mode is set to moving image (step S101: Yes), and the process proceeds to step S102.

  In step S102, the imaging device 10 determines whether or not a selection operation has been performed on the marking list 14c. Specifically, it is determined whether or not a predetermined operation (for example, tap operation) is performed on any of the tag names displayed in the marking list 14c shown in FIG.

  When the selection operation on the marking list 14c is not detected (step S102: No), the process returns to step S101. On the other hand, when the selection operation on the marking list 14c is detected (step S102: Yes), the imaging device 10 registers the selected tag as a tag at the start of moving image shooting (step S103).

  Note that the tag registration at the start of moving image shooting may be manually performed immediately after the start of moving image shooting (see steps S105 to S107 described later). In that case, steps S102 and S103 may be omitted.

  In subsequent step S104, the image capturing apparatus 10 starts capturing a moving image in response to a predetermined operation on the operation input unit 13 (e.g., operation on a capture button). As a result, the moving image shooting screen illustrated in FIG. 8 is displayed on the display unit 14, and the image 14d captured by the shooting unit 12 is recorded as a moving image. The user moves inside the real estate property while shooting a video.

  In step S105, the imaging device 10 determines whether or not a selection operation has been performed on the marking list 14c (step S105). Specifically, it is determined whether or not a predetermined operation (for example, tap operation) is performed on any of the tag names displayed in the marking list 14c shown in FIG.

  When the selection operation on the marking list 14c is not detected (step S105: No), the imaging device 10 stands by as it is. On the other hand, when the selection operation on the marking list 14c is detected (step S105: Yes), the imaging apparatus 10 subsequently determines whether or not an operation (for example, a tap operation) on the registration button 14f is detected (step S106). . FIG. 8 shows a state where the tag “bedroom” is selected. When the operation on the registration button 14f is not detected (step S106: No), the process returns to step S105.

  On the other hand, when the operation on the registration button 14f is detected (step S106: Yes), the imaging device 10 performs the marking process (step S107). That is, the tag selected in step S105 is added to the image data of the frame generated at the timing when the registration button 14f is operated. The frame to which the tag is added may be only a frame generated while the registration button 14f is operated, or a frame in a predetermined range including the frame generated while the registration button 14f is operated. It may be.

  The user performs marking processing (selection and addition of tags) at any time while shooting a moving image. As a result, the selected tags T1 to T8 are added to the frames captured at a plurality of points on the moving image capturing route R1.

  Here, the user can arbitrarily select the tag to be subjected to the marking process from the marking list. Therefore, the same tag may be given to frames that form different moving images or to frames that have different shooting positions (time) even in the same moving image. For example, as shown in FIG. 5, a tag T3 (for example, a “corridor” tag) is selected in the moving image shooting route R1 to perform marking processing, and the same tag T3 is selected in another moving image shooting route R4 to perform marking processing. By doing so, a frame with the same tag T3 exists in the two moving images. In this way, the frames to which the same tag T3 is attached are linked with the attached tag T3 as a key.

  In subsequent step S108, the imaging device 10 determines whether or not a moving image shooting end operation (for example, a shooting button re-pressing operation) has been detected. When the ending operation of moving image shooting is not detected (step S108: No), the process returns to step S105. On the other hand, when the end operation of moving image shooting is detected (step S108: Yes), the process proceeds to step S109 described later. For example, as shown in FIG. 5, when the shooting of the moving image shooting route R1 is finished and the next moving image shooting route R2 is taken, the moving image shooting is temporarily stopped at the end point (living room) of the moving image shooting route R1 (step S108). : Yes), moving to the start point (entrance of the kitchen) of the video shooting route R2 and restarting video shooting (step S109: No, S101: Yes). The order of shooting the moving images is not limited to this, and the shooting may be performed in any order.

  In step S109, the imaging device 10 determines whether to end the shooting operation of the moving image or the still image. For example, when the power of the image capturing apparatus 10 is turned off or a predetermined ending operation is performed, the image capturing operation ends as it is (step S109: Yes). On the other hand, when the shooting operation is not finished (step S109: No), the process returns to step S101.

  In step S101, when the shooting mode is not set to the moving image (step S101: No), that is, when the shooting mode is the still image mode, the still shooting screen illustrated in FIG. 9 is displayed on the display unit 14. .

  The imaging device 10 determines whether or not a selection operation to the marking list 14c has been detected (step S111). If the selection operation on the marking list 14c is not detected (step S111: No), the process proceeds to step S109.

  On the other hand, when the selection operation to the marking list 14c is detected (step S111: Yes), the imaging device 10 subsequently determines whether or not the operation (for example, tap operation) to the registration button 14f is detected (step S112). ). In the still image shooting mode, the registration button 14f also serves as a shooting button. When the operation on the registration button 14f (photographing button) is not detected (step S112: No), the process proceeds to step S109.

  On the other hand, when the operation on the registration button 14f (shooting button) is detected (step S112: Yes), the shooting device 10 shoots a still image and performs marking processing on the still image. Specifically, the image 14d captured by the image capturing unit 12 at the timing when the registration button 14f (shooting button) is tapped is recorded as a still image, and the selected tag is added to the image data. After that, the process proceeds to step S109.

  Here, as shown in steps S111 to S113, in the still image mode, the user cannot take a still image unless he or she selects any tag from the marking list. In other words, any tag is added to all still images. Therefore, all the still images are associated with one of the frames forming the moving image by the same tag. For example, it is assumed that the tag T5 (tag of "veranda") is subjected to the marking process during the shooting of the moving image shooting route R1, and then the tag T5 is selected to shoot the still image of the still image shooting point P6. In this case, the frame to which the tag T5 is added in the moving image and the still image to which the tag T5 is added are linked using the same tag T5 as a key.

  The image data generated by such a shooting operation is once stored in the image data storage unit 162, then uploaded to the server 20 via the communication network N, and subjected to image processing as necessary.

  Here, the shooting order of moving images on the moving image shooting routes R1 to R4 and the shooting order of still images at the still image shooting points P1 to P11 are not particularly limited. For example, the shooting of a still image may be started after the shooting of all moving images is completed, or vice versa. Alternatively, shooting of a moving image and shooting of a still image may be performed alternately. For example, after the moving image shooting on the moving image shooting route R1 is completed, the still image shooting at the still image shooting points P5, P6, and P7 is performed, and then the moving image shooting on the moving image shooting route R2 is performed.

  As described above, in the present embodiment, the marking list is registered in advance, and the tag selected from the marking list is added to the image data of the moving image or the still image by using the operation of the registration button 14f as a trigger. Therefore, even if a moving image and a still image are taken without worrying about the order of shooting, the marking process performed during the shooting causes the desired frames in the moving image to be separated from each other, or the desired frame and the still image in the moving image. Can be tied. Therefore, it is possible to acquire image data of moving images and still images in which the relationship between arbitrary images (frames) is organized.

  In the above description, the moving image shooting is performed a plurality of times (see FIG. 5), but the moving image shooting may be performed only once. For example, as shown in FIG. 10A, moving image shooting is performed along a moving image shooting route R20 while going around all the rooms like a single stroke, and marking processing is performed at any time (see tags T21 to T27). After that, the still image shooting / marking process may be performed at a place desired by the user (see still image shooting points P21 to P26 and tags T22 to T27).

  Here, marking processing using the same tag may be performed at different points in the moving image shooting route R20. For example, in FIG. 10A, the marking process by the tag T21 is performed immediately after entering the living room / kitchen from the entrance, and the marking process by the tag T21 is performed again when entering the living room / kitchen from the western room (2) again. Therefore, frames having different shooting times are linked to each other. As will be described later in detail, the tag is used as a mark when switching the images displayed on the display device 30. Therefore, as shown in FIG. 10A, in the moving image shot along the moving image shooting route R20, the frames to which the same tag T21 is attached are present at different positions, as shown in FIG. 10B, in chronological order. This means that there is an option of displaying a moving image and proceeding in the direction A or skipping a frame and proceeding in the direction B.

  Further, when shooting a moving image, not only one moving image shooting route may be passed once but also the same route may be reciprocated. In this case, by performing marking processing with the same tag at the same position on the forward path and the return path, it is possible to acquire images in different directions even at the same position. That is, an image with a wide field of view can be obtained. Therefore, by using a wide-angle lens with an angle of view of 180 ° or more and reciprocating the same route to shoot a moving image, it is possible to obtain a moving image corresponding to a spherical image.

Next, the operation of displaying the image photographed by the photographing device 10 on the display device 30 will be described. In the following, a case of displaying two moving images I _V1 and I _V2 and two still images I _S1 and I _S2 captured in a real estate property will be described. When displaying images on the display device 30, these moving image and still image image files are sequentially or collectively downloaded from the server 20. When sequentially downloading image files, there is an advantage that image display can be started quickly. On the other hand, when all the image files are downloaded in advance, there is an advantage that the display switching of the images becomes smooth. Which method should be used may be appropriately selected according to the communication environment and other conditions. In the present embodiment, it is the former.

11A to 11C are schematic diagrams for explaining the image display switching operation in the display device 30. 11A to 11C show an image display switching operation between the moving images I _V1 , I _V2 and the still image I _S1 .

In FIGS. 11A to 11C, it is assumed that all of the moving images I _V1 , I _V2 and the still image I _S1 are spherical images in which a subject at 360 ° around the photographing device 10 is captured. Further, the display range V indicates a range of the moving images I _V1 , I _V2 and the still image I _S1 that are displayed at one time on the display unit 33 (each display area 33a, 33b) of the display device 30. The position of the display range V with respect to the moving images I _V1 and I _V2 and the still image I _S1 changes according to the detection result of the orientation and orientation detection unit 34 (that is, the orientation and orientation of the head of the user wearing the display device 30). To do. In other words, for the user who views the image, the scenery that comes into view changes as the head moves.

  The display range V may be determined in consideration of the viewing angle of a general person. For example, a general view angle of a person who is stationary is about 180 ° to the left and right (about 90 ° on one side), but when exercising (for example, walking), other than the central region of the view (left and right edges). Visibility of parts). Therefore, a range narrower than 180 ° (for example, 150 °) is set as the display range V while displaying a moving image, and 180 ° or a wider range is set as the display range V while displaying a still image. It may be adjusted as appropriate. In other words, in the case of a moving image, only a predetermined range centered on the moving direction of the photographer is photographed, and in the case of a still image, the user can stop, look back, and look around. The shooting method may be different between moving images and still images, such as shooting a spherical image. As a result, it is possible to create a content that allows the user to naturally view the image while suppressing the total size of the image file.

When displaying the moving image I _V1 on the display unit 33, the display device 30 reads and reproduces the image file of the moving image I _V1 (see the solid line in FIG. 11A), while the image files of the moving image I _V2 and the still image I _S1 are not read. It is displayed (see the broken line in FIG. 11A). Then, when displaying the still image I _S1 reads the image file of the still image I _S1, switches the display to the still image I _S1 to pause playback of the video I _V1 (see FIG. 11B). Next, when the reproduction of the moving image I _V1 is resumed, the pause of the moving image I _V1 is released, the still image I _S is hidden, and the display is switched to the moving image I _V1 (see FIG. 11A). As a result, the display of the moving image I _V1 can be resumed from the timing at which the moving image I _V1 was temporarily stopped.

Further, when switching the display from the video I _V1 video I _V2 is (see the solid line in FIG. 11C) starts the reproduction reads the image file of the moving image I _V2, pause video playback I _V1. Then, the image file of the moving image I _V1 is closed (see the broken line in FIG. 11C). This is because the user does not always reproduce the same part of the moving picture I _V1 after switching the display from the moving picture I _V1 to the moving picture I _V2 .

  FIG. 12 is a flowchart showing the operation of the display device 30. 13 to 15 are schematic diagrams showing display examples of images on the display device 30. Hereinafter, a moving image or a still image that is displayed on the display unit 33 of the display device 30 and is three-dimensionally recognized by stereoscopic viewing using parallax is also referred to as a VR moving image or a VR still image. Displaying a VR moving image or a VR still image on the display unit 30 is also referred to as VR display.

  First, in step S200, the display device 30 opens the image file of the moving image read from the server 20 and reproduces the VR moving image. For example, in the case of displaying an image of a real estate property, as illustrated in FIG. 13, a moving image starting from a scene entering the entrance (for example, a moving image of the moving image capturing route R1 shown in FIG. 5) is first displayed, and the like. Set it.

  In subsequent step S201, the display device 30 acquires the time (reproduction time) from the start of the reproduction of the moving image to the present time, in other words, the frame number of the moving image.

  In subsequent step S202, the display device 30 determines whether or not the current reproduction time is within the reproduction time of the marked frame. That is, it is determined whether or not a tag is added to the currently displayed frame. If it is not within the reproduction time of the marked frame (step S202: No), the process proceeds to step S219.

  In step S219, the display device 30 determines whether to end displaying the image. For example, when the display device 30 is powered off, a predetermined input operation for stopping the display of the image is performed, or the VR moving image being displayed reaches the final frame, the image display is terminated. to decide. When it is determined that the image display is to be ended (step S219: Yes), the display device 30 ends the image display. On the other hand, when it is determined that the display of the image is not ended (step S219: No), the process returns to step S200.

  On the other hand, when the current reproduction time is within the reproduction time of the marked frame (step S202: Yes), the display device 30 searches for an image associated with the frame being displayed (image to be associated with). (Step S203). That is, the frame in the still image or the moving image to which the same tag as the tag attached to the frame being displayed is added is searched.

  In subsequent step S204, the display device 30 determines whether or not the image of the linked destination is included in the image (moving image) file currently being displayed. When the image of the link destination is not included in the image file currently displayed (step S204: No), the display device 30 reads the image file including the image of the link destination from the server 20 (step S205).

  Subsequently, the display device 30 displays the tag attached to the image (frame) currently displayed on the screen (step S206). For example, FIG. 13 shows a scene in which a tag T named “Reception Room” is displayed. Alternatively, an icon-shaped tag T'may be displayed instead of the tag T in which the text is described.

  In subsequent step S207, display device 30 determines whether the displayed tag is selected. For example, the operation detection unit 362 detects that the user has directed his / her line of sight to the tag for a predetermined time or more, or a predetermined operation (for example, pressing an operation button) has been performed on the operation input unit 32. , Etc., it is determined that the tag is selected.

  When the displayed tag is not selected (step S207: No), the process proceeds to step S219. In this case, unless it is determined that the display of the image should be ended (step S219: No), the process returns to step S200, and the moving image is continuously reproduced. For example, in the scene shown in FIG. 13, if the tag T of “reception room” is not selected, the VR video continues to be played back, and the route to the living room (see FIG. 14) in the back of the corridor is continuously displayed. To be done.

  On the other hand, when the displayed tag is selected (step S207: Yes), the display device 30 temporarily stops the reproduction of the VR moving image currently being displayed (step S208).

  In subsequent step S209, display device 30 determines whether or not the image file read in step S205 is a moving image. When the image file is not a moving image, that is, when it is a still image (step S209: No), the display device 30 displays the VR still image on the display unit 33 based on the read image file (step S210).

  In subsequent step S211, the display device 30 displays the end tag on the screen on which the VR still image is displayed, and determines whether or not the end tag is selected (step S212). The method of determining whether or not a tag has been selected is the same as in step S207.

  When the end tag is not selected (step S212: No), the process returns to step S210. In this case, the same VR still image continues to be displayed. On the other hand, when the end tag is selected (step S212: Yes), the display device 30 hides the VR still image (step S213), and the process proceeds to step S219. In this case, unless it is determined to end the display of the image (step S219: No), the process returns to step S200, and the temporarily stopped moving image (see step S208) is continuously reproduced.

  If the image file read in step S205 is a moving image (step S209: Yes), the display device 30 jumps to the frame associated with the tag selected in step S207 among the read image files. Yes (step S214). Then, the display on the display unit 33 is switched to the VR moving image of the linked destination (step S215). After that, the process returns to step S200 via step S219 (step S219: No), and the VR moving image is reproduced from the jumped frame.

  Further, in step S204, when the image of the linked destination of the frame being displayed is included in the image (moving image) file being displayed (step S204: Yes), the display device 30 adds the image to the image (frame) being displayed. The displayed tags are displayed on the screen (step S216).

  In subsequent step S217, display device 30 determines whether or not the displayed tag is selected. The method of determining whether or not a tag has been selected is the same as in step S207.

  When the tag is selected (step S217: Yes), the display device 30 skips to the frame to which the string is attached (step S218). After that, the process proceeds to step S200, and the VR moving image is reproduced from the skipped frame.

  The operation of the display device 30 described above will be described in more detail with reference to FIGS. 16 to 18B. 16 to 18B are timing charts showing display switching timings of images on the display device 30.

As shown in FIG. 16, after the reproduction of the moving image I _V1 is started, when the time t1 is reached, the reproduction time of the frame with the tag M1 is reached (step S202: Yes). Since this tag M1 is associated with an image file (still image I _S1 ) different from the moving image I _V1 (step S204: No), the image file of the still image I _S1 is read (step S205). However, when the tag M1 displayed on the screen is not selected (steps S206 and S207: No), the display of the moving image I _V1 is continued (step S201).

On the other hand, when the time t2 is reached after the reproduction of the moving image I _V1 is started, the reproduction time of the frame with the tag M2 is reached (step S202: Yes). Since this tag M2 is associated with an image file (still image I _S2 ) different from the moving image I _V1 (step S204: No), the image file of the still image I _S2 is read (step S205). Then, when the tag M2 displayed on the screen is selected (steps S206 and S207: Yes), the display of the moving image I _V1 is temporarily stopped (step S208). Then, since the image file read in step S205 is the image file of the still image I _S2 (step S209: No), VR display of this still image I _S2 is performed (step S210). Then, when the end tag displayed on the screen is selected at time t4 (steps S211, S212: Yes), the still image (still image I _S2 ) that has been displayed until then is hidden (step S213), The display of the moving image I _V1 is restarted (step S219: No, S200). At this time, the moving image I _V1 is reproduced from the continuation paused at the time t3.

In addition, as shown in FIG. 17, after the reproduction of the moving image I _V1 is started, when the time t5 is reached, the reproduction time of the frame with the tag M3 is reached (step S202: Yes). Since this tag M3 is associated with an image file (moving image I _V2 ) different from the moving image I _V1 (step S204: No), the image file of the moving image I _V2 is read (step S205). Then, when the tag M3 displayed on the screen is selected (steps S206 and S207: Yes), the display of the moving image I _V1 is stopped (step S208) and the frame of the moving image I _V2 to which the tag M3 is linked is jumped to. (Step S209: Yes, S214). Then, the display on the display unit 33 is switched to the moving image I _V2 , and the VR moving image is reproduced from the jumped frame (step S219: No, S200). After that, the image file of the moving image I _V1 is temporarily closed. The same applies to the processing after the reproduction time of the frame with the tag M4 is reached after the reproduction of the moving image I _V2 is started.

Further, as shown in FIG. 18A, it is assumed that tags M5 are attached to frames near times t6 and t9 and tags M6 are attached to frames near times t7 and t8 with respect to the frames forming the moving image I _V1. . In this case, as shown in FIG. 18B, when the time t6 is reached after the reproduction of the moving image I _V1 is started, the reproduction time of the frame with the tag M5 is reached (step S202: Yes). This tag M5 is associated with the frame in the image file of the same moving image I _V1 (step S204: Yes). Therefore, the tag M5 is displayed (step S216), and when this tag M5 is selected (step S217: Yes), the display on the display unit 33 is the frame of the tied destination, that is, from the time t9 when the same tag M5 is added. To the frame (step S218), and the moving image I _V1 is reproduced from there (step S200).

  As described above, according to the first embodiment of the present invention, the user can move in a three-dimensional space composed of a moving image and a still image with a real feeling.

  Further, according to the first embodiment of the present invention, an image obtained by shooting a subject is used as a material and requires less time and effort than before, and can be operated or operated by a user when displaying an image. It is possible to create data for generating a corresponding event and display an image based on this data.

  More specifically, according to the first embodiment of the present invention, a moving image or a still image is captured while marking with a preset tag, so that when image data for VR display is generated, image display switching is performed. It becomes unnecessary to perform the editing work for embedding the information necessary for generating the event in the image data. Therefore, it is possible to generate image data for VR display in a short time at a lower work cost than the conventional one.

  Further, according to the first embodiment of the present invention, since the subject is photographed while marking with the preset tag, the photographing order is determined in consideration of the display order of the images, and the time series is adjusted after the photographing. There is no need to edit the file. That is, the user may shoot the subjects in a desired order or an order that is easy to perform.

  Also, according to the first embodiment of the present invention, when switching the display between the images linked by the tags, after starting to read the image file of the image that may be displayed next, Since the tag is displayed on the screen, it is possible to switch images smoothly.

  In addition, according to the first embodiment of the present invention, image data for VR display of the 3D space is generated using a two-dimensional moving image or a still image obtained by shooting a subject, and thus 3DCG is used. There is no need to perform high-level and high-cost image processing as in the case of creating. Therefore, it is not necessary to work for a long time using a high-spec device, and it is possible to easily create image data for VR display at a lower cost than before.

  Next, a modification of the first embodiment of the present invention will be described. FIG. 19 is a flowchart showing the operation of the display device 30 in the modification of the first embodiment. FIG. 21 is a schematic diagram showing a display example on the display device 30 in the present modification. In the first embodiment, when displaying a series of images, the display of moving images is started, but the order of displaying moving images and still images is not limited to this.

  First, in step S230, the display device 30 determines whether the image file read from the server 20 is a moving image file. When the image file is a moving image file (step S230: Yes), the process proceeds to step S200. Subsequent steps S200 to S215 and S216 to S218 are the same as those in the first embodiment. However, if the read image file is a still image file in step S209 (step S209: No), the process proceeds to step S231.

  In step S240, the display device 30 determines whether or not the VR moving image being displayed (step S202: No, step S207: No) or the VR moving image whose display has been switched (see step S215) has ended, that is, It is determined whether the final frame has been reached. Then, if the VR moving image is not yet finished (step S240: No), the process proceeds to step S200. On the other hand, when the VR moving image has ended (step S240: Yes), the process proceeds to step S241.

  In step S241, the display device 30 further determines whether to end the display of the image. For example, when the power of the display device 30 is turned off, or when a predetermined input operation for stopping the display of the image is performed, it is determined that the display of the image is ended. When it is determined that the image display is to be ended (step S241: Yes), the display device 30 ends the image display. On the other hand, when it is determined that the display of the image is not ended (step S241: No), the process returns to step S230.

  On the other hand, if the image file read from the server 20 is not a moving image file (step S230: No), that is, if it is a still image file, the display device 30 displays the still image on the screen based on the image file. It is displayed (step S231). For example, FIG. 20 shows a still image showing the entrance hall at the entrance of a building and the doors of the three conference rooms visible from the vicinity of the entrance.

  Subsequently, in step S232, the display device 30 displays a tag for jumping to the image file associated with the image file of the VR still image on the screen on which the VR still image is displayed. For example, when the three image files respectively representing the insides of the three conference rooms are linked to the image file of the still image shown in FIG. 20, the display device 30 displays the names of the three conference rooms (for example, A Tags T (a), T (b), and T (c) representing the conference room, the B conference room, and the C conference room, respectively, are displayed on the screen. The image file representing the inside of each conference room may be a moving image file or a still image file. Further, the marking process may be performed on the image file representing the inside of the conference room as in the first embodiment.

  In subsequent step S233, display device 30 determines whether any of the tags displayed on the screen has been selected. The method of determining whether or not a tag has been selected is the same as in step S207.

  When no tag is selected (step S233: No), the display device 30 continues to display the VR still image (step S231). On the other hand, when any of the displayed tags is selected (step S233: Yes), the display device 30 reads the image file corresponding to the selected tag from the server 20 (step S234). After that, the process proceeds to step S241. After that, when the display of the image is not ended (step S241: No), the display device 30 determines whether the image file read in step S234 is a moving image file (step S230), and according to the determination result. Then, the moving image or the still image is displayed (steps S200 and S231). Subsequent steps are as described above.

  As described above, according to this modification, the user can move to a desired place in a three-dimensional space formed by images by selecting a tag displayed on a still image.

  In the above modification, the image file corresponding to the selected tag is read after the tag is selected (step S233: Yes), but all the tags displayed before the tag is selected. The image file corresponding to may be read. In this case, it is possible to shorten the time from when the tag is selected until the image based on the image file corresponding to the selected tag is displayed.

(Second embodiment)
Next, a second embodiment of the present invention will be described. The configuration of the image processing system according to this embodiment is the same as that of the first embodiment as a whole (see FIG. 1), and a part of the operation of the image capturing apparatus 10 is different from that of the first embodiment.

  FIG. 21 is a flowchart showing the operation of the image capturing apparatus 10 according to the second embodiment of the present invention. FIG. 22 is a schematic diagram illustrating a screen displayed on the display unit 14 of the image capturing apparatus 10.

  First, in step S300, the imaging device 10 determines whether or not the shooting mode is set to moving image. When the shooting mode is set to the moving image (step S300: Yes), the shooting device 10 starts the moving image shooting according to a predetermined operation (for example, an operation on the shooting button) on the operation input unit 13 (step S301). . As a result, the image captured by the photographing unit 12 is recorded as a moving image.

  In subsequent step S302, the image capturing apparatus 10 acquires the position information output from the position detection unit 15 and registers it as the moving image shooting start position.

  In the following step S303, the imaging device 10 determines whether or not an operation (for example, a tap operation) on the registration button 14f (see FIG. 22) is detected (step S303). When the operation to the registration button 14f is not detected (step S303: No), the imaging device 10 waits for the operation to the registration button while continuing to shoot the moving image.

  On the other hand, when the operation to the registration button 14f is detected (step S303: Yes), the imaging device 10 acquires the position information output from the position detection unit 15 and registers it as the image identification information (step S304). Specifically, the marking processing unit 174 creates a list in which information (for example, a frame number) for identifying a frame generated at the timing when the operation of the registration button 14f is detected is associated with the position information, and is stored in the storage unit 16. Remember. Alternatively, the position information may be directly embedded in the image data of the generated frame. The position information registered here may be a relative position with respect to the initial position registered in step S302.

  In subsequent step S305, the image capturing apparatus 10 determines whether or not a moving image capturing end operation (for example, a shooting button re-pressing operation) has been detected. When the ending operation of moving image shooting is not detected (step S305: No), the process returns to step S303. On the other hand, when the end operation of moving image shooting is detected (step S305: Yes), the process proceeds to step S306.

  In step S306, the imaging device 10 acquires the position information output from the position detection unit 15 and registers it as the moving image shooting end position.

  In step S307, the imaging device 10 determines whether to end the moving image or still image shooting operation. For example, when the power of the image capturing apparatus 10 is turned off or a predetermined ending operation is performed, the image capturing operation ends as it is (step S307: Yes). On the other hand, if the shooting operation is not to be ended (step S307: No), the process returns to step S300.

  Further, in step S300, when the shooting mode is not set to the moving image (step S300: No), that is, when the shooting mode is the still image mode, the shooting device 10 detects the operation of the still image shooting button. It is determined whether or not (step S311). When the operation of the shooting button is detected (step S311: Yes), the shooting device 10 shoots a still image (step 312). As a result, the image captured by the photographing unit 12 is recorded as a still image.

In the following step S313, the imaging device 10 acquires the position information output from the position detection unit 15 and registers it as image identification information (step S313). Specifically, the marking processing unit 174 adds position information to the image data of the still image captured in step S312. The position information at this time may be a relative position with respect to the initial value registered in step S302.
After that, the process proceeds to step S307.

  The moving image and still image captured in this manner are linked to each other by using the position information given to each image data as identification information. When used as the identification information, the position information does not need to be pinpoint, and for example, the position of another image data is within a predetermined range (for example, several tens of cm to several meters) centered on the position information given to certain image data. When the information is included, it is determined that the images have the same identification information.

  As described above, according to the second embodiment of the present invention, the image is marked with the position information that is automatically detected, so that it is possible to save the trouble of manually performing the marking process at the time of shooting.

(Third Embodiment)
Next, a third embodiment of the present invention will be described. The configuration of the image processing system according to this embodiment is the same as that of the first embodiment as a whole (see FIG. 1), and in this embodiment, image data of a moving image or a still image generated by the image capturing apparatus 10 is also included. In addition, it differs from the first and second embodiments in that 3D model image data is constructed.

  FIG. 23 is a flowchart showing the operation of the image capturing apparatus 10 according to the third embodiment of the present invention, and shows the operation for acquiring an image for constructing image data of a 3D model. However, the image data generated by the imaging device 10 may be two-dimensional moving image or still image data, as in the first and second embodiments. In the following, the operation when shooting a moving image will be described.

  First, in step S401, the imaging device 10 starts imaging for a 3D image. Specifically, the image capturing device 10 starts capturing a moving image in response to a predetermined operation (for example, operation on a capture button) on the operation input unit 13.

  In subsequent step S402, the imaging device 10 acquires the position information output from the position detection unit 15 and registers it as the moving image shooting start position.

  In subsequent step S403, the imaging device 10 determines whether or not an operation (for example, a tap operation) on a predetermined registration button (for example, the registration button 14f illustrated in FIG. 22) is detected. When the operation to the registration button 14f is not detected (step S403: No), the imaging device 10 waits for the operation to the registration button while continuing to shoot the moving image.

  On the other hand, when the operation on the registration button is detected (step S403: Yes), the imaging device 10 acquires the position information output from the position detection unit 15 and registers it as image identification information (step S404). Specifically, the marking processing unit 174 performs marking by adding position information to the image data of the image (frame) being photographed. The position information at this time may be a relative position with respect to the initial position registered in step S402.

  In the following step S405, the imaging device 10 determines whether or not a shooting end operation (for example, a shooting button re-pressing operation) has been detected. When the shooting end operation is not detected (step S405: No), the process returns to step S403. On the other hand, when the shooting end operation is detected (step S405: Yes), the process ends.

  The image data generated in this way is uploaded to the server 20. The server 20 executes image processing for VR-displaying the 3D space on the uploaded image data, as in the first and second embodiments. In addition to this, in the present embodiment, the server 20 constructs the 3D model image data based on the uploaded image data. As a method for constructing 3D model image data from a plane image, various known methods can be used. As an example, a 3D model is constructed by detecting the movement of a common object appearing in a plurality of frames in a moving image and estimating the depth of the object based on the principle that the greater the distance moved, the closer to the front. The method is known.

  FIG. 24 is a schematic diagram showing a display example of a 3D model constructed based on image data of a moving image or a still image captured by the image capturing apparatus 10. Tags corresponding to the position information added to the image data at the time of shooting are embedded in various places of this 3D model. As a result, each part of the 3D model is associated with the VR-displayed moving image or still image. For example, the entrance P30 in the 3D model is associated with the frame showing the entrance in the moving image, and the toilet P31 in the 3D model is associated with the frame showing the toilet in the moving image.

FIG. 25 is a flowchart showing the operation of the display device 30 according to the third embodiment of the present invention.
First, in step S410, the display device 30 downloads the image data of the 3D model from the server 20 and displays it on the display unit 33.

  In subsequent step S411, the display device 30 acquires the position information of the 3D model that is currently in the user's visual field.

  In the following step S412, the display device 30 determines whether the position information acquired in step S411 is within the range marked at the time of shooting (the range in which the position information is registered in step S404 of FIG. 23). . If it is not within the marked range (step S412: No), the process proceeds to step S433.

  In step S433, the display device 30 determines whether to end the operation. This determination method is the same as step S219 in FIG. When it is determined that the operation ends (step S433: Yes), the operation ends. On the other hand, when it is determined that the operation is not ended (step S433: No), the process returns to step S410.

  On the other hand, when the position information acquired in step S411 is within the marked range (step S412: Yes), the display device 30 searches for a stringed destination associated with the acquired position information (step S413). .

  When the linked destination exists in the 3D model (step S414: Yes), the display device 30 displays the tag on the 3D model being displayed (step S415). For example, in the case of FIG. 24, the tag T30 is displayed at the entrance P30.

  In subsequent step S416, display device 30 determines whether or not the displayed tag is selected. The method of determining whether or not a tag has been selected is the same as in step S207 of FIG. When the displayed tag is selected (step S416: Yes), the display device 30 causes the area to be displayed in the field of view of the user to jump to the position in the 3D model associated with the selected tag (step S417). ). Then, a process transfers to step S410. In this case, in step S410, the 3D model of the jump destination area is displayed. For example, in FIG. 24, when the tag T30 displayed at the entrance P30 is associated with the tag embedded in the corridor P32, the orientation of the 3D model changes, and the corridor P32 is displayed so as to face the user side. . On the other hand, when the tag is not selected in step S416 (step S416: No), the process directly returns to step S410. In this case, the conventional 3D model is displayed.

  In addition, in step S414, when the linked destination does not exist in the 3D model (step S414: No), the display device 30 reads the image file of the linked still image or moving image from the server 20 (step S418). Then, the display device 30 displays the tag on the 3D model being displayed (step S419).

  In subsequent step S420, display device 30 determines whether the displayed tag has been selected. The method of determining whether or not a tag has been selected is the same as in step S207 of FIG. When the tag is not selected (step S420: No), the process proceeds to step S433.

  On the other hand, when the displayed tag is selected (step S420: Yes), the display device 30 subsequently determines whether the image file read in step S418 is a moving image (step S421).

  When the image file is not a moving image (step S421: No), that is, when it is a still image, the display device 30 displays the VR still image on the display unit 33 based on the read image file (step S422).

  In the following step S423, the display device 30 displays the end tag on the screen on which the VR still image is displayed, and determines whether the end tag is selected (step S424). The method of determining whether or not a tag has been selected is the same as in step S207.

  When the end tag is not selected (step S424: No), the process returns to step S422. In this case, the VR still image continues to be displayed. On the other hand, when the end tag is selected (step S424: Yes), the display device 30 displays the 3D model again (step S425) and hides the VR still image (step S426). Then, a process transfers to step S433.

  On the other hand, when the image file read in step S421 is a moving image (step S421: Yes), the display device 30 jumps to the frame associated with the tag selected in step S420 among the read image files. Yes (step S427). Then, the display on the display unit 33 is switched to the VR moving image of the linked destination (step S428), and the VR moving image of the linked destination is reproduced (step S429).

  Here, after the reproduction of the VR moving image is started, similar to the display operation in the first embodiment, a jump to a frame to which a string is attached in a different VR moving image, a frame skip in the same VR image, The linked still image may be displayed / hidden (see FIG. 12).

  After that, when the playback of the VR video of the linked destination ends (or pauses) (step S430), the display device 30 displays the 3D model again (step S431) and hides the VR video (step S432). . Then, a process transfers to step S433.

  As described above, according to the third embodiment of the present invention, the 3D model is constructed based on the image data of the moving image or the still image generated by the imaging device 10, and the position information is added to the 3D model. Since it is associated with a moving image or a still image that is embedded and VR displayed, the subject can be displayed in various forms. For example, it is possible to use a 3D model to intuitively grasp the whole picture of a subject (for example, a real estate property), check a moving line with a VR moving image, and check details of the subject with a VR still image.

  In the present embodiment, the image data of the 3D model is constructed in the server 20, but it is not limited to this. For example, the image processing unit 172 of the image capturing apparatus 10 may construct image data of a 3D model, or the image data of a moving image or a still image generated in the image capturing apparatus 10 may be transferred to a computer dedicated to constructing a 3D model, and this computer may be used. In, the 3D model image data may be constructed.

(Fourth Embodiment)
Next, a fourth embodiment of the present invention will be described. FIG. 26 is a block diagram showing the schematic arrangement of the image processing apparatus according to the fourth embodiment of the present invention.

  The image processing device 60 shown in FIG. 26 has a function of shooting a subject and generating image data of a moving image or a still image, and a function of performing VR display based on the image data. Such an image processing device 60 can be configured using a general-purpose smartphone or the like, like the image capturing device 10 and the display device 30 shown in FIG. Of the units shown in FIG. 26, the configurations and operations of the image capturing unit 61, the operation input unit 62, the display unit 63, and the position detecting unit 65 are as follows: the image capturing unit 12, the operation input unit 13, the display unit 14, and The configuration and operation of the position detection unit 15 are the same as each other. The configuration and operation of the orientation / posture detection unit 64 shown in FIG. 26 are the same as the configuration and operation of the orientation / posture detection unit 34 shown in FIG. The storage unit 66 shown in FIG. 26 includes a program storage unit 661, an image data storage unit 662, and a marking list storage unit 663, like the image capturing apparatus 10 shown in FIG. The processor 67 shown in FIG. 26 includes a photographing control unit 671, an image processing unit 672, a marking list registration unit 673, a marking processing unit 674, and the processor 36 shown in FIG. The display control unit 361 and the operation detection unit 676 that function in the same manner as the respective units of FIG.

  In this way, by realizing the functions of the image capturing apparatus 10 and the display apparatus 30 by the same device, it is not necessary to upload and download the image data via the server 20, and the image of the linked destination can be smoothly displayed. There is also an advantage that the display can be switched.

  In the first to fourth embodiments and the modified examples described above, a moving image, a still image, or a 3D model is displayed on the display device 30 (or the image processing device 60) that the user wears on the head. However, the display form of the image is not limited to this. For example, these images may be displayed on the screen of a tablet terminal, a personal computer, or the like. In this case, the tag displayed on the screen may be selected by using a pointing device such as a mouse or a touch panel.

  In addition, in the first to fourth embodiments and the modified examples described above, a moving image or a still image is three-dimensionally displayed on the display device 30 (or the image processing device 60) that the user wears on his / her head. However, a part of the omnidirectional image may be displayed on the display device 30 (or the image processing device 60) as a panoramic image or may be displayed as a planar two-dimensional image. Also, a panoramic image or a two-dimensional planar image may be displayed on a tablet terminal, a personal computer, or the like.

  The present invention is not limited to the above-described first to fourth embodiments and modifications, but can be achieved by appropriately combining a plurality of constituent elements disclosed in the above-described first to fourth embodiments and modifications. , Various inventions can be formed. For example, some constituent elements may be formed by excluding all the constituent elements shown in the first to fourth embodiments and the modified examples, and in the first to fourth embodiments and the modified examples. It may be formed by appropriately combining the components shown.

1 Image Processing System 10 Imaging Device 11, 21, 31 Communication Unit 12, 61 Imaging Unit 13, 32, 62 Operation Input Unit 14, 33, 63 Display Unit 15, 65 Position Detection Unit 16, 35, 66 Storage Unit 17, 22 , 36, 67 processor 20 server 23 image data storage unit 24 property information database 30 display device 33a, 33b area 34, 64 orientation and posture detection unit 37 holder 60 image processing device 161, 351, 661 program storage unit 162, 352, 662 Image data storage unit 163, 663 Marking list storage unit 171, 671 Imaging control unit 172, 221, 672 Image processing unit 173, 673 Marking list registration unit 174, 674 Marking processing unit 361, 675 Display control unit 362, 676 Operation detection unit

Claims (15)

A shooting unit that generates image data of a moving image or a still image by shooting a subject,
A marking processing unit that adds identification information to a frame or a still image forming a moving image generated by the photographing unit according to a predetermined trigger,
An image processing unit that generates image data of a moving image or a still image for display based on the image data of the moving image or the still image generated by the photographing unit,
An image data storage unit that stores the image data generated by the image processing unit;
An image processing system including. A marking list storage unit that stores one or more types of identification information,
An operation input unit for selecting one piece of identification information from the one or more kinds of identification information according to an operation made from the outside,
Further equipped with,
The marking processing unit adds the identification information selected by the operation input unit to the frame or still image generated by the imaging unit at a timing according to the trigger,
The image processing system according to claim 1. Further comprising a position detection unit that detects the position of the imaging unit and outputs position information,
The marking processing unit, the position information output from the position detection unit at the timing according to the trigger, the identification information for the frame or still image generated by the imaging unit at the timing according to the trigger As,
The image processing system according to claim 1.   The image processing system according to any one of claims 1 to 3, wherein the trigger is an operation performed on a predetermined operation unit during shooting of a moving image.   The image processing system according to any one of claims 1 to 3, wherein the trigger is an operation on an operation unit that is operated when a still image is captured. A display unit capable of displaying a moving image or a still image based on the image data stored in the image data storage unit;
A display control unit for controlling image display on the display unit,
An operation detection unit that detects an operation performed from the outside,
Further equipped with,
The display control unit causes the display unit to display information indicating that the identification information is added when the frame or still image of the moving image being displayed on the display unit is provided with the identification information, When the operation detection unit detects an operation of selecting the information, the image display on the display unit is a moving image to which the same identification information as the identification information added to the frame or still image of the moving image being displayed is added. Switch to the frame or still image of
The image processing system according to claim 1. The image data storage unit stores image data of a plurality of moving images,
Each of the plurality of moving images includes a frame to which the same identification information is added,
The display control unit switches image display on the display unit between the plurality of moving images,
The image processing system according to claim 6. The moving image includes a plurality of frames that are generated at different times and have the same identification information,
The display control unit switches image display on the display unit between the plurality of frames,
The image processing system according to claim 6. The moving image includes a frame to which the same identification information as the identification information added to the still image is added,
The display control unit switches the image display on the display unit between the still image and a frame to which the same identification information is added,
The image processing system according to claim 6. The image processing unit further constructs image data of a three-dimensional model on the basis of the image data of the moving image or still image generated by the photographing unit, and the marking processing unit causes the moving image or still image to be displayed. Based on the identification information given to the image, give information representing the position to each place of the model,
The display control unit, based on the information given to each part of the model and the identification information given to the moving image or the still image, an image in the display unit between the model and the moving image or the still image. Switch display,
The image processing system according to claim 6.   The second image processing unit further includes a second image processing unit that generates image data of a moving image or a still image displayed three-dimensionally based on the image data of the moving image or the still image generated by the photographing unit. 10. The image processing system according to any one of items 10 to 10. Step (a) of generating image data of a moving image or a still image by shooting a subject,
A step (b) of adding identification information to a frame or a still image that constitutes the moving image generated by the image capturing unit according to a predetermined trigger;
Step (c) of generating image data of a moving image or a still image for display based on the image data of the moving image or the still image generated in step (a),
Storing step (d) the image data generated in step (c),
An image processing method including. A program to be executed by a photographing device, comprising: a photographing unit that generates image data of a moving image or a still image by photographing a subject; a storage unit; and an operation input unit that receives an operation performed from the outside.
The storage unit stores one or more types of identification information in advance,
Selecting one piece of identification information from the one or more types of identification information according to an operation on the operation input unit;
A step of adding the one piece of identification information to the frame or still image generated by the photographing unit at a timing according to a predetermined trigger;
A program to execute. A program to be executed by a photographing device, comprising: a photographing unit that generates moving image or still image data by photographing a subject; and a position detection unit that detects the position of the photographing unit and outputs position information. ,
A step of outputting position information from the position detection unit at a timing according to a predetermined trigger,
A step of adding the position information as identification information to the frame or still image generated by the image capturing unit at a timing according to the trigger;
A program to execute. A program executed by a display device, comprising: a display unit capable of displaying a moving image or a still image; a display control unit for controlling image display on the display unit; and an operation detection unit for detecting an operation performed from the outside. ,
When predetermined identification information is added to a frame or still image of the moving image being displayed on the display unit, a step of displaying information indicating that the predetermined identification information is added on the display unit,
When the operation detection unit detects an operation of selecting the information, the image display on the display unit is provided with the same identification information as the predetermined identification information given to the frame or still image of the moving image being displayed. The step of switching to the frame or still image of the recorded video,
A program to execute.

Images