KR20090132317A - Method and system for providing additional information based on video images - Google Patents

Abstract

PURPOSE: A method and a system for providing additional information based on video images are provided to offer video and additional information to a user according to the virtual location and view points. CONSTITUTION: The space information provider(10) offers the virtual position which user demands and the image corresponding to the virtual view point with 360 degree which records is the moving picture information. According to the virtual location and view point, the additional information provider(30) indicates the additional information for the object within the offered video. The 360 degree video information comprises GPS coordinates data according to each frame.

Description

Method and system for providing additional information based on video images}

The present invention relates to a three-dimensional geographic information service, and more particularly, to a method and system for providing video-based additional information that can provide image information and additional information of a desired location by using a 360-degree video.

As the Internet is emerging as a new industry and various services and businesses are being activated, the functions and effects of multimedia are showing remarkable growth. One of the hot spots recently in the Internet field is 3D geographic information service. In other words, it provides a three-dimensional map on the Internet using spatial information built using Geographic Information System (GIS) technology such as aerial photography and precision surveying. The service is getting attention recently.

In addition to the three-dimensional map, a service that provides a panorama of the photographic image of the main road has appeared. However, such a method of implementing a geographic information service is implemented by synthesizing image information photographed by a plurality of cameras, which is expensive and has a problem in that a production period is long.

In addition, the conventional 360-degree image service only provides a 360-degree image at a specific point linked to the map information, but does not actively provide a 360-degree image at the changed position in real time according to the change of the user's position, The additional information is also not provided in real time.

The technical problem to be achieved by the present invention is to provide a video-based additional information providing method and system that can provide a video to the user according to the user's virtual location and the virtual view by using 360-degree video information and effectively provide the additional information accordingly. There is.

According to an aspect of the present invention, there is provided a video-based additional information providing method comprising: providing an image corresponding to a virtual position and a virtual viewpoint requested by a user with 360-degree video information photographed in advance; And displaying additional information for an object in the provided image according to the virtual position and the virtual viewpoint.

Here, the 360 degree video information may have GPS coordinate data for each frame, and the providing of the image may provide an image corresponding to the virtual position by using the GPS coordinate data.

The video-based additional information providing method may further include displaying an avatar indicating the virtual location and the virtual viewpoint on a map image prepared in advance.

In the displaying of the additional information, the additional information may be displayed according to the virtual position and the virtual viewpoint indicated by the avatar.

In the displaying of the additional information, the additional information may be displayed by being superimposed on the provided image. In this case, the additional information may be displayed by being superimposed on the reference position of the object. You can decide. The size at which the additional information is displayed may be determined according to a difference between the GPS coordinates of the point at which the current image frame is obtained and the coordinates of the reference position of the object. The difference between the GPS coordinates of the acquired point and the coordinates of the reference position of the object and the central orientation information of the current image frame may be determined.

The additional information display step may change the position and size at which the additional information is displayed according to the change of the current image frame due to the change in the virtual position.

In order to solve the above technical problem, the video-based additional information providing system according to the present invention includes a spatial information providing unit for providing an image corresponding to a virtual position and a virtual viewpoint requested by the user with 360-degree video information photographed in advance; And an additional information providing unit displaying additional information for an object in the provided image according to the virtual location and the virtual viewpoint.

In order to solve the above technical problem, there is provided a computer-readable recording medium having recorded thereon a program for executing the video-based additional information providing method according to the present invention.

According to the present invention described above, by using the 360-degree video information can provide an image to the user according to the user's virtual position and the virtual view point and effectively provide additional information accordingly.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description and the accompanying drawings, the substantially identical components are represented by the same reference numerals, and thus redundant description will be omitted. In addition, in the following description of the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

1 is a block diagram of a video-based additional information providing system according to an embodiment of the present invention. Referring to FIG. 1, the video-based additional information system includes a spatial information providing unit 10, a video DB 20, an additional information providing unit 30, an additional information DB 40, and a user interface 50. .

The spatial information providing unit 10 provides an image corresponding to a virtual position and a virtual viewpoint requested by the user with 360-degree video information photographed in advance. The spatial information DB 20 is constructed with spatial information consisting of 360-degree video information having GPS coordinate data for each frame, and the spatial information providing unit 10 uses the GPS coordinate data to determine a virtual position and a virtual viewpoint requested by the user. Provides a video corresponding to the.

The acquisition of 360 degree video information with GPS coordinate data is described below. FIG. 2A illustrates an example of a 360-degree video camera that is currently commercially available, and FIG. 2B shows a 360-degree spherical image captured by the illustrated camera in a plan view. The 360-degree video camera as shown in FIG. 2A is provided with a plurality of lenses (for example, six lenses) in a single camera, and acquires 360-degree image information of the surrounding environment as shown in FIG. It allows you to acquire images and videos. In order to obtain a 360 degree video of the surroundings of the road, as shown in FIG. 3, a 360 degree video camera 1 is mounted on the vehicle, and then the surrounding environment of the driven road is photographed while driving the road. Then, the GPS coordinate data of each frame of the 360 degree video acquired through the 360 degree video camera 1 is acquired through the GPS receiver 2 and then matched with the corresponding frame of the 360 degree video as shown in FIG. The spatial information DB 20 is stored for each path. The 360 degree video stored in the spatial information DB 20 is appropriately modified to be provided to the user. For example, 360-degree video frames are extracted at a constant ratio (for example, 1 frame per 1m) to generate a uniform 360-degree video, and the segmented by dividing the uniformed video according to the branching point of the road or a predetermined reference distance. 5 is a diagram illustrating a video segmented and segmented. Referring to FIG. 5, a video is segmented so that a neighboring segment and a predetermined region overlap at a segmentation point of a segment for smooth screen processing of a connection portion between segments. In this case, as shown in FIG. 6, the overlapping frames in the video of each segment are matched with information about the overlapping area ID and stored in the spatial information DB 20. In addition, as the photographing progresses along the road as shown in FIG. 6, the photographing progression direction also changes according to the bending direction of the road. Therefore, the center direction information of each frame of the curved area in the segment (the user may Information about the direction to be viewed in actual driving) is stored together. In FIG. 6, the central direction is set to 0 degrees in the true north direction, and the angle of the direction that the user faces when proceeding along the road is displayed in the range of 0 degrees to 360 degrees.

As described above, the spatial information providing unit 10 displays 360-degree video information provided in the spatial information DB 20 and displays an image corresponding to a virtual position and a virtual viewpoint requested by the user through the user interface 50.

Hereinafter, the image display of the spatial information providing unit 10 will be described in detail. 7 illustrates an example of a screen provided by the spatial information providing unit 10 to the user interface 50 through a web browser. Referring to FIG. 7, a map image prepared in advance is displayed at the bottom of the web browser, and an avatar (a red arrow shape) indicating a virtual location and a virtual viewpoint requested by the user is displayed. The map image at the bottom displays a section in which the spatial information is constructed in the spatial information DB 20 with a blue line to guide a path through which the user searches for a virtual space. The avatar displayed on the map image indicates the virtual position and the virtual viewpoint requested by the user in the position and direction of the arrow. The movement of the virtual location means that the user converts location information located in the virtual space into new location information, wherein the spatial information providing unit 10 changes the location of the avatar according to the user's keyboard or mouse operation. Accordingly, the current frame is changed to a frame corresponding to the new position to provide a continuous photorealistic image. The virtual viewpoint can also be operated with the user's keyboard or mouse. Since each frame of the movie holds all 360-degree image information at the shooting point, such as left-right panning, vertical rotation, and zoom level adjustment of the virtual viewpoint. Processing is possible within this currently visible frame. The virtual position movement may have a value of forward or backward, and the left and right rotations of the user's viewpoint may have a value from 0 degrees (north north direction) to 360 degrees.

FIG. 8 shows how an avatar's movement in a virtual space appears on a map image according to a user's input. In order to realize a virtual space, unique identification information is allocated to each branch of each distance where a 360-degree video is captured, and unique identification information can be allocated to distances at each branch. As shown in FIG. 8, when the distance where the current avatar is located is referred to as 1st distance and passes through a fork in front of it, the user may enter any one of 2, 3, and 4th distances. The map image on the right side of FIG. 8 shows a result of the avatar entering the distance from the distance 1 to the distance 2 through the fork.

The additional information providing unit 30 displays additional information for the object in the image displayed on the user interface according to the virtual position and the virtual viewpoint requested by the user. Here, the additional information is information on buildings, roads, malls, and other objects existing in the image, such as address, phone number, number of recommendation, and advertisement information. Or a symbol indicating a link for providing the information.

The additional information about each of the objects in the virtual space is stored in the additional information DB 40 and inquired by the additional information providing unit 30. On the other hand, such additional information may be provided from various other DBs connected to the additional information providing unit 30 in addition to the additional information DB (40).

FIG. 9 is a flowchart illustrating a process in which the additional information providing unit 30 inquires and displays additional information about necessary objects from the additional information DB 40 according to an embodiment of the present invention.

When a movement command of the virtual position and the virtual viewpoint occurs by the manipulation of the user's keyboard or the mouse in step 910, the spatial information providing unit 10 moves to the avatar on the map image with the new virtual position and the virtual viewpoint in step 920. Move and search the corresponding video information in the spatial information DB 20 to display an image corresponding to the virtual position and the virtual viewpoint on the user interface 50.

If the avatar has not entered the new distance through the branch point, the process returns to step 910 (step 930), and steps 910 and 920 are repeated.

If the avatar has entered the new distance through the branch point, the process proceeds to step 940 where the additional information providing unit 30 inquires additional information about the objects included in the video of the new distance from the additional information DB 40 and the user interface. Overlaid on the video screen of (50). The information that can be used to search for additional information may use unique identification information of the corresponding distance, identification information of a branch point corresponding to both ends of the distance, or coordinate information of an area including the corresponding distance. The additional information DB 40 stores such identification information or coordinate information, and an object belonging thereto and corresponding additional information. The side information consists of the coordinates (eg, (X, Y) coordinates) of the reference position of the associated object with the content of the information.

10 is a diagram illustrating in detail how additional information is displayed on the user interface 50. As shown, the additional information is displayed by being superimposed on the reference position of the object (for example, when the object is a mall). The additional information may be displayed in various forms such as a speech bubble and an icon. As such, in order to display the additional information at a specific position on the image by overlapping, the location where the additional information is displayed must be determined.

FIG. 11 is a reference diagram for explaining a relationship between one image frame constituting a video and an arbitrary object (for example, a building). Referring to FIG. 11, an image frame has GPS coordinates (x, y) and center orientation information C of a point at which the image is acquired. The GPS coordinates (x, y) soon become the coordinates of the center position of the current video frame. Therefore, if the difference between the GPS coordinates (x, y) and the coordinates (X, Y) of the reference position of the building is obtained, the vector T from the center position of the image can be obtained. Then, the relative bearing Θ of the building based on the center direction can be obtained using the vector indicating the center direction C and the vector T.

Θ can be obtained according to the following equation.

The value of Θ may be used to determine the location where the additional information should overlap in the image. That is, since the image frame has the center orientation information, the additional information may be displayed at a position Θ away from the point corresponding to the center orientation information.

In addition, the size D = | T | of the vector T corresponds to the distance from the center position of the image to the reference position of the building. Accordingly, the size of the icon, speech bubble, or the like for displaying additional information may be adjusted using the size D of the vector T to indicate perspective. For example, according to a predetermined criterion, the speech bubble displaying additional information is relatively large when the D value is small, and the speech balloon is relatively large when the D value is large.

FIG. 12 is a diagram illustrating a state in which a vector and a relative orientation change to a reference position of a building according to a change of a virtual position of a user. In FIG. 12, a red circle indicates a reference position of a point at which each image frame is acquired, and a blue circle indicates a reference position of a building to which additional information is to be displayed. When the user changes the virtual position by operating the mouse or the keyboard, the position of the avatar (image information acquisition reference position) and the direction of looking at the building from the position change accordingly. Accordingly, the image frame is replaced and displayed at regular intervals in the user interface 50, and thus, the vector and the relative orientation to the building to which the additional information is displayed are changed. The additional information providing unit 30 displays the additional information by changing its position and size according to the vector and the relative orientation to the corresponding building that changes as the image frame changes. Therefore, the display position and size of the additional information can be adaptively and continuously reflected according to the virtual position and the virtual viewpoint of the user.

Meanwhile, the above-described embodiments of the present invention can be written as a program that can be executed in a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium. The computer-readable recording medium may include a magnetic storage medium (eg, ROM, floppy disk, hard disk, etc.), an optical reading medium (eg, CD-ROM, DVD, etc.) and a carrier wave (eg, internet). Storage medium).

So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

1 is a block diagram of a video-based additional information providing system according to an embodiment of the present invention.

2 is a view showing a plan view of a video camera and a 360-degree spherical image captured by the camera that is currently commercially available.

3 is a diagram illustrating a state in which a 360 degree video camera and a GPS receiver are mounted on a vehicle.

4 is a diagram illustrating a state in which frames and GPS coordinate data of a video photographed for each shooting path are matched and stored.

5 is a diagram illustrating a video segmented and segmented.

FIG. 6 is a diagram illustrating a state in which center orientation information for each frame of a road curved portion is additionally stored.

7 illustrates an example of a screen provided by the spatial information providing unit 10 to the user interface 50 through a web browser.

FIG. 8 shows how an avatar's movement in a virtual space appears on a map image according to a user's input.

9 is a flowchart illustrating a process of inquiring and displaying additional information about necessary objects according to an embodiment of the present invention.

10 is a diagram illustrating in detail how additional information is displayed on the user interface 50.

11 is a reference diagram for explaining a relationship between one image frame constituting a video and an arbitrary object.

FIG. 12 is a diagram illustrating a state in which a vector and a relative orientation change to a reference position of a building according to a change of a virtual position of a user.

Claims (19)

Providing an image corresponding to a virtual position and a virtual viewpoint requested by a user with 360-degree video information photographed in advance; And And displaying additional information for an object in the provided video according to the virtual position and the virtual view point. The method of claim 1, The 360-degree video information has GPS coordinate data for each frame, and the providing of the video includes providing the video corresponding to the virtual position by using the GPS coordinate data. The method of claim 1, And displaying the avatar representing the virtual location and the virtual viewpoint on a map image prepared in advance. The method of claim 3, The additional information display step of displaying the additional information according to the virtual position and the virtual view represented by the avatar, characterized in that the video-based additional information providing method. The method of claim 1, The additional information display step includes displaying the additional information superimposed in the provided image. The method of claim 5, The additional information display step includes displaying the additional information superimposed on a reference position of the object. The method of claim 6, The displaying of the additional information comprises: determining a location and a size at which the additional information is displayed using the GPS coordinates of the point at which the current image frame is acquired and the coordinates of the reference position of the object. The method of claim 7, wherein The size of the additional information to be displayed is determined based on the difference between the coordinates of the GPS coordinates of the point where the current image frame is obtained and the reference position of the object. The method of claim 7, wherein The position at which the additional information is to be displayed is determined based on a difference between the coordinates of the GPS coordinates of the point at which the current image frame is acquired, the coordinates of the reference position of the object, and the central orientation information of the current image frame. How to Provide. The method of claim 7, wherein And displaying the additional information by changing the position and size at which the additional information is displayed according to the change of the current image frame due to the change in the virtual position. A computer-readable recording medium having recorded thereon a program for executing the video-based additional information providing method according to any one of claims 1 to 10. A spatial information providing unit which provides an image corresponding to a virtual position and a virtual viewpoint requested by the user with 360-degree video information previously photographed; And And an additional information providing unit displaying additional information for an object in the provided image according to the virtual position and the virtual viewpoint. The method of claim 12, The 360-degree video information has GPS coordinate data for each frame, and the spatial information providing unit provides an image corresponding to the virtual position by using the GPS coordinate data. The method of claim 12, The spatial information providing unit displays an avatar indicating the virtual location and the virtual viewpoint on a map image prepared in advance. The method of claim 14, The additional information providing unit displays the additional information according to the virtual position and the virtual viewpoint indicated by the avatar. The method of claim 12, The additional information providing unit displays the additional information superimposed within the provided image. The method of claim 16, The additional information providing unit displays the additional information superimposed on the reference position of the object. The method of claim 17, The additional information providing unit determines a location and size at which the additional information is displayed by using the GPS coordinates of the point where the current image frame is obtained and the coordinates of the reference position of the object. The method of claim 18, The additional information providing unit changes the position and size at which the additional information is displayed according to the change of the current image frame due to the change of the virtual position.

Images