KR101842210B1 - Contents providing method through eye tracking and apparatus thereof - Google Patents

Abstract

The present invention relates to a method and an apparatus for providing content based on eye tracking. More specifically, the present invention relates to a method and an apparatus for obtaining a region of interest by tracking an eye of a user from a sensor of a wearable apparatus, and providing content on the obtained region of interest. According to the present invention, the content of a place where the eye of the user remains can be provided through eye tracking without an additional control even in outdoor trekking or a situation where both hands of the user are not free.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a content tracking method,

More particularly, the present invention relates to a method and apparatus for acquiring a region of interest by tracking a user's gaze from a sensor of a wearable device and providing content for the region of interest .

Eye tracking technology is being used to make life easier for modern people by analyzing pupil movement and size change. Since gaze is the most sensitive and responsive sensory organ of the five senses, it is the most effective means of interacting with information devices. It is used not only in various fields such as automobile, medical, market research and usability analysis, game, exercise, It is also possible to provide contents with high information transmission effect through user analysis.

Background Art of the present invention is disclosed in Korean Patent Laid-Open Publication No. 10-2013-0043366.

The present invention automatically acquires a region of interest of a user by automatically controlling a camera provided in a wearable device based on a line-of-sight tracking technology, and provides content for the region of interest.

According to an aspect of the present invention, there is provided a method of providing a line-tracking-based content through a single wearable device, the method comprising: tracking a user's position, movement direction and gaze direction from a wearable device; Or moving to the right, acquiring a field of view to which the user is to gaze; designating the field of view as an area of interest when the moved line of sight is maintained for a predetermined period of time; The method comprising the steps of: retracing a viewing direction; activating the camera of the wearable device in a continuous shooting mode when the viewing direction moves from the front to the designated region of interest, until the viewing direction in the designated region of interest faces the front Continuously photographing the visual range of the user, And transmitting the generated panoramic image to a terminal associated with the wearable device, the method comprising the steps of: transmitting the photographed image to a content providing module; generating a panoramic image by combining the continuously shot images with the content providing module; .

According to an aspect of the present invention, there is provided a method of providing a content of a plurality of users' eyesight tracking through a multiple wearable device, the method comprising: tracking a position, a movement direction and a gaze direction of a user from each of a first wearable device and a second wearable device, Acquiring a field of view in which the user gazes when the visual direction moves to the left or right with reference to the direction of movement; if the moved visual direction is maintained for a predetermined time, Region and a second region of interest, and retraining the direction of the user's gaze at a point in time when the first region of interest and the second region of interest are designated, wherein the gaze direction is the first region of interest and the second region of interest If the content is not moved to the area, Wherein the content providing module generates a panoramic image by combining the first ROI and the second ROI if the first ROI and the second ROI are similar at a point in time when the location information and the viewing direction are similar, And transmitting the generated panoramic image to the first terminal and the second terminal respectively associated with the first wearable device and the second wearable device.

According to an aspect of the present invention, a content providing apparatus includes a communication unit that performs a network connection with at least one wearable device or a terminal, a position information obtaining unit that obtains position information of the user, a sensor that tracks a movement direction and a gaze direction of the user, A visual field range in which the user gazes is obtained when the visual direction moves to the left or right relative to the movement direction, and when the moved visual direction is maintained for a predetermined time, the visual range is designated as a region of interest A control unit for activating a continuous shooting mode of the camera when the gaze direction of the user is retraced at a point in time when the gaze region is moved and the gaze direction moves from the front to the designated gaze region, Of the user ' s viewing range until the < RTI ID = 0.0 > direction < / RTI > It may include shooting and continuous shooting an image taking portion for storing parts of the memory, to generate a panoramic image by combining the images continuously photographed image generation unit for transmitting to the terminal or a wearable device, in conjunction with the communication unit.

The present invention can provide the contents of the place where the gaze stays through the gaze tracking without any special operation even when the outdoor trekking or both hands are not free.

Further, the present invention can extract statistical data on the number of persons and time of people who are interested in the area based on the same interest area designated by a plurality of users through interlocking of multiple wearable devices, The related contents can be produced or updated.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an exemplary view schematically illustrating a configuration of a wearable device according to an embodiment of the present invention; FIG.
FIG. 2 and FIG. 3 are flowcharts for explaining a method for providing a user's gaze-tracking-based content in a personal running through a single wearable device according to an embodiment of the present invention.
FIG. 4 and FIG. 5 are flowcharts for explaining a method of providing gaze-tracking-based content for a plurality of users during a group travel through a multi-wearable device according to an embodiment of the present invention;
6 is a block diagram for explaining a content providing apparatus according to an embodiment of the present invention;

While the present invention has been described in connection with certain exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and similarities. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

FIG. 1 is an exemplary view schematically illustrating a configuration of a wearable device according to an embodiment of the present invention.

A wearable device may be referred to as various terms such as a smart glass, an HMD or a see through display.

The wearable device may be in a form that can be closely attached to the head of the user, and may be configured to provide various information to the user through the display.

The wearable device may include a camera 120 and a sensor 160 and may provide content based on the captured image and sensing information generated by the camera 120 and the sensor 160. [

The wearable device may receive inputs that are used to control operations from the user via the user interface 180.

The wearable device may include various components necessary for interoperating with the wearable device of the terminal or at least one other user through the content providing module 10 to share the content.

FIG. 2 and FIG. 3 are flowcharts for explaining a method of providing a user's gaze-tracking-based content in a personal running through a single wearable device according to an embodiment of the present invention. Depending on the embodiment, at least one of the steps shown in FIG. 2 may be omitted, and the order of the steps shown may vary.

In step 210, the content providing device tracks the user's position, movement direction, and gaze direction through the sensor 160 of a single wearable device.

In step 220, the content providing apparatus obtains a viewing range in which the user gazes when the gaze direction tracked in step 210 moves leftward or rightward with respect to the direction of the user's movement.

In step 230, the content providing apparatus designates the field of view as a region of interest if the direction of the line of sight moved in Step 220 is maintained for a predetermined time.

In step 240, the content providing apparatus retraces the user ' s gaze direction for a predetermined time at the point in time when the region of interest is specified.

In step 250, the content providing apparatus activates the camera 120 in the continuous shooting mode when the retraced direction of the eyes is moved from the front to the designated attention area in step 240.

In step 260, the content providing apparatus consecutively photographs the user's field of view until the gaze direction is directed to the front in the designated area of interest in step 250, and transmits the successively photographed image to the content providing module 10.

In step 270, the content providing apparatus combines the consecutively captured images to generate a panoramic image and transmits the panoramic image to a terminal linked to the content providing apparatus. At this time, the content providing apparatus transmits the local information corresponding to the panoramic image to the terminal interlocked with the content providing apparatus.

Referring to FIG. 3, when the user's gaze direction moves to the left or right with reference to the user's movement direction, the content providing apparatus obtains the user's view range in the moved gaze direction, as shown in 301 of FIG. At this time, if the moved line of sight is maintained for a predetermined time, the field of view is designated as a region of interest.

When the region of interest is designated, the content providing apparatus retraces the direction of the line of sight of the user for a predetermined time at a point of time when the region of interest is designated. 3, if the direction of the user's eyes retraced is moved to the region of interest, the user's viewing range is changed until the user's gaze direction is directed to the front, that is, Continuous shooting. Finally, the panoramic image can be generated by combining the continuously shot images.

FIG. 4 and FIG. 5 are flowcharts for explaining a method for providing a plurality of user's gaze-tracking-based contents in a group driving through a multi-wearable device according to an embodiment of the present invention. Depending on the embodiment, at least one of the steps shown in FIG. 4 may be omitted, and the order of the steps shown may vary.

In the embodiment described with reference to FIG. 4, it is assumed that wearable devices possessed by at least one other user and a user group are formed. The formation of the user group can be performed through an authentication procedure with a server connected through a wireless wide area network. The formation of such a user group can use various conventional methods.

In step 410, the content providing device tracks the user's position, movement direction, and gaze direction from each of the first wearable device and the second wearable device.

In step 420, the content providing apparatus obtains a viewing range in which the user gazes when the gaze direction moves leftward or rightward with respect to the movement direction.

In step 430, the content providing apparatus designates the field of view range as the first ROI and the second ROI if the moved line of sight is maintained for a predetermined time.

In step 440, the content providing apparatus retrains the direction of the user's gaze at a point in time when the region of interest is specified.

In step 450, the content providing device transmits the first ROI and the second ROI to the content providing module if the gaze direction does not move from the front to the first ROI and the second ROI.

In step 460, the content providing device generates the panoramic image by combining the first ROI and the second ROI if the first ROI and the second ROI are similar to each other at the designated time.

In step 470, the content providing apparatus transmits the generated panoramic image to the first terminal and the second terminal, respectively, associated with the first wearable device and the second wearable device.

In performing the steps 410 to 470, the content providing apparatus according to an exemplary embodiment of the present invention may specify the first region of interest from the first wearable device and designate the second region of interest from the second wearable device In parallel. In addition, the content providing apparatus can provide the local information based on the panorama image as well as the location information corresponding to the panorama image.

As described above, the content providing apparatus according to an exemplary embodiment of the present invention can extract and store a region of interest through line-of-sight tracking without operating a separate wearable device. In addition, when the group driving is finished, the ROI data of the individual users can be acquired in the riding route, and the ROI data of a plurality of users can be acquired.

Referring to FIG. 5, if the gaze direction moves to the left or the right with reference to the movement direction of the first user, as shown in 501 of FIG. 5 based on the gaze tracking of a plurality of users, Of the field of view. At this time, if the moved line of sight is maintained for a predetermined time, the field of view is designated as a region of interest. In the same manner, the field of view of the second user as shown in 502 of FIG. 5 and the field of view of the third user as shown in 503 of FIG. 5 can be obtained. The view range of 501 to 503 in FIG. 5 is not an area in which the same user repeatedly gazed but is a region in which a plurality of users gazed at a similar point of view. Therefore, a panorama image is generated by designating the region as a region of interest. The generated panoramic image can provide the local information to the user when the local information exists based on the location information. If the local information does not exist, the information of the corresponding region can be updated.

6 is a block diagram illustrating a content providing apparatus according to an embodiment of the present invention. Depending on the embodiment, at least one of the components shown in Fig. 6 may be omitted.

The content providing apparatus includes a user input / output interface 610, a photographing section 620, a sensor section 630, a position information obtaining section 640, a communication section 650, a control section 660, a memory section 670, (680), which may be connected to each other via a bus (BUS).

The user input / output interface 610 may receive input from a user and display various information.

The photographing unit 620 can perform photographing when a user has input or an event that is set, thereby generating a photographing image. The generated photographed image may be provided to the content providing module 10 or the user terminal.

The sensor unit 630 tracks the movement direction and the viewing direction of the user. For example, it is possible to track the user's movement direction and gaze direction from the gyro sensor and the acceleration sensor.

The location information obtaining unit 640 obtains the location information of the user through the GPS signal.

The communication unit 650 can connect at least one wearable device to at least one terminal through a network, and exchange data in a connected state.

The control unit 660 obtains a field of view in which the user gazes when the direction of the user's gaze moves left or right based on the direction in which the user moves from the sensor unit 630 and the position information obtaining unit 640. [

The controller 660 designates the field of view as an area of interest when the user's line of sight is maintained for a predetermined time, and retraces the line of sight of the user at the point of time when the area of interest is designated.

The control unit 660 activates the continuous shooting mode of the camera when the viewing direction moves from the front to the designated ROI.

The control unit 660 can provide the local information that coincides with the location information of the panoramic image stored in the memory unit 670.

The memory unit 670 stores at least one of a user's interest area, a panorama image, map data, and local information on a building and a place under the control of the control unit 660.

The image generating unit 680 combines the continuous images captured by the photographing unit 620 to generate a panoramic image, and stores the generated panoramic image in the memory unit 670. At this time, the panoramic image includes positional information and time information at a time point when the panoramic image is designated as an area of interest to be combined. The panoramic image described above can be transmitted to the terminal interlocked with the wearable device through the communication unit 650.

The present invention has been described above with reference to the embodiments thereof. Many embodiments other than the above-described embodiments are within the scope of the claims of the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The disclosed embodiments should, therefore, be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

10: Content providing module
120: camera
140:
160: Sensor
180: User interface

Claims (8)

A method for providing a gaze tracking-based content through a single wearable device,
Tracking the user's position, movement direction and gaze direction from the wearable device;
Obtaining a visual range in which the gaze direction is based on the movement direction;
Designating the field of view as an area of interest and re-tracing the direction of the user's line of sight at the point of time when the area of interest is designated;
Activating the camera of the wearable device in a continuous shooting mode when the viewing direction moves from the front to the designated ROI;
Continuously photographing the user's field of view until the gaze direction is directed to the front of the designated area of interest, and transmitting the continuously photographed image to the content providing module; And
Generating a panoramic image by combining the continuously photographed images with the content providing module, and transmitting the generated panoramic image to a terminal interlocked with the wearable device.
The method of claim 1,
Wherein the step of acquiring the field of view in which the gaze direction is based on the movement direction,
The moving direction is set as a front face, and when the viewing direction of the user moves from the front side to the left side or right side, the viewing range to be gazed by the user is obtained.
The method of claim 1,
Wherein the step of designating the field of view as a region of interest and re-tracing the direction of the user's line of sight at the point of time the region of interest is designated
Wherein the visual field is designated as a region of interest if the direction of the user's gaze is moved from the front to the left or right and the direction moved for a predetermined time is maintained.
The method of claim 1,
Wherein the content providing module generates a panoramic image by combining the continuously photographed images and transmitting the generated panoramic image to the terminal associated with the wearable device,
And transmitting the local information together with the panoramic image to the terminal through the panoramic image including the position information and the time information at the point of time when the region of interest is designated.
A method for providing a content based on eye-tracking of a plurality of users through a multi-wearable device,
Tracking the user's position, movement direction and gaze direction from each of the first wearable device and the second wearable device;
Acquiring a field of view in which the user gazes when the gaze direction moves leftward or rightward with respect to the movement direction;
The visual range is designated as a first area of interest and a second area of interest when the moved direction of sight is maintained for a predetermined period of time and the visual direction of the user is designated at a designated time of the first area of interest and the area of interest, Tracking;
Transmitting the first ROI and the second ROI to the content providing module if the viewing direction does not move from the front to the first ROI and the second ROI; And
Generating a panoramic image by combining the first ROI and the second ROI, and transmitting the generated panoramic image to a first terminal and a second terminal interlocked with the first wearable device and the second wearable device, respectively, / RTI >
The method of claim 5,
Generating a panoramic image by combining the first ROI and the second ROI, and transmitting the generated panoramic image to a first terminal and a second terminal interlocked with the first wearable device and the second wearable device, respectively, ,
Wherein the content providing module combines the first ROI and the second ROI when the first ROI and the second ROI are similar to each other at a specified time point.
A communication unit for performing a network connection with at least one wearable device or terminal;
A location information acquisition unit for acquiring location information of a user;
A sensor unit for tracking a movement direction and a viewing direction of the user;
Acquiring a field of view in which the user gazes when the gaze direction moves leftward or rightward with respect to the movement direction and designates the field of view as an area of interest when the moved gaze direction is maintained for a predetermined time, A control unit for retracing the direction of the user's gaze at a point in time when the point of interest is designated and activating a continuous shooting mode of the camera when the point of sight moves from the front to the designated point of interest; And
And a photographing unit for continuously photographing the user's field of view until the gaze direction is directed to the front of the designated area of interest, and storing the continuously photographed image as a memory unit.
8. The method of claim 7,
An image generating unit for generating a panoramic image by combining the continuously photographed images and transmitting the panoramic image to a wearable device or a terminal connected to the communication unit; And
Further comprising a memory unit for storing the continuously photographed image, the panorama image, the map data, and the local information about the building and the place.

Images