KR101865329B1 - HMD headset apparatus and input method thereof - Google Patents

Abstract

The HMD headset apparatus of the present invention includes an electroencephalogram sensor for acquiring a user's electroencephalogram (EEG) signal, a gravity sensor for sensing gravity, a gyro sensor for sensing a change in orientation, a display unit And obtaining information on a user's eye flicker by analyzing the brain conduction signal, acquiring information on a user's head motion through the gravity sensor and the gyro sensor, and using the blinking information and the head motion information on the display unit And controls the movement of the cursor, which is the input point to be displayed, and the click operation.
According to the present invention, it is possible to control the contents of the HMD headset device without using a hand using an electroencephalogram signal, thereby contributing to the convenience of the user.

Description

[0001] The present invention relates to an HMD headset apparatus and an input method thereof,

The present invention relates to an input device for an HMD headset, and more particularly, to a device for performing click and cursor movement functions of a computer mouse without using a hand movement using data obtained from an electroencephalogram sensor, a gravity sensor, and a gyro sensor.

The present invention is based on the results of a research result (IITP-2016-R0992-15-1008) of the Future Creation Science Department, "Giga KOREA Project" and Future Development Research Program of ICT Research Center of the Institute of Creation Sciences and Information and Communication Technology Promotion Center will be.

HMD (Head Mounted Display) is a next-generation image display device that can be used for medical devices that can enjoy large-sized images on a large screen, .

Recently, various wearable devices have been developed in accordance with the trend of weight reduction and miniaturization of digital devices, and HMDs are also widely used. The HMD can be combined with augmented reality technology, N screen technology, etc., beyond the simple display function, to provide various convenience to the user.

In the case of an HMD headset, it is in close contact with the user's eyes to see the display device. The HMD headset is worn in the form of a spectacle or a hat. An input device for controlling the displayed content or window screen is a touch pad, .

Conventional HMD headsets require a separate controller using the hands or use the hands, which can cause physical pain in the arm or wrist during long operation. In addition, some of the devices are wired to prevent free movement. There is a problem in that it is unsuitable for everyday use because it is unsuitable for fashion when worn out, and there is not a lot of contents to enjoy, and the interest after purchasing falls sharply. In this way, since the input device must be operated by using the hand to control the screen while wearing the HMD headset, it is possible to feel a different feeling or inconvenience than usual, and it is more inconvenient There is a problem.

Korean Patent Publication No. 10-2012-0015637

SUMMARY OF THE INVENTION The present invention is conceived to solve the above-mentioned problems, and it is an object of the present invention to provide an HMD headset device capable of controlling contents of an HMD headset without using a hand by using an EEG information acquisition sensor, a gravity sensor and a gyro sensor, The purpose is to provide.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

In order to achieve the above object, an HMD headset device of the present invention includes an electroencephalogram sensor for acquiring a user's electroencephalogram (EEG) signal, a gravity sensor for sensing gravity, a gyro sensor for sensing a change in orientation, A display unit for displaying a signal and information on a user's eye flicker by analyzing the brain wave signal, obtaining information on a user's head motion through the gravity sensor and the gyro sensor, And controls the movement of the cursor, which is an input point displayed on the display unit, and the click operation.

The controller analyzes the EEG signal received through the EEG sensor and controls the user to perform a click operation at a point where the cursor is located when it is determined that the user is blinking.

The controller may move the cursor displayed on the screen by using the user's head movement information received through the gravity sensor and the gyro sensor.

The EEG sensor periodically sends the acquired EEG signal to the controller. The controller determines whether the user wears the HMD headset device using the Poor signal among the EEG signals received from the EEG sensor. The blinking information of the user can be obtained by using the signal.

The control unit initializes signal values for a pitch and a roll direction received from the gravity sensor when the user determines that the user wears an HMD headset, The position of the cursor can be controlled according to the signal.

The control unit obtains the vertical movement of the user's head from the pitch signal of the gyro sensor, obtains the left and right rotation of the user's head from the yaw signal of the gyro sensor, The position can be moved.

The controller analyzes the signal inputted from the sensors to control the cursor movement and the click operation, and controls the size and position of the window of the application executed on the screen of the display unit through the operation of the cursor.

The control unit may move, rotate, or control the distance between the user's eyes and the window according to the execution of the cursor.

In an input method in an HMD headset apparatus of the present invention, a method of acquiring a user's EEG (electro-encephalogram) signal, acquiring information about a head movement of a user, analyzing the EEG signal, And controlling the movement of the cursor, which is an input point displayed on the screen, and the click operation using the blink information and the head motion information.

If the user's eye flicker is determined to be caused by analyzing the electroencephalogram signal, the controller can control to perform a click operation at a point where the cursor is located.

The cursor displayed on the screen can be moved using the received user's head motion information.

Using the Poor signal, the user can determine whether or not the user wears the HMD headset device, and obtain blinking information of the user using the Blink signal from the brain conduction signal.

The application window displayed on the screen can be moved, rotated, or the distance between the user's eyes and the window can be controlled according to the execution of the cursor operation.

According to the present invention, it is possible to control the contents of the HMD headset device without using a hand using an electroencephalogram signal, thereby contributing to the convenience of the user. In particular, the present invention is expected to be more useful to users with disabilities.

1 is a block diagram illustrating an internal configuration of an HMD headset apparatus according to an embodiment of the present invention.
2 is a flowchart illustrating a method of performing a click operation using an EEG signal in an HMD headset apparatus according to an embodiment of the present invention.
FIG. 3 is a flowchart illustrating a method of performing cursor movement and screen movement in the HMD headset apparatus according to an embodiment of the present invention.
4 is an exemplary screen of an HMD headset apparatus according to an embodiment of the present invention.
FIG. 5 is a view illustrating an example of controlling the window size in the HMD headset according to an exemplary embodiment of the present invention. Referring to FIG.
6 is a view for explaining window position control in an HMD headset apparatus according to an embodiment of the present invention.
7 is a view for explaining window rotation control in an HMD headset apparatus according to an embodiment of the present invention.
8 is a view for explaining window distance control in an HMD headset apparatus according to an embodiment of the present invention.
9 is a diagram for explaining a control method using a gyro sensor in an HMD headset apparatus according to an embodiment of the present invention.
10 is a view for explaining an active state control method in an HMD headset apparatus according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted in an ideal or overly formal sense unless expressly defined in the present application Do not.

In the following description of the present invention with reference to the accompanying drawings, the same components are denoted by the same reference numerals regardless of the reference numerals, and redundant explanations thereof will be omitted. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

The present invention relates to an HMD headset device capable of recognizing a user's control signal without using a hand.

1 is a block diagram illustrating an internal configuration of an HMD headset apparatus according to an embodiment of the present invention.

1, an HMD headset 100 according to an exemplary embodiment of the present invention includes an electroencephalogram sensor 110, a gravity sensor 120, a gyro sensor 130, a controller 140, and a display unit 150 .

The electroencephalogram sensor 110 serves to acquire a user's electroencephalogram (EEG) signal. The EEG sensor 110 can acquire various EEG signals such as Blink, Poor, Attention, Meditation, Delta, Low Alpha, High Alpha, Low Beta, High Beta, Low Gamma, High Gamma and Connect State.

In the present invention, the Poor signal is used for whether or not the headset is worn and for communication connection,

The Blink signal is used for cursor clicks.

The gravity sensor 120 serves to sense gravity.

The gyro sensor 130 serves to sense a change in azimuth.

The control unit 140 processes the signals of the electroencephalogram sensor 110, the gravity sensor 120, and the gyro sensor 130.

The controller 140 obtains information on the movement of the user's head through the gravity sensor 120 and the gyro sensor 130 and obtains information on the user's eye blinking through the electroencephalogram sensor 110. [ The control unit 140 executes a cursor movement instruction and a click instruction, which are input points displayed on the display unit 150, using the user's head motion information and eye blinking information.

The display unit 150 outputs a video signal.

In the present invention, the control unit 140 analyzes the signals input from the sensors 110 to 130 to control the cursor movement and the click operation, and controls the operation of the application executed on the screen of the display unit 150 You can control the size and position of the window.

At this time, the controller 140 can move or rotate the window or control the distance between the user's eyes and the window according to the execution of the cursor operation.

2 is a flowchart illustrating a method of performing a click operation using an EEG signal in an HMD headset apparatus according to an embodiment of the present invention.

Referring to FIG. 2, the controller 140 determines a connection state of the electroencephalogram signal received by the electroencephalogram sensor 110 with the electroencephalogram sensor 110 using a Poor signal. When the controller 140 is connected to the electroencephalogram sensor 110, it is assumed that the user wears the HMD headset. At this time, the electroencephalogram sensor 110 periodically sends an electroencephalogram signal to the control unit 140, and the control unit 140 analyzes the electroencephalogram signal and performs the input control command.

Of the EEG signals, the Blink signal is a signal that detects the movement of a fine head, especially a signal that detects blinking of the eyes.

The control unit 140 analyzes the Blink signal in the EEP signal S240 received from the EEG sensor 110 (S210).

As a result of the analysis, if there is a Blink signal (S220), the control unit 140 executes a click command (S230). For example, when the user wearing the HMD headset device wants to generate a click event, if the cursor position displayed on the display unit 150 is located in the window region or the icon region of the content, the Blink signal is blinked And perform a click operation.

FIG. 3 is a flowchart illustrating a method of performing cursor movement and screen movement in the HMD headset apparatus according to an embodiment of the present invention.

3, when the user wears the HMD headset device, the controller 140 initializes signal values for the pitch and roll directions received from the gravity sensor 120 (S310).

After the initialization, the control unit 140 receives the signal of movement from the gyro sensor 130 and controls the position of the window or the cursor (S320 to S350). At this time, the control unit 140 acquires the vertical movement of the user's head from the pitch signal of the gyro sensor 130 and obtains the left and right rotation of the user's head from the yaw signal of the gyro sensor 130 .

4 is an exemplary screen of an HMD headset apparatus according to an embodiment of the present invention. 4 (a) is an example of an initial screen, and FIG. 4 (b) is an example of a screen executed by a user.

4, in a state where the user wears the HMD headset device 100, the controller 140 detects movement of the user's head on the initial screen (a), and moves the cursor to the corresponding position . The control unit 140 has an executable application icon near the moved cursor. When receiving the Blink signal from the electroencephalogram sensor 110, the controller 140 performs a click operation at the cursor position.

FIG. 5 is a view illustrating an example of controlling the window size in the HMD headset according to an exemplary embodiment of the present invention. Referring to FIG.

5, when the size of the window of the HMD headset 100 is changed, a user button of a window such as a window size adjustment button, a window distance adjustment button, a window movement button, a close button, Move it properly so that the size does not change.

In Fig. 5, the bar-shaped button indicating the name of the window is divided by the name of the window and the button. The name part moves in proportion to the size change of the window like the user button of the window. Since the size of the bar shape button should not change when the size of the window content changes up and down, the size of the bar shape button must be changed when the size of the window contents changes to the left and right according to the size change of the window contents. do.

6 is a view for explaining window position control in an HMD headset apparatus according to an embodiment of the present invention.

Referring to FIG. 6, three methods ((a), (b), and (c)) will be exemplified as to how to move the window displayed on the display unit 150 in the HMD headset apparatus according to the present invention.

The controller 140 processes signals received from the gravity sensor 120 and the gyro sensor 130 to acquire information on the head position of the user and uses the information to determine the direction in which the head of the user is facing the front, As shown in FIG.
In FIG. 6 (a), the first method allows the position of the window to follow the user's line of sight while the front of the window always sees the user, allowing the user to perform a desired operation immediately after the window is moved.

In Fig. 6 (b), the second method allows the position of the window to follow the user's line of sight, but limits the angle between the window and the user. That is, the angle between the user and the window immediately after creation of the window is 90 degrees, and when the user starts moving, the angle decreases. Then, when the angle reaches a certain angle, the angle does not decrease any more, and the window follows the user's gaze while maintaining the fixed angle.

In Fig. 6 (c), the third method moves the position of the window on a specific plane by the user's operation without following the user's line of sight. Here, a certain restriction is placed on the angle between the window and the user so that the window goes too far and the user is not forced to press the button.

7 is a view for explaining window rotation control in an HMD headset apparatus according to an embodiment of the present invention.

As shown in FIG. 7, in the rotation control of the window, a separate rotation button is required to rotate the window. In other words, the window should be rotated in a desired direction so that it can be easily used, and it is necessary to implement such that the rotation of the window dragged to a desired position can be easily manipulated.

8 is a view for explaining window distance control in an HMD headset apparatus according to an embodiment of the present invention.

Referring to FIG. 8, since the minimum distance of the window in the distance control of the window is related to the size of the window, it is necessary to limit the size of the window considering the size of the window. In addition, the maximum distance of the window should not be inconvenient for the user to use. The maximum distance of the window is fixed to an absolute value so that it is not too far away.

8A and 8B, in the case of the first method, the absolute value of the Z value must be shifted in the window distance. However, in the case of the second method and the third method, the relative Z value (B).

Referring to FIG. 8 (c), since the minimum distance of the window needs to consider the size of the window, the angle (? ₁ ) with the button at the edge is limited so that there is no inconvenience in pressing the remote button.

Referring to Fig. 8 (d), the minimum distance of the window should be determined in consideration of the angle ([theta] ₂ ) with the button at the edge when it is near.

9 is a diagram for explaining a control method using a gyro sensor in an HMD headset apparatus according to an embodiment of the present invention.

In order to move, rotate, and control the size in the control of the window in FIG. 9, the concept of increasing and decreasing the coordinate value, the rotation value, the magnitude value and the like is required. Therefore, the gyro sensor 130) is used. That is, if the rotation in one direction increases the value through the gyro sensor 130, the rotation in the opposite direction reduces the value.

10 is a view for explaining an active state control method in an HMD headset apparatus according to an embodiment of the present invention.

Referring to FIG. 10, in the control of the window, the active state of the movement, rotation, and size control is performed by clicking the control button, and controls the window using the value of the gyro sensor 130 due to the rotation of the user's head.

When the user inputs desired information, the user blinks his / her eyes regardless of the direction of the line of sight and releases the active state. At this time, in releasing the active state, the HMD headset apparatus 100 allows the eye flickering during the active state to be recognized only for releasing the active state so that no other button is depressed.

While the present invention has been described with reference to several preferred embodiments, these embodiments are illustrative and not restrictive. It will be understood by those skilled in the art that various changes and modifications may be made therein without departing from the spirit of the invention and the scope of the appended claims.

110 electroencephalogram sensor 120 gravity sensor
130 gyro sensor 140 controller
150 Display unit 100 HMD headset device

Claims (13)

An electroencephalogram sensor for acquiring a user's electroencephalogram (EEG) signal;
A gravity sensor for sensing gravity;
A gyro sensor for sensing azimuth change;
A display unit for displaying a video signal; And
Wherein the gravity sensor and the gyro sensor are used to obtain information on the user's head movement and to display on the display unit using the eye blinking information and the head movement information, And a control unit for controlling the movement of the cursor, which is the input point, and the click operation,
Wherein the controller analyzes the EEG signal received through the EEG sensor and controls the user to perform a click operation at a point where the cursor is located when the user is determined to have an eye blink,
The EEG sensor periodically sends the acquired EEG signal to the controller. The controller determines whether the user wears the HMD headset device using the Poor signal among the EEG signals received from the EEG sensor, Signal to obtain user's blinking information,
The controller analyzes the signals inputted from the sensors to control the cursor movement and the click operation,
The control unit controls the size and position of the window of the application executed on the screen of the display unit through the operation of the cursor, and sets or clears the active state in which the window can be moved, rotated, or scaled on the screen of the display unit When the control button is clicked, the control button is activated and the window is moved or rotated according to the execution of the cursor, or the distance between the user's eyes and the window is controlled, The control unit releases the active state,
The control unit processes signals received from the gravity sensor and the gyro sensor to acquire information about the head position of the user, determines the direction in which the head of the user is heading toward the user's gaze direction,
Wherein the control unit causes the position of the window to follow the line of sight of the user so that the front of the window always faces the user in the movement of the window.
delete delete delete delete delete delete delete delete delete delete delete delete

Images