CN111831119A - Eyeball tracking method and device, storage medium and head-mounted display equipment - Google Patents
Eyeball tracking method and device, storage medium and head-mounted display equipment Download PDFInfo
- Publication number
- CN111831119A CN111831119A CN202010664802.8A CN202010664802A CN111831119A CN 111831119 A CN111831119 A CN 111831119A CN 202010664802 A CN202010664802 A CN 202010664802A CN 111831119 A CN111831119 A CN 111831119A
- Authority
- CN
- China
- Prior art keywords
- vision sensor
- sensor module
- dynamic vision
- imaging
- display screen
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 210000005252 bulbus oculi Anatomy 0.000 title claims abstract description 128
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000003384 imaging method Methods 0.000 claims abstract description 98
- 238000013507 mapping Methods 0.000 claims abstract description 22
- 238000004364 calculation method Methods 0.000 claims abstract description 9
- 210000001508 eye Anatomy 0.000 claims description 42
- 230000004913 activation Effects 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 20
- 238000012937 correction Methods 0.000 claims description 2
- 230000008859 change Effects 0.000 description 16
- 239000011521 glass Substances 0.000 description 12
- 238000010586 diagram Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000003190 augmentative effect Effects 0.000 description 4
- 206010034960 Photophobia Diseases 0.000 description 3
- 208000013469 light sensitivity Diseases 0.000 description 3
- 238000009877 rendering Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000007654 immersion Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 210000001747 pupil Anatomy 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0179—Display position adjusting means not related to the information to be displayed
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0179—Display position adjusting means not related to the information to be displayed
- G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Optics & Photonics (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Position Input By Displaying (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The embodiment of the application discloses an eyeball tracking method, an eyeball tracking device, a storage medium and head-mounted display equipment, wherein event stream data which is shot and output by a dynamic vision sensor module is collected; acquiring imaging information of the eyeballs in the dynamic vision sensor module according to the event stream data; calculating the imaging position coordinates of the eyeballs on the dynamic vision sensor module according to the imaging information; according to the coordinate mapping relation and the imaging position coordinate, the fixation position coordinate of the eyeball on the display screen is obtained through calculation, and based on the coordinate mapping relation and the imaging position coordinate, when the user changes the fixation direction, the dynamic vision sensor can perform imaging according to the movement of the eyeball and calculate the fixation position coordinate of the eyeball on the display screen.
Description
Technical Field
The application relates to the technical field of electronic equipment, in particular to an eyeball tracking method and device, a storage medium and head-mounted display equipment.
Background
With the development of human-computer interaction technology, the eyeball tracking technology is more and more widely applied in the field of human-computer interaction. The eyeball tracking technology is a technology for positioning the sight of human eyes, integrates knowledge in subject fields such as mechanics, electronics and optics, and has good application prospects in the fields such as artificial intelligence and machine vision, for example, fatigue monitoring, virtual reality, communication auxiliary tools and the like.
In a conventional eyeball tracking technology, sensing acquisition of an eyeball tracking image is generally realized by adopting an infrared supplementary lighting emitting and camera receiving mode, but the mode has the defect of poor real-time performance.
Disclosure of Invention
The embodiment of the application provides an eyeball tracking method, an eyeball tracking device, a storage medium and head-mounted display equipment, which can improve the instantaneity of eyeball tracking.
In a first aspect, an embodiment of the present application provides an eyeball tracking method, including:
acquiring event stream data which is shot and output by the dynamic vision sensor module to the eyes of the user;
acquiring imaging information of the eyeballs in the dynamic vision sensor module according to the event stream data;
calculating the imaging position coordinates of the eyeballs on the dynamic vision sensor module according to the imaging information;
and calculating to obtain the fixation position coordinate of the eyeball on the display screen according to the coordinate mapping relation between the display screen and the dynamic vision sensor module and the imaging position coordinate.
In a second aspect, an embodiment of the present application further provides an eyeball tracking device, including:
the data acquisition module is used for acquiring event stream data which is shot and output by the dynamic vision sensor module to the eyes of the user;
the imaging acquisition module is used for acquiring the imaging information of the eyeballs in the dynamic vision sensor module according to the event stream data;
the first calculation module is used for calculating the imaging position coordinates of the eyeballs on the dynamic vision sensor module according to the imaging information;
and the second calculation module is used for calculating and obtaining the fixation position coordinate of the eyeball on the display screen according to the coordinate mapping relation between the display screen and the dynamic vision sensor module and the imaging position coordinate.
In a third aspect, embodiments of the present application further provide a storage medium having a computer program stored thereon, where the computer program is executed on a computer, so as to enable the computer to execute the eyeball tracking method provided in any embodiment of the present application.
In a fourth aspect, embodiments of the present application further provide a head-mounted display device, including a processor and a memory, where the memory has a computer program, and the processor is configured to execute the eye tracking method according to any of the embodiments of the present application by calling the computer program.
According to the technical scheme provided by the embodiment of the application, the eyes of a user are shot through the dynamic vision sensor module, event stream data output by the dynamic vision sensor is collected, imaging information of eyeballs in the dynamic vision sensor module is obtained according to the event stream data, imaging position coordinates of the eyeballs are calculated according to the imaging information, the fixation position coordinates of the eyeballs on the display screen are calculated according to the coordinate mapping relation between the display screen and the dynamic vision sensor and the imaging position coordinates, when the user fixes the display screen and changes the fixation direction, the dynamic vision sensor can carry out imaging according to the movement of the eyeballs, the change of the fixation direction of the user is determined according to the change of the imaging position, and the fixation position coordinates of the eyeballs on the display screen are calculated, so that the scheme only captures the characteristic of target movement change, the method greatly reduces the redundancy of information, reduces the data volume, has the characteristic of low time delay, can efficiently acquire the fixation position coordinate, and improves the instantaneity of eyeball tracking.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a first flowchart of an eyeball tracking method according to an embodiment of the present disclosure.
Fig. 2 is an imaging schematic diagram of a dynamic vision sensor module according to an embodiment of the present application.
Fig. 3 is a schematic structural diagram of an eyeball tracking device according to an embodiment of the present application.
Fig. 4 is a schematic structural diagram of a head-mounted display device according to an embodiment of the present disclosure.
Fig. 5 is a second structural schematic diagram of a head-mounted display device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without inventive step, are within the scope of the present application.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
An implementation subject of the eyeball tracking method may be the eyeball tracking device provided in the embodiment of the application, or a head-mounted display device integrated with the eyeball tracking device, wherein the eyeball tracking device may be implemented in a hardware or software manner. The head-mounted display device may be an AR (augmented Reality) glasses, a VR (Virtual Reality) glasses, or the like.
Referring to fig. 1, fig. 1 is a first flowchart illustrating an eyeball tracking method according to an embodiment of the present disclosure. The specific process of the eyeball tracking method provided by the embodiment of the application can be as follows:
in 101, event stream data which is shot and output by the dynamic vision sensor module to the eyes of the user is collected.
For convenience of explanation, the following describes embodiments of the present application with AR glasses integrated with an eye tracking device as an execution body. The augmented reality is a technology for increasing the perception of a user to the real world through information provided by a system, the augmented reality overlays contents such as virtual objects, scenes or prompt information generated by a computer to the real scene for display so as to enhance or modify the perception of the real world environment or data representing the real world environment, and the user can observe the effect of the augmented reality or mixed reality after the contents such as the virtual objects and the real world are overlaid by wearing head-mounted display equipment such as AR glasses in the real world environment. In the virtual reality technology, a virtual environment is simulated through a computer so as to give people an environmental immersion feeling, and a user can view a virtual object by wearing head-mounted display equipment such as VR glasses and the like.
AR glasses and VR glasses are all near-to-eye display device, and its control mode is not so convenient as the control to portable electronic equipment such as smart mobile phone, and if these equipment have possessed eyeball tracking function, the user can come the interactive operation of controlgear through the rotation of eyes, improves interactive efficiency. For example, studies have shown that the focus resolution area of human eyes can only be 10 ° of the pupil facing position, within which the user can clearly distinguish the displayed content, and outside which the user does not recognize the content in detail. Therefore, if the head-mounted display device tracks the eye gaze focusing position of the user, i.e., the gaze position, the gaze focusing position may be heavily rendered, while other positions may reduce the amount of rendering. Therefore, the image rendering workload of the GPU is reduced, the calculation amount is favorably reduced, and the load and the power consumption of a system are reduced.
Based on this, this scheme sets up a dynamic vision sensor module on head mounted display device, and this dynamic vision sensor module can be according to following position setting on head mounted display device: when the user wears the head-mounted display equipment, the eyeballs are located in the field of view of the dynamic vision sensor module.
The dynamic vision sensor module comprises a dynamic vision sensor and a lens module, the lens module shoots an external object, light is converted through the lens module to form an image on a pixel array of the dynamic vision sensor, when the position of the external object changes, the position of the image on the pixel array changes along with the change, and when the position of the image moves, the light sensitivity of a corresponding pixel also changes. For example, when the motion vision sensor module shoots an eye of a user, the position of an image reflected on the pixel array of the motion vision sensor moves when the eyeball of the user rotates, and the light sensitivity of the corresponding pixel changes.
The dynamic vision sensor module is an Event-driven (Event-drive) photoelectric sensor, the output of the sensor no longer has the concept of frame, but is an Event (Event), each pixel unit in the dynamic vision sensor module independently senses the change of light intensity in space, when the light sensitivity of the pixel changes and the pixel point with the light intensity change exceeding a threshold value is considered as an activated pixel point, the dynamic vision sensor module packs and encodes and outputs the information of the row and column positions, the time stamps and the like of the activated pixel point, the related information of the activated pixel points is the Event, and a plurality of activated pixel points form an Event stream.
When the position of an external object moves, the corresponding pixel position in the pixel array of the dynamic vision sensor module is changed in light intensity, and then the corresponding event is output, namely, the dynamic vision sensor module only captures the characteristic of target motion change, so that the redundancy of information is greatly reduced, the data volume is reduced, the computing resource and the bandwidth can be saved in practical application, and the dynamic vision sensor module is very suitable for being used in the field of machine vision. The scheme of this application embodiment utilizes dynamic vision sensor module to have high dynamic range, be good at catching high-speed object, can filter characteristics such as background and carry out eyeball tracking. And compared with the conventional mode of adopting infrared light supplement emission and camera receiving, the dynamic vision sensor module adopts a passive acquisition mode, does not need to actively emit infrared rays to irradiate the eyeballs of people, reduces safety risks, has the characteristics of low time delay and low power consumption, can save electric quantity, can efficiently acquire fixation position coordinates, and improves the real-time performance of eyeball tracking
In addition, it should be noted that, when the resolution of the dynamic vision sensor module is higher, the imaging of the moving object that can be obtained is clearer, the outline of the object that can be identified is clearer, and the details of the object can be embodied more, so that the scheme of the application can select the dynamic vision sensor module with higher resolution when applied, so as to improve the accuracy of eyeball tracking.
In this application embodiment, when the eyeball that is located the visual field scope of dynamic vision sensor module rotated, this kind of motion information was caught to the dynamic vision sensor module, gathered the event stream data of dynamic vision sensor module output.
At 102, imaging information of the eyeball in the dynamic vision sensor module is obtained according to the event flow data.
After the event stream data output by the dynamic vision sensor is acquired, the event stream data is extracted and calculated to obtain the imaging information of the eyeballs.
When the eyeball rotates, the light intensity of the corresponding pixel in the pixel array of the dynamic vision sensor module changes, and a corresponding event is output, wherein the output event comprises the following information { x, y, p, t } of four dimensions. Where t denotes an activation timestamp, x denotes a row position of an activated pixel, y denotes a column position of an activated pixel, and p denotes an activation polarity, where p-1 denotes an "on" polarity, and p-1 denotes an "off" polarity, and when an increase in light intensity sensed by a pixel exceeds a threshold, the pixel is activated, generating an "on" event, and when a decrease in light intensity sensed by a pixel exceeds a threshold, the pixel is activated, generating an "off" event.
And each activated pixel in the pixel array of the dynamic vision sensor module outputs a corresponding event according to the format to form event stream data. The event stream selects the activated pixels according to the principle of 'activation first and reading first' and reads information. And the activation polarities are not distinguished, and the detected activation pixel points are identified.
In addition, when the user wears the head-mounted display device, the parts except the eyeballs are static relative to the head-mounted display device, when the user changes the watching direction of the display screen, the eyeballs can rotate, and therefore the eyeball part of the eyes of the user is imaged on the dynamic vision sensor module according to the event stream data.
For example, in one embodiment, "acquiring imaging information of an eye on the dynamic data sensor from the event stream data" includes: determining a target tracking time point from the event stream data; and acquiring a plurality of activated pixel points corresponding to the target tracking time point from the event stream data, and forming imaging information of eyeballs on the dynamic data sensor by the plurality of activated pixel points.
Since the image of the eyeball part on the dynamic vision sensor module corresponds to a plurality of activated pixels, at the time t, the coordinates of the plurality of corresponding activated pixels, (x0, y0), (x1, y1) … … (xn, yn) can be obtained from the event stream data.
In addition, since the eyeball may be continuously rotated, event stream data composed of events may be continuously output, for example, an event may be output once at each interval t0, when new event data is received, a time point closest to the current time is determined from the event stream data as a target tracking time point, in such a manner that as time goes by, the target tracking time point gradually goes by, event data corresponding to each target tracking time point is obtained, an eyeball gaze position corresponding to each target tracking time point is calculated from the event data corresponding to each target tracking time point, and dynamic tracking of the eyeball gaze position is further achieved.
Referring to fig. 2, fig. 2 is an imaging schematic diagram of a dynamic vision sensor module according to an embodiment of the present application. Each square box corresponds to a pixel on the pixel array of the dynamic vision sensor module, and fig. 2 is a 7 × 7 pixel array, where a gray pixel is an activated pixel point and represents a detected moving object. Where t2 is t1+ t0, and t0 is the minimum time interval for detecting the dynamic vision sensor module, i.e. the minimum time interval of the event stream. From time t1 to time t2, the position of the activated pixel points on the pixel array as a result of the motion of the object changes as shown in FIG. 2. That is, the image obtained at the time t1 is composed of t1 gray pixels, and the image obtained at the time t2 is composed of t2 gray pixels.
It will be appreciated that the example of figure 2 simplifies the pixel array and the size of the image for the convenience of the reader, and in practice the number of pixels in the pixel array is much greater than that shown in figure 2.
In 103, calculating the coordinates of the imaging position of the eyeball on the dynamic vision sensor module according to the imaging information.
And after the imaging information of the eyeball is acquired, calculating the imaging position coordinate of the eyeball on the dynamic vision sensor module according to the imaging information. For example, in an embodiment, the calculating the coordinates of the imaging position of the eyeball on the dynamic vision sensor module according to the imaging information includes: carrying out feature point identification on the imaging information, and obtaining target activation pixel points forming an iris outline from the plurality of activation pixel points; and calculating the central position coordinates of the target activation pixel points, and taking the central position coordinates as the imaging position coordinates of the eyeballs.
In the embodiment, taking the time t2 as an example, all the activation pixels corresponding to the time t2 are obtained, feature extraction is performed on the activation pixels, target activation pixels capable of representing and forming an iris outline are obtained, and the center position coordinates of the target activation pixels are calculated.
In an embodiment, the target activation pixel points constituting the iris outline in the imaging information may be identified according to a pre-trained preset convolutional neural network, wherein the preset convolutional neural network may be obtained by training a sample imaging image, and the sample imaging image has labeling information of the activation pixel points constituting the iris outline.
And 104, calculating to obtain the fixation position coordinate of the eyeball on the display screen according to the coordinate mapping relation between the display screen and the dynamic vision sensor module and the imaging position coordinate.
After acquiring the coordinates of the imaging position of the eyeball on the dynamic vision sensor module, performing coordinate conversion processing according to a coordinate mapping relation between the display screen and the dynamic vision sensor module, wherein the coordinate mapping relation can be stored in the head-mounted display device after calibration when the head-mounted display device leaves a factory, or can be calibrated by a user according to needs.
For example, in an embodiment, before acquiring event stream data output by the dynamic vision sensor module for shooting eyes of the user, the method further includes: displaying calibration information on the display screen, and outputting prompt information, wherein the prompt information is used for prompting a user to watch the calibration information; acquiring a first position coordinate of an eyeball when a user watches the calibration information according to the dynamic vision sensor module; acquiring a plurality of first position coordinates according to the calibration information of a plurality of different positions; and calculating to obtain a coordinate mapping relation between the display screen and the dynamic vision sensor module according to the respective second position coordinates of the plurality of calibration information and the plurality of first position coordinates.
In this embodiment, the user, when using the device for the first time, controls the device to enter a calibration mode, which in turn displays a plurality of calibration information, e.g., calibration icons, at different locations on the display screen. Wherein the user is prompted to gaze at each calibration icon as it is displayed. After a user watches the calibration icon, a confirmation instruction can be triggered, after the equipment receives the confirmation instruction, the first position coordinates of eyeballs when the user watches the calibration information are obtained according to the dynamic vision sensor module, each calibration information is provided with a corresponding second position coordinate, when the calibration is completed according to the plurality of calibration information displayed at different positions of the display screen according to the mode, a plurality of groups of first position coordinates and second position coordinates can be obtained, and the coordinate mapping relation between the display screen and the dynamic vision sensor module can be obtained through calculation according to the plurality of groups of first position coordinates and second position coordinates.
Wherein, in some embodiments, after calculating the gaze location coordinates of the eye on the display screen, the method further comprises: and controlling the display of the display screen according to the target display content corresponding to the fixation position coordinates on the display screen.
After the gaze position coordinates of the user gazing at the display screen are obtained, the display of the display screen can be controlled in various ways based on the gaze position coordinates, for example, the target display content corresponding to the gaze position coordinates of the user is subjected to key rendering and the like.
In particular implementation, the present application is not limited by the execution sequence of the described steps, and some steps may be performed in other sequences or simultaneously without conflict.
As can be seen from the above, the eyeball tracking method provided in the embodiment of the present application captures an eye of a user through the dynamic vision sensor module, collects event stream data output by the dynamic vision sensor, obtains imaging information of the eyeball in the dynamic vision sensor module according to the event stream data, calculates an imaging position coordinate of the eyeball according to the imaging information, calculates a gaze position coordinate of the eyeball on the display screen according to a coordinate mapping relationship between the display screen and the dynamic vision sensor and the imaging position coordinate, and according to the scheme, when the user gazes at the display screen and changes a gaze direction, the dynamic vision sensor can image according to a movement of the eyeball, determines a change of the gaze direction of the user according to a change of the imaging position, and further calculates the gaze position coordinate of the eyeball on the display screen, so that the scheme captures only a characteristic of a change of a target movement, the method greatly reduces the redundancy of information, reduces the data volume, has the characteristic of low time delay, can efficiently acquire the fixation position coordinate, and improves the instantaneity of eyeball tracking.
An eye tracking device is also provided in one embodiment. Referring to fig. 3, fig. 3 is a schematic structural diagram of an eyeball tracking device 300 according to an embodiment of the present application. The eyeball tracking apparatus 300 is applied to a head-mounted display device, and the eyeball tracking apparatus 300 includes a data acquisition module 301, an imaging acquisition module 302, a first calculation module 303 and a second calculation module 304, as follows:
the data acquisition module 301 is configured to acquire event stream data that is output by the dynamic vision sensor module when the eyes of the user are photographed;
an imaging acquisition module 302, configured to acquire imaging information of an eyeball in the dynamic vision sensor module according to the event stream data;
the first calculating module 303 is configured to calculate coordinates of an imaging position of an eyeball on the dynamic vision sensor module according to the imaging information;
and a second calculating module 304, configured to calculate, according to the coordinate mapping relationship between the display screen and the dynamic vision sensor module and the imaging position coordinate, a gaze position coordinate of the eyeball on the display screen.
In some embodiments, the imaging acquisition module 302 is further configured to determine a target tracking time point from the event stream data;
and acquiring a plurality of activated pixel points corresponding to the target tracking time point from the event stream data, and forming imaging information of eyeballs on the dynamic data sensor by the plurality of activated pixel points.
In some embodiments, the first calculating module 303 is further configured to perform feature point identification on the imaging information, and obtain target activated pixel points forming an iris outline from the plurality of activated pixel points; and calculating the central position coordinates of the target activation pixel points, and taking the central position coordinates as the imaging position coordinates of the eyeballs.
In some embodiments, the eye tracking apparatus 300 further comprises:
and the display control module is used for controlling the display of the display screen according to the target display content corresponding to the fixation position coordinates on the display screen.
In some embodiments, the eye tracking apparatus 300 further comprises:
the coordinate correction module is used for displaying calibration information on the display screen and outputting prompt information, and the prompt information is used for prompting a user to watch the calibration information;
acquiring a first position coordinate of an eyeball when a user watches the calibration information according to the dynamic vision sensor module;
acquiring a plurality of first position coordinates according to the calibration information of a plurality of different positions;
and calculating to obtain a coordinate mapping relation between the display screen and the dynamic vision sensor module according to the respective second position coordinates of the plurality of calibration information and the plurality of first position coordinates.
It should be noted that the eyeball tracking device provided in the embodiment of the present application and the eyeball tracking method in the above embodiment belong to the same concept, and any method provided in the embodiment of the eyeball tracking method can be implemented by the eyeball tracking device, and the specific implementation process thereof is described in the embodiment of the eyeball tracking method, and is not described herein again.
As can be seen from the above, the eyeball tracking apparatus provided in the embodiment of the present application captures an eye of a user through the dynamic vision sensor module, collects event stream data output by the dynamic vision sensor, obtains imaging information of the eyeball in the dynamic vision sensor module according to the event stream data, calculates an imaging position coordinate of the eyeball according to the imaging information, calculates a gaze position coordinate of the eyeball on the display screen according to a coordinate mapping relationship between the display screen and the dynamic vision sensor and the imaging position coordinate, and according to the scheme, when the user gazes at the display screen and changes a gaze direction, the dynamic vision sensor can image according to a movement of the eyeball, determines a change of the gaze direction of the user according to a change of the imaging position, and further calculates the gaze position coordinate of the eyeball on the display screen, so that the scheme captures only a characteristic of a change of a target movement, the method greatly reduces the redundancy of information, reduces the data volume, has the characteristic of low time delay, can efficiently acquire the fixation position coordinate, and improves the instantaneity of eyeball tracking.
The embodiment of the application also provides head-mounted display equipment. The head mounted display device may be AR glasses or the like. Referring to fig. 4, fig. 4 is a schematic view illustrating a first structure of a head-mounted display device according to an embodiment of the present disclosure. The head mounted display device 400 comprises a processor 401 and a memory 402, and a display screen 403 connected thereto. The processor 401 and the memory 402 are electrically connected to the display 403, and the processor 401 is electrically connected to the memory 402.
The processor 401 is a control center of the head-mounted display apparatus 400, connects various parts of the entire head-mounted display apparatus using various interfaces and lines, and performs various functions of the head-mounted display apparatus and processes data by running or calling a computer program stored in the memory 402 and calling data stored in the memory 402, thereby performing overall monitoring of the head-mounted display apparatus.
The display screen 403 may be used to display information input by or provided to the user as well as various graphical user interfaces of the head mounted display device, which may be comprised of images, text, icons, video, and any combination thereof. Wherein, in some embodiments, the head mounted display device may be AR glasses and the display screen 403 is a lens of the AR glasses.
In this embodiment, the processor 401 in the head-mounted display device 400 loads instructions corresponding to one or more processes of the computer program into the memory 402 according to the following steps, and the processor 401 runs the computer program stored in the memory 402, so as to implement various functions:
acquiring event stream data which is shot and output by the dynamic vision sensor module to the eyes of the user;
acquiring imaging information of the eyeballs in the dynamic vision sensor module according to the event stream data;
calculating the imaging position coordinates of the eyeballs on the dynamic vision sensor module according to the imaging information;
and calculating to obtain the fixation position coordinate of the eyeball on the display screen according to the coordinate mapping relation between the display screen and the dynamic vision sensor module and the imaging position coordinate.
In some embodiments, please refer to fig. 5, and fig. 5 is a second structural diagram of a head-mounted display device according to an embodiment of the present disclosure. The head mounted display device 400 further includes: radio frequency circuit 404, display screen 403, control circuit 405, input unit 406, audio circuit 407, sensor 408, power supply 409 and dynamic vision sensor module 410. The processor 401 is electrically connected to the rf circuit 404, the display 403, the control circuit 405, the input unit 406, the audio circuit 407, the sensor 408, and the power source 409.
The radio frequency circuit 404 is used for transceiving radio frequency signals to communicate with a network device or other head-mounted display device through wireless communication.
The control circuit 405 is electrically connected to the display 403, and is configured to control the display 403 to display information.
The input unit 406 may be used to receive input numbers, character information, or user characteristic information (e.g., fingerprint), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. The input unit 406 may include a fingerprint recognition module.
The sensor 408 is used to collect external environmental information. The sensors 408 may include one or more of ambient light sensors, acceleration sensors, gyroscopes, etc.
The power supply 409 is used to power the various components of the head mounted display device 400. In some embodiments, the power source 409 may be logically connected to the processor 401 through a power management system, so that functions of managing charging, discharging, and power consumption are implemented through the power management system.
The dynamic vision sensor module 410 is used for collecting the eye image of the wearer, and when the user wears the head-mounted display device, the eyes of the user are located in the field of view of the dynamic vision sensor module. Therein, in some embodiments, the dynamic vision sensor module 410 may be mounted at the lower edge of the display screen. Alternatively, in some other embodiments, the head-mounted display device is AR glasses, the AR glasses include a frame, and the dynamic vision sensor module 40 may be mounted on a lower frame or a side frame of the frame.
Although not shown in the drawings, the head-mounted display device 400 may further include a bluetooth module or the like, which is not described in detail herein.
In this embodiment, the processor 401 in the head-mounted display device 400 loads instructions corresponding to one or more processes of the computer program into the memory 402 according to the following steps, and the processor 401 runs the computer program stored in the memory 402, so as to implement various functions:
acquiring event stream data which is shot and output by the dynamic vision sensor module to the eyes of the user;
acquiring imaging information of the eyeballs in the dynamic vision sensor module according to the event stream data;
calculating the imaging position coordinates of the eyeballs on the dynamic vision sensor module according to the imaging information;
and calculating to obtain the fixation position coordinate of the eyeball on the display screen according to the coordinate mapping relation between the display screen and the dynamic vision sensor module and the imaging position coordinate.
In some embodiments, when acquiring imaging information of an eye on the dynamic data sensor from the event stream data, the processor 401 performs:
determining a target tracking time point from the event stream data;
and acquiring a plurality of activated pixel points corresponding to the target tracking time point from the event stream data, and forming imaging information of eyeballs on the dynamic data sensor by the plurality of activated pixel points.
In some embodiments, when calculating the imaging position coordinate of the eyeball on the dynamic vision sensor module according to the imaging information, the processor 401 executes:
carrying out feature point identification on the imaging information, and obtaining target activation pixel points forming an iris outline from the plurality of activation pixel points;
and calculating the central position coordinates of the target activation pixel points, and taking the central position coordinates as the imaging position coordinates of the eyeballs.
In some embodiments, after the calculating the gaze location coordinates of the eyeball on the display screen, the processor 401 further performs:
and controlling the display of the display screen according to the target display content corresponding to the fixation position coordinates on the display screen.
In some embodiments, before acquiring the event stream data output by the dynamic vision sensor module for shooting the eyes of the user, the processor 401 further performs:
displaying calibration information on the display screen, and outputting prompt information, wherein the prompt information is used for prompting a user to watch the calibration information;
acquiring a first position coordinate of an eyeball when a user watches the calibration information according to the dynamic vision sensor module;
acquiring a plurality of first position coordinates according to the calibration information of a plurality of different positions;
and calculating to obtain a coordinate mapping relation between the display screen and the dynamic vision sensor module according to the respective second position coordinates of the plurality of calibration information and the plurality of first position coordinates.
From the above, the embodiment of the present application provides a head-mounted display device, where the head-mounted display device photographs eyes of a user through a dynamic vision sensor module, collects event stream data output by the dynamic vision sensor, obtains imaging information of eyeballs in the dynamic vision sensor module according to the event stream data, calculates imaging position coordinates of the eyeballs according to the imaging information, calculates gaze position coordinates of the eyeballs on a display screen according to a coordinate mapping relationship between the display screen and the dynamic vision sensor and the imaging position coordinates, and by using the scheme, when the user gazes at the display screen and changes a gaze direction, the dynamic vision sensor can image according to movement of the eyeballs, determines a change of the gaze direction of the user according to the change of the imaging position, and further calculates the gaze position coordinates of the eyeballs on the display screen, therefore, the scheme only captures the characteristics of target motion change, greatly reduces the redundancy of information, reduces the data volume, has the characteristic of low time delay, can efficiently acquire the fixation position coordinates, and improves the real-time performance of eyeball tracking.
An embodiment of the present application further provides a storage medium, where a computer program is stored, and when the computer program runs on a computer, the computer executes the eyeball tracking method according to any one of the above embodiments.
It should be noted that, all or part of the steps in the methods of the above embodiments may be implemented by hardware related to instructions of a computer program, which may be stored in a computer-readable storage medium, which may include, but is not limited to: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.
Furthermore, the terms "first", "second", and "third", etc. in this application are used to distinguish different objects, and are not used to describe a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or modules is not limited to only those steps or modules listed, but rather, some embodiments may include other steps or modules not listed or inherent to such process, method, article, or apparatus.
The eyeball tracking method, device, storage medium and head-mounted display device provided by the embodiment of the application are described in detail above. The principle and the implementation of the present application are explained herein by applying specific examples, and the above description of the embodiments is only used to help understand the method and the core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
1. An eye tracking method, comprising:
acquiring event stream data which is shot and output by the dynamic vision sensor module to the eyes of the user;
acquiring imaging information of the eyeballs in the dynamic vision sensor module according to the event stream data;
calculating the imaging position coordinates of the eyeballs on the dynamic vision sensor module according to the imaging information;
and calculating to obtain the fixation position coordinate of the eyeball on the display screen according to the coordinate mapping relation between the display screen and the dynamic vision sensor module and the imaging position coordinate.
2. The eye tracking method according to claim 1, wherein said obtaining imaging information of an eye on said dynamic data sensor based on said event stream data comprises:
determining a target tracking time point from the event stream data;
and acquiring a plurality of activated pixel points corresponding to the target tracking time point from the event stream data, and forming imaging information of eyeballs on the dynamic data sensor by the plurality of activated pixel points.
3. The method for tracking an eyeball according to claim 2, wherein the calculating of coordinates of an imaging position of the eyeball on the dynamic vision sensor module according to the imaging information comprises:
carrying out feature point identification on the imaging information, and obtaining target activation pixel points forming an iris outline from the plurality of activation pixel points;
and calculating the central position coordinates of the target activation pixel points, and taking the central position coordinates as the imaging position coordinates of the eyeballs.
4. The eye tracking method of claim 1, wherein after said calculating the gaze location coordinates of the eye on the display screen, further comprising:
and controlling the display of the display screen according to the target display content corresponding to the fixation position coordinates on the display screen.
5. The eye tracking method according to claim 1, wherein before acquiring the event stream data that is output by the dynamic vision sensor module for capturing the eye of the user, the method further comprises:
displaying calibration information on the display screen, and outputting prompt information, wherein the prompt information is used for prompting a user to watch the calibration information;
acquiring a first position coordinate of an eyeball when a user watches the calibration information according to the dynamic vision sensor module;
acquiring a plurality of first position coordinates according to the calibration information of a plurality of different positions;
and calculating to obtain a coordinate mapping relation between the display screen and the dynamic vision sensor module according to the respective second position coordinates of the plurality of calibration information and the plurality of first position coordinates.
6. An eye tracking device, comprising:
the data acquisition module is used for acquiring event stream data which is shot and output by the dynamic vision sensor module to the eyes of the user;
the imaging acquisition module is used for acquiring the imaging information of the eyeballs in the dynamic vision sensor module according to the event stream data;
the first calculation module is used for calculating the imaging position coordinates of the eyeballs on the dynamic vision sensor module according to the imaging information;
and the second calculation module is used for calculating and obtaining the fixation position coordinate of the eyeball on the display screen according to the coordinate mapping relation between the display screen and the dynamic vision sensor module and the imaging position coordinate.
7. The eye tracking device of claim 6, wherein said imaging acquisition module is further configured to:
determining a target tracking time point from the event stream data;
and acquiring a plurality of activated pixel points corresponding to the target tracking time point from the event stream data, and forming imaging information of eyeballs on the dynamic data sensor by the plurality of activated pixel points.
8. An eye tracking device according to claim 7, further comprising:
the coordinate correction module is used for displaying calibration information on the display screen and outputting prompt information, and the prompt information is used for prompting a user to watch the calibration information;
acquiring a first position coordinate of an eyeball when a user watches the calibration information according to the dynamic vision sensor module;
acquiring a plurality of first position coordinates according to the calibration information of a plurality of different positions;
and calculating to obtain a coordinate mapping relation between the display screen and the dynamic vision sensor module according to the respective second position coordinates of the plurality of calibration information and the plurality of first position coordinates.
9. A storage medium having stored thereon a computer program for causing a computer to perform the eye tracking method according to any one of claims 1 to 5 when the computer program runs on the computer.
10. A head-mounted display device comprises a display screen, a processor and a memory, wherein the processor and the memory are connected with the display screen, and the memory stores computer programs; the processor is adapted to perform the eye tracking method according to any one of claims 1 to 5 by calling the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010664802.8A CN111831119A (en) | 2020-07-10 | 2020-07-10 | Eyeball tracking method and device, storage medium and head-mounted display equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010664802.8A CN111831119A (en) | 2020-07-10 | 2020-07-10 | Eyeball tracking method and device, storage medium and head-mounted display equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111831119A true CN111831119A (en) | 2020-10-27 |
Family
ID=72900532
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010664802.8A Pending CN111831119A (en) | 2020-07-10 | 2020-07-10 | Eyeball tracking method and device, storage medium and head-mounted display equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111831119A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112148200A (en) * | 2020-11-27 | 2020-12-29 | 亿昇（深圳）眼镜科技有限公司 | Intelligent glasses system |
| CN112669344A (en) * | 2020-12-24 | 2021-04-16 | 北京灵汐科技有限公司 | Method and device for positioning moving object, electronic equipment and storage medium |
| CN113262464A (en) * | 2021-04-21 | 2021-08-17 | 青岛小鸟看看科技有限公司 | Dynamic change method and device of virtual reality scene and electronic equipment |
| CN113467619A (en) * | 2021-07-21 | 2021-10-01 | 腾讯科技（深圳）有限公司 | Picture display method, picture display device, storage medium and electronic equipment |
| CN113891002A (en) * | 2021-11-12 | 2022-01-04 | 维沃移动通信有限公司 | Shooting method and device |
| CN114646457A (en) * | 2022-05-20 | 2022-06-21 | 南昌虚拟现实研究院股份有限公司 | Eye tracking accuracy testing method and system |
| CN114760440A (en) * | 2022-03-22 | 2022-07-15 | 浙江大学 | Visual monitoring glasses |
| CN114791673A (en) * | 2021-01-26 | 2022-07-26 | 精工爱普生株式会社 | Display method, display device, and recording medium |
| CN114954265A (en) * | 2022-05-25 | 2022-08-30 | 中国第一汽车股份有限公司 | Intelligent visual tracking vehicle-mounted display screen device and control method |
| WO2022247646A1 (en) * | 2021-05-25 | 2022-12-01 | 北京芯海视界三维科技有限公司 | Timing controllers and display device |
| CN115984950A (en) * | 2022-12-28 | 2023-04-18 | 北京字跳网络技术有限公司 | Sight line detection method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105812777A (en) * | 2015-01-21 | 2016-07-27 | 成都理想境界科技有限公司 | Binocular AR head-mounted display device and information display method therefor |
| CN110196640A (en) * | 2019-05-31 | 2019-09-03 | 维沃移动通信有限公司 | A kind of method of controlling operation thereof and terminal |
| US20200026349A1 (en) * | 2018-07-19 | 2020-01-23 | Samsung Electronics Co., Ltd | System and method for hybrid eye tracker |
-
2020
- 2020-07-10 CN CN202010664802.8A patent/CN111831119A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105812777A (en) * | 2015-01-21 | 2016-07-27 | 成都理想境界科技有限公司 | Binocular AR head-mounted display device and information display method therefor |
| US20200026349A1 (en) * | 2018-07-19 | 2020-01-23 | Samsung Electronics Co., Ltd | System and method for hybrid eye tracker |
| CN110196640A (en) * | 2019-05-31 | 2019-09-03 | 维沃移动通信有限公司 | A kind of method of controlling operation thereof and terminal |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112148200A (en) * | 2020-11-27 | 2020-12-29 | 亿昇（深圳）眼镜科技有限公司 | Intelligent glasses system |
| CN112669344A (en) * | 2020-12-24 | 2021-04-16 | 北京灵汐科技有限公司 | Method and device for positioning moving object, electronic equipment and storage medium |
| CN112669344B (en) * | 2020-12-24 | 2024-05-28 | 北京灵汐科技有限公司 | Method and device for positioning moving object, electronic equipment and storage medium |
| WO2022135511A1 (en) * | 2020-12-24 | 2022-06-30 | 北京灵汐科技有限公司 | Method and apparatus for positioning moving object, and electronic device and storage medium |
| CN114791673B (en) * | 2021-01-26 | 2023-12-15 | 精工爱普生株式会社 | Display method, display device, and recording medium |
| CN114791673A (en) * | 2021-01-26 | 2022-07-26 | 精工爱普生株式会社 | Display method, display device, and recording medium |
| CN113262464A (en) * | 2021-04-21 | 2021-08-17 | 青岛小鸟看看科技有限公司 | Dynamic change method and device of virtual reality scene and electronic equipment |
| US11782505B1 (en) | 2021-04-21 | 2023-10-10 | Qingdao Pico Technology Co., Ltd. | Dynamic changing method and apparatus for virtual reality scene, and electronic device |
| WO2022247646A1 (en) * | 2021-05-25 | 2022-12-01 | 北京芯海视界三维科技有限公司 | Timing controllers and display device |
| CN113467619A (en) * | 2021-07-21 | 2021-10-01 | 腾讯科技（深圳）有限公司 | Picture display method, picture display device, storage medium and electronic equipment |
| CN113467619B (en) * | 2021-07-21 | 2023-07-14 | 腾讯科技（深圳）有限公司 | Picture display method and device, storage medium and electronic equipment |
| CN113891002A (en) * | 2021-11-12 | 2022-01-04 | 维沃移动通信有限公司 | Shooting method and device |
| CN114760440A (en) * | 2022-03-22 | 2022-07-15 | 浙江大学 | Visual monitoring glasses |
| CN114646457B (en) * | 2022-05-20 | 2022-08-12 | 南昌虚拟现实研究院股份有限公司 | Eye movement tracking accuracy testing method and system |
| CN114646457A (en) * | 2022-05-20 | 2022-06-21 | 南昌虚拟现实研究院股份有限公司 | Eye tracking accuracy testing method and system |
| CN114954265A (en) * | 2022-05-25 | 2022-08-30 | 中国第一汽车股份有限公司 | Intelligent visual tracking vehicle-mounted display screen device and control method |
| CN115984950A (en) * | 2022-12-28 | 2023-04-18 | 北京字跳网络技术有限公司 | Sight line detection method and device, electronic equipment and storage medium |
| CN115984950B (en) * | 2022-12-28 | 2024-03-12 | 北京字跳网络技术有限公司 | Sight line detection method, device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111831119A (en) | Eyeball tracking method and device, storage medium and head-mounted display equipment | |
| CN110647865B (en) | Face gesture recognition method, device, equipment and storage medium | |
| CN106873778B (en) | Application operation control method and device and virtual reality equipment | |
| CN108170279B (en) | Eye movement and head movement interaction method of head display equipment | |
| US12041380B2 (en) | Event-based IR camera | |
| CN111263066B (en) | Composition guiding method, composition guiding device, electronic equipment and storage medium | |
| CN108076290B (en) | Image processing method and mobile terminal | |
| CN115167723A (en) | Wearable device capable of eye tracking | |
| CN112911182A (en) | Game interaction method, device, terminal and storage medium | |
| KR20170100641A (en) | Virtual representations of real-world objects | |
| CN108431667A (en) | Information processing apparatus, information processing method, and program | |
| CN115209057B (en) | Shooting focusing method and related electronic equipment | |
| KR20160046495A (en) | Method and device to display screen in response to event related to external obejct | |
| CN111698564B (en) | Information recommendation method, device, equipment and storage medium | |
| US20210409626A1 (en) | Patch tracking image sensor | |
| CN113467619B (en) | Picture display method and device, storage medium and electronic equipment | |
| CN107174824B (en) | Special effect information processing method and device, electronic equipment and storage medium | |
| CN112257552B (en) | Image processing method, device, equipment and storage medium | |
| CN110263617B (en) | Three-dimensional face model obtaining method and device | |
| CN109978996B (en) | Method, device, terminal and storage medium for generating expression three-dimensional model | |
| CN112000221A (en) | Method for automatically detecting skin, method for automatically guiding skin care and makeup and terminal | |
| CN109189225A (en) | Display interface method of adjustment, device, wearable device and storage medium | |
| US11328187B2 (en) | Information processing apparatus and information processing method | |
| CN111589138B (en) | Action prediction method, device, equipment and storage medium | |
| CN112907702A (en) | Image processing method, image processing device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201027 |