[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

WO2020062523A1 - 注视点判断方法和装置、电子设备和计算机存储介质 - Google Patents

注视点判断方法和装置、电子设备和计算机存储介质 Download PDF

Info

Publication number
WO2020062523A1
WO2020062523A1 PCT/CN2018/117287 CN2018117287W WO2020062523A1 WO 2020062523 A1 WO2020062523 A1 WO 2020062523A1 CN 2018117287 W CN2018117287 W CN 2018117287W WO 2020062523 A1 WO2020062523 A1 WO 2020062523A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
feature points
eye
face
dimensional coordinates
Prior art date
Application number
PCT/CN2018/117287
Other languages
English (en)
French (fr)
Inventor
刘庭皓
王权
钱晨
Original Assignee
北京市商汤科技开发有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京市商汤科技开发有限公司 filed Critical 北京市商汤科技开发有限公司
Priority to MYPI2019007790A priority Critical patent/MY201730A/en
Priority to SG11202000065TA priority patent/SG11202000065TA/en
Priority to KR1020197038765A priority patent/KR102347336B1/ko
Priority to JP2019570498A priority patent/JP6900516B2/ja
Priority to US16/729,424 priority patent/US11295474B2/en
Publication of WO2020062523A1 publication Critical patent/WO2020062523A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
    • G07C9/00—Individual registration on entry or exit
    • G07C9/00174—Electronically operated locks; Circuits therefor; Nonmechanical keys therefor, e.g. passive or active electrical keys or other data carriers without mechanical keys
    • G07C9/00563—Electronically operated locks; Circuits therefor; Nonmechanical keys therefor, e.g. passive or active electrical keys or other data carriers without mechanical keys using personal physical data of the operator, e.g. finger prints, retinal images, voicepatterns
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • the present disclosure relates to the field of computer technology but is not limited to the field of computer technology, and in particular, to a method and an apparatus for determining a gaze point, an electronic device, and a computer storage medium.
  • Face recognition is a new biometric technology that has emerged in recent years with the rapid development of computer vision, pattern recognition, neural networks and artificial intelligence. Face recognition is mainly based on the geometric information such as the position and shape of the face's various organs provided by the feature points of the face for facial recognition. Therefore, in the process of face recognition, the location of the feature points of the face is very important. At present, the localization of facial feature points can obtain high accuracy through deep learning.
  • the embodiment of the present disclosure provides a technical solution for determining a gaze point.
  • a method for determining a gaze point including:
  • a gaze point judgment device including:
  • An obtaining unit configured to obtain two-dimensional coordinates of eye feature points of at least one eye of a human face in an image, wherein the eye feature points include feature points of a central area of the eyeball; and two based on the feature points of the central area of the eyeball; Obtaining three-dimensional coordinates of a feature point of a corresponding eyeball center region in a three-dimensional face model corresponding to the face in the image in a preset three-dimensional coordinate system;
  • the judging unit is configured to obtain a pair of coordinates based on the two-dimensional coordinates of the feature points other than the feature points in the central area of the eyeball and the three-dimensional coordinates of the feature points in the central area of the eyeball in a preset three-dimensional coordinate system. A judgment result of an eye gaze point position of the human face in the image.
  • an electronic device including the device described in any one of the above embodiments.
  • an electronic device including:
  • Memory configured to store executable instructions
  • a processor configured to communicate with the memory to execute the executable instructions to complete the method described in any one of the above embodiments.
  • a computer program including computer-readable code, and when the computer-readable code runs on a device, a processor in the device executes a program for implementing any of the foregoing. Instructions for the methods described in the embodiments.
  • a computer program product for storing computer-readable instructions that, when executed, cause a computer to perform operations of the method described in any one of the foregoing embodiments.
  • two-dimensional coordinates of eye feature points of at least one eye of a human face in an image are obtained, where the eye feature points include an eyeball
  • the center area feature point is based on the two-dimensional coordinates of the center point feature of the eyeball to obtain the three-dimensional coordinates of the corresponding center point feature points of the eyeball in the three-dimensional face model corresponding to the face in the image in a preset three-dimensional coordinate system.
  • the two-dimensional coordinates of the feature points other than the feature points in the center of the eyeball and the three-dimensional coordinates of the feature points in the center of the eyeball in a preset three-dimensional coordinate system are used to obtain the judgment result of the position of the eye fixation point of the face in the image.
  • the learning of the more detailed information around the face and eyes in the image can achieve a more accurate judgment of the eye state, and thus can obtain more accurate eye state information.
  • FIG. 1 is a flowchart of a gaze point judgment method according to some embodiments of the present disclosure
  • FIGS. 2A and 2B are schematic diagrams of an application embodiment of a gaze point judgment method of the present disclosure
  • FIG. 3 is a flowchart of obtaining coordinates of feature points of faces and eyes of each image in a preset data set according to a preset data set according to an embodiment of the present disclosure
  • FIG. 4 is a flowchart of training a neural network according to the coordinates of feature points of faces and eyes of each image in a preset data set according to an embodiment of the present disclosure
  • FIG. 5 is a schematic structural diagram of a gaze point judgment device according to some embodiments of the present disclosure.
  • FIG. 6 is a schematic structural diagram of an obtaining unit in a gaze point judgment device according to some embodiments of the present disclosure
  • FIG. 7 is a schematic structural diagram of an electronic device according to some embodiments of the present disclosure.
  • Embodiments of the present disclosure may be applied to a computer system / server, which may operate with many other general or special purpose computing system environments or configurations.
  • Examples of well-known computing systems, environments, and / or configurations suitable for use with computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, based on Microprocessor systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above, and so on.
  • a computer system / server may be described in the general context of computer system executable instructions, such as program modules, executed by a computer system.
  • program modules may include routines, programs, target programs, components, logic, data structures, and so on, which perform specific tasks or implement specific abstract data types.
  • the computer system / server can be implemented in a distributed cloud computing environment. In a distributed cloud computing environment, tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on a local or remote computing system storage medium including a storage device.
  • FIG. 1 is a flowchart of a gaze point determination method according to some embodiments of the present disclosure.
  • the method is executed by a server or a terminal device.
  • the terminal device includes a mobile phone, a computer, and a vehicle-mounted device.
  • the method includes:
  • the image for determining the gaze point may be obtained from an image acquisition device or a storage device.
  • the image acquisition device may include a video camera, a camera, a scanner, and the like.
  • the storage device It may include: a hard disk, an optical disk, a floppy disk, and the like.
  • the embodiment of the present disclosure does not limit the manner of obtaining the image for determining the gaze point.
  • the eye feature points include feature points in the center of the eyeball.
  • the eye feature points further include: eyelid line feature points and eyeball contour feature points, which are not limited in the embodiments of the present disclosure.
  • the two-dimensional coordinates of the eye feature points of one eye of the face in the image can be obtained to determine the fixation points of the eyes of the face in the image.
  • the two-dimensional coordinates of the eye feature points of the two eyes of the face in the image can be obtained to determine the fixation points of the eyes of the face in the image.
  • two-dimensional coordinates of facial feature points in the image may be obtained by performing feature extraction on the image, and then based on the two-dimensional coordinates of facial feature points in the image, eyes of at least one eye of the human face in the image may be acquired.
  • the two-dimensional coordinates of the feature points For example, you can perform feature extraction on an image to obtain the two-dimensional coordinates of the 106 feature points of the face in the image, and then based on the two-dimensional coordinates of the 106 feature points, obtain the eye feature points of at least one eye of the face in the image. Two-dimensional coordinates.
  • a rectangular image containing the corresponding eye area may be intercepted from the image, and then the rectangular image may be subjected to feature extraction to obtain at least one of the faces in the image.
  • Two-dimensional coordinates of the eye feature points of only the eyes may be intercepted from the image, and then the rectangular image may be subjected to feature extraction to obtain at least one of the faces in the image.
  • a rectangular image including the corresponding eye area can be intercepted from the image, and then the feature extraction of the rectangular image can be performed to obtain the person in the image.
  • Two-dimensional coordinates of the eye feature points of the face and one eye can be performed to obtain the person in the image.
  • a rectangular image containing the corresponding eye area can be intercepted from the image, and then the rectangular image is mirror-processed to the rectangular image.
  • Feature extraction is performed on the rectangular image after mirror processing, and two-dimensional coordinates of eye feature points of two eyes of a face in the image are obtained.
  • feature extraction may be performed on the image through a neural network or other machine learning methods.
  • the neural network may be a convolutional neural network. The embodiment of the present disclosure does not limit the method for performing feature extraction on an image.
  • a corresponding three-dimensional face model can be generated based on the face in the image, and then based on the two-dimensional coordinates of the feature points of the central area of the eyeball, the corresponding feature points of the central area of the eyeball in the three-dimensional face model are obtained in a preset three-dimensional Three-dimensional coordinates in a coordinate system.
  • the key points of the face in the image may be preset, and a corresponding three-dimensional face model of the face is generated according to the correspondence between the key points of the face in the image and the key points of the prior three-dimensional face model.
  • the plurality of key points may include facial outline key points, eye key points, eyebrow key points, lip key points, nose key points, etc. The types and number of key points are not limited in the embodiments of the present disclosure.
  • the preset three-dimensional coordinate system can be determined according to preset principles, for example: the coordinate origin of the three-dimensional coordinate system is the center point of the front camera of the mobile phone, and the positive direction of the X-axis of the three-dimensional coordinate system is along the coordinate origin.
  • the front camera of the mobile phone is horizontal to the left
  • the positive direction of the Y axis of the three-dimensional coordinate system is the vertical upward direction from the coordinate origin along the front camera of the mobile phone
  • the positive direction of the Z axis of the three-dimensional coordinate system is perpendicular to the front camera of the mobile phone.
  • the origin of the coordinates points in the direction outside the front camera of the phone.
  • the face and eyes in the image may be obtained according to the two-dimensional coordinates of the feature points other than the feature points of the central area of the eyeball and the three-dimensional coordinates of the feature points of the central area of the eyeball in a preset three-dimensional coordinate system. Gaze score, and then compare the gaze score with a preset threshold to obtain the judgment result of the eye gaze point position of the face in the image.
  • the neural network or other machines may be used according to the two-dimensional coordinates of the feature points other than the feature points of the central area of the eyeball and the three-dimensional coordinates of the feature points of the central area of the eyeball in a preset three-dimensional coordinate system. Learn the method to get the gaze score of the face and eyes in the image.
  • the neural network can adopt a simple network structure composed of a fully connected layer and a ReLU layer. The embodiment of the present disclosure does not limit the method for obtaining the gaze score of the face and eyes in the image.
  • the facial eyes in the image are obtained through a neural network.
  • the format adjustment is the adjustment of the data order, that is, the two-dimensional coordinates of the feature points other than the feature points of the eyeball center region among the feature points of the eye and the three-dimensional coordinates of the feature points of the eyeball center region in a preset three-dimensional coordinate system.
  • the order of data is adjusted so that it is consistent with the data order when training the neural network.
  • the judgment result may include: the eye fixation point of the human face in the image is within the preset area, and the eye fixation point of the human face in the image is outside the preset area.
  • the preset area may include: a part or all of the screen area, that is, the embodiments of the present disclosure may be used to determine whether the eye gaze point of a human face in the image is within the screen area, or to determine whether the human face in the image is Whether the eye gaze point is within a specific area of the screen.
  • the preset threshold may include: a difference between a true case rate and a false positive case rate, wherein the real case rate may include: an example in which an eye fixation point of a human face in an image is determined to be correct Rate, the false positive rate may include: the rate of incorrect judgments when the eye gaze point of a human face in the image is outside the preset area.
  • the image may be further processed according to the judgment result.
  • the eye fixation point of the human face in the image may be displayed in a preset area in response to the first preset display mode, and the eye fixation point of the human face in the image may be outside the preset area according to The second preset display mode displays the image.
  • the first preset display mode and the second preset display mode are configured with different color borders for the display image, for example, as shown in FIGS.
  • a red border is configured for the displayed image
  • a blue border is configured for the displayed image
  • the method according to the embodiments of the present disclosure may be used to identify facial expressions in an image, and may also determine whether a person in the image has a current situation by judging the position of a person's eyes and gaze point in the image. Informed, used for terminal payment, terminal lock, and terminal unlock to ensure the security of payment, lock, and unlock.
  • two-dimensional coordinates of eye feature points of at least one eye of a human face in an image are obtained, where the eye feature points include feature points of a central area of the eyeball, and are based on the central area of the eyeball.
  • the two-dimensional coordinates of the feature points are used to obtain the three-dimensional coordinates of the corresponding feature points of the center of the eyeball in the three-dimensional face model corresponding to the face in the image.
  • the feature points of the center of the eyeball are excluded from the feature points of the eye.
  • the two-dimensional coordinates of the external feature points and the three-dimensional coordinates of the feature points in the center of the eyeball are in a preset three-dimensional coordinate system to obtain the judgment result of the position of the eye fixation point of the face in the image.
  • the learning of information can achieve a more accurate judgment of the eye state, and thus can obtain more accurate eye state information.
  • a neural network is used to obtain an alignment.
  • the neural network used is not a traditional neural network, the neural network needs to be trained first, and because the input of the neural network is not a traditional image, but an image The coordinates of the facial feature points of the face in the face. Therefore, before training the neural network based on the images in the preset data set, the coordinates of the feature points of the face eyes in the image in the preset data set must also be obtained.
  • FIG. 3 and FIG. 4 are only for helping those skilled in the art to better understand the technical solutions of the present disclosure, and should not be construed as limiting the present disclosure. Those skilled in the art can perform various transformations on the basis of FIGS. 3 and 4, and such transformations should also understand a part of the technical solution disclosed.
  • the method includes:
  • the image is obtained from a preset data set, and each image in the preset data set is marked with a position of an eye fixation point of a face in the image, wherein the preset data set may use an existing face
  • the identification data set is not limited in the embodiment of the present disclosure.
  • feature extraction may be performed on the image to obtain the two-dimensional coordinates of the feature points of the face in the image, and then the corresponding A rectangular image of the eye area, and the rectangular image is mirror-processed.
  • the rectangular image and the rectangular image after the mirror-processed feature extraction are performed to obtain the two-dimensional coordinates and eye contour of the eyelid line feature points of the two eyes of the face in the image.
  • feature extraction may be performed on the image through a neural network or other machine learning methods.
  • the neural network may be a convolutional neural network.
  • the embodiment of the present disclosure does not limit the method for performing feature extraction on an image.
  • the feature extraction of the image can be performed through the God network to obtain the two-dimensional coordinates of the 106 feature points of the face in the image.
  • a corresponding three-dimensional face model can be generated based on the face in the image, and then based on the two-dimensional coordinates of the feature points of the central area of the eyeball, the corresponding feature points of the central area of the eyeball in the three-dimensional face model are obtained in a preset three-dimensional Three-dimensional coordinates in a coordinate system.
  • the key points of the face in the image may be preset, and a corresponding three-dimensional face model of the face is generated according to the correspondence between the key points of the face in the image and the key points of the prior three-dimensional face model.
  • the plurality of key points may include facial outline key points, eye key points, eyebrow key points, lip key points, nose key points, etc. The types and number of key points are not limited in the embodiments of the present disclosure.
  • the three-dimensional coordinates of the feature points of the central area of the two eyes in the three-dimensional face model corresponding to the human face in the image are obtained in a preset three-dimensional coordinate system.
  • the two-dimensional coordinates of the eyelid line feature points, the two-dimensional coordinates of the eyeball contour feature points, and the three-dimensional coordinates of the center point feature points of the eyeballs can be stored in a file according to a preset format.
  • the preset format may be the order of the two-dimensional coordinates of the eyelid line feature points, the two-dimensional coordinates of the eyeball contour feature points, and the three-dimensional coordinates of the feature points of the center of the eyeball in the preset three-dimensional coordinate system.
  • the pre- when the two-dimensional coordinates of the eyelid line feature point, the two-dimensional coordinates of the eyeball contour feature point, and the three-dimensional coordinates of the eyeball center area feature point in a preset three-dimensional coordinate system are stored in a file, the pre- Let the coordinates of the feature points of each face of the image in the data set be divided into a test set and a training set, respectively, for training and testing the neural network.
  • the method includes:
  • the neural network is used to obtain an image in the image.
  • a loss between the judgment result of the position of the eye fixation point of the face and the position of the eye fixation point of the face in the image labeled with the image corresponding to the training sample, and the parameters of the neural network are updated by back propagation based on the loss.
  • the training samples in the training set can be obtained according to the images in the preset data set, where each training sample in the training set consists of the two-dimensional coordinates of the eyelid line feature points, the two-dimensional coordinates of the eyeball contour feature points, and The feature points in the center of the eyeball are composed of three-dimensional coordinates in a preset three-dimensional coordinate system.
  • Each image in the preset data set is marked with the position of the eye's gaze point of the face in the image.
  • the preset data set can use existing face recognition data. This is not limited in the embodiments of the present disclosure.
  • the neural network can adopt a simple network structure composed of a fully connected layer and a ReLU layer.
  • a neural network training method such as a gradient descent method may be used to train the neural network, which is not limited in the embodiments of the present disclosure.
  • the neural network After training the neural network a preset number of times, according to the two-dimensional coordinates of the eyelid line feature points of the test sample in the test set, the two-dimensional coordinates of the eyeball contour feature points, and the feature points of the central area of the eyeball in a preset three-dimensional coordinate system.
  • the accuracy of the judgment result of the position of the eye fixation point of the face in the image is obtained by the neural network in the three-dimensional coordinates in the image, and the training of the neural network is stopped according to the accuracy rate.
  • the test samples in the test set can be obtained according to the images in the preset data set, where each test sample in the test set consists of the two-dimensional coordinates of the eyelid line feature points, the two-dimensional coordinates of the eyeball contour feature points, and The feature points in the center of the eyeball are composed of three-dimensional coordinates in a preset three-dimensional coordinate system.
  • Each image in the preset data set is labeled with the position of the eye's gaze point of the face in the image.
  • the test set and training set can be based on the same preset data set. It can also be obtained according to different preset data sets, which is not limited in the embodiment of the present disclosure.
  • the preset data set can be an existing face recognition data set, which is not limited in the embodiments of the present disclosure.
  • the task of the neural network is relatively simple, overfitting is easy to occur.
  • the network can be terminated in advance. training.
  • a preset threshold for judging the position of the eye gaze point of a human face in the image may be determined according to the test set.
  • the gaze score of the image corresponding to each test sample can be obtained by inputting the test sample into the neural network, and then comparing each gaze score with the corresponding image annotation, and determining the gaze score greater than the current threshold as the gaze preset area, Determine that the gaze score is less than or equal to the current threshold value as a non-gaze preset area, and use True_positive to indicate that the score is greater than the current threshold value and mark it as positive (fixation), and False_negative to indicate that the score is less than or equal to the current threshold value and mark it as positive (fixation).
  • False_positive indicates that the score is greater than the current threshold and marked as negative (non-fixation)
  • True_negative indicates that the score is less than or equal to the current threshold and marked as negative (non-fixation).
  • the true rate is called the first rate; the false positive rate is called the second rate.
  • the "true” and “false positive” are names that distinguish two interest rates.
  • FIG. 5 is a schematic structural diagram of a gaze point judging device according to some embodiments of the present disclosure.
  • the device is set to be executed by a server or a terminal device.
  • the terminal device includes a mobile phone, a computer, and a vehicle-mounted device.
  • the device includes: The obtaining unit 510 and the determining unit 520. among them,
  • the obtaining unit 510 is configured to obtain two-dimensional coordinates of eye feature points of at least one eye of a human face in an image.
  • the image for determining the gaze point may be obtained from an image acquisition device or a storage device.
  • the image acquisition device may include a video camera, a camera, a scanner, and the like.
  • the storage device It may include: a hard disk, an optical disk, a floppy disk, and the like.
  • the embodiment of the present disclosure does not limit the manner of obtaining the image for determining the gaze point.
  • the eye feature points include feature points in the center of the eyeball.
  • the eye feature points further include: eyelid line feature points and eyeball contour feature points, which are not limited in the embodiments of the present disclosure.
  • the obtaining unit 510 may fix the eyes of the face in the image by obtaining the two-dimensional coordinates of the eye feature points of one eye of the face in the image. Click to judge.
  • the obtaining unit 510 may determine the fixation points of the eyes of the human face in the image by acquiring the two-dimensional coordinates of the eye feature points of the two eyes of the human face in the image.
  • the obtaining unit 510 may obtain a two-dimensional coordinate of a feature point of a face in the image by performing feature extraction on the image, and then obtain at least one face in the image based on the two-dimensional coordinate of the feature point of the face in the image.
  • the two-dimensional coordinates of the eye's feature points may perform feature extraction on the image, obtain the two-dimensional coordinates of the 106 feature points of the face in the image, and then obtain the eyes of at least one eye of the face in the image based on the two-dimensional coordinates of the 106 feature points.
  • the two-dimensional coordinates of the feature points may obtain a two-dimensional coordinate of a feature point of a face in the image by performing feature extraction on the image, and then obtain at least one face in the image based on the two-dimensional coordinate of the feature point of the face in the image.
  • the obtaining unit 510 may include a truncation subunit 511 and an extraction subunit 512.
  • the interception subunit 511 can intercept a rectangular image including the corresponding eye area from the image according to the two-dimensional coordinates of the feature points of the face and eye area in the image, and the extraction subunit 512 can perform feature extraction on the rectangular image to obtain the image. Two-dimensional coordinates of eye feature points of at least one eye of a human face.
  • the interception sub-unit 511 may intercept a rectangular image including the corresponding eye area from the image according to the two-dimensional coordinates of the feature points of one eye area of the face in the image, and the extraction sub-unit 512 may perform an operation on the rectangular image. Feature extraction is performed to obtain two-dimensional coordinates of eye feature points of one eye of a face in an image.
  • the obtaining unit 510 further includes: a mirroring sub-unit 513, and the intercepting sub-unit 511 may intercept a feature including corresponding eye regions from the image according to the two-dimensional coordinates of the feature points of one eye region of the face in the image.
  • a rectangular image, the mirroring sub-unit 513 can mirror the rectangular image; the extraction sub-unit 512 can perform feature extraction on the rectangular image and the mirror-processed rectangular image, and obtain two of the eye feature points of the two eyes of the face in the image Dimensional coordinates.
  • features can be extracted from the image through a neural network or other machine learning methods.
  • the neural network may be a convolutional neural network. The embodiment of the present disclosure does not limit the method for performing feature extraction on an image.
  • the obtaining unit 510 is further configured to obtain, based on the two-dimensional coordinates of the feature points of the central area of the eyeball, the three-dimensional coordinates of the feature points of the corresponding central area of the eyeball in the three-dimensional face model corresponding to the face in the image in a preset three-dimensional coordinate system.
  • the obtaining unit 510 may generate a corresponding three-dimensional face model according to the human face in the image, and then obtain the corresponding feature point of the central eye area in the three-dimensional face model based on the two-dimensional coordinates of the feature point of the central eye area. Three-dimensional coordinates in a preset three-dimensional coordinate system.
  • the key points of the face in the image may be preset, and the obtaining unit 510 generates a corresponding three-dimensional face corresponding to the key points of the face in the image and the key points of the prior three-dimensional face model.
  • Face model for example, multiple key points may include facial outline key points, eye key points, eyebrow key points, lips key points, nose key points, etc. The types and number of key points are not limited in the embodiments of the present disclosure. .
  • the preset three-dimensional coordinate system can be determined according to preset principles, for example: the coordinate origin of the three-dimensional coordinate system is the center point of the front camera of the mobile phone, and the positive direction of the X-axis of the three-dimensional coordinate system is along the coordinate origin.
  • the front camera of the mobile phone is horizontal to the left
  • the positive direction of the Y axis of the three-dimensional coordinate system is the vertical upward direction from the coordinate origin along the front camera of the mobile phone
  • the positive direction of the Z axis of the three-dimensional coordinate system is perpendicular to the front camera of the mobile phone.
  • the origin of the coordinates points in the direction outside the front camera of the phone.
  • the judging unit 520 is configured to obtain, based on the two-dimensional coordinates of the feature points other than the feature points in the central area of the eyeball, and the three-dimensional coordinates of the feature points in the central area of the eyeball in a preset three-dimensional coordinate system, the Judgment result of eye gaze point position.
  • the judging unit 520 may obtain the image in the image according to the two-dimensional coordinates of the feature points other than the feature points of the central area of the eyeball and the three-dimensional coordinates of the feature points of the central area of the eyeball in a preset three-dimensional coordinate system.
  • the gaze score of the human face and eyes, and then the gaze score is compared with a preset threshold to obtain a judgment result of the position of the eye gaze point of the human face in the image.
  • the judging unit 520 may use a neural network according to the two-dimensional coordinates of the feature points other than the feature points of the central area of the eyeball and the three-dimensional coordinates of the feature points of the central area of the eyeball in a preset three-dimensional coordinate system.
  • the neural network can adopt a simple network structure composed of a fully connected layer and a ReLU layer.
  • the embodiment of the present disclosure does not limit the method for obtaining the gaze score of the face and eyes in the image.
  • the device further includes an adjustment unit, and in the obtaining unit 510, based on the two-dimensional coordinates of the feature points in the center area of the eyeball, the corresponding feature points in the center area of the eyeball in the three-dimensional face model corresponding to the face in the image are obtained in After setting the three-dimensional coordinates in the three-dimensional coordinate system, the adjusting unit can preset the two-dimensional coordinates of the feature points other than the feature points of the eyeball center area and the feature points of the eyeball center area in the preset three-dimensional coordinates according to a preset format.
  • the format of the three-dimensional coordinates in the system is adjusted, so that the judging unit 520 can adjust the two-dimensional coordinates of the feature points other than the feature points of the eyeball center area and the feature points of the eyeball center area in the preset three-dimensional coordinates according to the adjusted feature points
  • the three-dimensional coordinates in the system are used to obtain the gaze score of the face and eyes in the image through a neural network.
  • the format adjustment is the adjustment of the data order, that is, the two-dimensional coordinates of the feature points other than the feature points of the eyeball center region among the feature points of the eye and the three-dimensional coordinates of the feature points of the eyeball center region in a preset three-dimensional coordinate system
  • the order of data is adjusted so that it is consistent with the data order when training the neural network.
  • the judgment result may include: the eye fixation point of the human face in the image is within the preset area, and the eye fixation point of the human face in the image is outside the preset area.
  • the judgment unit 520 compares the fixation score with a preset threshold, and obtains a determination result that the eye fixation point of the face in the image is within a preset area according to the fixation score being greater than the preset threshold; and according to the fixation score is less than or equal to the preset threshold
  • the threshold can be used to obtain the judgment result of the eye fixation point of the face in the image outside the preset area.
  • the preset area may include: a part or all of the screen area, that is, the embodiments of the present disclosure may be used to determine whether the eye gaze point of a human face in the image is within the screen area, or to determine whether the human face in the image is Whether the eye gaze point is within a specific area of the screen.
  • the preset threshold may include: a difference between a true case rate and a false positive case rate, wherein the real case rate may include: an example in which an eye fixation point of a human face in an image is determined to be correct Rate, the false positive rate may include: the rate of incorrect judgments when the eye gaze point of a human face in the image is outside the preset area.
  • the apparatus further includes: a processing unit. After comparing the fixation score with a preset threshold to obtain a judgment result of the position of the eye fixation point of the face in the image, the processing unit may further perform corresponding processing on the image according to the judgment result. In one example, the processing unit may respond to the eye gaze point of the human face in the image within the preset area, and display the image in the first preset display manner, and may respond to the eye gaze point of the human face in the image outside the preset area. , Displaying the image according to the second preset display mode. In one example, the first preset display mode and the second preset display mode are configured with borders of different colors for the display image. For example, as shown in FIGS.
  • the processing unit responds to the eye gaze of the human face in the image.
  • Point within the preset area configure a red border for the displayed image
  • in response to the eye gaze point of the human face in the image outside the preset area configure a blue border for the displayed image.
  • the device according to the embodiment of the present disclosure may be used to recognize facial expressions in an image, and may also determine whether a person in the image is aware of the current situation by judging the position of a face's eye fixation point in the image. Informed, used for terminal payment, terminal lock, and terminal unlock to ensure the security of payment, lock, and unlock.
  • two-dimensional coordinates of eye feature points of at least one eye of a human face in an image are obtained, where the eye feature points include feature points of a central area of the eyeball, and are based on the central area of the eyeball.
  • the two-dimensional coordinates of the feature points are used to obtain the three-dimensional coordinates of the corresponding feature points of the center of the eyeball in the three-dimensional face model corresponding to the face in the image.
  • the feature points of the center of the eyeball are excluded from the feature points of the eye.
  • the two-dimensional coordinates of the external feature points and the three-dimensional coordinates of the feature points in the center of the eyeball are in a preset three-dimensional coordinate system to obtain the judgment result of the position of the eye fixation point of the face in the image.
  • the learning of information can achieve a more accurate judgment of the eye state, and thus can obtain more accurate eye state information.
  • FIG. 7 illustrates a schematic structural diagram of an electronic device 700 suitable for implementing a terminal device or a server in the embodiment of the present application.
  • the electronic device 700 includes one or more processors and a communication unit.
  • the one or more processors are, for example, one or more central processing units (CPUs) 701, and / or one or more acceleration units 713.
  • the acceleration units 713 may include, but are not limited to, GPUs, FPGAs, and other types.
  • the processor can perform various appropriate actions based on executable instructions stored in read-only memory (ROM) 702 or executable instructions loaded from storage section 708 into random access memory (RAM) 703 And processing.
  • the communication unit 712 may include, but is not limited to, a network card.
  • the network card may include, but is not limited to, an IB (Infiniband) network card.
  • the processor may communicate with the read-only memory 702 and / or the random access memory 703 to execute executable instructions. It is connected to the communication unit 712 and communicates with other target devices via the communication unit 712, thereby completing the operation corresponding to any of the methods provided in the embodiments of the present application, for example, acquiring eye feature points of at least one eye of a human face in an image.
  • Two-dimensional coordinates wherein the eye feature points include feature points in a central area of the eyeball; and based on the two-dimensional coordinates of feature points in the central area of the eyeball, obtaining correspondences in the three-dimensional face model corresponding to the face in the image
  • the three-dimensional coordinates of the feature points of the center of the eyeball in a preset three-dimensional coordinate system; according to the two-dimensional coordinates of the feature points of the feature points of the eye other than the feature points of the center of the eyeball and the feature points of the center of the eyeball at A three-dimensional coordinate in a three-dimensional coordinate system is preset to obtain a judgment result of an eye gaze point position of the human face in the image.
  • the RAM 703 can also store various programs and data required for the operation of the device.
  • the CPU 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704.
  • ROM 702 is an optional module.
  • the RAM 703 stores executable instructions, or writes executable instructions to the ROM 702 at runtime, and the executable instructions cause the central processing unit 701 to perform operations corresponding to the foregoing communication method.
  • An input / output (I / O) interface 705 is also connected to the bus 704.
  • the communication unit 712 may be provided in an integrated manner, or may be provided with a plurality of sub-modules (for example, a plurality of IB network cards) and connected on a bus link.
  • the following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), and the speaker; a storage section 708 including a hard disk And a communication section 709 including a network interface card such as a LAN card, a modem, and the like.
  • the communication section 709 performs communication processing via a network such as the Internet.
  • the driver 710 is also connected to the I / O interface 705 as needed.
  • a removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read out therefrom is installed into the storage section 708 as needed.
  • FIG. 7 is only an optional implementation manner.
  • the number and types of components in FIG. 7 can be selected, deleted, added, or replaced according to actual needs.
  • separate or integrated settings can also be used.
  • the acceleration unit 713 and CPU701 can be set separately or the acceleration unit 713 can be integrated on the CPU701.
  • the communication unit 712 can be set separately or integrated on the CPU701 or acceleration unit 713, and so on.
  • embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code for performing a method shown in a flowchart, and the program code may include a corresponding
  • the instructions corresponding to the method steps provided in the embodiments of the present application are executed, for example, obtaining two-dimensional coordinates of eye feature points of at least one eye of a human face in an image, wherein the eye feature points include feature points of a central area of the eyeball; Obtain the two-dimensional coordinates of the feature points of the central area of the eyeball, and obtain the three-dimensional coordinates of the feature points of the corresponding central area of the eyeball in the three-dimensional face model corresponding to the face in the image in a preset three-dimensional coordinate system; The two-dimensional coordinates of the feature points in the eye feature points other than the feature points in
  • the computer program may be downloaded and installed from a network through the communication section 709, and / or installed from a removable medium 711.
  • a central processing unit (CPU) 701 the above-mentioned functions defined in the method of the present application are executed.
  • an embodiment of the present disclosure further provides a computer program program product for storing computer-readable instructions that, when executed, cause a computer to execute any of the foregoing possible implementation manners. Gaze point judgment method.
  • the computer program product may be specifically implemented by hardware, software, or a combination thereof.
  • the computer program product is embodied as a computer storage medium.
  • the computer program product is embodied as a software product, such as a Software Development Kit (SDK) and the like.
  • SDK Software Development Kit
  • an embodiment of the present disclosure further provides a gaze point judgment method and a corresponding device, an electronic device, a computer storage medium, a computer program, and a computer program product, wherein the method includes: The first device sends a gaze point determination instruction to the second device, and the instruction causes the second device to execute the gaze point determination method in any of the foregoing possible embodiments; the first device receives a result of the gaze point determination sent by the second device.
  • the fixation point judgment instruction may be specifically a call instruction.
  • the first device may instruct the second device to perform fixation point judgment by means of a call. Accordingly, in response to receiving the call instruction, the second device may execute the above-mentioned instruction. Steps and / or processes in any embodiment of the gaze point judgment method.
  • a plurality may refer to two or more, and “at least one” may refer to one, two, or more.
  • the methods and apparatus of the present disclosure may be implemented in many ways.
  • the methods and apparatuses of the present disclosure may be implemented by software, hardware, firmware or any combination of software, hardware, firmware.
  • the above order of the steps of the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specifically stated.
  • the present disclosure may also be implemented as programs recorded in a recording medium, which programs include machine-readable instructions for implementing the method according to the present disclosure.
  • the present disclosure also covers a recording medium storing a program for executing a method according to the present disclosure.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Ophthalmology & Optometry (AREA)
  • Evolutionary Computation (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Medical Informatics (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
  • Eye Examination Apparatus (AREA)

Abstract

一种注视点判断方法和装置、电子设备和计算机存储介质，其中，方法包括：获取图像中人脸至少一只眼睛的眼部特征点的二维坐标(102)，其中，所述眼部特征点包括眼球中心区域特征点；基于所述眼球中心区域特征点的二维坐标，获取所述图像中的所述人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标(104)；根据所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，得到对所述图像中所述人脸的眼睛注视点位置的判断结果(106)。

Description

注视点判断方法和装置、电子设备和计算机存储介质
相关申请的交叉引用
本申请基于申请号为201811141352.3、申请日为2018年09月28日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本申请作为参考。
技术领域
本公开涉及计算机技术领域但不限于计算机技术领域，尤其是一种注视点判断方法和装置、电子设备和计算机存储介质。
背景技术
人脸识别是近年来随着计算机视觉、模式识别、神经网络和人工智能等技术快速发展而出现的一种新的生物识别技术。人脸识别主要是基于人脸特征点提供的人脸各器官的位置、形状以及人脸外轮廓等几何信息进行身份识别，因此在人脸识别过程中，人脸特征点的定位非常重要。目前，人脸特征点的定位可以通过过深度学习获得很高的精度。
发明内容
本公开实施例提供一种注视点判断技术方案。
根据本公开实施例的一个方面，提供一种注视点判断方法，包括：
获取图像中人脸至少一只眼睛的眼部特征点的二维坐标，其中，所述眼部特征点包括眼球中心区域特征点；
基于所述眼球中心区域特征点的二维坐标，获取所述图像中的所述人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标；
根据所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，得到对所述图像中所述人脸的眼睛注视点位置的判断结果。
根据本公开实施例的另一个方面，提供一种注视点判断装置，包括：
获取单元，配置为获取图像中人脸至少一只眼睛的眼部特征点的二维坐标，其中，所述眼部特征点包括眼球中心区域特征点；以及基于所述眼球中心区域特征点的二维坐标，获取所述图像中的所述人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标；
判断单元，配置为根据所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，得到对所述图像中所述人脸的眼睛注视点位置的判断结果。
根据本公开实施例的又一个方面，提供的一种电子设备，包括上述任 一实施例所述的装置。
根据本公开实施例的再一个方面，提供的一种电子设备，包括：
存储器，配置为存储可执行指令；以及
处理器，配置为与所述存储器通信以执行所述可执行指令从而完成上述任一实施例所述的方法。
根据本公开实施例的再一个方面，提供的一种计算机程序，包括计算机可读代码，当所述计算机可读代码在设备上运行时，所述设备中的处理器执行用于实现上述任一实施例所述方法的指令。
根据本公开实施例的再一个方面，提供的一种计算机程序产品，用于存储计算机可读指令，所述指令被执行时使得计算机执行上述任一实施例所述的方法的操作。
基于本公开上述实施例提供的注视点判断方法和装置、电子设备和计算机存储介质，通过获取图像中人脸至少一只眼睛的眼部特征点的二维坐标，其中，眼部特征点包括眼球中心区域特征点，基于眼球中心区域特征点的二维坐标，获取图像中的人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标，根据眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标，得到对图像中人脸的眼睛注视点位置的判断结果，利用对图像中人脸眼睛周围更精细信息的学习，可以实现对眼睛状态更准确的判断，从而可以获得更精准的眼睛状态信息。
下面通过附图和实施例，对本公开的技术方案做进一步的详细描述。
附图说明
构成说明书的一部分的附图描述了本公开的实施例，并且连同描述一起用于解释本公开的原理。
参照附图，根据下面的详细描述，可以更加清楚地理解本公开，其中：
图1为本公开一些实施例的注视点判断方法的流程图；
图2A和图2B为本公开注视点判断方法的一个应用实施例的示意图；
图3为本公开实施例根据预设数据集获取预设数据集中各图像的人脸眼部特征点的坐标的流程图；
图4为本公开实施例根据预设数据集中各图像的人脸眼部特征点的坐标对神经网络进行训练的流程图；
图5为本公开一些实施例的注视点判断装置的结构示意图；
图6为本公开一些实施例的注视点判断装置中获取单元的结构示意图；
图7为本公开一些实施例的电子设备的结构示意图。
具体实施方式
现在将参照附图来详细描述本公开的各种示例性实施例。应注意到： 除非另外具体说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本公开的范围。
同时，应当明白，为了便于描述，附图中所示出的各个部分的尺寸并不是按照实际的比例关系绘制的。
以下对至少一个示例性实施例的描述实际上仅仅是说明性的，决不作为对本公开及其应用或使用的任何限制。
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论，但在适当情况下，所述技术、方法和设备应当被视为说明书的一部分。
应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步讨论。
本公开实施例可以应用于计算机系统/服务器，其可与众多其它通用或专用计算系统环境或配置一起操作。适于与计算机系统/服务器一起使用的众所周知的计算系统、环境和/或配置的例子包括但不限于：个人计算机系统、服务器计算机系统、瘦客户机、厚客户机、手持或膝上设备、基于微处理器的系统、机顶盒、可编程消费电子产品、网络个人电脑、小型计算机系统﹑大型计算机系统和包括上述任何系统的分布式云计算技术环境，等等。
计算机系统/服务器可以在由计算机系统执行的计算机系统可执行指令(诸如程序模块)的一般语境下描述。通常，程序模块可以包括例程、程序、目标程序、组件、逻辑、数据结构等等，它们执行特定的任务或者实现特定的抽象数据类型。计算机系统/服务器可以在分布式云计算环境中实施，分布式云计算环境中，任务是由通过通信网络链接的远程处理设备执行的。在分布式云计算环境中，程序模块可以位于包括存储设备的本地或远程计算系统存储介质上。
图1为本公开一些实施例的注视点判断方法的流程图，该方法由服务器或者终端设备执行，例如：终端设备包括手机、计算机、车载设备等，如图1所示，该方法包括：
102，获取图像中人脸至少一只眼睛的眼部特征点的二维坐标。
在本公开实施例中，用于进行注视点判断的图像可以是从图像采集设备获取的，也可以是从存储设备获取，例如：图像采集设备可以包括：摄像机、照相机、扫描仪等，存储设备可以包括：硬盘、光盘、软盘等，本公开实施例对进行注视点判断的图像的获取方式不作限定。其中，眼部特征点包括眼球中心区域特征点，在一些实施例中，眼部特征点还包括：眼睑线特征点和眼球轮廓特征点，本公开实施例不以此为限。
在一个例子中，由于通常人的双眼是同时注视同一个方向，因此可以通过获取图像中人脸一只眼睛的眼部特征点的二维坐标，来对图像中人脸双眼的注视点进行判断。在另一个例子中，可以通过获取图像中人脸两只 眼睛的眼部特征点的二维坐标，来对图像中人脸双眼的注视点进行判断。
在一些实施例中，可以通过对图像进行特征提取，获取图像中人脸特征点的二维坐标，然后基于图像中人脸特征点的二维坐标，获取图像中人脸至少一只眼睛的眼部特征点的二维坐标。例如：可以对图像进行特征提取，获取图像中人脸的106个特征点的二维坐标，然后基于这106个特征点的二维坐标，获取图像中人脸至少一只眼睛的眼部特征点的二维坐标。
在一些实施例中，可以根据图像中人脸眼睛区域的特征点的二维坐标，从图像中截取包含对应的眼睛区域的矩形图像，然后对矩形图像进行特征提取，获取图像中人脸至少一只眼睛的眼部特征点的二维坐标。
在一个例子中，可以根据图像中人脸一只眼睛区域的特征点的二维坐标，从图像中截取包含对应的眼睛区域的一张矩形图像，然后对矩形图像进行特征提取，获取图像中人脸一只眼睛的眼部特征点的二维坐标。
在另一个例子中，可以根据图像中人脸一只眼睛区域的特征点的二维坐标，从图像中截取包含对应的眼睛区域的一张矩形图像，然后对矩形图像进行镜像处理，对矩形图像和镜像处理后的矩形图像进行特征提取，获取图像中人脸两只眼睛的眼部特征点的二维坐标。
在一些实施例中，可以通过神经网络或者其它机器学习的方法对图像进行特征提取。在一个例子中，神经网络可以采用卷积神经网络。本公开实施例对对图像进行特征提取的方法不作限定。
104，基于眼球中心区域特征点的二维坐标，获取图像中的人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标。
在一些实施例中，可以根据图像中的人脸生成对应的三维人脸模型，然后基于眼球中心区域特征点的二维坐标，获取三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标。在一些实施例中，可以预设图像中人脸的关键点，根据图像中人脸的关键点与先验三维人脸模型的关键点之间的对应关系，生成人脸对应的三维人脸模型，例如，多个关键点可以包括脸部外轮廓关键点、眼部关键点、眉毛关键点、嘴唇关键点、鼻子关键点等，本公开实施例对关键点的类型和数量不作限定。
在一些实施例中，预设三维坐标系可以根据预设的原则确定，例如：三维坐标系的坐标原点为手机前置摄像头的中心点，三维坐标系的X轴正方向为从坐标原点沿着手机前置摄像头水平向左的方向，三维坐标系的Y轴正方向为从坐标原点沿着手机前置摄像头竖直向上的方向，三维坐标系的Z轴正方向为与手机前置摄像头垂直从坐标原点指向手机前置摄像头外侧的方向。
106，根据眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标，得到对图像中人脸的眼睛注视点位置的判断结果。
在一些实施例中，可以根据眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标，得到图像中人脸眼睛的注视分数，然后将注视分数与预设阈值进行比较，得到对图像中人脸的眼睛注视点位置的判断结果。在一些实施例中，根据眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标，可以通过神经网络或者其它机器学习的方法，得到图像中人脸眼睛的注视分数。在一个例子中，神经网络可以采用由全连接层和ReLU层组成的简单的网络结构。本公开实施例对获得图像中人脸眼睛的注视分数的方法不作限定。
在一些实施例中，在基于眼球中心区域特征点的二维坐标，获取图像中的人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标之后，还可以按照预设的格式，对眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标的格式进行调整，从而根据格式调整后的眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标，通过神经网络得到图像中人脸眼睛的注视分数。在一个例子中，格式调整为数据顺序的调整，即对眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标的顺序进行调整，使其与对神经网络进行训练时的数据顺序保持一致。
在一些实施例中，判断结果可以包括：图像中人脸的眼睛注视点在预设区域内，和图像中人脸的眼睛注视点在预设区域外。通过将注视分数与预设阈值进行比较，根据注视分数大于预设阈值，可以得到图像中人脸的眼睛注视点在预设区域内的判断结果；而根据注视分数小于或等于预设阈值，可以得到图像中人脸的眼睛注视点在预设区域外的判断结果。在一个例子中，预设区域可以包括：屏幕区域的一部分或者全部，即本公开实施例可以用来判断图像中人脸的眼睛注视点是否在屏幕区域内，或者用来判断图像中人脸的眼睛注视点是否在屏幕的特定区域内。
在一些实施例中，预设阈值可以包括：真正例率与假正例率的差值，其中，真正例率可以包括：图像中人脸的眼睛注视点在预设区域内时判断正确的例率，假正例率可以包括：图像中人脸的眼睛注视点在预设区域外时判断错误的例率。
在一些实施例中，在将注视分数与预设阈值进行比较，得到对图像中人脸的眼睛注视点位置的判断结果之后，还可以根据判断结果，对图像进行相应的处理。在一个例子中，可以响应于图像中人脸的眼睛注视点在预设区域内，按照第一预设展示方式展示图像，可以响应于图像中人脸的眼睛注视点在预设区域外，按照第二预设展示方式展示图像。在一个例子中，第一预设展示方式和第二预设展示方式分别为为展示图像配置不同颜色的 边框，例如：如图2A和2B所示，响应于图像中人脸的眼睛注视点在预设区域内，为展示的图像配置红色边框，响应于图像中人脸的眼睛注视点在预设区域外，为展示的图像配置蓝色边框。
在一些实施例中，本公开实施例的方法可以用于对图像中人脸的表情进行识别，也可以通过对图像中人脸眼睛注视点位置的判断，确定图像中的人对当前的情况是否知情，以用于终端支付、终端锁定、终端解锁，保证支付、锁定、解锁的安全性。
基于本公开上述实施例提供的注视点判断方法，通过获取图像中人脸至少一只眼睛的眼部特征点的二维坐标，其中，眼部特征点包括眼球中心区域特征点，基于眼球中心区域特征点的二维坐标，获取图像中的人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标，根据眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标，得到对图像中人脸的眼睛注视点位置的判断结果，利用对图像中人脸眼睛周围更精细信息的学习，可以实现对眼睛状态更准确的判断，从而可以获得更精确的眼睛状态信息。
在上述各实施例中，当根据眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标，通过神经网络得到对图像中人脸的眼睛注视点位置的判断结果时，由于所采用的神经网络并非传统的神经网络，需要先对该神经网络进行训练，又由于该神经网络的输入并非传统的图像，而是图像中人脸眼部特征点的坐标，因此，在根据预设数据集中的图像对该神经网络进行训练之前，还需要先获取预设数据集中图像的人脸眼部特征点的坐标。
下面将结合图3和图4的实施例，详细描述根据预设数据集获取预设数据集中各图像的人脸眼部特征点的坐标的流程，和根据预设数据集中各图像的人脸眼部特征点的坐标对神经网络进行训练的流程。
应理解，图3和图4所示的例子仅仅是为了帮助本领域技术人员更好地理解本公开的技术方案，而不应理解成对本公开的限定。本领域技术人员可以在图3和图4的基础上进行各种变换，而这种变换也应理解成本公开技术方案的一部分。
如图3所示，该方法包括：
302，获取图像中人脸两只眼睛的眼睑线特征点的二维坐标、眼球轮廓特征点的二维坐标和眼球中心区域特征点的二维坐标。
在本公开实施例中，图像是从预设数据集中获取的，预设数据集中的各图像均标注有图像中人脸的眼睛注视点位置，其中，预设数据集可以采用现有的人脸识别数据集，本公开实施例对此不作限定。
在一些实施例中，可以对图像进行特征提取，获取图像中人脸特征点的二维坐标，然后根据图像中人脸一只眼睛区域的特征点的二维坐标，从 图像中截取包含对应的眼睛区域的一张矩形图像，并对矩形图像进行镜像处理，对矩形图像和镜像处理后的矩形图像进行特征提取，获取图像中人脸两只眼睛的眼睑线特征点的二维坐标、眼球轮廓特征点的二维坐标和眼球中心区域特征点的二维坐标。
在一些实施例中，可以通过神经网络或者其它机器学习的方法对图像进行特征提取。在一个例子中，神经网络可以采用卷积神经网络。本公开实施例对对图像进行特征提取的方法不作限定。例如：可以通过神将网络对图像进行特征提取，获取图像中人脸的106个特征点的二维坐标。
304，基于眼球中心区域特征点的二维坐标，获取图像中的人脸对应的三维人脸模型中两只眼睛的眼球中心区域特征点在预设三维坐标系中的三维坐标。
在一些实施例中，可以根据图像中的人脸生成对应的三维人脸模型，然后基于眼球中心区域特征点的二维坐标，获取三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标。在一些实施例中，可以预设图像中人脸的关键点，根据图像中人脸的关键点与先验三维人脸模型的关键点之间的对应关系，生成人脸对应的三维人脸模型，例如，多个关键点可以包括脸部外轮廓关键点、眼部关键点、眉毛关键点、嘴唇关键点、鼻子关键点等，本公开实施例对关键点的类型和数量不作限定。
在一些实施例中，在基于眼球中心区域特征点的二维坐标，获取图像中的人脸对应的三维人脸模型中两只眼睛的眼球中心区域特征点在预设三维坐标系中的三维坐标之后，还可以按照预设的格式，将眼睑线特征点的二维坐标、眼球轮廓特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标存处于文件中。在一个例子中，预设的格式可以为眼睑线特征点的二维坐标、眼球轮廓特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标的顺序。
在一些实施例中，在将眼睑线特征点的二维坐标、眼球轮廓特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标存储于文件中时，可以将预设数据集中各图像人脸眼部特征点的坐标分为测试集和训练集，以分别用来对神经网络进行训练和测试。
如图4所示，该方法包括：
402，根据训练集中的训练样本的眼睑线特征点的二维坐标、眼球轮廓特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标，通过神经网络得到对图像中人脸的眼睛注视点位置的判断结果与训练样本对应的图像标注的图像中人脸的眼睛注视点位置之间的损失，并基于该损失通过反向传播更新神经网络的参数。
在本公开实施例中，可以根据预设数据集中的图像获得的训练集中的训练样本，其中，训练集中的各训练样本由眼睑线特征点的二维坐标、眼球轮廓特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三 维坐标组成，预设数据集中的各图像均标注有图像中人脸的眼睛注视点位置，预设数据集可以采用现有的人脸识别数据集，本公开实施例对此不作限定。
在一个例子中，神经网络可以采用由全连接层和ReLU层组成的简单的网络结构。在一些实施例中，可以采用梯度下降法等神经网络的训练方法对神经网络进行训练，本公开实施例对此不作限定。
404，在经过预设次数对神经网络的训练后，根据测试集中的测试样本的眼睑线特征点的二维坐标、眼球轮廓特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标，通过神经网络得到对图像中人脸的眼睛注视点位置的判断结果的准确率，根据该准确率停止对神经网络的训练。
在本公开实施例中，可以根据预设数据集中的图像获得的测试集中的测试样本，其中，测试集中的各测试样本由眼睑线特征点的二维坐标、眼球轮廓特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标组成，预设数据集中的各图像均标注有图像中人脸的眼睛注视点位置，测试集和训练集可以根据同一个预设数据集获得，也可以根据不同的预设数据集获得，本公开实施例对此不作限定，预设数据集可以采用现有的人脸识别数据集，本公开实施例对此不作限定。
在一些实施例中，由于神经网络的任务比较简单，很容易出现过拟合的现象，当通过训练集获得的损失不变，而通过测试集获得的准确率开始上升时，可以提前终止网络的训练。
在上述各实施例中，在完成对神经网络的训练后，可以根据测试集确定对图像中人脸的眼睛注视点位置判断的预设阈值。
可以通过将测试样本输入神经网络，获取神经网络对各测试样本对应的图像的注视分数，然后将各注视分数与对应的图像的标注做比较，将注视分数大于当前阈值确定为注视预设区域，将注视分数小于或等于当前阈值确定为非注视预设区域，并以True_positive表示分数大于当前阈值且标记为正(注视)，以False_negative表示分数小于或等于当前阈值且标记为正(注视)，以False_positive表示分数大于当前阈值且标记为负(非注视)，以True_negative表示分数小于或等于当前阈值且标记为负(非注视)对各测试样本得到的结果进行记录，并根据记录的结果得到真正例率(True positive rate，Tpr)和假正例率(False positive rate，Fpr)，其中Tpr＝True_positive/(True_positive+False_negative)，表示图像中人脸的眼睛注视预设区域时判断正确的例率，Fpr＝False_positive/(True_negative+False_positive)，表示图像中人脸的眼睛非注视预设区域时判断错误的例率，取最大的Mx＝Tpr-Fpr时候的阈值作为预设阈值，并存储总准确率。
在一些实施例中，所述真正例率可以称之为第一例率；所述假正例率 可以称之为第二例率。所述“真正”和“假正”为区分两个利率的名称。
图5为本公开一些实施例的注视点判断装置的结构示意图，该装置设置于服务器或者终端设备执行，例如：终端设备包括手机、计算机、车载设备等，如图5所示，该装置包括：获取单元510和判断单元520。其中，
获取单元510，配置为获取图像中人脸至少一只眼睛的眼部特征点的二维坐标。
在本公开实施例中，用于进行注视点判断的图像可以是从图像采集设备获取的，也可以是从存储设备获取，例如：图像采集设备可以包括：摄像机、照相机、扫描仪等，存储设备可以包括：硬盘、光盘、软盘等，本公开实施例对进行注视点判断的图像的获取方式不作限定。其中，眼部特征点包括眼球中心区域特征点，在一些实施例中，眼部特征点还包括：眼睑线特征点和眼球轮廓特征点，本公开实施例不以此为限。
在一个例子中，由于通常人的双眼是同时注视同一个方向，因此获取单元510可以通过获取图像中人脸一只眼睛的眼部特征点的二维坐标，来对图像中人脸双眼的注视点进行判断。在另一个例子中，获取单元510可以通过获取图像中人脸两只眼睛的眼部特征点的二维坐标，来对图像中人脸双眼的注视点进行判断。
在一些实施例中，获取单元510可以通过对图像进行特征提取，获取图像中人脸特征点的二维坐标，然后基于图像中人脸特征点的二维坐标，获取图像中人脸至少一只眼睛的眼部特征点的二维坐标。例如：获取单元510可以对图像进行特征提取，获取图像中人脸的106个特征点的二维坐标，然后基于这106个特征点的二维坐标，获取图像中人脸至少一只眼睛的眼部特征点的二维坐标。
在一些实施例中，如图6所示，获取单元510可以包括：截取子单元511和提取子单元512。其中，截取子单元511可以根据图像中人脸眼睛区域的特征点的二维坐标，从图像中截取包含对应的眼睛区域的矩形图像，提取子单元512可以对矩形图像进行特征提取，获取图像中人脸至少一只眼睛的眼部特征点的二维坐标。
在一个例子中，截取子单元511可以根据图像中人脸一只眼睛区域的特征点的二维坐标，从图像中截取包含对应的眼睛区域的一张矩形图像，提取子单元512可以对矩形图像进行特征提取，获取图像中人脸一只眼睛的眼部特征点的二维坐标。
在另一个例子中，获取单元510还包括：镜像子单元513，截取子单元511可以根据图像中人脸一只眼睛区域的特征点的二维坐标，从图像中截取包含对应的眼睛区域的一张矩形图像，镜像子单元513可以对矩形图像进行镜像处理；提取子单元512可以对矩形图像和镜像处理后的矩形图像进行特征提取，获取图像中人脸两只眼睛的眼部特征点的二维坐标。
在一些实施例中，可以通过神经网络或者其它机器学习的方法对图像 进行特征提取。在一个例子中，神经网络可以采用卷积神经网络。本公开实施例对对图像进行特征提取的方法不作限定。
获取单元510，还用于基于眼球中心区域特征点的二维坐标，获取图像中的人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标。
在一些实施例中，获取单元510可以根据图像中的人脸生成对应的三维人脸模型，然后基于眼球中心区域特征点的二维坐标，获取三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标。在一些实施例中，可以预设图像中人脸的关键点，获取单元510根据图像中人脸的关键点与先验三维人脸模型的关键点之间的对应关系，生成人脸对应的三维人脸模型，例如，多个关键点可以包括脸部外轮廓关键点、眼部关键点、眉毛关键点、嘴唇关键点、鼻子关键点等，本公开实施例对关键点的类型和数量不作限定。
在一些实施例中，预设三维坐标系可以根据预设的原则确定，例如：三维坐标系的坐标原点为手机前置摄像头的中心点，三维坐标系的X轴正方向为从坐标原点沿着手机前置摄像头水平向左的方向，三维坐标系的Y轴正方向为从坐标原点沿着手机前置摄像头竖直向上的方向，三维坐标系的Z轴正方向为与手机前置摄像头垂直从坐标原点指向手机前置摄像头外侧的方向。
判断单元520，配置为根据眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标，得到对图像中人脸的眼睛注视点位置的判断结果。
在一些实施例中，判断单元520可以根据眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标，得到图像中人脸眼睛的注视分数，然后将注视分数与预设阈值进行比较，得到对图像中人脸的眼睛注视点位置的判断结果。在一些实施例中，判断单元520根据眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标，可以通过神经网络或者其它机器学习的方法，得到图像中人脸眼睛的注视分数。在一个例子中，神经网络可以采用由全连接层和ReLU层组成的简单的网络结构。本公开实施例对获得图像中人脸眼睛的注视分数的方法不作限定。
在一些实施例中，该装置还包括调整单元，在获取单元510基于眼球中心区域特征点的二维坐标，获取图像中的人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标之后，调整单元可以按照预设的格式，对眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标的格式进行调整，从而判断单元520可以根据格式调整后的眼部特征点中除眼球 中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标，通过神经网络得到图像中人脸眼睛的注视分数。在一个例子中，格式调整为数据顺序的调整，即对眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标的顺序进行调整，使其与对神经网络进行训练时的数据顺序保持一致。
在一些实施例中，判断结果可以包括：图像中人脸的眼睛注视点在预设区域内，和图像中人脸的眼睛注视点在预设区域外。判断单元520通过将注视分数与预设阈值进行比较，根据注视分数大于预设阈值，可以得到图像中人脸的眼睛注视点在预设区域内的判断结果；而根据注视分数小于或等于预设阈值，可以得到图像中人脸的眼睛注视点在预设区域外的判断结果。在一个例子中，预设区域可以包括：屏幕区域的一部分或者全部，即本公开实施例可以用来判断图像中人脸的眼睛注视点是否在屏幕区域内，或者用来判断图像中人脸的眼睛注视点是否在屏幕的特定区域内。
在一些实施例中，预设阈值可以包括：真正例率与假正例率的差值，其中，真正例率可以包括：图像中人脸的眼睛注视点在预设区域内时判断正确的例率，假正例率可以包括：图像中人脸的眼睛注视点在预设区域外时判断错误的例率。
在一些实施例中，该装置还包括：处理单元。处理单元在将注视分数与预设阈值进行比较，得到对图像中人脸的眼睛注视点位置的判断结果之后，还可以根据判断结果，对图像进行相应的处理。在一个例子中，处理单元可以响应于图像中人脸的眼睛注视点在预设区域内，按照第一预设展示方式展示图像，可以响应于图像中人脸的眼睛注视点在预设区域外，按照第二预设展示方式展示图像。在一个例子中，第一预设展示方式和第二预设展示方式分别为为展示图像配置不同颜色的边框，例如：如图2A和2B所示，处理单元响应于图像中人脸的眼睛注视点在预设区域内，为展示的图像配置红色边框，响应于图像中人脸的眼睛注视点在预设区域外，为展示的图像配置蓝色边框。
在一些实施例中，本公开实施例的装置可以用于对图像中人脸的表情进行识别，也可以通过对图像中人脸眼睛注视点位置的判断，确定图像中的人对当前的情况是否知情，以用于终端支付、终端锁定、终端解锁，保证支付、锁定、解锁的安全性。
基于本公开上述实施例提供的注视点判断装置，通过获取图像中人脸至少一只眼睛的眼部特征点的二维坐标，其中，眼部特征点包括眼球中心区域特征点，基于眼球中心区域特征点的二维坐标，获取图像中的人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标，根据眼部特征点中除眼球中心区域特征点外的特征点的二维坐标和眼球中心区域特征点在预设三维坐标系中的三维坐标，得到对图像中人 脸的眼睛注视点位置的判断结果，利用对图像中人脸眼睛周围更精细信息的学习，可以实现对眼睛状态更准确的判断，从而可以获得更精确的眼睛状态信息。
本公开实施例还提供了一种电子设备，例如可以是移动终端、个人计算机(PC)、平板电脑、服务器等。下面参考图7，其示出了适于用来实现本申请实施例的终端设备或服务器的电子设备700的结构示意图：如图7所示，电子设备700包括一个或多个处理器、通信部等，所述一个或多个处理器例如：一个或多个中央处理单元(CPU)701，和/或一个或多个加速单元713等，加速单元713可包括但不限于GPU、FPGA、其他类型的专用处理器等，处理器可以根据存储在只读存储器(ROM)702中的可执行指令或者从存储部分708加载到随机访问存储器(RAM)703中的可执行指令而执行各种适当的动作和处理。通信部712可包括但不限于网卡，所述网卡可包括但不限于IB(Infiniband)网卡，处理器可与只读存储器702和/或随机访问存储器703中通信以执行可执行指令，通过总线704与通信部712相连、并经通信部712与其他目标设备通信，从而完成本申请实施例提供的任一项方法对应的操作，例如，获取图像中人脸至少一只眼睛的眼部特征点的二维坐标，其中，所述眼部特征点包括眼球中心区域特征点；基于所述眼球中心区域特征点的二维坐标，获取所述图像中的所述人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标；根据所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，得到对所述图像中所述人脸的眼睛注视点位置的判断结果。
此外，在RAM 703中，还可存储有装置操作所需的各种程序和数据。CPU701、ROM702以及RAM703通过总线704彼此相连。在有RAM703的情况下，ROM702为可选模块。RAM703存储可执行指令，或在运行时向ROM702中写入可执行指令，可执行指令使中央处理单元701执行上述通信方法对应的操作。输入/输出(I/O)接口705也连接至总线704。通信部712可以集成设置，也可以设置为具有多个子模块(例如多个IB网卡)，并在总线链接上。
以下部件连接至I/O接口705：包括键盘、鼠标等的输入部分706；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分707；包括硬盘等的存储部分708；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分709。通信部分709经由诸如因特网的网络执行通信处理。驱动器710也根据需要连接至I/O接口705。可拆卸介质711，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器710上，以便于从其上读出的计算机程序根据需要被安装入存储部分708。
需要说明的，如图7所示的架构仅为一种可选实现方式，在具体实践过程中，可根据实际需要对上述图7的部件数量和类型进行选择、删减、 增加或替换；在不同功能部件设置上，也可采用分离设置或集成设置等实现方式，例如加速单元713和CPU701可分离设置或者可将加速单元713集成在CPU701上，通信部712可分离设置，也可集成设置在CPU701或加速单元713上，等等。这些可替换的实施方式均落入本公开公开的保护范围。
在一些实施例中，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括有形地包含在机器可读介质上的计算机程序，计算机程序包含用于执行流程图所示的方法的程序代码，程序代码可包括对应执行本申请实施例提供的方法步骤对应的指令，例如，获取图像中人脸至少一只眼睛的眼部特征点的二维坐标，其中，所述眼部特征点包括眼球中心区域特征点；基于所述眼球中心区域特征点的二维坐标，获取所述图像中的所述人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标；根据所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，得到对所述图像中所述人脸的眼睛注视点位置的判断结果。在这样的实施例中，该计算机程序可以通过通信部分709从网络上被下载和安装，和/或从可拆卸介质711被安装。在该计算机程序被中央处理单元(CPU)701执行时，执行本申请的方法中限定的上述功能。
在一个或多个可选实施方式中，本公开实施例还提供了一种计算机程序程序产品，用于存储计算机可读指令，该指令被执行时使得计算机执行上述任一可能的实现方式中的注视点判断方法。
该计算机程序产品可以具体通过硬件、软件或其结合的方式实现。在一个可选例子中，该计算机程序产品具体体现为计算机存储介质，在另一个可选例子中，该计算机程序产品具体体现为软件产品，例如软件开发包(Software Development Kit，SDK)等等。
在一个或多个可选实施方式中，本公开实施例还提供了一种注视点判断方法及其对应的装置、电子设备、计算机存储介质、计算机程序以及计算机程序产品，其中，该方法包括：第一装置向第二装置发送注视点判断指示，该指示使得第二装置执行上述任一可能的实施例中的注视点判断方法；第一装置接收第二装置发送的注视点判断的结果。
在一些实施例中，该注视点判断指示可以具体为调用指令，第一装置可以通过调用的方式指示第二装置执行注视点判断，相应地，响应于接收到调用指令，第二装置可以执行上述注视点判断方法中的任意实施例中的步骤和/或流程。
应理解，本公开实施例中的“第一”、“第二”等术语仅仅是为了区分，而不应理解成对本公开实施例的限定。
还应理解，在本公开中，“多个”可以指两个或两个以上，“至少一个” 可以指一个、两个或两个以上。
还应理解，对于本公开中提及的任一部件、数据或结构，在没有明确限定或者在前后文给出相反启示的情况下，一般可以理解为一个或多个。
还应理解，本公开对各个实施例的描述着重强调各个实施例之间的不同之处，其相同或相似之处可以相互参考，为了简洁，不再一一赘述。
可能以许多方式来实现本公开的方法和装置。例如，可通过软件、硬件、固件或者软件、硬件、固件的任何组合来实现本公开的方法和装置。用于所述方法的步骤的上述顺序仅是为了进行说明，本公开的方法的步骤不限于以上具体描述的顺序，除非以其它方式特别说明。此外，在一些实施例中，还可将本公开实施为记录在记录介质中的程序，这些程序包括用于实现根据本公开的方法的机器可读指令。因而，本公开还覆盖存储用于执行根据本公开的方法的程序的记录介质。
本公开的描述是为了示例和描述起见而给出的，而并不是无遗漏的或者将本公开限于所公开的形式。很多修改和变化对于本领域的普通技术人员而言是显然的。选择和描述实施例是为了更好说明本公开的原理和实际应用，并且使本领域的普通技术人员能够理解本公开从而设计适于特定用途的带有各种修改的各种实施例。

Claims (35)

  1. 一种注视点判断方法，包括：
    获取图像中人脸至少一只眼睛的眼部特征点的二维坐标，其中，所述眼部特征点包括眼球中心区域特征点；
    基于所述眼球中心区域特征点的二维坐标，获取所述图像中的所述人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标；
    根据所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，得到对所述图像中所述人脸的眼睛注视点位置的判断结果。
  2. 根据权利要求1所述的方法，其中，所述眼部特征点还包括：眼睑线特征点和眼球轮廓特征点。
  3. 根据权利要求1或2所述的方法，其中，所述获取图像中人脸至少一只眼睛的眼部征点的二维坐标，包括：
    对所述图像进行特征提取，获取所述图像中所述人脸特征点的二维坐标；
    基于所述图像中所述人脸特征点的二维坐标，获取所述图像中所述人脸至少一只眼睛的眼部特征点的二维坐标。
  4. 根据权利要求3所述的方法，其中，所述基于所述图像中所述人脸特征点的二维坐标，获取所述图像中所述人脸至少一只眼睛的眼部特征点的二维坐标，包括：
    根据所述图像中所述人脸眼睛区域的特征点的二维坐标，从所述图像中截取包含对应的眼睛区域的矩形图像；
    对所述矩形图像进行特征提取，获取所述图像中所述人脸至少一只眼睛的眼部特征点的二维坐标。
  5. 根据权利要求4所述的方法，其中，所述根据所述图像中所述人脸眼睛区域的特征点的二维坐标，从所述图像中截取包含对应的眼睛区域的矩形图像，包括：
    根据所述图像中所述人脸一只眼睛区域的特征点的二维坐标，从所述图像中截取包含对应的眼睛区域的一张矩形图像；
    所述对所述矩形图像进行特征提取，获取所述图像中所述人脸至少一只眼睛的眼部特征点的二维坐标，包括：
    对所述矩形图像进行特征提取，获取所述图像中所述人脸一只眼睛的眼部特征点的二维坐标；或者，
    对所述矩形图像进行镜像处理；
    对所述矩形图像和所述镜像处理后的矩形图像进行特征提取，获取所述图像中所述人脸两只眼睛的眼部特征点的二维坐标。
  6. 根据权利要求1至5中任意一项所述的方法，其中，所述基于所述眼球中心区域特征点的二维坐标，获取所述图像中的所述人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标，包括：
    根据所述图像中的所述人脸生成对应的三维人脸模型；
    基于所述眼球中心区域特征点的二维坐标，获取所述三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标。
  7. 根据权利要求6所述的方法，其中，所述根据所述图像中的所述人脸生成对应的三维人脸模型，包括：
    根据所述图像中所述人脸的关键点与先验三维人脸模型的关键点之间的对应关系，生成所述人脸对应的三维人脸模型。
  8. 根据权利要求1至7中任意一项所述的方法，其中，所述根据所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，得到对所述图像中所述人脸的眼睛注视点位置的判断结果，包括：
    根据所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维
    睛注视点在预设区域外时判断错误的例率。
  9. 根据权利要求8所述的方法，其中，所述将所述注视分数与预设阈值进行比较，得到对所述图像中所述人脸的眼睛注视点位置的判断结果，包括：
    根据所述注视分数大于所述预设阈值，得到所述图像中所述人脸的眼睛注视点在预设区域内；或者，
    根据所述注视分数小于或等于所述预设阈值，得到所述图像中所述人脸的眼睛注视点在预设区域外。
  10. 根据权利要求9所述的方法，其中，所述预设阈值包括：真正例率与假正例率的差值；其中，所述真正例率包括：图像中人脸的眼睛注视点在预设区域内时判断正确的例率；所述假正例率包括：图像中人脸的眼
    根据所述判断结果，对所述图像进行相应的处理。
  11. 根据权利要求9或10所述的方法，其中，所述预设区域包括：屏幕区域的一部分或者全部。
  12. 根据权利要求9至11中任意一项所述的方法，其中，所述将所述注视分数与预设阈值进行比较，得到对所述图像中所述人脸的眼睛注视点位置的判断结果之后，还包括：坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，得到所述图像中所述人脸眼睛的注视分数；
    将所述注视分数与预设阈值进行比较，得到对所述图像中所述人脸的眼睛注视点位置的判断结果。
  13. 根据权利要求12所述的方法，其中，所述根据所述判断结果，对所述图像进行相应的处理，包括：
    响应于所述图像中所述人脸的眼睛注视点在预设区域内，按照第一预设展示方式展示所述图像；
    响应于所述图像中所述人脸的眼睛注视点在预设区域外，按照第二预设展示方式展示所述图像。
  14. 根据权利要求8至13中任意一项所述的方法，其中，所述根据所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，得到所述图像中所述人脸眼睛的注视分数，包括：
    根据所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，通过神经网络得到所述图像中所述人脸眼睛的注视分数。
  15. 根据权利要求14所述的方法，其中，所述基于所述眼球中心区域特征点的二维坐标，获取所述图像中的所述人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标之后，还包括：
    按照预设的格式，对所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标的格式进行调整；
    所述根据所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，通过神经网络得到所述图像中所述人脸眼睛的注视分数，包括：
    根据所述格式调整后的所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，通过神经网络得到所述图像中所述人脸眼睛的注视分数。
  16. 根据权利要求1至15中任意一项所述的方法，其中，应用于以下任意一项或任意组合：表情识别、终端支付、终端锁定、终端解锁。
  17. 一种注视点判断装置，包括：
    获取单元，配置为获取图像中人脸至少一只眼睛的眼部特征点的二维坐标，其中，所述眼部特征点包括眼球中心区域特征点；以及基于所述眼球中心区域特征点的二维坐标，获取所述图像中的所述人脸对应的三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标；
    判断单元，配置为根据所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，得到对所述图像中所述人脸的眼睛注视点位置的判断结果。
  18. 根据权利要求17所述的装置，其中，所述眼部特征点还包括：眼睑线特征点和眼球轮廓特征点。
  19. 根据权利要求18所述的装置，其中，所述获取单元，配置为对所 述图像进行特征提取，获取所述图像中所述人脸特征点的二维坐标；基于所述图像中所述人脸特征点的二维坐标，获取所述图像中所述人脸至少一只眼睛的眼部特征点的二维坐标。
  20. 根据权利要求19所述的装置，其中，所述获取单元包括：
    截取子单元，配置为根据所述图像中所述人脸眼睛区域的特征点的二维坐标，从所述图像中截取包含对应的眼睛区域的矩形图像；
    提取子单元，配置为对所述矩形图像进行特征提取，获取所述图像中所述人脸至少一只眼睛的眼部特征点的二维坐标。
  21. 根据权利要求20所述的装置，其中，所述截取子单元，配置为根据所述图像中所述人脸一只眼睛区域的特征点的二维坐标，从所述图像中截取包含对应的眼睛区域的一张矩形图像；
    所述获取单元还包括：镜像子单元，配置为对所述矩形图像进行镜像处理；
    所述提取子单元，配置为对所述矩形图像进行特征提取，获取所述图像中所述人脸一只眼睛的眼部特征点的二维坐标；或者，用于对所述矩形图像和所述镜像处理后的矩形图像进行特征提取，获取所述图像中所述人脸两只眼睛的眼部特征点的二维坐标。
  22. 根据权利要求17至21中任意一项所述的装置，其中，所述获取单元，配置为根据所述图像中的所述人脸生成对应的三维人脸模型；基于所述眼球中心区域特征点的二维坐标，获取所述三维人脸模型中对应的眼球中心区域特征点在预设三维坐标系中的三维坐标。
  23. 根据权利要求22所述的装置，其中，所述获取单元，配置为根据所述图像中所述人脸的关键点与先验三维人脸模型的关键点之间的对应关系，生成所述人脸对应的三维人脸模型。
  24. 根据权利要求17至23中任意一项所述的装置，其中，所述判断单元，配置为根据所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，得到所述图像中所述人脸眼睛的注视分数；将所述注视分数与预设阈值进行比较，得到对所述图像中所述人脸的眼睛注视点位置的判断结果。
  25. 根据权利要求24所述的装置，其中，所述判断单元，配置为根据所述注视分数大于所述预设阈值，得到所述图像中所述人脸的眼睛注视点在预设区域内；或者，根据所述注视分数小于或等于所述预设阈值，得到所述图像中所述人脸的眼睛注视点在预设区域外。
  26. 根据权利要求25所述的装置，其中，所述预设阈值包括：真正例率与假正例率的差值；其中，所述真正例率包括：图像中人脸的眼睛注视点在预设区域内时判断正确的例率；所述假正例率包括：图像中人脸的眼睛注视点在预设区域外时判断错误的例率。
  27. 根据权利要求25或26所述的装置，其中，所述预设区域包括： 屏幕区域的一部分或者全部。
  28. 根据权利要求25至27中任意一项所述的装置，其中，还包括：
    处理单元，配置为根据所述判断结果，对所述图像进行相应的处理。
  29. 根据权利要求28所述的装置，其中，所述处理单元，配置为响应于所述图像中所述人脸的眼睛注视点在预设区域内，按照第一预设展示方式展示所述图像；响应于所述图像中所述人脸的眼睛注视点在预设区域外，按照第二预设展示方式展示所述图像。
  30. 根据权利要求24至29中任意一项所述的装置，其中，所述判断单元，配置为根据所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，通过神经网络得到所述图像中所述人脸眼睛的注视分数。
  31. 根据权利要求30所述的装置，其中，还包括：
    调整单元，配置为按照预设的格式，对所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标的格式进行调整；
    所述判断单元，配置为根据所述格式调整后的所述眼部特征点中除所述眼球中心区域特征点外的特征点的二维坐标和所述眼球中心区域特征点在预设三维坐标系中的三维坐标，通过神经网络得到所述图像中所述人脸眼睛的注视分数。
  32. 根据权利要求17至31中任意一项所述的装置，其中，应用于以下任意一项或任意组合：表情识别、终端支付、终端锁定、终端解锁。
  33. 一种电子设备，包括权利要求17至32中任意一项所述的装置。
  34. 一种电子设备，包括：
    存储器，配置为存储可执行指令；以及
    处理器，配置为与所述存储器通信以执行所述可执行指令从而完成权利要求1至16中任意一项所述的方法。
  35. 一种计算机存储介质，用于存储计算机可读取的指令，其中，所述指令被执行时实现权利要求1至16中任意一项所述的方法。
PCT/CN2018/117287 2018-09-28 2018-11-23 注视点判断方法和装置、电子设备和计算机存储介质 WO2020062523A1 (zh)

Priority Applications (5)

Application Number Priority Date Filing Date Title
MYPI2019007790A MY201730A (en) 2018-09-28 2018-11-23 Gaze point determination method and apparatus, electronic device, and computer storage medium
SG11202000065TA SG11202000065TA (en) 2018-09-28 2018-11-23 Gaze point determination method and apparatus, electronic device, and computer storage medium
KR1020197038765A KR102347336B1 (ko) 2018-09-28 2018-11-23 응시점 판단 방법 및 장치, 전자 기기 및 컴퓨터 저장 매체
JP2019570498A JP6900516B2 (ja) 2018-09-28 2018-11-23 注視点判定方法および装置、電子機器ならびにコンピュータ記憶媒体
US16/729,424 US11295474B2 (en) 2018-09-28 2019-12-29 Gaze point determination method and apparatus, electronic device, and computer storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811141352.3A CN109389069B (zh) 2018-09-28 2018-09-28 注视点判断方法和装置、电子设备和计算机存储介质
CN201811141352.3 2018-09-28

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/729,424 Continuation US11295474B2 (en) 2018-09-28 2019-12-29 Gaze point determination method and apparatus, electronic device, and computer storage medium

Publications (1)

Publication Number Publication Date
WO2020062523A1 true WO2020062523A1 (zh) 2020-04-02

Family

ID=65418277

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/117287 WO2020062523A1 (zh) 2018-09-28 2018-11-23 注视点判断方法和装置、电子设备和计算机存储介质

Country Status (8)

Country Link
US (1) US11295474B2 (zh)
JP (1) JP6900516B2 (zh)
KR (1) KR102347336B1 (zh)
CN (1) CN109389069B (zh)
MY (1) MY201730A (zh)
SG (1) SG11202000065TA (zh)
TW (1) TWI714225B (zh)
WO (1) WO2020062523A1 (zh)

Families Citing this family (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018033137A1 (zh) * 2016-08-19 2018-02-22 北京市商汤科技开发有限公司 在视频图像中展示业务对象的方法、装置和电子设备
US11113842B2 (en) * 2018-12-24 2021-09-07 Samsung Electronics Co., Ltd. Method and apparatus with gaze estimation
CN111435432B (zh) * 2019-01-15 2023-05-26 北京市商汤科技开发有限公司 网络优化方法及装置、图像处理方法及装置、存储介质
CN110335306B (zh) * 2019-04-23 2022-04-15 华中科技大学 注视点预测方法及装置
CN110051319A (zh) * 2019-04-23 2019-07-26 七鑫易维(深圳)科技有限公司 眼球追踪传感器的调节方法、装置、设备及存储介质
CN110428390B (zh) * 2019-07-18 2022-08-26 北京达佳互联信息技术有限公司 一种素材展示方法、装置、电子设备和存储介质
CN110531853B (zh) * 2019-08-23 2021-01-19 西安交通大学 一种基于人眼注视点检测的电子书阅读器控制方法及系统
CN111126137B (zh) * 2019-11-18 2021-02-19 珠海格力电器股份有限公司 一种交互控制方法、装置、终端及计算机可读介质
CN111027492B (zh) * 2019-12-12 2024-01-23 广东智媒云图科技股份有限公司 一种肢体特征点连线的动物作画方法及装置
CN113208591B (zh) * 2020-01-21 2023-01-06 魔门塔(苏州)科技有限公司 一种眼睛开闭距离的确定方法及装置
US11921917B2 (en) 2020-04-07 2024-03-05 Eyetech Digital Systems, Inc. Compact eye-tracking camera systems and methods
US10996753B1 (en) 2020-04-07 2021-05-04 Eyetech Digital Systems, Inc. Multi-mode eye-tracking with independently operable illuminators
CN113723144A (zh) * 2020-05-26 2021-11-30 华为技术有限公司 一种人脸注视解锁方法及电子设备
CN113815623B (zh) * 2020-06-11 2023-08-08 广州汽车集团股份有限公司 一种视觉追踪人眼注视点的方法、车辆预警方法及装置
CN111985384B (zh) * 2020-08-14 2024-09-24 深圳地平线机器人科技有限公司 获取脸部关键点的3d坐标及3d脸部模型的方法和装置
CN114078278A (zh) * 2020-08-20 2022-02-22 Oppo广东移动通信有限公司 注视点定位方法与装置、电子设备和存储介质
CN112949551B (zh) * 2021-03-19 2024-08-27 科大讯飞股份有限公司 一种眼部关键信息确定方法、装置、设备及存储介质
CN113596345B (zh) * 2021-08-09 2023-01-17 荣耀终端有限公司 参数的调整方法、显示的控制方法、电子设备及介质
CN113885708A (zh) * 2021-10-22 2022-01-04 Oppo广东移动通信有限公司 电子设备的屏幕控制方法、装置、电子设备以及存储介质
KR102567390B1 (ko) * 2021-11-03 2023-08-16 주식회사 유니온커뮤니티 동공 움직임을 이용한 위조 홍채 판별장치 및 그 방법
TWI819439B (zh) * 2021-12-21 2023-10-21 財團法人工業技術研究院 視線追蹤方法及應用其之視線追蹤裝置
US12002290B2 (en) 2022-02-25 2024-06-04 Eyetech Digital Systems, Inc. Systems and methods for hybrid edge/cloud processing of eye-tracking image data
EP4322106B8 (en) * 2022-06-30 2024-10-23 Contemporary Amperex Technology (Hong Kong) Limited Defect detection method and apparatus
CN118351602A (zh) * 2024-06-14 2024-07-16 杭州海康威视数字技术股份有限公司 一种虹膜识别设备及方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100080418A1 (en) * 2008-09-29 2010-04-01 Atsushi Ito Portable suspicious individual detection apparatus, suspicious individual detection method, and computer-readable medium
CN103870796A (zh) * 2012-12-13 2014-06-18 汉王科技股份有限公司 一种人眼视线估计的方法与装置
CN104978548A (zh) * 2014-04-02 2015-10-14 汉王科技股份有限公司 一种基于三维主动形状模型的视线估计方法与装置
CN105740846A (zh) * 2016-03-02 2016-07-06 河海大学常州校区 一种基于深度相机的水平视角估计及校准方法
CN106598221A (zh) * 2016-11-17 2017-04-26 电子科技大学 基于眼部关键点检测的3d视线方向估计方法

Family Cites Families (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4675492B2 (ja) * 2001-03-22 2011-04-20 本田技研工業株式会社 顔画像を使用した個人認証装置
US7306337B2 (en) * 2003-03-06 2007-12-11 Rensselaer Polytechnic Institute Calibration-free gaze tracking under natural head movement
JP4603975B2 (ja) * 2005-12-28 2010-12-22 株式会社春光社 コンテンツ注目評価装置及び評価方法
JP2007265367A (ja) * 2006-03-30 2007-10-11 Fujifilm Corp 視線検出方法および装置ならびにプログラム
JP4829141B2 (ja) * 2007-02-09 2011-12-07 株式会社東芝 視線検出装置及びその方法
TWI476729B (zh) * 2010-11-26 2015-03-11 Inst Information Industry Dimensional image and three - dimensional model of the combination of the system and its computer program products
WO2012107892A2 (en) * 2011-02-09 2012-08-16 Primesense Ltd. Gaze detection in a 3d mapping environment
US8824779B1 (en) * 2011-12-20 2014-09-02 Christopher Charles Smyth Apparatus and method for determining eye gaze from stereo-optic views
JP2013252301A (ja) * 2012-06-07 2013-12-19 Toyota Central R&D Labs Inc 眼球中心推定装置及びプログラム
KR101351745B1 (ko) * 2012-06-29 2014-01-16 인텔렉추얼디스커버리 주식회사 ３차원 얼굴 생성 장치 및 방법
TWI471808B (zh) * 2012-07-20 2015-02-01 Pixart Imaging Inc 瞳孔偵測裝置
JP6234762B2 (ja) * 2013-10-09 2017-11-22 アイシン精機株式会社 眼部検出装置、方法、およびプログラム
US9727972B2 (en) * 2013-12-30 2017-08-08 Xiaomi Inc. Method and terminal for generating thumbnail of image
JP6256165B2 (ja) * 2014-04-09 2018-01-10 富士通株式会社 視線検出装置、視線検出プログラム及び視線検出方法
KR102212209B1 (ko) * 2014-04-10 2021-02-05 삼성전자주식회사 시선 추적 방법, 장치 및 컴퓨터 판독가능한 기록 매체
US9864430B2 (en) * 2015-01-09 2018-01-09 Microsoft Technology Licensing, Llc Gaze tracking via eye gaze model
CN105989326B (zh) * 2015-01-29 2020-03-03 北京三星通信技术研究有限公司 人眼三维位置信息的确定方法和装置
JP6525635B2 (ja) * 2015-02-25 2019-06-05 キヤノン株式会社 画像処理装置、画像処理方法及びプログラム
EP3345127A4 (en) * 2015-08-31 2019-08-21 SRI International METHOD AND SYSTEM FOR MONITORING DRIVING BEHAVIORS
CN108139813A (zh) * 2015-10-19 2018-06-08 鸥利研究所股份有限公司 视线输入装置、视线输入方法以及视线输入程序
JP2018101212A (ja) * 2016-12-19 2018-06-28 矢崎総業株式会社 車載器および顔正面度算出方法
CN106843500B (zh) * 2017-02-27 2020-07-07 南通大学 基于眼动追踪技术的认知水平康复训练系统
CN107679447A (zh) * 2017-08-17 2018-02-09 平安科技（深圳）有限公司 面部特征点检测方法、装置及存储介质
CN107679448B (zh) * 2017-08-17 2018-09-25 平安科技（深圳）有限公司 眼球动作分析方法、装置及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100080418A1 (en) * 2008-09-29 2010-04-01 Atsushi Ito Portable suspicious individual detection apparatus, suspicious individual detection method, and computer-readable medium
CN103870796A (zh) * 2012-12-13 2014-06-18 汉王科技股份有限公司 一种人眼视线估计的方法与装置
CN104978548A (zh) * 2014-04-02 2015-10-14 汉王科技股份有限公司 一种基于三维主动形状模型的视线估计方法与装置
CN105740846A (zh) * 2016-03-02 2016-07-06 河海大学常州校区 一种基于深度相机的水平视角估计及校准方法
CN106598221A (zh) * 2016-11-17 2017-04-26 电子科技大学 基于眼部关键点检测的3d视线方向估计方法

Also Published As

Publication number Publication date
US20200134868A1 (en) 2020-04-30
JP6900516B2 (ja) 2021-07-07
KR20200037147A (ko) 2020-04-08
TWI714225B (zh) 2020-12-21
MY201730A (en) 2024-03-14
US11295474B2 (en) 2022-04-05
SG11202000065TA (en) 2020-05-28
CN109389069B (zh) 2021-01-05
JP2021502618A (ja) 2021-01-28
CN109389069A (zh) 2019-02-26
TW202013244A (zh) 2020-04-01
KR102347336B1 (ko) 2022-01-05

Similar Documents

Publication Publication Date Title
WO2020062523A1 (zh) 注视点判断方法和装置、电子设备和计算机存储介质
US11775056B2 (en) System and method using machine learning for iris tracking, measurement, and simulation
US11341769B2 (en) Face pose analysis method, electronic device, and storage medium
US11551482B2 (en) Facial recognition-based authentication
CN106897658B (zh) 人脸活体的鉴别方法和装置
US11270142B2 (en) Methods and apparatuses for processing image and identity verification, electronic devices, and storage media
CN108229301B (zh) 眼睑线检测方法、装置和电子设备
WO2019029486A1 (zh) 人脸图像处理方法、装置和电子设备
CN109359548A (zh) 多人脸识别监控方法及装置、电子设备及存储介质
CN109960974A (zh) 人脸关键点检测方法、装置、电子设备及存储介质
CN105740688B (zh) 解锁的方法和装置
EP3338217A1 (en) Feature detection and masking in images based on color distributions
EP3647992A1 (en) Face image processing method and apparatus, storage medium, and electronic device
CN108734078B (zh) 图像处理方法、装置、电子设备、存储介质及程序
CN108388889B (zh) 用于分析人脸图像的方法和装置
CN109241934A (zh) 用于生成信息的方法和装置
CN106778574A (zh) 用于人脸图像的检测方法和装置
WO2023034251A1 (en) Spoof detection based on challenge response analysis
CN109584145A (zh) 卡通化方法和装置、电子设备和计算机存储介质
CN113688737A (zh) 人脸图像处理方法、装置、电子设备、存储介质及程序
CN111199176B (zh) 人脸身份检测方法及装置
US11195301B1 (en) Estimation of head yaw in an image
CN111488778A (zh) 图像处理方法及装置、计算机系统和可读存储介质
CN111368644B (zh) 图像处理方法、装置、电子设备及存储介质
JP2021061048A (ja) 情報処理装置、情報処理方法、およびプログラム

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2019570498

Country of ref document: JP

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18934572

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 08/07/2021)

122 Ep: pct application non-entry in european phase

Ref document number: 18934572

Country of ref document: EP

Kind code of ref document: A1