CN111357006A - Fatigue prompting method and terminal - Google Patents
Fatigue prompting method and terminal Download PDFInfo
- Publication number
- CN111357006A CN111357006A CN201780096860.1A CN201780096860A CN111357006A CN 111357006 A CN111357006 A CN 111357006A CN 201780096860 A CN201780096860 A CN 201780096860A CN 111357006 A CN111357006 A CN 111357006A
- Authority
- CN
- China
- Prior art keywords
- terminal
- user
- fatigue
- camera
- eye
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 230000001815 facial effect Effects 0.000 claims abstract description 17
- 238000004590 computer program Methods 0.000 claims description 16
- 230000015654 memory Effects 0.000 claims description 16
- 238000005286 illumination Methods 0.000 claims description 13
- 238000003860 storage Methods 0.000 claims description 9
- 230000001502 supplementing effect Effects 0.000 claims description 6
- 239000000284 extract Substances 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 208000003464 asthenopia Diseases 0.000 abstract description 26
- 238000001514 detection method Methods 0.000 abstract description 9
- 239000013589 supplement Substances 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 19
- 238000010586 diagram Methods 0.000 description 11
- 238000012545 processing Methods 0.000 description 9
- 238000004422 calculation algorithm Methods 0.000 description 5
- 230000001413 cellular effect Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 230000009191 jumping Effects 0.000 description 4
- 239000008280 blood Substances 0.000 description 3
- 210000004369 blood Anatomy 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000005236 sound signal Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 235000002673 Dioscorea communis Nutrition 0.000 description 1
- 241000544230 Dioscorea communis Species 0.000 description 1
- 208000003556 Dry Eye Syndromes Diseases 0.000 description 1
- 206010013774 Dry eye Diseases 0.000 description 1
- 208000035753 Periorbital contusion Diseases 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000005802 health problem Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 238000010079 rubber tapping Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000010897 surface acoustic wave method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- User Interface Of Digital Computer (AREA)
- Telephone Function (AREA)
Abstract
The application relates to the field of terminal equipment, in particular to a fatigue prompting method. In the fatigue prompting method, a terminal comprises a visible light camera, an infrared camera and an ambient light sensor. When the user uses the terminal, the visible light camera and the infrared camera are started to collect facial images of the user, and the facial images are used for detecting the fatigue state of the eyes of the user. When the light condition is not good, the terminal automatically supplements light or adjusts the image contrast, and the detection accuracy is improved. And when the terminal detects the eye fatigue of the user, the terminal sends out a notice to remind the user of paying attention. Through the scheme provided by the application, the accuracy of detecting the eye fatigue by the terminal can be improved.
Description
The present application relates to a facial feature detection technology, and in particular, to a fatigue prompting method and a terminal.
With the rapid development of smart phones, more and more people have severe dependence on smart phones, and the health of users is affected while the smart phones provide convenience. Among them, eye fatigue and discomfort are a health problem of great concern. When a user frequently watches the mobile phone, eyes are tired, and discomfort such as red blood streak, black eye circles, dry eyes and the like appears. In the prior art, the human face image is collected through the camera, and the fatigue state of the user is judged according to the eye characteristics, so that the accuracy is difficult to ensure.
Disclosure of Invention
The application provides a fatigue prompting method and a terminal, which can improve the accuracy of fatigue judgment.
In a first aspect, an embodiment of the present application provides a method for fatigue indication. The method is applied to the terminal and comprises the following steps: the terminal comprises a visible light camera, an infrared camera and an ambient light sensor, and the ambient light sensor detects the first illumination condition; according to a preset time interval, the visible light camera and the infrared camera automatically acquire facial images of a user at the same time under the first illumination condition; the terminal extracts eye features according to the collected face image; the terminal determines the level of the fatigue state of the user according to the eye features; when the level of the fatigue state of the user exceeds a first threshold value, the terminal presents a first notice, and the first notice is used for prompting the user to take a rest. Therefore, facial images are simultaneously acquired through the visible light camera and the infrared camera, and the distance between the face of the user and the terminal can be determined while the eye features of the user are determined by combining the illumination conditions of the environment where the terminal is located, so that the fatigue judgment accuracy is improved.
In one possible implementation, the method includes: the terminal further comprises an infrared light source, when the first illumination condition does not exceed a second threshold value, the infrared light source emits infrared light, and the infrared light is used for supplementing light to ambient light. When the illumination condition is poor, more details of the face image can be collected through supplementary lighting, and the fatigue judgment accuracy is further improved.
In another possible implementation manner, the method further includes: the terminal further comprises an exposure adjusting module, and when the exposure value of the collected face image is smaller than a third threshold value, the exposure adjusting module adjusts the exposure value. Through adjusting the exposure value of gathering the image, avoid exposing the inaccurate overexposure that leads to or the problem of exposure lack, can improve the degree of accuracy that fatigue judged.
In another possible implementation manner, the method further includes: the terminal presents a first notice to prompt the user to have a rest and automatically starts an eye protection mode.
In another possible implementation manner, the method further includes: the eye protection mode comprises that the terminal adjusts at least one of the following options according to the level of the fatigue state of the user: screen brightness, font size, screen contrast, and display background color.
In another possible implementation manner, the method further includes: and the terminal determines the distance from the face of the user to the terminal according to the acquired face image, and when the distance is smaller than a fourth threshold value, the terminal presents a second notice which is used for prompting the user to get away from the screen.
In a second aspect, the present invention provides a terminal, including: an ambient light sensor for detecting a first lighting condition of an environment in which the terminal is located; the system comprises a visible light camera and an infrared camera, wherein the visible light camera and the infrared camera are used for automatically acquiring facial images of a user at the same time under the first illumination condition according to a preset time interval; the extraction module is used for extracting eye features according to the collected face image; the judging module is used for determining the level of the fatigue state of the user according to the eye features; the notification module is used for sending out a first notification when the level of the fatigue state of the user exceeds a first threshold, and the first notification is used for prompting the user to take a rest.
In one possible implementation, the terminal includes: the infrared light source is used for emitting infrared light when the first illumination condition does not exceed a second threshold value, and the infrared light is used for supplementing light to ambient light.
In another possible implementation manner, the terminal further includes: an adjustment unit for adjusting at least one of the following options: screen brightness, font size, screen contrast, and display background color.
In another possible implementation manner, the terminal further includes: an exposure adjustment module that adjusts the exposure value of the captured face image when the exposure value is less than a third threshold value.
In another possible implementation manner, the terminal further includes: the judging module is also used for determining the distance between the face of the user and the terminal; the notification module is further configured to present a second notification when the distance is smaller than a third threshold, where the second notification is used to prompt the user to leave the screen.
In a third aspect, an embodiment of the present invention provides a terminal, including: one or more processors; one or more memories in which one or more computer programs are stored, the one or more computer programs comprising instructions which, when executed by the one or more processors, cause the terminal to perform any of the methods of the first aspect described above.
In a fourth aspect, an embodiment of the present invention provides a computer program product including instructions, which, when run on an electronic device, causes the electronic device to perform the method according to the first aspect.
In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, which includes instructions that, when executed on an electronic device, cause the electronic device to perform the method according to the first aspect.
In a sixth aspect, an embodiment of the present invention further provides a data processing system, which includes modules for executing the methods provided in the first aspect.
Fig. 1 is a partial block diagram of a terminal according to an embodiment of the present invention;
fig. 2 is a flowchart of a fatigue indication method according to an embodiment of the present invention;
FIG. 3 is a flow chart of a method for identifying fatigue levels according to an embodiment of the present invention;
FIG. 4 is a flowchart of a method for measuring a distance from an eye of a user to a screen according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating a method for detecting eye fatigue of a user during driving according to an embodiment of the present invention;
fig. 6 is a schematic diagram of a terminal device according to an embodiment of the present invention.
Embodiments of the present invention will be described below with reference to the accompanying drawings.
The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the examples of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The character "/" herein generally indicates that the former and latter associated objects are in an "or" relationship.
It should be understood that although the terms first, second, third, etc. may be used to describe various messages, requests, and terminals in embodiments of the present invention, these messages, requests, and terminals should not be limited by these terms. These terms are only used to distinguish messages, requests and terminals from one another. For example, a first terminal may also be referred to as a second terminal, and similarly, a second terminal may also be referred to as a first terminal, without departing from the scope of embodiments of the present invention.
The fatigue prompting method and the terminal provided by the embodiment of the invention are used for reminding a user of paying attention to eye fatigue. The terminal may be, for example: a Mobile phone, a tablet computer, a laptop computer, a digital camera, a Personal Digital Assistant (PDA), a navigation Device, a Mobile Internet Device (MID) or a wearable Device, etc.
Fig. 1 is a block diagram of a partial structure of a terminal according to an embodiment of the present invention. The terminal is described by taking a mobile phone 100 as an example, and referring to fig. 1, the mobile phone 100 includes: radio Frequency (RF) circuit 110, power supply 120, processor 130, memory 140, input unit 150, display unit 160, sensor 170, audio circuit 180, and wireless fidelity (Wi-Fi) module 190. Those skilled in the art will appreciate that the handset configuration shown in fig. 1 is not intended to be limiting and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
The following describes the components of the mobile phone 100 in detail with reference to fig. 1:
the RF circuit 110 may be used for transmitting and receiving information or for receiving and transmitting signals during a call. For example: RF circuitry 110 may send downlink data received from the base station to processor 130 for processing and may send uplink data to the base station. In general, RF circuits include, but are not limited to, an RF chip, an antenna, at least one Amplifier, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, a radio frequency switch, and the like. In addition, the RF circuitry 110 may also communicate wirelessly with networks and other devices. The wireless communication may use any communication standard or protocol, including but not limited to Global System for Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), and the like.
The memory 140 may be used to store software programs and modules, and the processor 130 executes various functional applications and data processing of the mobile phone 100 by operating the software programs and modules stored in the memory 140. The memory 140 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone 100, and the like. Further, the memory 140 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. The memory 140 may also store a knowledge base, a tag base, and an algorithm base.
The input unit 150 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the cellular phone 100. Specifically, the input unit 150 may include a touch panel 151 and other input devices 152. The touch panel 151, also referred to as a touch screen, may collect a touch operation performed by a user on or near the touch panel 151 (e.g., an operation performed by the user on or near the touch panel 151 using any suitable object or accessory such as a finger or a stylus), and drive a corresponding connection device according to a preset program. Alternatively, the touch panel 151 may include two parts, a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 130, and can receive and execute commands sent by the processor 130. In addition, the touch panel 151 may be implemented in various types, such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave. The input unit 150 may include other input devices 152 in addition to the touch panel 151. In particular, other input devices 152 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
The display unit 160 may be used to display information input by or provided to the user and various menus of the cellular phone 100. The display unit 160 may include a display panel 161, and optionally, the display panel 161 may be configured in the form of a Liquid Crystal Display (LCD), an electromechanical laser display (OLED), or the like. Further, the touch panel 151 may cover the display panel 161, and when the touch panel 151 detects a touch operation thereon or nearby, the touch panel transmits the touch operation to the processor 130 to determine the type of the touch event, and then the processor 130 provides a corresponding visual output on the display panel 161 according to the type of the touch event. Although the touch panel 151 and the display panel 161 are shown in fig. 1 as two separate components to implement the input and output functions of the mobile phone 100, in some embodiments, the touch panel 151 and the display panel 161 may be integrated to implement the input and output functions of the mobile phone 100.
The handset 100 may also include at least one sensor 170, such as a light sensor, motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor that adjusts the brightness of the display panel 161 according to the brightness of ambient light, and a proximity sensor that turns off the display panel 161 and/or the backlight when the mobile phone 100 is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), detect the magnitude and direction of gravity when stationary, and can be used for applications of recognizing the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), and the like. The mobile phone 100 may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which are not described herein again.
The audio circuitry 180, speaker 181, microphone 182 may provide an audio interface between a user and the handset 100. The audio circuit 180 may transmit the electrical signal converted from the received audio data to the speaker 181, and the electrical signal is converted into a sound signal by the speaker 181 and output; on the other hand, the microphone 182 converts the collected sound signals into electrical signals, which are received by the audio circuit 180 and converted into audio data, which are then output to the RF circuit 110 for transmission to, for example, another cell phone, or to the memory 140 for further processing.
Wi-Fi belongs to a short-distance wireless transmission technology, and the mobile phone 100 can help a user to receive and send emails, browse webpages, access streaming media and the like through the Wi-Fi module 190, and provides wireless broadband internet access for the user. Although fig. 1 shows the Wi-Fi module 190, it is understood that it does not belong to the essential constitution of the cellular phone 100, and may be omitted entirely as needed within the scope not changing the essence of the invention.
The processor 130 is a control center of the mobile phone 100, connects various parts of the entire mobile phone by using various interfaces and lines, and performs various functions of the mobile phone 100 and processes data by operating or executing software programs and/or modules stored in the memory 140 and calling data stored in the memory 140, thereby implementing various services based on the mobile phone. Optionally, processor 130 may include one or more processing units; preferably, the processor 130 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 130.
In embodiments of the present invention, processor 130 may execute program instructions stored in memory 140 to implement the methods illustrated in the following embodiments.
The handset 100 also includes a power supply 120 (e.g., a battery) for powering the various components, which may be logically coupled to the processor 130 via a power management system to manage charging, discharging, and power consumption via the power management system.
Although not shown, the mobile phone 100 may further include a camera, a bluetooth module, etc., which will not be described herein.
Fig. 2 is a flowchart of a fatigue prompting method according to an embodiment of the present invention, where the fatigue prompting method specifically includes the following steps:
a terminal 201 collects a face image of a user;
202, extracting eye features according to the collected face image of the user;
203, detecting whether the eye fatigue level of the user reaches or exceeds a first threshold value according to the eye features, if the eye fatigue level of the user does not exceed the first threshold value, jumping to a step 204, and if the eye fatigue level of the user exceeds the first threshold value, jumping to a step 205;
204, the terminal continues detecting or stops detecting;
205 the terminal presents a first notification.
In step 201, the terminal simultaneously acquires facial images of a user through a visible light camera and an infrared camera, and the two cameras work simultaneously to synchronously output frames.
Optionally, the visible light camera and the infrared camera are both front-facing cameras arranged on one side of the terminal display screen, wherein the visible light camera is a main camera, and the infrared camera is an auxiliary camera.
Optionally, the terminal further includes an ambient light sensor, where the ambient light sensor may detect an illumination condition of an environment where the terminal is located, and when the ambient light sensor of the terminal detects that the illumination condition is less than or equal to a certain threshold, the terminal performs light supplement during image acquisition through the auxiliary light source.
Optionally, the terminal comprises an infrared light source serving as an auxiliary light source, which may emit infrared light. In some embodiments, the infrared light source may take the form of an infrared soft light and is disposed on a side of the terminal display. The infrared light source is automatically started to carry out active light supplement in a dark environment, infrared images with rich details of human eyes can be obtained, and human eye detection and identification in a dark environment are improved. Meanwhile, infrared light emitted by the infrared soft light is invisible light, and the influence on a user cannot be generated in the active light supplementing process.
Further, in some embodiments, the terminal automatically collects the facial images of the user according to a certain preset time interval, and the time interval is preset before the terminal leaves a factory or can be set by the user. For example, every 1 minute, the visible light camera and the infrared camera automatically capture facial images of the user at the same time. Here, the setting of the time interval to 1 minute is only an illustrative example and does not limit the present application, and the time interval may be set to 30 seconds, 5 minutes, 10 minutes, or other values.
Further, in some embodiments, the terminal further includes an exposure adjustment module that adjusts the exposure value when the exposure value of the captured facial image is less than a preset threshold. Through adjusting the exposure value of gathering the image, avoid exposing the inaccurate overexposure that leads to or the problem of exposure lack, can improve the degree of accuracy that fatigue judged.
In step 202, the terminal extracts eye features in the face image of the user according to the face images collected by the visible light camera and the infrared camera. The method comprises the steps that a visible light camera and an infrared camera respectively collect a frame of image, the two frames of image are read and fused, the eyes in the face image are identified through an algorithm, and eye features are extracted. Specifically, the algorithm may be any algorithm capable of extracting eye features from a facial image, such as the Adaboost algorithm.
The eye features extracted in step 202 may include at least one of the following options: the number of blinks, the degree of eye openness, the distribution of red blood streak, etc.
In step 203, the eye features extracted in step 202 are analyzed to determine the user fatigue level. For example, the fatigue level is determined by analyzing the degree of opening and closing of the eyes, assuming that the degree of opening and closing of the eyes of the user in the normal state is 100%, when the degree of opening and closing of the eyes is 80%, the user is considered to be in a light fatigue state, and when the degree of opening and closing of the eyes is 60%, the user is considered to be in a moderate fatigue state.
Further, in some embodiments, the fatigue level is determined by analyzing the degree of eye openness and the number of blinks, and when the degree of eye openness is 80%, and when the distribution number of red blood filaments reaches 20%, the user is considered to be in a light fatigue state.
Further, in some embodiments, the determination of whether to remind the user of the need to take a break is made by detecting whether the user's fatigue level reaches a first threshold. For example, the user is reminded to rest when the user's fatigue level reaches a first threshold, which may be a light fatigue state or a medium fatigue state.
In step 204, when the eye fatigue level of the user does not reach the first threshold, the terminal may continue to detect the eye fatigue state of the user or stop detecting according to a preset time interval.
In step 205, when the terminal detects that the eye fatigue level of the user reaches the first threshold, the terminal presents a first notification. The first notification may be presented in the form of a pop-up box, a sound, or a vibration, etc. Specifically, when the terminal detects that the eye fatigue level of the user reaches the first threshold value, a popup box appears on the terminal screen, and the popup box comprises text information and prompts the user to feel eye fatigue and take a rest.
Further, in some embodiments, the terminal presents a first notification and automatically turns on an eye-protection mode.
Further, in some embodiments, when the user aims the terminal at the face, the infrared camera and the visible light camera are automatically turned on to collect the face image of the user and identify the fatigue state of the eyes. And when the terminal is moved away from the vicinity of the face of the user, the infrared camera and the visible light camera stop collecting the face image of the user.
Optionally, in some embodiments, the terminal has an option to turn on eye fatigue detection, and the user can select to turn on or off. And when the user selects to close, the terminal does not detect the eye fatigue state of the user any more.
FIG. 3 provides a method for determining fatigue level according to the present invention, comprising the following steps:
301, the terminal detects the eye fatigue state of a user;
302 the terminal determining a percentage of the user's eye fatigue status that exceeds the first threshold;
303, determining the eye fatigue level of the user and sending a first notice;
304 the terminal automatically triggers an eye protection mode according to the fatigue level.
In step 302, the terminal determines the fatigue level of the user's eyes according to the percentage of the fatigue state of the user's eyes exceeding the first threshold. For example, when the number of blinks of the user exceeds a first threshold value of ten percent while the degree of opening of the eyes of the user exceeds the first threshold value of ten percent, the eye fatigue level of the user reaches a first level. The fatigue level can be set by a terminal or can be set by a user in a self-defined way.
The first notification in step 303 is the same as the above embodiment, and is not described herein again.
In step 304, the terminal automatically triggers an eye protection mode of the terminal according to eye fatigue grades of a user, the eye protection modes of the terminal have different grades, and the eye protection modes of different grades can correspond to different fatigue grades.
Further, in an eye protection mode, the terminal adjusts at least one of the following options according to the level of the fatigue state of the user: screen brightness, font size, screen contrast, and display background color.
Further, in the embodiment of the present invention, the eye protection mode is set by the terminal, and the level of the eye protection mode is automatically adjusted according to the detected fatigue level of the eyes of the user.
Fig. 4 provides a method for measuring the distance from the eyes of the user to the screen according to the present invention, which comprises the following steps:
401 calculating image depth information according to the collected image;
402 obtaining a first distance according to the image depth information;
403, judging whether the first distance is smaller than a second threshold value, if not, jumping to step 504, and if so, jumping to step 405;
404, the terminal does not prompt and works normally;
the terminal 405 issues a second warning message.
In step 401, the collected images are visible light images and infrared images of the face of the user collected by the front-facing double cameras, and depth calculation is performed according to the visible light images and the infrared images;
in step 402, obtaining a first distance from the human eyes to the terminal screen according to the depth calculation result in step 401;
in step 403, it is determined whether the first distance from the human eyes to the terminal screen is less than a certain set threshold, for example, when the first distance is less than 30cm, the user is reminded to watch too close.
FIG. 5 provides a method for detecting eye fatigue of a user during driving, comprising the steps of:
501, a terminal enters a driving mode, and front double cameras of the terminal are started;
502 the terminal detects the eye fatigue state of the user and determines the fatigue level;
503 the terminal issues a third notification.
In step 501, the driving mode may be a driving mode carried by the terminal itself, or a driving mode determined by the terminal to enter the driving mode by the user turning on software with a navigation function during driving. After the terminal enters a driving mode, the terminal automatically opens the front double cameras, namely the visible light camera and the infrared camera, and detects the eye fatigue state of the user.
Optionally, in some embodiments, the terminal enters a driving mode, and the front dual cameras are always turned on and are not turned off when the face of the user leaves.
Optionally, in some embodiments, the terminal performs eye fatigue detection when the user views the navigation route.
Optionally, in some embodiments, if the user selects to turn off the eye fatigue detection function, the eye fatigue detection function is not turned on when the terminal enters the driving mode.
Optionally, in some embodiments, when the terminal determines that the user is driving the vehicle, the third notification sent by the terminal is an audible alarm, and the volume of the alarm may be adjusted according to different levels of fatigue.
As shown in fig. 6, a terminal 600 provided by the present invention includes an ambient light sensor 601, a visible light camera and an infrared camera 602, an extraction module 603, a determination module 604, a notification module 605, an infrared unit 606, an adjustment unit 607, and an exposure adjustment module 608.
The ambient light sensor 601 is configured to detect a first lighting condition of an environment in which the terminal is located; the visible light camera and the infrared camera 602 are used for automatically acquiring facial images of a user at the same time under the first lighting condition according to a preset time interval; the extracting module 603 is configured to extract an eye feature according to the acquired facial image; the determining module 604 is configured to determine a level of the fatigue state of the user according to the eye feature; the notification module 605 is configured to send a first notification when the level of the fatigue state of the user exceeds a first threshold, where the first notification is used to prompt the user to take a rest; when the first illumination condition does not exceed a second threshold value, the infrared light source 606 emits infrared light, and the infrared light is used for supplementing ambient light; the adjusting unit 607 is configured to adjust at least one of the following options: screen brightness, font size, screen contrast and display background color; an exposure adjustment module 608, the exposure adjustment module 608 adjusting the exposure value of the captured facial image when the exposure value is less than a third threshold.
Further, in some embodiments, the determining module 604 is further configured to determine a distance between the face of the user and the terminal.
Further, in some embodiments, the notification module 605 is further configured to present a second notification when the distance is less than a third threshold, the second notification being configured to prompt the user to move away from the screen.
Further, in some embodiments, the ambient light sensor 601, the visible light camera, and the infrared camera 602 are on the display side of the terminal.
The embodiment of the present invention further provides a computer storage medium, configured to store computer software instructions for implementing the method for prompting fatigue of a terminal shown in the foregoing embodiment, where the computer storage medium includes program codes designed to execute the foregoing method embodiment.
The embodiment of the invention also provides a computer program product. The computer program product comprises computer software instructions which can be loaded by a processor for implementing the method in the above-described method embodiments.
While the invention has been described in connection with various embodiments, other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a review of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Those skilled in the art will recognize that, in one or more of the examples described above, the functions described in this invention may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (14)
- A fatigue prompting method is applied to a terminal, wherein the terminal comprises a visible light camera, an infrared camera and an ambient light sensor, and the method comprises the following steps:the ambient light sensor detects a first lighting condition;according to a preset time interval, the visible light camera and the infrared camera automatically acquire facial images of a user at the same time under the first illumination condition;the terminal extracts eye features according to the collected face image;the terminal determines the level of the fatigue state of the user according to the eye features;when the level of the fatigue state of the user exceeds a first threshold value, the terminal presents a first notice, and the first notice is used for prompting the user to take a rest.
- The method of claim 1, wherein the terminal further comprises an infrared light source, the method further comprising:when the first illumination condition does not exceed a second threshold value, the infrared light source emits infrared light, and the infrared light is used for supplementing light to ambient light.
- The method of claim 1, wherein the terminal presenting the first notification comprises:the terminal prompts the user to have a rest and automatically starts an eye protection mode.
- The method of claim 3, wherein the eye-shielding mode comprises:the terminal adjusts at least one of the following options according to the level of the fatigue state of the user:screen brightness, font size, screen contrast, and display background color.
- The method according to any one of claims 1-4, further comprising:and the terminal determines the distance from the face of the user to the terminal according to the acquired face image, and when the distance is smaller than a third threshold value, the terminal presents a second notice which is used for prompting the user to get away from the screen.
- The method according to any one of claims 1 to 5,the terminal further comprises an exposure adjusting module, and when the exposure value of the collected face image is smaller than a fourth threshold value, the exposure adjusting module adjusts the exposure value.
- A terminal, characterized in that the terminal comprises:an ambient light sensor for detecting a first lighting condition of an environment in which the terminal is located;the system comprises a visible light camera and an infrared camera, wherein the visible light camera and the infrared camera are used for automatically acquiring facial images of a user at the same time under the first illumination condition according to a preset time interval;the extraction module is used for extracting eye features according to the collected face image;the judging module is used for determining the level of the fatigue state of the user according to the eye features;the notification module is used for sending out a first notification when the level of the fatigue state of the user exceeds a first threshold, and the first notification is used for prompting the user to take a rest.
- The terminal of claim 7, further comprising an infrared light source, wherein when the first lighting condition does not exceed a second threshold, the infrared light source emits infrared light, and wherein the infrared light is used for supplementing ambient light.
- The terminal according to claim 7 or 8, characterized in that the terminal further comprises an adjusting unit for adjusting at least one of the following options:screen brightness, font size, screen contrast, and display background color.
- The terminal according to any of claims 7-9, further comprising an exposure adjustment module that adjusts the exposure value of the captured facial image when the exposure value is less than a fourth threshold.
- A terminal according to any of claims 7-10, wherein the determining means is further adapted to determine the distance of the user's face from the terminal;the notification module is further configured to present a second notification when the distance is smaller than a third threshold, where the second notification is used to prompt the user to leave the screen.
- A terminal, comprising:a touch screen, wherein the touch screen comprises a touch sensitive surface and a display;one or more processors;a memory;a plurality of application programs;and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions which, when executed by the terminal, cause the terminal to perform the method of any of claims 1-6.
- A computer program product comprising instructions for causing an electronic device to perform the method according to any of claims 1-6 when the computer program product is run on the electronic device.
- A computer-readable storage medium comprising instructions that, when executed on an electronic device, cause the electronic device to perform the method of any of claims 1-6.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2017/111150 WO2019095155A1 (en) | 2017-11-15 | 2017-11-15 | Fatigue notification method and terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111357006A true CN111357006A (en) | 2020-06-30 |
Family
ID=66538354
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201780096860.1A Pending CN111357006A (en) | 2017-11-15 | 2017-11-15 | Fatigue prompting method and terminal |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111357006A (en) |
| WO (1) | WO2019095155A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112506338A (en) * | 2020-11-30 | 2021-03-16 | 崔刚 | Equipment control method, device and equipment based on full vision |
| CN113297734A (en) * | 2021-05-21 | 2021-08-24 | 南京邮电大学 | Visual cone-based simulation evaluation method for assembly process in limited space |
| CN113709308A (en) * | 2021-08-31 | 2021-11-26 | 维沃移动通信有限公司 | Usage monitoring method and device for electronic equipment |
| CN114120395A (en) * | 2021-11-02 | 2022-03-01 | 深圳技术大学 | Driving behavior monitoring method and device and computer readable storage medium |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110633665B (en) * | 2019-09-05 | 2023-01-10 | 卓尔智联(武汉)研究院有限公司 | Identification method, device and storage medium |
| CN116844219A (en) * | 2023-06-29 | 2023-10-03 | 深圳卓创智能科技有限公司 | Display method, device, equipment and storage medium for preventing eyestrain |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN203552276U (en) * | 2013-10-23 | 2014-04-16 | 深圳市中控生物识别技术有限公司 | Facial recognition device applicable to wide-dynamic-range ambient light |
| CN105187727A (en) * | 2015-06-17 | 2015-12-23 | 广州市巽腾信息科技有限公司 | Image information acquisition device, image acquisition method and use of image information acquisition device |
| CN105279498A (en) * | 2015-10-28 | 2016-01-27 | 广东欧珀移动通信有限公司 | Eyeball identification method, device and terminal |
| CN106168854A (en) * | 2016-06-30 | 2016-11-30 | 北京小米移动软件有限公司 | The based reminding method of sight protectio and device |
| CN106897725A (en) * | 2015-12-18 | 2017-06-27 | 西安中兴新软件有限责任公司 | A kind of method and device for judging user's asthenopia |
| CN106952450A (en) * | 2017-05-11 | 2017-07-14 | 赵可成 | One kind utilizes dual camera device for resisting fatigue driving |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106445092A (en) * | 2015-08-13 | 2017-02-22 | 中兴通讯股份有限公司 | Vision protection processing method and apparatus for terminal |
| CN105512613A (en) * | 2015-11-26 | 2016-04-20 | 中山大学 | Smartphone-based eye fatigue detection method |
| TWI729983B (en) * | 2016-04-05 | 2021-06-11 | 鴻海精密工業股份有限公司 | Electronic device, system and method for adjusting display device |
| CN106200004A (en) * | 2016-07-15 | 2016-12-07 | 宇龙计算机通信科技(深圳)有限公司 | Based reminding method, intelligent glasses and onboard system |
| CN106792177A (en) * | 2016-12-28 | 2017-05-31 | 海尔优家智能科技（北京）有限公司 | A kind of TV control method and system |
-
2017
- 2017-11-15 WO PCT/CN2017/111150 patent/WO2019095155A1/en active Application Filing
- 2017-11-15 CN CN201780096860.1A patent/CN111357006A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN203552276U (en) * | 2013-10-23 | 2014-04-16 | 深圳市中控生物识别技术有限公司 | Facial recognition device applicable to wide-dynamic-range ambient light |
| CN105187727A (en) * | 2015-06-17 | 2015-12-23 | 广州市巽腾信息科技有限公司 | Image information acquisition device, image acquisition method and use of image information acquisition device |
| CN105279498A (en) * | 2015-10-28 | 2016-01-27 | 广东欧珀移动通信有限公司 | Eyeball identification method, device and terminal |
| CN106897725A (en) * | 2015-12-18 | 2017-06-27 | 西安中兴新软件有限责任公司 | A kind of method and device for judging user's asthenopia |
| CN106168854A (en) * | 2016-06-30 | 2016-11-30 | 北京小米移动软件有限公司 | The based reminding method of sight protectio and device |
| CN106952450A (en) * | 2017-05-11 | 2017-07-14 | 赵可成 | One kind utilizes dual camera device for resisting fatigue driving |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112506338A (en) * | 2020-11-30 | 2021-03-16 | 崔刚 | Equipment control method, device and equipment based on full vision |
| CN112506338B (en) * | 2020-11-30 | 2024-04-16 | 崔刚 | Full-vision-based equipment control method, device and equipment |
| CN113297734A (en) * | 2021-05-21 | 2021-08-24 | 南京邮电大学 | Visual cone-based simulation evaluation method for assembly process in limited space |
| CN113297734B (en) * | 2021-05-21 | 2024-03-29 | 南京邮电大学 | Visual cone-based simulation evaluation method for assembly process in limited space |
| CN113709308A (en) * | 2021-08-31 | 2021-11-26 | 维沃移动通信有限公司 | Usage monitoring method and device for electronic equipment |
| CN114120395A (en) * | 2021-11-02 | 2022-03-01 | 深圳技术大学 | Driving behavior monitoring method and device and computer readable storage medium |
| CN114120395B (en) * | 2021-11-02 | 2024-09-13 | 深圳技术大学 | Driving behavior monitoring method and device and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2019095155A1 (en) | 2019-05-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111357006A (en) | Fatigue prompting method and terminal | |
| WO2016041340A1 (en) | An indication method and mobile terminal | |
| US20170124837A1 (en) | Communication method, apparatus, system and computer-readable medium for wearable device | |
| CN108597186B (en) | Drowning alarm method based on user behavior and wearable device | |
| CN108429888B (en) | Light supplementing method and mobile terminal | |
| CN109164477B (en) | Application positioning method and mobile terminal | |
| CN107592615B (en) | Positioning method and device | |
| CN108307106B (en) | Image processing method and device and mobile terminal | |
| AU2018299499B2 (en) | Method for iris recognition and related products | |
| CN106445596B (en) | Method and device for managing setting items | |
| CN107734170B (en) | Notification message processing method, mobile terminal and wearable device | |
| CN107767839A (en) | Brightness adjusting method and related product | |
| CN110784915B (en) | Power consumption control method of electronic equipment and electronic equipment | |
| CN111477334A (en) | Target area reminding method and electronic equipment | |
| CN108681413B (en) | Control method of display module and mobile terminal | |
| CN106408869B (en) | Terminal and method and device for detecting human body falling | |
| CN108460599B (en) | Mobile payment method and mobile terminal | |
| CN108551525B (en) | State determination method of movement track and mobile terminal | |
| CN110351418B (en) | Screen control method and device, mobile terminal and computer readable medium | |
| CN107085718B (en) | Method for detecting human face and device, computer equipment, computer readable storage medium | |
| CN108062195A (en) | The reminding method and mobile terminal of a kind of notification event | |
| CN108491539A (en) | terminal control method, device, storage medium and electronic equipment | |
| CN112114879A (en) | Terminal wake-up method, terminal and storage medium | |
| CN107895108B (en) | Operation management method and mobile terminal | |
| CN108632469A (en) | The position indicating method and mobile terminal of mobile terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200630 |
|
| RJ01 | Rejection of invention patent application after publication |