CN113143274B - Camera-based emotional early warning method - Google Patents

Abstract

The invention discloses a camera-based emotion pre-warning method, which comprises the steps of obtaining a human body photo in real time through a camera, finding one or more human faces existing in an image through a human face detection algorithm, finding a face hairless area by utilizing a relative geometric relation, measuring and calculating the image brightness change of the area to obtain a continuous array, further obtaining the heart beat interval of a monitored person, obtaining the emotion condition of the monitored person through an emotion judgment model in combination with the facial expression of the monitored person, and further judging whether an alarm is needed or not.

Description

Camera-based emotional early warning method

Technical field

The invention relates to the field of emotion monitoring and early warning, and in particular to a camera-based emotion early warning method.

Background technique

In people's daily work and life, the impact of emotional state on work performance is relatively obvious, especially work that lasts for a long time or requires a high degree of concentration, and is more obviously affected by the emotional state of the worker. For example, students' listening state in class will actually be affected by their own emotional states. If teachers can provide appropriate guidance based on students' emotional states, teaching efficiency will certainly be improved.

For some particularly important jobs, such as staff in the control room of a nuclear power plant, they need to be highly concentrated and judge the operating status of the nuclear power plant in real time. If the emotional state of the staff can be known in real time, when the emotional state is unstable or obviously tired, Timely reminders for shift changes will definitely further enhance the safety of nuclear power plants and eliminate safety hazards caused by human negligence from the root.

There are also some special positions, such as bus drivers, if their emotions are in an extreme state, it will pose a potential threat to public safety. If the driver's emotional state can be known in time, and reminders or alarms can be issued when necessary, social security can be further enhanced. .

However, there is currently no technical solution that can effectively measure the emotions of normal workers. Although facial expressions can be judged through facial recognition, facial expressions cannot fully reflect the emotional state of the human body, so the level of research in this area is It still needs to be further improved.

In addition, the data measured by the existing technology are single-dimensional, that is, the degree of tension, or what some studies call the "frustrated" degree. In the field of psychology, these are collectively called the degree of emotional arousal (i.e., from calm to agitated). (dimension of change), however, emotion is not only one dimension. Emotional valence (i.e., the nature of the positive-negative dimension of emotion) is also a very important discriminant index. Without the positive and negative judgment of emotion, emotion discrimination is impossible. complete. For example, the same very agitated state can have two extremes of rage (negative) and ecstasy (positive). The same very calm state can also have two very different states of despair (negative) and tranquility.

Due to the above reasons, the inventor of the present invention conducted in-depth research on existing emotion recognition and judgment methods in order to design a camera-based emotion warning method that can solve the above problems.

Contents of the invention

In order to overcome the above problems, the inventor conducted intensive research and designed a camera-based emotional early warning method. In this method, human body photos are obtained in real time through the camera, and one or more people present in the image are found by using a face detection algorithm. Based on the human face, we use the relative geometric relationship to find the facial hair-free area, measure the image brightness changes in this area, and obtain a continuous array, and then obtain the heart beat interval of the monitored person, combined with the facial expression of the monitored person, through the emotion The judgment model obtains the emotional state of the monitored person, and then judges whether an alarm is needed, thereby completing the present invention.

Specifically, the purpose of the present invention is to provide the following aspects: a camera-based emotion early warning method, which method includes the following steps:

Step 1: Obtain images and photos containing the face of the monitored person through real-time shooting with the camera;

Step 2: Use facial recognition to distinguish the identity information of each monitored person in the image and photo, and set up a corresponding independent storage unit for each monitored person;

Step 3: Obtain the heart beat intervals and facial expressions of the monitored person by reading the images and photos, and store them in an independent storage unit;

Step 4: Input heart beat intervals and facial expressions into the emotion judgment model in real time to determine the emotional state of the person being monitored;

Step 5: When the monitored person's emotion is in a warning state, an alarm message is issued.

Among them, the step 3 includes the following sub-steps:

Sub-step a, filter the image photos obtained in step 1, and delete the image photos that cannot be used to read the heart beat interval;

Sub-step b, in the remaining image photos, use relative geometric relationships to find the hairless part of the face as the detection area;

Sub-step c, measure the image brightness change of this area in the continuous image photos, and obtain a continuous array,

Preferably, the array is a curve describing human heartbeat activity. When the average brightness is higher, it represents cardiac diastole, that is, the trough of the ECG wave. When the average brightness is lower, it represents the contraction of the heart, that is, the ECG wave peak; the time interval between the two peaks. That is the heart beat interval.

Wherein, the emotion judgment model is obtained through the following sub-steps:

Sub-step 1, collect physiological data and facial expressions through the collection device, the physiological data includes heart beat intervals, and convert the physiological data into sympathetic nerve activity indicators and parasympathetic nerve activity indicators;

Sub-step 2, set the emotional arousal tag and the emotional valence tag, record the specific emotional arousal level in the emotional arousal tag, record the specific emotional valence in the emotional valence tag, and combine the neural activity index data, facial expression data and The combination of emotion tags is the basic data;

Sub-step 3: Adjust the format of the basic data to obtain basic data in a unified format, and determine whether the basic data in a unified format meets the requirements;

Sub-step 4: Select available data from basic data in a unified format that meets the requirements;

Sub-step 5: Obtain the emotion judgment model based on the available data in sub-step 4.

Wherein, each comprehensive nerve activity index includes one or more of the following data: sympathetic nerve activity index, parasympathetic nerve activity index, quotient of sympathetic nerve activity index and parasympathetic nerve activity index, sympathetic nerve activity index The sum of the activity index and the parasympathetic nerve activity index, and the difference between the sympathetic nerve activity index and the parasympathetic nerve activity index.

Among them, in sub-step 3, judging whether the basic data in a unified format meets the requirements includes the following sub-steps:

Sub-step 1: randomly divide all basic data in a unified format into learning groups and testing groups according to predetermined proportions;

Sub-step 2, use the data in the learning group to flush the model, then use each data in the test group to verify the model one by one, and record the verification results of each data in the test group respectively;

Sub-step 3, repeat sub-step 1 and sub-step 2, in which the basic data in the unified format that were once assigned to the inspection group are no longer allocated to the inspection group, ensuring that each basic data in the unified format has been assigned to the inspection group. In the testing group, the models that have been washed out by the data in the learning group are verified until all verification results corresponding to the basic data in a unified format are obtained;

Sub-step 4: Calculate the total pass rate of all unified format basic data verification results. When the total pass rate is greater than 85%, the unified format basic data meets the requirements. Otherwise, delete the unified format basic data and repeat. Sub-step 1 and sub-step 2.

Among them, obtaining available data in sub-step 4 includes the following sub-steps:

Sub-step a, repeat sub-steps 1-3 multiple times. Each time you repeat sub-step 1, you will get a test group composed of different basic data in a unified format; so that each basic data in a unified format corresponds to multiple Verify the results, and then calculate the average pass rate corresponding to each basic data in a unified format;

Sub-step b: Find and hide the unified format basic data of 1 case with the lowest average pass rate, use the remaining unified format basic data to perform sub-steps 1-4 again, and observe whether the overall pass rate is improved compared to before hiding the data. , if the total pass rate increases, delete the hidden basic data in a unified format, and perform sub-step c; if the total pass rate does not increase, restore the hidden data, select and hide the second lowest average pass rate For basic data in a unified format, repeat the above process until the overall pass rate increases;

Sub-step c, after the overall pass rate is improved, use the remaining basic data in a unified format as the basis, repeat sub-step a and sub-step b, and find that the overall pass rate is improved, then use the current remaining basic data in a unified format. As the basis, continue to repeat sub-step a and sub-step b until the total pass rate reaches more than 90%, or until the deleted basic data in a unified format reaches 30% of the total basic data in a unified format, at this time the remaining Basic data in a unified format is usable data.

Wherein, in sub-step 5, the emotion judgment model includes the emotional arousal prediction model and the emotional valence prediction model;

In the process of obtaining the emotional judgment model, the comprehensive neural activity indicator data, facial expression data and emotional arousal data in each available data are spliced into a data segment, which is used as learning material to obtain the emotional arousal prediction model through machine learning;

The comprehensive neural activity indicator data, facial expression data and emotional valence data in each available data are spliced into a data segment and used as learning materials to obtain an emotional valence prediction model through machine learning.

The emotional state in step 4 includes emotional agitation and emotional valence.

Among them, in step 5, the emotional state of the monitored person is compared with the average emotional agitation value and the average emotional valence value in the calm state,

When monitoring personnel are engaged in important, difficult or high-intensity operations,

The level of emotional agitation is 1.5 standard deviations higher than the average level of emotional agitation in the calm state.

Or the level of emotional agitation is 1 standard deviation below the average level of emotional agitation in a calm state,

Or when its emotional valence is 1.5 standard deviations lower than the average emotional valence in a calm state, an alarm message is issued;

When monitoring personnel are engaged in ordinary work,

The level of emotional agitation is 1.5 standard deviations higher than the average level of emotional agitation in the calm state.

Or the level of emotional agitation is 1 standard deviation below the average level of emotional agitation in a calm state,

Or when the emotional valence is 1.5 standard deviations lower than the standard average emotional valence value in the calm state, an alarm message is issued;

When the monitored person's emotional agitation is 2 standard deviations higher than the average emotional agitation value in the calm state, and his or her emotional valence is 2 standard deviations lower than the average emotional valence in the calm state, the person being monitored is responsible for the public Potential threats to security occur and an alarm message is issued.

The beneficial effects of the present invention include:

(1) The camera-based emotion early warning method provided by the present invention can set different alarm conditions according to the different work properties of the monitored person, thereby expanding the application scope of the method;

(2) According to the camera-based emotion warning method provided by the present invention, an emotion judgment model based on a large number of sample washes is provided, which can accurately and timely calculate the emotional state of the monitored person based on image information;

(3) The camera-based emotion warning method provided by the present invention adopts a two-dimensional emotion evaluation scheme, which can not only measure the arousal of emotions, but also estimate the emotional valence. Compared with the previous 2 classification or 4 classification Emotion assessment technology, this technology can output 100 kinds of emotion assessments of different intensities and properties. The results are more real, close to common sense and easier to be understood by people, so they are more usable in actual production and life;

(4) The camera-based emotion warning method provided by the present invention can determine in real time whether the person being monitored is suitable for participating in important, difficult, and high-intensity tasks; it can also determine in real time whether the person being monitored is experiencing intense psychological activities and respond to Public safety is a threat.

Description of the drawings

Figure 1 shows the overall logic diagram of a camera-based emotion early warning method according to a preferred embodiment of the present invention;

Figure 2 shows the changes in the emotional agitation level of a monitored person within a day obtained through the camera-based emotion warning method in the embodiment of the present invention;

Figure 3 shows the changes in emotional valence of a monitored person within a day obtained through the camera-based emotion warning method in the embodiment of the present invention;

Figure 4 shows an evaluation display interface of a monitored person's emotional status within a day in the embodiment of the present invention.

Detailed ways

The present invention will be further described in detail below through the drawings and examples. Through these descriptions, the features and advantages of the present invention will become more apparent.

The word "exemplary" as used herein means "serving as an example, example, or illustrative." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or superior to other embodiments. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

According to the camera-based emotion warning method provided by the present invention, as shown in Figure 1; the method includes the following steps:

Step 1: Obtain images and photos containing the face of the monitored person through real-time shooting with the camera;

Step 2: Distinguish the identity information of each monitored person in the image and photo, and set up a corresponding independent storage unit for each monitored person;

Step 3: Obtain the heart beat intervals and facial expressions of the monitored person by reading the images and photos, and store them in an independent storage unit;

Step 4: Input heart beat intervals and facial expressions into the emotion judgment model in real time to determine the emotional state of the person being monitored;

Step 5: When the monitored person's emotion is in a warning state, an alarm message is issued.

In a preferred embodiment, in step 1, the camera can be placed near the working position of the person being monitored. For example, if the person being monitored is an operator in the control room of a nuclear power plant, the camera can be placed near the display screen, such as the person being monitored. If the monitoring person is a student, the camera can be placed near the blackboard. If the monitored person is a bus driver, the camera can be placed near the front window. That is, the camera is preferably placed in the direction of the face of the monitored person, as far as possible. The face of the monitored person is captured in multiple places. The camera has an adjustable focal length and can capture multiple monitored persons at the same time.

In a preferred embodiment, in step 2, the identity information of each monitored person in the image photo is distinguished through face recognition, and a corresponding independent storage unit is set for each monitored person; the person The face recognition method can be processed using the open source openface face recognition tool.

In a preferred embodiment, step 3 includes the following sub-steps:

Sub-step a, filter the image photos obtained in step 1, and delete the image photos that cannot be used to read the heart beat interval;

Among them, specifically, the images that need to be deleted include: ① In the 1000 millisecond time window, the average change in facial image brightness is greater than 1 (brightness range is 0-255); ② Images whose faces are not captured by openface; ③ If the facial contour displacement exceeds 1% of the vertical resolution of the picture within 1000 milliseconds (for example, if the vertical resolution is 1080 pixels, in principle, if the contour displacement exceeds 10 pixels, the picture during this period will be deleted).

Sub-step b, in the remaining image photos, use relative geometric relationships to find the hairless part of the face as the detection area; that is, use the forehead and cheek as the detection area.

Sub-step c, measure the image brightness change of this area in the continuous image photos, and obtain a continuous array,

Preferably, the array is a curve describing human heartbeat activity. When the average brightness is higher, it represents cardiac diastole, that is, the trough of the ECG wave. When the average brightness is lower, it represents the contraction of the heart, that is, the ECG wave peak; the time interval between the two peaks. That is the heart beat interval.

Use the Butterworth filter to filter the original signal, retain the 0.5-2Hz band signal, and find the peaks and troughs in the filtered signal. The peak is the point in the signal window of 500 milliseconds that is greater than the value on both sides, and the trough is 500 milliseconds. Points in the signal bed whose values are smaller than the two sides;

Preferably in this application, generating a set of data requires continuous collection of images for at least 7 seconds at a sampling frequency of no less than 25 Hz.

Preferably, the data collected by using a camera to collect continuous image data is in color or black and white. Black and white images can be used directly. For color images, it is preferred to use the green channel among red, green and blue. In addition, through color coordinate system transformation, the red, green, and blue channels can be converted into HSV or HLS color space. In the HSV color space, the V channel (i.e., brightness channel) value is used as the input index, and in the HLS color space, the L channel (also known as Luminance channel) value as input indicator. The time frequency of image collection should be above 25Hz, that is, at least 25 frames of continuous images should be collected per second.

In a preferred implementation, the emotion judgment model is obtained through the following sub-steps:

Sub-step 1, collect physiological data and facial expressions through the collection device, the physiological data includes heart beat intervals, and convert the physiological data into sympathetic nerve activity indicators and parasympathetic nerve activity indicators; the heart beat interval is also Called the R-R interval.

Sub-step 2: Set the emotional arousal label and the emotional valence label, select the specific emotional arousal level in the emotional arousal label, select the specific emotional valence in the emotional valence label, and combine the neural activity index data, facial expression data and The combination of emotion tags is the basic data;

Sub-step 3: Adjust the format of the basic data to obtain basic data in a unified format, and determine whether the basic data in a unified format meets the requirements;

Sub-step 4: Select available data from basic data in a unified format that meets the requirements;

Sub-step 5: Obtain the emotion judgment model based on the available data in step 4.

In a preferred embodiment, the collection device includes a wearable bracelet, a smart watch and a camera. Preferably, the collection equipment may also include massage chairs, treadmills, etc. After collecting physiological data and recording tag data through the collection device, all data can be transmitted to a remote server in real time for statistical storage, or a memory chip can be integrated into the collection device for real-time storage and calculation processing.

Preferably, in sub-step 1, two sets of data are converted and output based on the collected heart beat intervals, which are the activity indicators of the sympathetic nerves and the activity indicators of the parasympathetic nerves, thereby making the scheme in this application more delicate. time granularity.

In sub-step 1, the two nerves jointly influence the heart beat and the periodic proactive influence of nerve activities on each other ultimately constitutes heart rate variability.

Preferably, the emotional arousal tag is provided with a plurality of numerical values that can represent the degree of emotional arousal. The corresponding numerical value can be selected according to the actual situation. Preferably, the emotional arousal tag is provided with 5-10 numerical gears. According to Participants choose the numerical range closest to their actual situation. The emotional arousal label represents the degree of emotional arousal. The lowest value represents complete calmness, and the larger the value, the more excited the emotion.

The emotional valence tag is provided with a plurality of numerical values that can represent the emotional valence, and the corresponding value can be selected according to the actual situation. Preferably, the emotional valence tag is provided with 2-10 numerical gears, depending on the participation. Choose the numerical range closest to the user's actual situation. The emotional valence label represents the degree of positivity and negativity of the emotion. The lowest value represents the most negative value, and the larger the value, the more positive the emotion. The data formats in the two emotional valence tags with the same numerical gear are unified, and the data formats in the two emotional arousal tags with the same numerical gear are unified.

Preferably, a standardized emotional arousal score is used as the original label score in the emotional arousal label;

Preferably, the PANAS standard score is used as the original label score in the emotion valence label, where positive emotion: average score is 29.7, standard deviation: 7.9; negative emotion: average score is 14.8, standard deviation is 5.4.

Further preferably, in both the emotional arousal label and the emotional valence label, the range of plus or minus 1.96 standard deviations of the numerical range is divided into 10 parts according to the frequency of the data distribution.

Preferably, in sub-step 2, the emotion tags include emotional arousal tags and emotional valence tags, and the two tags can be provided separately or simultaneously in the form of coordinates or charts. The emotional arousal tag is used to record emotional arousal data, and the emotional valence tag is used to record emotional valence data.

Preferably, in sub-step 2, the comprehensive nerve activity index is related to the activity index of the sympathetic nerve and the activity index of the parasympathetic nerve, and each comprehensive nerve activity index contains one or more of the following data: sympathetic Nerve activity index, parasympathetic nerve activity index, quotient of sympathetic nerve activity index and parasympathetic nerve activity index, sum of sympathetic nerve activity index and parasympathetic nerve activity index, sympathetic nerve activity index and parasympathetic nerve activity index The difference and so on.

Preferably, the collection frequency of the comprehensive neural activity index data is relatively high, and 60-90 or even more sets of the comprehensive neural activity index data can be provided every minute. Each time the comprehensive neural activity index data is collected, try to capture the current scene through the camera. Facial expressions, because facial expressions play an auxiliary role in model processing, if facial expressions are collected more comprehensively, the model effect will be enhanced.

The collection frequency of the emotion tags is relatively low, and can be collected once an hour, or 2-5 times a day. Each time the emotion tags are collected, the facial expression at that time is collected through the camera. Therefore, each emotion label data corresponds to multiple comprehensive neural activity indicator data. Combining an emotion label data, facial expression data and multiple corresponding comprehensive neural activity indicator data together can form a basic data. Among them, each emotion label data includes emotional arousal data and emotional valence data.

Preferably, the numerical ranges in the emotional valence tag and the emotional arousal tag may be the same or different. Problems of mismatch or data misalignment may occur during data statistics. For this reason, in sub-step 3, adjust the basis The format of the data mainly includes adjusting the numerical values and numerical gears in the emotion label data; specifically, first set the number of standard numerical gears. For example, if it is set to 5 numerical gears, then adjust the numerical gears in the basic data to 5, and then adjust the gear value selected in the basic data to the gear value in the case of 5 numerical gears according to the proportion, and round up when it is not divisible.

In a preferred embodiment, determining whether the basic data in a unified format meets the requirements in sub-step 3 includes the following sub-steps:

Sub-step 1: randomly divide all basic data in a unified format into two groups according to a predetermined ratio, namely the learning group and the testing group; preferably, the ratio can be 8 to 9:1, and more preferably, in the learning group The ratio of the number of data to the number of data in the test group is 8:1;

Sub-step 2, use the data in the learning group to flush the model, then use each data in the test group to verify the model one by one, and record the verification results of each data in the test group respectively. Preferably, the verification results include verification Passed and failed verification; among them, passing verification means bringing the comprehensive neural activity indicator data and facial expression data of the basic data in a unified format in the inspection group into the model, and the obtained emotion label data is consistent with the emotion label in the basic data. The data is consistent, that is, the degree of emotional agitation and emotional valence are consistent; verification failure means that the comprehensive neural activity index data and facial expression data of the basic data in the test group are brought into the model, and the obtained emotion label data is consistent with the basic data The emotion label data in is inconsistent, that is, the degree of emotional agitation and/or the valence of the emotion are inconsistent;

Sub-step 3, repeat the above sub-step 1 and sub-step 2 multiple times, in which the basic data in the unified format that was once assigned to the inspection group will no longer be assigned to the inspection group to ensure that the basis of each unified format The data has been verified in the inspection group to verify the model washed by the data in the learning group, until all basic data in a unified format have obtained corresponding verification results;

Sub-step 4: Calculate the total pass rate of the verification results of all basic data in a unified format. The total pass rate of the verification results of all basic data in a unified format is the number of verification passes and the number of all basic data in a unified format. ratio; when the total pass rate is not greater than 85%, it is considered that these basic data in a unified format do not meet the basic requirements, all are given up, and sub-steps 1 and 2 are repeated to obtain new basic data; when the result in sub-step 4 is When the total pass rate is greater than 70%, it is considered that these basic data in a unified format meet the usage requirements and can be processed in the next step.

In a preferred embodiment, obtaining available data in sub-step 4 includes the following sub-steps:

In sub-step a, the gradient method is used to eliminate outlier data for each model-parameter combination and screen out models with high ecological utility. Specifically, sub-steps 1-3 in sub-step 3 are repeated multiple times, and each time sub-step 1 is repeated, a test group composed of different basic data in a unified format is obtained, that is, all test groups are Different; preferably, the sub-steps 1-3 are repeated 8-10 times, so that each basic data in a unified format corresponds to multiple verification results, and then each corresponding basic data in a unified format is solved separately. Average pass rate; the average pass rate corresponding to the basic data in a unified format is the ratio of the number of verification results that have passed in the verification results corresponding to the basic data in a unified format to the total number of verification results corresponding to the basic data in a unified format.

Sub-step b, find and hide one case of basic data in a unified format with the lowest average pass rate. When there are multiple cases of basic data in a unified format with the same and lowest average pass rate, just hide any one of them, and the hidden data , before being restored, it will no longer participate in any calculation processing; find and use the remaining basic data in a unified format to perform sub-steps 1-4 again, and observe whether the total pass rate is improved compared to before hiding the data. If the total pass rate is improved , then delete the hidden basic data in a unified format, and perform sub-step c; if the total pass rate does not increase, restore the hidden data, select and hide the basic data in a unified format with the second lowest average pass rate, Among them, if there are multiple basic data in a unified format with the same and lowest average pass rate, then you can select other basic data in a unified format with the lowest hit rate; repeat the above process until the total pass rate increases;

Sub-step c, after the overall pass rate is improved, use the remaining basic data in a unified format as the basis, repeat sub-step a and sub-step b, and find that the overall pass rate is improved, then use the current remaining basic data in a unified format. As the basis, continue to repeat sub-step a and sub-step b until the total pass rate reaches more than 90%, preferably more than 92%; or when the deleted basic data in a unified format reaches 30% of the total basic data in a unified format So far, the remaining basic data in a unified format is available data.

Preferably, the model in sub-step 2 includes most supervised learning models, and the flushing process of the model includes comprehensive judgment of multiple supervised models. The specific flushing process includes but is not limited to linear regression, Support vector machine, gradient descent method, naive Bayes classification, decision tree classification, AdaBoost, XGBoost, multi-layer neural network and other flushing methods. Preferably, the average of two results that are close to each other among the results of the three models of multi-layer neural network with 3-4 layer structure, C4.5 decision tree, and XGBoost is used as the output value of each flush, that is, 3-4 The combination of layer-structured multi-layer neural network, C4.5 decision tree, and XGBoost is the most preferred model, that is, a model with high ecological utility. In this application, preferably, the neural network selects a one-dimensional convolutional neural network.

In step 5, in the process of obtaining the emotion judgment model, the comprehensive neural activity indicator data, facial expression data and emotional arousal data in each available data are spliced into a data segment, which is used as learning material to obtain the emotional arousal emotion through machine learning. judgment model;

The comprehensive neural activity indicator data, facial expression data and emotional valence data in each available data are spliced into a data segment as learning materials, and an emotional valence emotion judgment model is obtained through machine learning; the emotion judgment model includes the above Emotional arousal prediction model and emotional valence prediction model.

Preferably, in step 5, the learning process of the emotional arousal prediction model and the emotional valence prediction model is medium, while using comprehensive neural activity indicators, facial expression data and label data to establish a neural network that flushes the 3-4 layer structure, C4. 5. Three models of decision tree and The average of two close output values. For example, for a set of data, among the output results given by the three models, one is 8, one is 20, and one is 7. The output result 7 and the output result 8 are close to each other, then the output result 7 of the final model is, that is The average of 7 and 8, rounded down.

In a preferred embodiment, in sub-steps 1-5, 1,000 participants of all ages are tracked and collected for 2 weeks to 2 months to obtain tracking data. Participants' physiological data comes from wearable devices such as smart watches and scanning sensors, facial expression data comes from cameras, and rating data comes from participants' daily self-evaluations; physiological data uses a method of collecting 90 seconds of data every 10 minutes, 24 hours a day. Continuous tracking; in terms of scoring data for emotional arousal labels and emotional valence labels, participants are required to assess their agitation and emotional valence at least three times a day, and each time they fill in the emotional arousal labels and emotional valence labels, they are captured by camera Facial expression photos.

In a preferred embodiment, in step 4, based on the existing emotion judgment model, the emotion judgment model includes an emotion arousal prediction model and an emotion valence prediction model. By inputting the heart beat intervals and facial expressions in the storage unit into the two models, the corresponding emotional arousal and emotional valence can be obtained.

Specifically, the cardiac beat interval RRI is first converted into the sympathetic output and parasympathetic output of the comprehensive neural activity index:

Use the Laguerre function recursion formula, let its dependent variable be the closest RRI, and let its independent variable be the decomposition term X of 8 Laguerre recursion formulas. Each decomposition term consists of an unknown coefficient G, The inference coefficient φ is composed of an RRI value, and the overall estimated expression is as shown in the following equation (1):

Among them, S represents the upper limit of j, that is, the order of the Laguerre polynomial. This order determines how many past RRIs are used to fit an expression. The more orders, the more accurate the result. It is preferred to use 9; j represents the order of the orthogonal Laguerre discrete time function; g(j,t) represents the coefficient matrix obtained by combining the j-order Laguerre polynomial and the RRI interval time within the t time range. The coefficients in this coefficient matrix are The purpose of each RRI coefficient is to merge multiple RRIs into a recursive Laguerre polynomial, use past RRIs to fit the last RRI, and make multiple RRIs form a recursive relationship; F(t) represents the inclusion The calculated position number of the specific interval in the sequence of adjacent cardiac beat intervals; n represents the serial number of the RRI going back starting from this RRI; RR _F(t)-n represents any RRI, through the Laguerre polynomial obtained recursively; Represents the j-order orthogonal Laguerre discrete time function, which is obtained by the following formula (2);

α is a constant, its value is 0.2;

Starting from the closest RRI, take 8 RRIs in reverse time and substitute them into the above RRIs to obtain the RRI combination, forming RRI=∑(i∈0-2)Xi+∑(i∈3-8)Xi. Use Kalman autoregression to obtain 8 unknown coefficients G. Substituting ∑(i∈0-2)NiGi and ∑(i∈3-8)NiGi respectively represent the sympathetic and parasympathetic nerve output values in the comprehensive neural activity index. The matching coefficients N use constants 39, 10, -5, 28, -17, 6, 12, 6, -7, -6, -4 respectively.

The comprehensive neural activity index and facial expression input are brought into the emotional arousal prediction model, and the comprehensive neural activity index and facial expression are brought into the emotional valence prediction model. The following processing is performed in the two models:

The emotional arousal prediction model includes a multi-layer neural network model with a 3-4 layer structure, a C4.5 decision tree model and an XGBoost calculation module model. After receiving comprehensive neural activity indicators and facial expressions, the emotional arousal prediction model obtains 3- For the values output by the 4-layer multi-layer neural network model, C4.5 decision tree model and XGBoost calculation module model, select 2 closer values among these three output values and calculate the average of the two values. value as the output of the emotional arousal model.

The emotional valence prediction model also includes a 3-4 layer structure multi-layer neural network model, C4.5 decision tree model and XGBoost calculation module model. After receiving comprehensive neural activity indicators and facial expressions, the emotional valence prediction model obtains respectively From the values output by the multi-layer neural network model with a 3-4 layer structure, the C4.5 decision tree model and the XGBoost calculation module model, two closer values are selected from the three output values, and the two values are calculated. The average of the values is used as the output of the emotional valence prediction model.

Finally, the corresponding degree of emotional arousal and emotional valence are obtained, which is the emotional state of the person being monitored.

In a preferred embodiment, in step 5, the emotional state of the monitored person is compared with the average emotional agitation value and the average emotional value value in the calm state,

When monitoring personnel are engaged in important, difficult or high-intensity operations,

The level of emotional agitation is 1.5 standard deviations higher than the average level of emotional agitation in the calm state.

Or the level of emotional agitation is 1 standard deviation below the average level of emotional agitation in a calm state,

Or when its emotional valence is 1.5 standard deviations lower than the average emotional valence in a calm state, an alarm message is issued;

The calm state described in this application is a data set collected from at least 100 participants in a calm state. It is a collection of all emotional values collected. The average and standard deviation of this data set are calculated, so that the average value and standard deviation as the basis for judgment, set the critical value for triggering the alarm at the mean plus 1.5 or 1 standard deviation; that is, among the emotion values obtained through the emotion judgment model, if the obtained emotional value value is higher than the average The emotional valence value plus 1 to 1.5 times the standard deviation, or the emotional valence value obtained is lower than the average emotional valence value minus 1 to 1.5 times the standard deviation, or the emotional agitation value obtained is higher than the average agitation value plus 1 to 1.5 times the standard deviation, or If the obtained emotional agitation value is lower than the average agitation value minus 1 to 1.5 times the standard deviation, then it is selected whether to trigger the alarm according to different personnel conditions.

The important, difficult or high-intensity operations refer to those that require long-term exposure to hazardous environments or potentially hazardous working environments, such as high-altitude operations, operations of construction machinery such as drivers and cranes, and control of important facilities such as power plants. Maintenance work, etc.;

The above-mentioned ordinary jobs refer to occupations that are not in high-risk or potentially hazardous environments and have average labor intensity, such as clerks, editors, service industry practitioners, students, teachers, librarians, etc.

The level of emotional agitation is 1.5 standard deviations higher than the average level of emotional agitation in the calm state.

Or the level of emotional agitation is 1 standard deviation below the average level of emotional agitation in a calm state,

Or when its emotional valence is 1.5 standard deviations lower than the average emotional valence in a calm state, an alarm message is issued;

When the monitored person's emotional agitation is 2 standard deviations higher than the average emotional agitation value in the calm state, and his or her emotional valence is 2 standard deviations lower than the average emotional valence in the calm state, the person being monitored is responsible for the public Potential threats to security occur and an alarm message is issued. Among them, the potential threats mainly involve people who may endanger the safety of others in public places under excessive emotions, such as bus drivers, crane excavator drivers, passengers in train station or airport waiting rooms, etc.

The average emotional agitation value is the intermediate value representing calmness in the emotional arousal label in sub-step 2, and the average emotional valence value is the intermediate value representing calmness in the emotional valence label in sub-step 2.

Example

Establish an emotion judgment model. Specifically, 100 participants are selected and all participants are continuously tracked for a month. The participants live within the shooting range of the camera for at least 12 hours a day, and at least 8 hours of them can The face is captured by the camera, and the participant's facial expression and the brightness changes of the hairless part of the face are obtained in real time through the camera, and the brightness changes are converted into heart beat interval data in real time, and then the heart beat interval data is converted into sympathetic nerves Activity indicators and parasympathetic activity indicators. In addition, participants recorded the degree of emotional agitation in the emotional arousal label and the emotional valence in the emotional valence label three times a day. The labels included 10 numerical gears, including , participants recorded their average emotional agitation and emotional valence in the morning every morning, recorded their average emotional agitation and emotional valence in the afternoon every afternoon, and recorded their average emotional agitation and emotional valence in the evening every evening. .

A total of 610,820 pieces of RRI data and 415,128 frames of facial expression data were obtained. The RRI data were then converted into sympathetic nerve activity indicators and parasympathetic nerve activity indicators. At the same time, 9,000 records containing emotional arousal labels and emotional valence labels were obtained through data collection. , combine an emotion label data and multiple corresponding comprehensive neural activity index data into one basic data, forming a total of 9000 basic data.

To obtain the total pass rate, all 9000 pieces of basic data are randomly divided into 9 parts, one part is used as the inspection group, and the other parts are used as the learning group. The model is flushed through the learning group, and then the data in the inspection group is used to verify the model, and each part is obtained. Verify the results of the test group data, and then use the data in other parts as the test group. Repeat the above steps for a total of 9 times to ensure that each data has been assigned to the test group, that is, each data has the corresponding Verify the results and find that the total pass rate is 88%. If it is higher than 85%, you can proceed to the next step.

Eliminate abnormal data in the basic data to obtain usable data. Specifically,

Find the average passing rate, re-divide all basic data into 9 parts, one part is used as the inspection group, and the other parts are used as the learning group. The model is flushed through the learning group, and then the data in the inspection group is used to verify the model, and the value of each data is obtained. Verify the results; then redistribute the inspection group and learning group, and repeat the above process at least 81 times to ensure that each basic data is divided into the inspection group at least 9 times, that is, each basic data has obtained 9 corresponding verification results. , and then obtain the average passing rate of each basic data;

Find and hide the basic data of 1 case with the lowest average pass rate, use the remaining 8999 pieces of basic data to perform the above process of calculating the average pass rate and the total pass rate again, and observe whether the total pass rate has improved compared with before hiding the data. If the total pass rate If the pass rate increases, delete the hidden basic data in a unified format; if the total pass rate does not improve, restore the hidden data, select and hide the basic data with the second lowest average pass rate, and repeat the above formula to calculate the total pass rate process until the overall pass rate increases;

After the hit rate increases, delete the hidden data, continue to perform the above process of finding the average pass rate based on the remaining basic data, calculate the average pass rate corresponding to each basic data, and find and hide the lowest average pass rate data, and then calculate the total pass rate based on the data with the lowest average pass rate, and continue to repeat the above elimination process.

After the hit rate increases, delete the hidden data and continue to repeat the above process based on the remaining basic data.

It stops when the deleted data reaches 2700, and the remaining data is the available data.

Emotional arousal prediction models and emotional valence prediction models are obtained based on the available data. Specifically,

Use the available data to flush the one-dimensional convolutional neural network to get the one-dimensional convolutional neural network model. Use the available data to flush the C4.5 decision tree to get the C4.5 decision tree model. Use the available data to flush the XGBoost computing module to get the XGBoost computing module model. The three models are combined to form an emotion judgment model; when the emotion judgment model receives new comprehensive neural activity indicators and facial expression information, it copies the received information into three copies and transmits them to the one-dimensional convolutional neural network model and C4 respectively. .5 Decision tree model and XGBoost calculation module model. The output value of the emotion judgment model is the mean of two closer values among the three model outputs given by the three models, thus obtaining the emotional arousal prediction model and emotional valence prediction. model, that is, the emotion judgment model is obtained.

Select 50 monitored personnel working in the main control room of the nuclear power plant, and obtain images and photos containing the faces of the monitored personnel through real-time camera shooting;

Distinguish the identity information of each monitored person in images and photos through facial recognition, and set up a corresponding independent storage unit for each monitored person;

The heart beat intervals and facial expressions of the monitored person are obtained by reading the image photos, and are stored in an independent storage unit;

Input heart beat intervals and facial expressions into the emotion judgment model in real time to determine the emotional status of the person being monitored;

After continuing for 12 hours, the emotional changes of the 50 monitored persons within 12 hours were obtained. The emotional status of one of them is shown in Figures 2 and 3; among them, Figure 2 shows the emotional arousal of the monitored person, that is, The change curve of the degree of emotional agitation. The abscissa in the figure represents time, and the ordinate represents the degree of emotional agitation. The higher the value, the stronger the degree of agitation. In Figure 2, the dotted line in the middle represents the average emotional agitation value;

Figure 3 shows the change curve of the monitored person's emotional valence. The abscissa in the figure represents time, and the ordinate represents the emotional valence. The higher the value, the greater the emotional valence; in Figure 3, the dotted line in the middle represents average emotional valence value;

The emotional agitation level of the 50 monitored persons has always been within the range of plus or minus 0.7 standard deviations of the average agitation value, and their emotional valence remains within the range of plus or minus 1 standard deviation of the average emotional valence value in the calm state. There is no need to issue Alarm information.

The monitored person is then asked to self-evaluate his/her emotional changes within 14 hours. Figure 4 shows the self-evaluation of a monitored person. The scoring method uses a sliding bar interface. The specific evaluation plan is: 6 hours after getting up. Give your own emotional state evaluation value in the morning period within 6-10 hours after getting up. The emotional state evaluation value is the emotional state evaluation value in the period from the first evaluation to the second evaluation; the emotional state evaluation value is given in the evening period from 10 days after getting up to before going to bed. That is, the emotional evaluation value during the period from the second evaluation to the third evaluation is given;

The self-evaluated emotional status of 50 monitored persons was counted and compared with the emotional status obtained by the emotional judgment model, and it was found that the matching rate reached 85%.

It can be seen from the above results that the camera-based emotion warning method provided by this application can timely and accurately determine the emotional changes of the monitored person.

The present invention has been described above with reference to preferred embodiments, but these embodiments are only exemplary and serve an illustrative purpose. On this basis, various substitutions and improvements can be made to the present invention, which all fall within the protection scope of the present invention.

Claims (5)

1. A camera-based emotion early warning method, characterized in that the method includes the following steps: Step 1: Obtain images and photos containing the face of the monitored person through real-time shooting with the camera; Step 2: Use facial recognition to distinguish the identity information of each monitored person in the image and photo, and set up a corresponding independent storage unit for each monitored person; Step 3: Obtain the heart beat intervals and facial expressions of the monitored person by reading the images and photos, and store them in an independent storage unit; Step 4: Input heart beat intervals and facial expressions into the emotion judgment model in real time to determine the emotional state of the person being monitored; Step 5: When the monitored person's emotions are in a warning state, an alarm message is issued; The step 3 includes the following sub-steps: Sub-step a, filter the image photos obtained in step 1, and delete the image photos that cannot be used to read the heart beat interval; Sub-step b, in the remaining image photos, use relative geometric relationships to find the hairless part of the face as the detection area; Sub-step c, measure the image brightness change of this area in the continuous image photos, and obtain a continuous array, This array is a curve describing human heartbeat activity. When the average brightness is higher, it represents cardiac diastole, which is the trough of the ECG wave. When the average brightness is lower, it represents the contraction of the heart, which is the ECG peak. The time interval between the two peaks is the heart rate. beat interval; The emotion judgment model is obtained through the following sub-steps: Sub-step 1, collect physiological data and facial expressions through the collection device, the physiological data includes heart beat intervals, and convert the physiological data into sympathetic nerve activity indicators and parasympathetic nerve activity indicators; Sub-step 2, set the emotional arousal tag and the emotional valence tag, record the specific emotional arousal level in the emotional arousal tag, record the specific emotional valence in the emotional valence tag, and combine the neural activity index data, facial expression data and The combination of emotion tags is the basic data; Sub-step 3: Adjust the format of the basic data to obtain basic data in a unified format, and determine whether the basic data in a unified format meets the requirements; Sub-step 4: Select available data from basic data in a unified format that meets the requirements; Sub-step 5: Obtain the emotion judgment model based on the available data in sub-step 4; Each comprehensive nerve activity index includes one or more of the following data: sympathetic nerve activity index, parasympathetic nerve activity index, quotient of sympathetic nerve activity index and parasympathetic nerve activity index, sympathetic nerve activity index The sum of the activity indicators of the parasympathetic nerves and the difference between the activity indicators of the sympathetic nerves and the activity indicators of the parasympathetic nerves; In sub-step 5, the emotion judgment model includes an emotional arousal prediction model and an emotional valence prediction model; In the process of obtaining the emotional judgment model, the comprehensive neural activity indicator data, facial expression data and emotional arousal data in each available data are spliced into a data segment, which is used as learning material to obtain the emotional arousal prediction model through machine learning; The comprehensive neural activity indicator data, facial expression data and emotional valence data in each available data are spliced into a data segment and used as learning materials to obtain an emotional valence prediction model through machine learning. 2. The camera-based emotion warning method according to claim 1, characterized in that: In sub-step 3, judging whether the basic data in a unified format meets the requirements includes the following sub-steps: Sub-step 1: randomly divide all basic data in a unified format into learning groups and testing groups according to predetermined proportions; Sub-step 2, use the data in the learning group to flush the model, then use each data in the test group to verify the model one by one, and record the verification results of each data in the test group respectively; Sub-step 3, repeat sub-step 1 and sub-step 2, in which the basic data in the unified format that were once assigned to the inspection group are no longer allocated to the inspection group, ensuring that each basic data in the unified format has been assigned to the inspection group. In the testing group, the models that have been washed out by the data in the learning group are verified until all verification results corresponding to the basic data in a unified format are obtained; Sub-step 4: Calculate the total pass rate of all unified format basic data verification results. When the total pass rate is greater than 85%, the unified format basic data meets the requirements. Otherwise, delete the unified format basic data and repeat. Sub-step 1 and sub-step 2. 3. The camera-based emotion warning method according to claim 2, characterized in that: Obtaining usable data in sub-step 4 includes the following sub-steps: Sub-step a, repeat sub-steps 1-3 multiple times. Each time you repeat sub-step 1, you will get a test group composed of different basic data in a unified format; so that each basic data in a unified format corresponds to multiple Verify the results, and then calculate the average pass rate corresponding to each basic data in a unified format; Sub-step b: Find and hide the unified format basic data of 1 case with the lowest average pass rate, use the remaining unified format basic data to perform sub-steps 1-4 again, and observe whether the overall pass rate is improved compared to before hiding the data. , if the total pass rate increases, delete the hidden basic data in a unified format, and perform sub-step c; if the total pass rate does not increase, restore the hidden data, select and hide the second lowest average pass rate For basic data in a unified format, repeat the above process until the overall pass rate increases; Sub-step c, after the overall pass rate is improved, use the remaining basic data in a unified format as the basis, repeat sub-step a and sub-step b, and find that the overall pass rate is improved, then use the current remaining basic data in a unified format. As the basis, continue to repeat sub-step a and sub-step b until the total pass rate reaches more than 90%, or until the deleted basic data in a unified format reaches 30% of the total basic data in a unified format, at this time the remaining Basic data in a unified format is usable data. 4. The camera-based emotion warning method according to claim 1, characterized in that, The emotional state in step 4 includes emotional agitation and emotional valence. 5. The camera-based emotion warning method according to claim 1, characterized in that: In step 5, the emotional state of the monitored person is compared with the average emotional agitation value and the average emotional valence value in the calm state, When monitoring personnel are engaged in important, difficult or high-intensity operations, The level of emotional agitation is 1.5 standard deviations higher than the average level of emotional agitation in the calm state. Or the level of emotional agitation is 1 standard deviation below the average level of emotional agitation in a calm state, Or when its emotional valence is 1.5 standard deviations lower than the average emotional valence in a calm state, an alarm message is issued; When monitoring personnel are engaged in ordinary work, The level of emotional agitation is 1.5 standard deviations higher than the average level of emotional agitation in the calm state. Or the level of emotional agitation is 1 standard deviation below the average level of emotional agitation in a calm state, Or when its emotional valence is 1.5 standard deviations lower than the average emotional valence in a calm state, an alarm message is issued; When the monitored person's emotional agitation is 2 standard deviations higher than the average emotional agitation value in the calm state, and his or her emotional valence is 2 standard deviations lower than the average emotional valence in the calm state, the person being monitored is responsible for the public Potential threats to security occur and an alarm message is issued.