KR102033320B1 - Glasses type multi modal apparatus for estimating emotion - Google Patents

Abstract

Embodiments are provided with a spectacle body including a front part and a leg, a wide-angle camera installed on the spectacle body and acquiring a part of the face of the user to obtain a local part of the facial part, and transmitting and receiving the local part of the facial part to the processing device. Including a portion, wherein the facial portion local image is related to the spectacle-type multi-modal device and the method using the same, characterized in that used to estimate the emotional state of the user.

Description

Glasses type multimodal device and method thereof for emotion estimation {GLASSES TYPE MULTI MODAL APPARATUS FOR ESTIMATING EMOTION}

The present invention relates to an eyeglass multimodal device for estimating a user's emotion and a method thereof, and more particularly, to an apparatus and method for estimating a user's emotion using a local facial image.

Recently, as technology for health care has developed, various wearable devices that can be worn on a body have been developed. The wearable device mainly utilizes a specific body part such as a hand, which can easily acquire a biosignal. For example, a system that acquires a bio signal such as a user's heart rate in a wearable form applied to a hand such as a 'smart ring' or a 'smart watch' provides a user's health care application.

In addition, the conventional glasses type smart device is used to acquire external information and provide it to the user, such as 'Google Glass', but it acquires a biosignal or image from the user to interpret the information about the user and derive the result. There was no equipment. That is, in the case of the eyeglass type device, even though it is in close contact with the user's body and used for a long time, there is a problem that no method of obtaining or using significant information from the user has been studied.

Korean Patent Application Publication No. 10-2016-0024412 United States Patent Application Publication US2013 / 0245396

In order to solve the above problems, there is a need for a method of acquiring and using blood flow or skin conductivity in a facial image or face of a user through a spectacle-type device.

The problem to be solved of the present invention is not limited to those mentioned above, another technical problem not mentioned will be clearly understood by those skilled in the art from the following description.

Eyeglass multi-modal device for emotion estimation according to an embodiment of the present invention is a spectacle body including a front portion and a leg, is installed on the spectacle body, a wide angle to obtain a facial image by photographing a part of the face of the user A camera and a transceiver for transmitting the acquired facial part image to the processing device, wherein the facial part image may be used to estimate the emotional state of the user.

In one preferred embodiment, a portion of the face portion is a spectacled multi-modal device for emotion estimation, characterized in that it comprises a part of one eye and one ball.

In a preferred embodiment, the wide-angle camera may be located at least one of the inside, the outside, the inside of the leg of the front portion.

In a preferred embodiment, when the wide-angle camera is located outside the front portion, the wide-angle camera may be fixed to the leg by rotating based on the hinge portion.

In a preferred embodiment, the wide-angle camera may be embedded in the front portion or the leg.

In a preferred embodiment, further comprising a GSR (Galvanic Skin Response) sensor is installed on the nose support portion of the front portion of the spectacle body for measuring the skin conductivity of at least one of the nose, the transceiver unit, the skin conductivity of the measured nose, etc. May be further transmitted to the processing apparatus.

In a preferred embodiment, the GSR sensor extends downward from one position of the leg of the spectacle body, located behind or below the ear of the user to further measure the reference skin conductivity, and the transceiver unit, the reference Skin conductivity may be further transmitted to the treatment device.

In a preferred embodiment, further comprising a photoplethysmogram (PPG) sensor for measuring the blood flow in the earlobe of the user, the transceiver unit may further transmit the measured blood flow to the processing device.

In a preferred embodiment, the processing device may be embedded in the spectacle multimodal device for emotion estimation or may be a stand alone device separate from the spectacle multimodal device for emotion estimation.

In one preferred embodiment, the treatment device, the facial region image, the skin conductivity of the nose and the like; Alternatively, the emotional state of the user may be estimated based on the blood flow amount.

In a preferred embodiment, the processing apparatus may estimate the emotional state of the user by giving weight to the skin conductivity of the nose or the blood flow rate when the resolution or brightness of the local facial image is below the threshold level.

In one preferred embodiment, the treatment apparatus is based on the facial region image, the skin conductivity of the nose or the like or the skin conductivity of the nose or the blood flow based on the quality of the data (quality) data on the blood flow Can be weighted.

In the emotion estimation method using the spectacle type multi-modal apparatus according to an embodiment of the present invention, a part of the face of the user is photographed by using a wide-angle camera installed on the spectacle body including the front part and the leg to obtain a facial part image. ; And transmitting the acquired facial part local image to a processing apparatus through a transceiver, wherein the facial part local image may be used to estimate the emotional state of the user.

According to an embodiment of the present invention, since the wearable device has a long wearing time and is in close contact with the user's body, there is an advantage in that useful and various information related to the user can be obtained for a long time. Based on such user information, various services may be implemented by the user, such as health care or emotional care.

The effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a block diagram of an eyeglass multimodal device for emotion estimation according to an embodiment of the present invention.
2 is a perspective view of an eyeglass multimodal device 100 for emotion estimation according to an embodiment of the present invention.
3 illustrates an installation position of the wide-angle camera 120 according to various embodiments of the present disclosure.
4 illustrates a case where the wide-angle camera 120 according to the exemplary embodiment of the present invention is installed outside the spectacle body 110.
5 shows an example of using the wide-angle camera 120 according to an embodiment of the present invention.
6A and 6B illustrate a facial local image (FIG. 6A) and a general full facial image (FIG. 6B) according to an embodiment of the present invention.
Referring to FIG. 7, a GSR sensor 130B for measuring reference skin conductivity according to an embodiment of the present invention is shown.
FIG. 8 illustrates a correlation between data measuring skin conductivity according to a user's face GSR position and data measuring skin conductivity of a finger.
9 shows a result of emotion classification according to GSR data in a face part and a finger of a user.
10 illustrates a step of obtaining a first sensor value and a second or third sensor value and estimating an emotional state of a user using the same according to an embodiment of the present invention.
11 is a graph comparing a result of classifying an emotional state and a random result based on a local facial image according to an embodiment of the present invention.
FIG. 12 shows a case in which only a GSR sensor is used, a case in which a GSR sensor and a PPG sensor are used together, and a random result as a result according to another embodiment of the present invention.
FIG. 13 is a flowchart illustrating a emotion monitoring method using an eyeglass multimodal device according to an embodiment of the present invention.

Embodiments described herein may have aspects that are wholly hardware, partly hardware and partly software, or wholly software. As used herein, "unit", "module", "device" or "system" and the like refer to hardware, a combination of hardware and software, or a computer related entity such as software. For example, parts, modules, devices, or systems herein refer to running processes, processors, objects, executables, threads of execution, programs, and / or computers. computer, but is not limited thereto. For example, both an application running on a computer and a computer may correspond to a part, module, device, system, or the like herein.

Embodiments have been described with reference to the flowchart presented in the drawings. Although the method is shown and described in a series of blocks for the sake of simplicity, the invention is not limited to the order of the blocks, and some blocks may occur in different order or simultaneously with other blocks than those shown and described herein. Various other branches, flow paths, and blocks may be implemented in order to achieve the same or similar results. In addition, not all illustrated blocks may be required for the implementation of the methods described herein. Furthermore, the method according to an embodiment of the present invention may be implemented in the form of a computer program for performing a series of processes, which may be recorded on a computer-readable recording medium.

Hereinafter, the configuration and characteristics of the present invention will be described using examples, but these examples are not intended to limit the present invention.

1 is a block diagram of an eyeglass multimodal device for emotion estimation according to an embodiment of the present invention. Referring to FIG. 1, the spectacle type multi-modal apparatus 100 for emotion estimation may include a spectacle body 110, a wide angle camera 120, and a transceiver 130. In another embodiment, the spectacle multimodal device 100 for emotion estimation may further include a GSR sensor 130 and / or a PPG sensor 140. The spectacled multi-modal apparatus 100 for emotion estimation may obtain body information for estimating an emotional state from a user and transmit it to the processing apparatus 200. In some cases, the spectacle type multi-modal apparatus 100 for emotion estimation may exchange data with the processing apparatus 200.

The processing apparatus 200 may be embedded in (included) the spectacle type multimodal apparatus 100 for emotion estimation or may be a stand alone device separately from the spectacle type multimodal apparatus 100 for emotion estimation.

FIG. 1 illustrates a case where the processing apparatus 200 is a device independent of the spectacle type multimodal apparatus 100 for emotion estimation. In this case, the spectacled multi-modal apparatus 100 and the processing apparatus 200 for emotion estimation may communicate through a wired or wireless network. The communication method can be any communication method conventionally available. For example, various communication protocols such as Bluetooth, Wi-Fi, 3G, and 4G may be used.

When the processing apparatus 200 is a device independent of the spectacle type multi-modal apparatus 100 for emotion estimation, the transceiver 150 may perform data communication with the processing apparatus 200.

2 is a perspective view of an eyeglass multimodal device 100 for emotion estimation according to an embodiment of the present invention. Referring to FIG. 2, the external shape of the spectacle type multi-modal apparatus 100 for emotion estimation may be configured as the spectacle body 110. Therefore, the user may wear the spectacle type multi-modal device 100 for emotion estimation, such as wearing glasses, on the face.

The spectacle body 110 may include a front portion 111 positioned in front of the user's eyes and a leg 112 to be worn on the user's ear. The legs may be one or more than two. The rim shape of the front portion 111 may be an upper, lower or side portion is open form. Typically, the rim refers to a space of the front portion 111 into which the lens can be inserted, and the rim and the rim are connected through a bridge.

2 shows that the upper part of the rim is open. In this case, since the upper part of the rim is opened, there is no shadow on the face, thereby reducing noise in face image analysis.

The wide-angle camera 120 is installed on the spectacle-shaped body 110 and may photograph a part of the face of the user to acquire a face image. The facial local image may be used to estimate the emotional state of the user.

3 illustrates an installation position of the wide-angle camera 120 according to various embodiments of the present disclosure. Referring to FIG. 3, the wide-angle camera 120 is installed on the spectacle body 110 to photograph the face of the user. The wide-angle camera 120 includes at least one of the inner side 120A, the outer side 120B, and the inner side 120C of the leg. It can be located at The inside is the direction toward the user and the outside is the opposite.

The wide angle camera 120 may be embedded in the front portion 111 or the leg 112. Therefore, it may not protrude as shown in FIG. In the case of Figure 2, the wide-angle camera 120 is installed on the outside of the front portion 111 is represented in a protruding form.

4 illustrates a case where the wide-angle camera 120 according to the exemplary embodiment of the present invention is installed outside the spectacle body 110. Referring to FIG. 4, when the wide angle camera 120 is installed outside the front portion 111, the wide angle camera 120 may be fixed to the leg 112 by rotating based on the hinge 122. If the wide-angle camera does not operate, it may be significant that the position of the wide-angle camera may be changed as it may obstruct the user's field of view. Referring to FIG. 4, the lens 121 of the wide-angle camera 120 may be positioned to face the user's face in the use mode and may be fixed to the leg in the non-use mode and face outward.

The wide angle camera 120 may include various image sensors such as a CCD sensor or an infrared sensor, but is not limited thereto. Any image recognition device for acquiring a face image may be included in the wide angle camera 120.

5 shows an example of using the wide-angle camera 120 according to an embodiment of the present invention. Referring to FIG. 5, the wide-angle camera 120 is located near the eyes of the user. Accordingly, the facial part image acquired by the wide-angle camera 120 may include one eye and one upper cheek. Preferably, the facial region image may include only one eye and one upper part of the cheek. That is, it may include only one eye and its periphery, not both eyes. The wide angle camera may be located under or above the eyes of the user so as not to obscure the user's field of view. In order to prevent the ball from covering the eyebrows, the wide-angle camera may be located under the eye as shown in FIG. 5.

 6A and 6B illustrate a facial local image (FIG. 6A) and a general full facial image (FIG. 6B) according to an embodiment of the present invention. In general, in order to estimate the emotional state by the user's facial expression, an image including the entire face part is used as shown in FIG. 6B. However, in the case of using the spectacle-shaped body as in the present invention, in order to acquire an image including all of the face parts, the camera must be very far from the face part (even when using a wide viewing angle camera), and as a result, the volume of the wearable device increases considerably. have. For example, the antenna should be straight out of the spectacle body as if it were an antenna.

The wide-angle camera of the present invention has an advantage in that it does not have to be far from the face part because it acquires a facial part image including only a part of the face part of the user. Therefore, the device can be miniaturized. Experimental results using the same algorithm to estimate the emotional state of the user from the entire face image as shown in Figure 6b and the emotional state of the user from the local face image (one eye 42 and the upper part of the ball 43) as shown in Figure 6a The result of estimating is estimated to be at the same level. The coil portion 41 of FIG. 6A may or may not be included in the facial local image.

As a result of the inventor's experiment, two kinds of emotions were classified for the above two kinds of images (facial local image and full facial image) based on PCA and Fisher LDA, and the accuracy of classification was 95.41% for the full facial image. In the case of 96.04%. These results show that in determining at least two emotion states, a meaningful result can be obtained even if only the local image is used without using the entire face region image of the user. Here, two kinds of emotional states are two kinds according to arousal and affirmation / valence. In addition, significant results can be obtained for four or more emotional states considering both domains beyond two emotional states.

The transceiver 150 may transmit the acquired facial part local image to the processing apparatus 200. The transceiver 150 may transmit data acquired by other sensors (eg, PPG and GSR) to the processing apparatus 200 in addition to the facial image.

The processing apparatus 200 may estimate the emotional state of the user based on the local facial image. To this end, it is possible to estimate the emotional state by analyzing the facial image on the basis of PCA, Fisher LDA algorithm, etc., but the method is not limited to the above algorithm, and any conventional face recognition algorithm may be used.

In one embodiment, the spectacled multi-modal device 100 for emotion estimation is installed in the nose portion of the front portion 111 of the spectacle body 110 to measure the skin conductivity of at least one nose, etc. (Galvanic Skin Response) ) May further include a sensor (130). At this time, the transceiver may further transmit the measured skin conductivity of the nose and the like to the processing apparatus.

Referring back to FIG. 2, the GSR sensor 130A may be installed at the nose support part (or bridge). The GSR sensor 130A may measure skin conductivity by contacting a user's nose. The information obtained from the GSR sensor may be used to measure the user's emotion according to arousal and affirmation / negation.

The processing device 200 may determine the skin conductivity value obtained from the nose by comparing the skin conductivity obtained from the nose and the like with reference skin conductivity. Here, the reference skin conductivity may be obtained from behind the user's ear or from below the ear (ie, the papillary portion).

Referring to FIG. 2, for this purpose, the GSR sensor 130B extends downward from one position of the leg of the spectacle body and may be located behind the user's ear or below the ear to measure the reference skin conductivity.

Referring to FIG. 7, a GSR sensor 130B for measuring reference skin conductivity according to an embodiment of the present invention is shown. Referring to FIG. 7, a structure of the GSR sensor 130B capable of moving the position on the leg 112 and changing the length of the leg extending to the foot to adjust the position of the GSR sensor 130B according to the size of the face or the ear of the user is provided. appear.

Referring to FIG. 7, in order to move the position of the body 131B of the GSR sensor 130 on the leg 112 (x-axis movement), the user may pull the latch 133B to release the fixing state and move the body 131B. have. In addition, the electrode unit 132B for measuring the reference skin conductivity is rotatable based on one end of the body 131B (r direction). Therefore, the position of the electrode for measuring the reference skin conductivity can be easily adjusted according to the position of the ear or the face of the user.

In general, the GSR signal acquisition for measuring the emotional state of the user is performed for the user's finger. However, the GSR sensor 130 of the present invention is installed on the spectacle body, and measures the conductivity of the skin around the user's nose and ears, not the finger.

FIG. 8 illustrates a correlation between data measuring skin conductivity according to a location of a face of a user (ears, nose, ears, and nose) and data measuring skin conductivity of a finger; FIG. The results of emotional classification according to GSR data in the nose) and fingers are shown. The ears may be earballs and the nose may be a nose, etc.

In detail, (a) of FIG. 8 is a result of emotional components based on data from which noise signals are removed from raw data, and FIG. 8 (b) shows emotion classification based on phasic data of GSR. One result is shown.

Referring to FIG. 8A, it can be seen that in the state where noise is removed, the emotional state based on the result of measuring skin conductivity only with respect to the ear is most relevant to the result with the finger.

Referring to the results of (a) and (b) of FIG. 8, it can be seen that obtaining the GSR signal at the 'ear and nose' together has the highest similarity with obtaining the GSR signal at the hand.

In addition, Figure 9 (a) is the emotional classification results according to the excitation (Arousal) and Figure 9 (b) is the emotional classification results according to the positive / negative (Valence). Referring to FIG. 9, when classifying emotions according to the degree of excitement and positive / negativeness, when classifying emotions through the GSR data of the ears and the nose, the accuracy is higher than that of other parts.

Also, in one embodiment, the spectacle multimodal device 100 for emotion estimation may further include a PPG (Photoplethysmography) sensor 140 together with or separately from the GSR sensor 130. In this case, the transceiver may further transmit the measured blood flow amount to the processing apparatus.

Referring back to FIG. 2, the PPG sensor 140 extends from a portion of the spectacle body 101 and has an end portion having a tong. The PPG sensor 140 may be attached to the ear ball of the user to measure blood flow. The processing device 200 may measure the emotional state of the user based on the measured blood flow change.

In one embodiment, the processing apparatus 200 may estimate the emotional state of the user based on the local facial image, 1) skin conductivity of the nose, or 2) blood flow. The blood flow may be measured in the ear ball of the user. In such an embodiment, the processing apparatus 200 may estimate the emotional state of the user based on the multi-modal signal rather than the uni-modal signal.

For clarity of description, in the present specification, the facial local image is referred to as a first sensor value, a skin conductivity of a nose, etc., as a second sensor value, and a blood flow rate as a third sensor value.

In order to minimize the signal processing load and consider the influence of the use environment, the processing apparatus 200 may use only the sensor values above the threshold without using the first sensor value or the third sensor value below the threshold value.

10 illustrates a step of obtaining a first sensor value and a second or third sensor value and estimating an emotional state of a user using the same according to an embodiment of the present invention. Referring to FIG. 10, in a data acquisition step S10, the wide-angle camera acquires a facial part image for analyzing a facial expression of a user (S11), and obtains a second sensor value or a third sensor value that is user's body response information. (S12). In the threshold processing step (S20), the processing apparatus 200 checks whether each of the first to third sensor values exceeds a predetermined threshold value, and ignores data that does not exceed the threshold value, and the data exceeding the threshold value. Proceed to the next step for. Step S20 may be omitted according to an embodiment.

In the feature extraction step (S30), the processing apparatus 200 extracts a first feature based on brightness intensity of each pixel in the facial region image, and extracts a second feature based on a change pattern of a second or third sensor value. do.

In the weighting operation S40, the processing apparatus 200 may weight the first and second features according to the data quality of the first to third sensor values. That is, the processing apparatus 200 weights the data of the facial area image, the skin conductivity of the nose, etc., or the blood flow rate based on the quality of the data on the facial area image, the skin conductivity of the nose, or the blood flow rate. May be given. Also, for example, in the weighting step S40, when the resolution or brightness of the local facial image is less than or equal to a threshold level, the processing apparatus 200 weights the skin conductivity (second sensor value) or blood flow (third sensor value) such as a nose. Subject's emotional state can be estimated. The weighting step S40 may be omitted.

In the feature conversion step S40 and the classification step S50, the processing apparatus 200 selects and classifies a meaningful feature according to an algorithm among various features according to the first feature and the second feature to estimate the emotional state of the user. Can be.

11 is a graph comparing a result of classifying an emotional state and a random result based on a local facial image according to an embodiment of the present invention. Referring to FIG. 11, the results of classifying the emotional state based on the facial region images compared to the random results show not only four types but also high accuracy in arousal and positive / negative degrees. In other words, it is possible to classify emotions with high accuracy only by using facial image of local area.

FIG. 12 shows a result according to another embodiment of the present invention, in which only a GSR sensor, a GSR sensor and a PPG sensor are used together, and a random result. Referring to FIG. 11, the results of the four classifications are not significantly different. However, the results of the arousal and the positive / negative degree are shown to be able to distinguish the emotional state with higher accuracy when the GSR and the PPG sensor are used at the same time.

FIG. 13 is a flowchart illustrating a emotion monitoring method using an eyeglass multimodal device according to an embodiment of the present invention. The emotion monitoring method using the spectacle multimodal device may be implemented by the components of the spectacle multimodal device 100 for emotion estimation described above. Referring to FIG. 13, in the emotion monitoring method using the multi-modal eyeglass device, acquiring a local image of a face part by capturing a part of a face part of a user through a wide-angle camera installed in a spectacle body including a front part and a leg (S100) And transmitting the acquired facial part local image to the processing apparatus through the transceiver (S200). Here, the local facial image may be used to estimate the emotional state of the user.

Although the present invention has been described by specific embodiments such as specific components and the like, but the embodiments and the drawings are provided to assist in a more general understanding of the present invention, the present invention is not limited to the above embodiments. For those skilled in the art, various modifications and variations can be made from these descriptions.

Accordingly, the spirit of the present invention should not be limited to the above-described embodiments, and all of the equivalents or equivalents of the claims, as well as the appended claims, fall within the scope of the spirit of the present invention. I will say.

Claims (13)

An eyeglass body including a front portion and a leg;
A wide-angle camera installed in the spectacle-shaped body and acquiring a local image of the facial part by capturing a part of the facial part of the user;
Transmitting and receiving unit for transmitting the acquired facial portion local image to the processing device; And
Is installed in the nose portion of the front of the spectacle-shaped body including a GSR (Galvanic Skin Response) sensor for measuring the skin conductivity of at least one nose,
The GSR sensor extends downward from one position of the leg of the spectacle body, and is located behind or below the ear of the user to further measure reference skin conductivity,
The transmitting and receiving unit further transmits the measured skin conductivity such as the nose and the reference skin conductivity to the processing device,
The facial region image, skin conductivity of the nose and the like and reference skin conductivity are used for estimating the emotional state of the user, the spectacle type multimodal device for emotion estimation.
The method of claim 1,
One portion of the face portion is a spectacle-type multi-modal device for emotion estimation, characterized in that it comprises a portion of one eye and one ball.
The method of claim 1,
The wide-angle camera is an eyeglass type multi-modal device for emotion estimation, characterized in that located on at least one of the inside, outside, the inside of the leg of the front portion.
The method of claim 3,
If the wide-angle camera is located outside the front portion, the wide-angle camera is a multi-modal device for emotion estimation, characterized in that the rotation is fixed to the hinge relative to the hinge portion.
The method of claim 1,
The wide-angle camera is an eyeglass type multi-modal device for emotion estimation, characterized in that embedded in the front portion or the leg.
delete delete The method of claim 1,
Further comprising a photoplethysmogram (PPG) sensor for measuring blood flow in the earlobe of the user,
The transceiving unit, the spectacles multi-modal device for emotion estimation, characterized in that for further transmitting the measured blood flow to the processing device.
The method of claim 1,
The processing apparatus is built in the spectacle type multi-modal apparatus for emotion estimation or is a stand-alone device separate from the spectacle type multi-modal apparatus for emotion estimation (eye-type multi-modal for emotion estimation) Device.
The method of claim 8,
The treatment device, the facial part local image,
Skin conductivity of the nose and the like; Or the blood flow amount; estimating the emotional state of the user based on the spectacle type multimodal device for emotion estimation.
The method of claim 10,
The processing device,
When the resolution or brightness of the local facial image is less than the threshold level, the multi-modal device for the estimation of emotion, characterized in that the weight of the skin conductance or blood flow of the nose and the like to estimate the emotional state of the user.
The method of claim 10,
The processing apparatus may weight the data of the facial area image, the skin conductivity of the nose or the like or the data of the blood flow based on the quality of the data of the facial area image, the skin conductivity of the nose, or the blood flow rate. Glasses type multi-modal device for emotion estimation, characterized in that.
Obtaining a facial image by capturing a part of the face of the user through a wide-angle camera installed on the spectacle body including the front part and the leg;
Measuring skin conductivity of at least one of the noses through a galvanic skin response (GSR) sensor installed at a nose portion of the front of the spectacle body;
Measuring, via the GSR sensor, a reference skin conductivity behind or below the ear of the user;
Transmitting the acquired facial region image, skin conductivity of the nose and the like and the reference skin conductivity through the transceiver, to the processing device,
The facial region image, skin conductivity of the nose and the like and reference skin conductivity are used to estimate the emotional state of the user emotion estimation method using the spectacle-type multi-modal device.

Images