JP2010003116A - Object deciding device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an object deciding device and a program for deciding whether or not an object is wearing a mounted object such as a mask or sun glasses with high reliability. <P>SOLUTION: A likelihood value p1 showing object-likeliness is calculated by using a machine learning system configured to learn to respond to the image of an object shown by a pickup image, the object wearing a mask and the object not wearing any mask (S104), and a likelihood value p2 showing object-likeliness wearing the mask is calculated by using the machine learning system configured to learn to respond to the image of the object wearing the mask (S106), and a likelihood value p3 showing object-likeliness is calculated by using the machine learning system configured to respond to the image of the object not wearing the mask (S108), and whether or not the object shown by the pickup image is wearing the mask is decided on the basis of the calculated three likelihood values p1, p2 and p3 (S110). <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an object determination device and a program, and more particularly to an object determination device and a program for determining whether or not an object is wearing an attachment.

The target person wears a mask, sunglasses, etc. when estimating the face state quantity such as face direction, blink, gaze, etc., using facial feature quantities such as mouth and nose extracted from the face image Then, the face state quantity cannot be estimated correctly. As a method for estimating the face orientation in consideration of the attachment on the face, for example, there is a technique described in Patent Document 1 below. In this technique, feature points such as eyes, mouth, and nose are extracted from the face image to estimate the face direction. If feature points of the eyes cannot be extracted a predetermined number of times, it is determined that sunglasses are worn, Face orientation is estimated using image features. If the feature amount of the nose and mouth cannot be extracted a predetermined number of times, it is determined that the mask is worn, and the face orientation is estimated using the image feature on the mask.
JP 2003-296712 A

  In the above conventional technique, when it is not possible to extract the feature amounts of the eyes, nose, and mouth, it is determined that the sunglasses and the max are worn, respectively. For example, the face orientation of the driver who drives the vehicle is estimated. In some cases, in the actual driving environment, the characteristics of eyes, nose, and mouth are often undetected due to local brightness changes on the face image due to fluctuations in the surrounding lighting environment, face posture of the subject, etc. Occurs. For this reason, it is determined that the user wears sunglasses and a mask because the eyes, nose, and mouth cannot be detected even though the sunglasses and mask are not worn. A state quantity such as the face orientation is estimated from the point, and as a result, a correct state quantity cannot be obtained, and conversely, a state quantity with a very large error is calculated.

  The present invention has been proposed in order to solve the above-described problem, and an object determination apparatus capable of determining with high reliability whether or not the object is wearing an object such as a mask or sunglasses, and The purpose is to provide a program.

  In order to achieve the above object, the object determination device of the invention of claim 1 is configured to capture an image of an object represented by an imaging unit for imaging the object and an image obtained by imaging by the imaging unit. The object-likeness is obtained by using an image captured by the means, an image of the object with the attached object, and a machine learning system learned to react to the image of the object without the attached object. A first likelihood value is calculated, and the wearing is performed using an image obtained by being picked up by the image pickup means, and a machine learning system that has learned to react to an image of the object on which the wearing object is attached. A second likelihood value indicating the likelihood of the object wearing the object is calculated, and learning is performed so as to react to the image obtained by imaging by the imaging unit and the image of the object not wearing the object. Using the machine learning system A likelihood value calculating means for calculating a third likelihood value indicating the likelihood of an object not wearing an object; a first likelihood value, a second likelihood value calculated by the likelihood value calculating means; And determining means for determining whether or not the object represented by the image obtained by imaging by the imaging means is wearing the wearing object based on the three likelihood values.

  In this way, the machine learning system learned to respond to the image of the object with the attachment and the image of the object without the attachment, and the image of the object with the attachment Three likelihood values are obtained using the machine learning system learned in the above and the machine learning system learned to react to the image of the object not wearing the attachment, and based on the three likelihood values. Since it is determined whether or not the object is wearing an attachment, the presence or absence of the attachment can be determined with high reliability. In particular, by using likelihood values obtained from three different types of machine learning systems as evaluation criteria, it is possible to reduce false detections more than when using one type of machine learning system.

  The object determination apparatus according to the invention of claim 2 assumes that the object is mounted on the object from an imaging unit for imaging the object and an image obtained by imaging with the imaging unit. An extraction unit that extracts a predetermined region that is estimated to be the presence of the wearing object, and an image of the region extracted by the extraction unit with respect to an object represented by an image captured by the imaging unit And learning to react to the image of the predetermined area in the image of the object with the wearing object and the image of the predetermined area in the image of the object not wearing the wearing object. A first likelihood value indicating the likelihood of an object is calculated using a machine learning system, and the predetermined area in the image of the area extracted by the extraction unit and the image of the object mounted with the attachment Image of A second likelihood value indicating the likelihood of the object wearing the wearing object is calculated using a machine learning system that learns to react, and the image of the region extracted by the extracting unit and the wearing object are attached. A third likelihood value indicating the likelihood of the object not wearing the attachment is calculated using a machine learning system that has learned to react to the image of the predetermined region in the image of the object that has not been attached. Obtained by the imaging means based on the likelihood value calculating means and the first likelihood value, the second likelihood value, and the third likelihood value calculated by the likelihood value calculating means. Determination means for determining whether or not the object represented by the image is wearing the wearing object.

  In this way, a predetermined area that is assumed to be present when the object is assumed to be mounted is extracted from the image obtained by imaging, and the extracted area is extracted. And the image of the predetermined area in the image of the object with the attached object and the image of the predetermined area in the image of the object without the attached object. A machine learning system, a machine learning system that learns to react to an image of the predetermined area in the image of the object on which the wearing object is mounted, and the predetermined image in the image of the object that does not wear the wearing object Three likelihood values are obtained using a machine learning system that has learned to react to an image of a given region, and it is determined whether or not the object is wearing an attachment based on the three likelihood values. You Because the way, like the invention described in claim 1, the presence or absence of the mounting of the mounting object can be determined reliably. Further, in addition to the effect of the invention described in claim 1, in the invention described in claim 2, the determination is made by paying attention to an image of a predetermined region in which it is estimated that a wearing object exists, not an image of the entire object. As a result, the processing time is shortened.

  As in the invention described in claim 3, the determination means of the object determination device according to claim 1 or 2 is configured such that the first likelihood value, the second likelihood value, and the third likelihood value are determined. Based on the size relationship, it may be determined whether or not the object represented by the image obtained by the imaging unit is wearing the wearing object.

  Further, as in the invention described in claim 4, the determination unit of the object determination device according to claim 1 or 2 includes the ratio between the first likelihood value and the second likelihood value, and the first likelihood. Based on the ratio between the degree value and the third likelihood value, it may be determined whether or not the object represented by the image captured by the imaging unit is wearing the attachment. .

  According to a fifth aspect of the present invention, there is provided a program for an object represented by an image picked up by an image pickup means for picking up an object on a computer. A first likelihood value that calculates a first likelihood value indicating the likelihood of an object using a machine learning system that has learned to react to an image of an object wearing the object and an image of an object not wearing the attachment The degree value calculating step and the object represented by the image captured by the image capturing unit are responsive to the image captured by the image capturing unit and the image of the object mounted with the mounted object. A second likelihood value calculating step for calculating a second likelihood value indicating the likelihood of the object on which the object is mounted using the machine learning system that has been learned, and an image obtained by being imaged by the imaging means. For the object to represent An object that is not attached to the wearing object using an image obtained by being picked up by the imaging means and a machine learning system that has learned to react to an image of the object that is not attached to the wearing object Based on the third likelihood value calculating step for calculating the third likelihood value indicating the likelihood and the calculated first likelihood value, second likelihood value, and third likelihood value, the imaging means A determination step of determining whether or not the object represented by the image obtained by imaging is mounted on the mounted object.

  Even with such a program, the same effect as that of the object determination device according to claim 1 can be obtained because it operates in the same manner as the object determination device according to claim 1. Note that the three steps of the first likelihood value calculation step, the second likelihood value calculation step, and the third likelihood value calculation step may be performed in any order.

  According to a sixth aspect of the present invention, there is provided a program according to a sixth aspect of the present invention, when it is assumed that the object is attached to the object from an image obtained by imaging with an imaging means for imaging the object. An extraction step for extracting a predetermined area estimated to exist, and an object represented by an image captured by the imaging means, and an image of the extracted area and the wearing object are mounted Using a machine learning system that has learned to react to an image of the predetermined area in the image of the object and an image of the predetermined area in the image of the object that is not mounted A first likelihood value calculating step for calculating a first likelihood value indicating physicality; an object image represented by an image captured by the imaging unit; and an image of the extracted region; A second likelihood value indicating the likelihood of the object wearing the wearing object is calculated using a machine learning system learned to react to the image of the predetermined area in the image of the object wearing the kimono. For the object represented by the second likelihood value calculating step and the image obtained by imaging by the imaging means, the image in the extracted area and the image of the object not wearing the wearing object in advance A third likelihood value calculating step of calculating a third likelihood value indicating the likelihood of an object not wearing the wearing object using a machine learning system that has learned to react to an image of a predetermined area; Based on the calculated first likelihood value, second likelihood value, and third likelihood value, is the object represented by the image captured by the imaging means wearing the wearing object? A determination step for determining whether or not Is a program for executing the.

  Such a program also acts in the same manner as the object determination device according to claim 2, and therefore the same effect as the object determination device according to claim 2 can be obtained. Note that the three steps of the first likelihood value calculation step, the second likelihood value calculation step, and the third likelihood value calculation step may be performed in any order.

  As described above, according to the present invention, it is possible to determine with high reliability whether or not an object is wearing an attachment.

  FIG. 1 is a schematic configuration diagram of an object determination device 10 according to an embodiment of the present invention. The target object determination apparatus 10 according to the present embodiment captures a person's face as the target object, and determines whether or not an attachment (here, a mask) is attached to the photographed person's face.

  As shown in FIG. 1, the object determination device 10 includes a camera 14 for photographing a subject 12, an image capturing device 16, and an image processing device 18.

  The image capturing device 16 includes an image memory, captures image data obtained by photographing with the camera 14, and stores the image data in the image memory. The image capturing device 16 reads out the image data from the image memory and inputs it to the image processing device 18.

  FIG. 2 is a functional configuration diagram of the image processing apparatus 18.

  The image processing device 18 is a device that performs processing for determining whether or not a mask is attached to the face of the subject 12 based on the image data input from the image capturing device 16, and includes a face region detection unit 20, The likelihood calculation unit 22 and the determination unit 24 are included.

  The face area detection unit 20 detects the face area of the subject 12 from the image of the image data input from the image capturing device 16. There are various techniques for detecting the face area. In this embodiment, H. Rowley et al. Using a neural network (hereinafter referred to as NN) which is a kind of classifier having excellent pattern recognition capability. A face detection method (see “Neural Network-based Face Detection”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No. 1, 1998, pp. 23-38) may be used. However, since the face area detection unit 20 does not determine whether or not the subject 12 is wearing a mask, the face of the subject 12 is detected regardless of whether or not the subject 12 is wearing a mask. NN is learned beforehand so that it can be done.

  More specifically, an NN configured by connecting an input layer, an intermediate layer, and an output layer is caused to learn a connection relationship using known input / output data. Here, by a learning method called back propagation, as shown in FIG. 3 (A), a large number of facial images without a mask, and masks of various shapes as shown in FIG. Excitement learning is performed by giving a teacher signal 1 so that the output layer of the NN outputs a signal of 1 for a large number of face images worn. On the other hand, a number of images completely different from the face (for example, landscape images) are prepared as non-face images, and suppression learning is performed by giving a teacher signal 0 to these images. This NN is referred to as a mask / non-mask face reaction NN. It should be noted that the output value of the mask / non-mask face reaction NN is learned to take a value in the range of 0 to 1.

  The likelihood calculating unit 22 has a first likelihood value p1 indicating the face-likeness regardless of whether or not the mask is attached to the region detected by the face region detecting unit 20, and a first likelihood indicating the face-likeness wearing the mask. A two-likelihood value p2 and a third likelihood value p3 indicating the likelihood of a face not wearing a mask are calculated.

  Specifically, for the first likelihood value p1 indicating the facial appearance regardless of whether or not a mask is worn, the mask / non-mask face reaction NN used in the face area detection of the face area detection unit 20 is expressed as follows: The output value when applied to the detected face area is defined as a first likelihood value p1. Accordingly, the first likelihood value p1 takes a value in the range of 0 to 1.

  In addition, for the second likelihood value p2 indicating the likelihood of a face wearing a mask, the face area detecting unit 20 uses an NN learned to react to an image of the face wearing the mask (hereinafter referred to as mask face reaction NN). The calculation is applied to the detected face area. Note that the mask face reaction NN also uses a number of masks having various shapes as shown in FIG. 3B by using the back-propagation method in the same manner as the mask / non-mask face reaction NN. Performing so-called excitement learning with a teacher signal of 1 for a face image, and at the same time, suppressing learning with a teacher signal of 0 for a number of face images not wearing a mask as shown in FIG. To do. By performing such learning, the output layer of the mask face reaction NN outputs a high output value for a face image wearing a mask and a low output value for a face image not wearing a mask. It can be expected that Note that the output value of the mask face reaction NN is learned to take a value in the range of 0 to 1. In the present embodiment, the output value when the mask face reaction NN is applied to the face area detected by the face area detection unit 20 is used as it is as the second likelihood value p2. Therefore, the second likelihood value p2 takes a value in the range of 0 to 1.

  For the third likelihood value p3 indicating the likelihood of a face not wearing a mask, NN (non-mask face reaction NN) learned to react only to a face image not wearing a mask is used as a face region. The calculation is applied to the face area detected by the detection unit 20. The non-mask face reaction NN is also applied to a large number of face images not wearing a mask as shown in FIG. 3A by using the back-propagation method in the same manner as the mask / non-mask face reaction NN. On the other hand, excitement learning is performed by setting the teacher signal to 1, and at the same time, suppression learning is performed by setting the teacher signal to 0 for a large number of face images wearing masks of various shapes as shown in FIG. To do. By performing such learning, the output layer of the non-mask face reaction NN outputs a high output value for a face image not wearing a mask and a low output value for a face image wearing a mask. You can expect to be. It should be noted that the output value of the non-mask face reaction NN is learned to take a value in the range of 0 to 1. In the present embodiment, the output value when the non-mask face reaction NN is applied to the face area detected by the face area detection unit 20 is used as it is as the third likelihood value p3. Accordingly, the third likelihood value p3 takes a value in the range of 0 to 1.

  The determination unit 24 uses the first likelihood value p1, the second likelihood value p2, and the third likelihood value p3 calculated by the likelihood calculation unit 22 to represent an object captured by the camera 14 It is determined whether a mask is attached to the face of the person 12.

  Each component constituting the image capturing device 16 and the image processing device 18 described above is realized by a computer including a CPU, a RAM, and a ROM. That is, the CPU realizes the functions of the above-described components by executing a program stored in a ROM or a predetermined storage device, and the processing described below is performed. Moreover, each component may be comprised by a separate computer, and may be comprised by one computer.

  Next, details of the mask mounting determination process executed in the present embodiment will be described.

  FIG. 4 is a flowchart showing the flow of the mask attachment determination process performed by the image capturing device 16 and the image processing device 18 of the object determination device 10.

  First, in step S100, the image capturing device 16 captures image data obtained by imaging with the camera 14 and temporarily stores it in the image memory. Then, the image data is read from the image memory and input to the face area detection unit 20 of the image processing device 18. In step S102, the face area detection unit 20 detects a face area from the image represented by the image data input from the image capturing device 16 using the mask / non-mask face reaction NN.

  In step S104, the likelihood calculating unit 22 calculates the first likelihood value p1 by applying the mask / non-mask face reaction NN to the face area detected by the face area detecting unit 20. As described above, in the mask / non-mask face reaction NN, 1 is output for a face image regardless of whether or not a mask is attached, and 0 is output for other images. Therefore, the first likelihood value p1 is a numerical value in the range of 0 to 1.

  In step S <b> 106, the likelihood calculating unit 22 calculates the second likelihood value p <b> 2 by applying the mask face reaction NN to the face area detected by the face area detecting unit 20. As described above, the mask face reaction NN is learned so that 1 is output for a face image wearing a mask, and 0 is output for other images. The second likelihood value p2 is a numerical value in the range of 0 to 1, and the second likelihood value p2 when applied to a face image wearing a mask is the second value when applied to a face image not wearing a mask. It can be expected that the value becomes larger than the likelihood value p2.

  In step S108, the likelihood calculating unit 22 calculates the third likelihood value p3 by applying the non-masked face reaction NN to the face area detected by the face area detecting unit 20. As described above, the non-mask face reaction NN is learned so that 1 is output for a face image without a mask and 0 is output for other images. Therefore, the third likelihood value p3 is also a numerical value in the range of 0 to 1, and the third likelihood value p3 when applied to the face image not wearing the mask is applied to the face image wearing the mask. It can be expected that the value becomes larger than the third likelihood value p3 at that time.

  In step S110, the determination unit 24 determines whether the subject 12 is wearing a mask based on the first likelihood value p1, the second likelihood value p2, and the third likelihood value p3.

  If the target face image is wearing a mask, the second likelihood value p2 can be expected to be close to 1, and the third likelihood value p3 can be expected to be close to 0. On the other hand, since the first likelihood value Pl is an output of the mask / non-mask face reaction NN learned to react to both the face image with the mask and the face image without the mask, the second likelihood value p2 Although not as high as possible (or possibly as high as the second likelihood value p2), it can be expected to be larger than the third likelihood value p3. Therefore, when the first likelihood value p1, the second likelihood value p2, and the third likelihood value p3 satisfy the magnitude relationship shown in the following equation (1), the determination unit 24 masks the detected face. It is determined that it is attached.

  p2 ≧ p1> p3 (1)

  On the other hand, if the target face image is not wearing a mask, the second likelihood value p2 is close to 0, the third likelihood value p3 is close to 1, and the first likelihood The value Pl is not as high as the third likelihood value p3 as in the case described above (or may be the same value as the third likelihood value p3), but the second likelihood value. It can be expected to be a large value compared to p2. Therefore, when the first likelihood value p1, the second likelihood value p2, and the third likelihood value p3 satisfy the magnitude relationship shown in the following formula (2), the determination unit 24 masks the detected face. It is determined that it is not installed.

  p3 ≧ p1> p2 (2)

  Also, if the magnitude relationship among the first likelihood value p1, the second likelihood value p2, and the third likelihood value p3 does not satisfy either of the expressions (1) and (2), the presence / absence of a mask is determined. Therefore, it is determined that there is not sufficient accuracy, and the determination is suspended.

  Thus, a mask / non-mask face reaction NN that reacts to a face image regardless of whether or not a mask is attached, a mask face reaction NN that reacts only to a face image with a mask attached, and only a face image without a mask attached By using three non-masked face reactions NN that react to 3 and obtaining three likelihood values, comparing the likelihood values of the three likelihood values makes it possible to make a more probable decision and to keep false detection low. Can do.

  The determination result of the mask wearing determination process can be used, for example, when a face state estimation process is performed. For example, when it is determined that the mask is worn, it is difficult to detect the feature amount of the mouth and nose from the image. Therefore, it is only necessary to perform a face state estimation process using another feature amount. Stable face state estimation can be realized by efficiently switching the processing procedure depending on whether or not the camera is worn.

  In addition, although the example which determines the presence or absence of mask wearing based on the magnitude relationship of the 1st likelihood value p1, the 2nd likelihood value p2, and the 3rd likelihood value p3 was demonstrated here, the 1st likelihood value The presence / absence of wearing a mask may be determined based on the ratio between p1 and the second likelihood value p2 and the ratio between the first likelihood value p1 and the third likelihood value p3.

Specifically, first, the ratio r1 between the first likelihood value p1 and the second likelihood value p2 is obtained by the following equation (3), and the ratio between the first likelihood value p1 and the third likelihood value p3 is calculated. r2 is obtained by the following equation (4).
r1 = second likelihood value p2 / first likelihood value p1 (3)
r2 = first likelihood value p1 / third likelihood value p3 (4)

Then, it is determined whether or not the following relational expression (5) is satisfied.
r1> th1 and r2> th2 (5)

  Here, th1 and th2 are threshold values for determination, and have a relationship of 0 <th1 ≦ th2. If the relational expression (5) is satisfied, the detected face image is determined to be a face wearing a mask.

Similarly, the ratio r3 between the first likelihood value p1 and the second likelihood value p2 is obtained by the following equation (6), and the ratio r4 between the first likelihood value p1 and the third likelihood value p3 is calculated as follows: It calculates | requires by Formula (7).
r3 = third likelihood value p3 / first likelihood value p1 (6)
r4 = first likelihood value p1 / second likelihood value p2 (7)

Then, it is determined whether or not the following relational expression (5) is satisfied.
r3> th3 and r4> th4 (8)

  Here, th3 and th4 are thresholds for determination, and have a relationship of 0 <th3 ≦ th4. When the relational expression (8) is satisfied, the detected face image is determined as a face not wearing a mask.

  If none of the relational expressions (5) and (8) correspond, it may be determined that there is not sufficient accuracy to determine the presence / absence of a mask, and the determination may be suspended.

  Thus, even if the ratio is used, whether or not the mask is mounted can be determined with high reliability as described above.

  In the present embodiment, an example of determining whether or not a mask is mounted has been described. However, even if it is determined whether or not sunglasses are mounted, the same processing as in the above embodiment may be performed. In other words, there are prepared an NN that reacts to a face image regardless of whether or not sunglasses are worn, an NN that reacts only to a face image wearing sunglasses, and an NN that reacts only to a face image not wearing sunglasses. The same effect can be obtained by obtaining three likelihood values using these three NNs and using them for the determination.

  Moreover, you may comprise so that the presence or absence of mounting | wearing of both a mask and sunglasses may be determined. That is, an NN that reacts to a face image regardless of whether or not a mask and sunglasses are worn, an NN that reacts only to a face image that wears both a mask and sunglasses, and a face that does not wear both a mask and sunglasses Similar effects can be obtained by preparing an NN that reacts only to an image, obtaining three likelihood values using these three NNs, and using them for determination.

  In the above-described embodiment, an example in which three different NNs are used to determine whether or not a mask is attached has been described. However, the present invention is not limited to this, and a machine learning system other than NN such as a support vector machine may be used. Good.

  In the above-described embodiment, the example in which the NN is applied to the entire face image to determine the wearing of the mask has been described. However, the wearing of the mask is determined using only the image of the lower half of the face. You may make it perform. Hereinafter, a modified example in which the mask wearing determination is performed using only the image of the lower half area of the face will be described.

  The configuration of the object determination device of this modification is the same as that of the object determination device 10 of the above embodiment. However, the following components function as described below.

  The face area detection unit 20 of the image processing device 18 detects the face area of the subject 12 from the image represented by the image data obtained by being photographed by the camera 14 as in the above embodiment. Further, image data of an image in the lower half area when the detected face area is divided into upper and lower halves is cut out and output to the likelihood calculating unit 22. The lower half area is an area where it is estimated that the mask exists in the face image when it is assumed that the subject 12 wears the mask.

  The likelihood calculation unit 22 attaches a first likelihood value p1 indicating the likelihood of a face to the image indicated by the image data detected and cut out by the face region detection unit 20 regardless of whether or not the mask is attached, and the mask. A first likelihood value p2 indicating the likelihood of a face and a third likelihood value p3 indicating the likelihood of a face not wearing a mask are calculated.

  However, the mask / non-mask face reaction NN, the mask face reaction NN, and the non-mask face reaction NN for obtaining the first likelihood value p1, the second likelihood value p2, and the third likelihood value p3 are not masked. Learning is performed using the lower half image of the wearing image and the lower half image of the mask wearing image.

  That is, the mask / non-mask face reaction NN uses the back-propagation method, as shown by the broken line in FIG. As indicated by a broken line in FIG. 5B, excitement learning is performed with a teacher signal as 1 for the lower half of many facial images wearing masks, and at the same time, a non-face completely different from the face The suppression learning is performed on the image with the teacher signal set to 0.

  In addition, the mask face reaction NN is also applied to the lower half of many face images wearing masks by using the back-propagation method as indicated by the broken line in FIG. At the same time as excitement learning is performed with a signal of 1, teacher signals are applied to the lower half images and non-face images of a large number of face images not wearing masks, as shown by the broken lines in FIG. Suppression learning is performed as 0.

  Further, the non-mask face reaction NN also uses the back-propagation method, as shown by the broken line in FIG. At the same time as performing the excitement learning with the teacher signal as 1, the teacher signal is applied to the lower half images and non-face images of the many face images wearing masks as shown by the broken lines in FIG. Suppression learning is performed with 0 being 0.

  Note that the NN used when the face area detection unit 20 detects the face area is learned so that it reacts to the entire face image regardless of whether or not a mask is attached in order to detect the entire face area. Therefore, in this modified example, the NN used by the face area detection unit 20 and the NN used by the likelihood calculation unit 22 to calculate the first likelihood value p1 are separately prepared and learned.

  FIG. 6A is a flowchart showing the flow of the mask attachment determination process performed by the image capturing device 16 and the image processing device 18 in this modification.

  Steps S200 to S202 are processing for detecting the face area by taking in the image data obtained by the camera 14 (see FIG. 6B), and will be described in the same manner as steps S100 to S102. Is omitted.

  In step S <b> 204, the face area detection unit 20 divides the detected face area of the target person 12 into upper and lower halves from the image represented by the image data input from the image capturing device 16, and the lower half area image data. Is output to the likelihood calculating unit 22 (see FIG. 6C).

  In step S206, the likelihood calculation unit 22 applies the mask / non-mask face reaction NN to the image indicated by the image data of the lower half region cut out by the face region detection unit 20, and uses the first likelihood value p1. calculate.

  In step S208, the likelihood calculating unit 22 applies the mask face reaction NN to the image indicated by the image data of the lower half region cut out by the face region detecting unit 20, and calculates the second likelihood value p2.

  In step S210, the likelihood calculating unit 22 applies the non-masked face reaction NN to the image indicated by the image data of the lower half region cut out by the face region detecting unit 20, and calculates the third likelihood value p3. .

  In step S212, the determination unit 24 determines whether the subject 12 is wearing a mask based on the first likelihood value p1, the second likelihood value p2, and the third likelihood value p3. Since the determination method is the same as that in step S110, description thereof is omitted.

  In this way, the processing time can be shortened by determining whether or not the mask is mounted using only the image of the lower half of the face.

  In this modification, the case where the presence / absence of wearing a mask is determined has been described, but the presence / absence of wearing sunglasses may be determined. In this case, the upper half area of the face image is cut out in the same procedure as in the above-described modification, and the sunglasses face / non-sunglass face reaction NN, the sunglasses face reaction NN, the non-sunglass face reaction NN, which are learned based on the upper half area of the face image. The sunglasses wearing correspondence NN is used to determine whether to wear sunglasses.

  In the above-described embodiment and modification, an example in which likelihood values are calculated in the order of the first likelihood value p1, the second likelihood value p2, and the third likelihood value p3 has been described. The calculation order is not particularly limited.

  In the above-described embodiment and modification, an example of determining whether a mask or the like is mounted on a person's face has been described. However, the present invention is not limited to this, and the present invention can be applied to various objects. It is.

It is a schematic block diagram of the target object determination apparatus which concerns on embodiment of this invention. It is a functional block diagram of an image processing apparatus. (A) is a figure which shows an example of the face image which is not wearing the mask, (B) is a figure which shows an example of the face image which is wearing the mask. It is a flowchart which shows the flow of the mask mounting | wearing determination process performed by the image capture device and image processing apparatus of a target object determination apparatus. (A) is a figure which shows the example of the cut-out area | region in the case of cutting out the lower half of the face image which is not wearing the mask, (B) is the case where the lower half of the face image which is wearing the mask is cut out It is a figure which shows the example of a cut-out area | region. (A) is a flowchart showing a flow of mask wearing determination processing performed by an image capturing device and an image processing device in a modified example, and (B) is a diagram showing an example of a face area detection result; C) is a diagram showing an image of a lower half area cut out from an image of a face area.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Target object determination apparatus 12 Subject person 14 Camera 16 Image capturing apparatus 18 Image processing apparatus 20 Face area detection part 22 Likelihood calculation part 24 Determination part

Claims (6)

An imaging means for imaging the object;
For an object represented by an image obtained by imaging with the imaging means, an image obtained by imaging with the imaging means, an image of an object with an attached object, and an object without the attached object The first likelihood value indicating the likelihood of an object is calculated using a machine learning system that has learned to respond to the image of the image, and the image obtained by being picked up by the image pickup means and the object to which the attachment is attached Calculating a second likelihood value indicating the likelihood of the object wearing the wearing object using a machine learning system learned to react to an image of the object, and an image obtained by being imaged by the imaging unit; A likelihood value for calculating a third likelihood value indicating the likelihood of the object not wearing the attachment using a machine learning system learned to react to an image of the object not wearing the attachment A calculation means;
Based on the first likelihood value, the second likelihood value, and the third likelihood value calculated by the likelihood value calculation means, the object represented by the image captured by the imaging means is the wearing Determining means for determining whether or not an object is attached;
An object determination apparatus including: An imaging means for imaging the object;
An extraction means for extracting a predetermined area that is estimated to be present when the object is assumed to be mounted from the image captured by the imaging means;
For the object represented by the image obtained by the imaging means, the image of the area extracted by the extraction means, the image of the predetermined area in the image of the object to which the attachment is attached, and the Calculating a first likelihood value indicating the likelihood of an object using a machine learning system that has learned to react to an image of the predetermined region in an image of an object that is not mounted; An object wearing the attachment using an image of the region extracted by the means and a machine learning system learned to react to the image of the predetermined region in the image of the object wearing the attachment A second likelihood value indicating physicality is calculated and reacts to the image of the region extracted by the extraction means and the image of the predetermined region in the image of the object not wearing the attachment. And likelihood value calculation means for calculating a third likelihood value indicating object likeness not wearing the wearable object by using the learned machine learning system as,
Based on the first likelihood value, the second likelihood value, and the third likelihood value calculated by the likelihood value calculation means, the object represented by the image captured by the imaging means is the wearing Determining means for determining whether or not an object is attached;
An object determination apparatus including: The determination unit is configured such that an object represented by an image captured by the imaging unit is based on a magnitude relationship between the first likelihood value, the second likelihood value, and the third likelihood value. The object determination apparatus according to claim 1, wherein it is determined whether or not an object is mounted. The determination means is obtained by being imaged by the imaging means based on a ratio between the first likelihood value and the second likelihood value and a ratio between the first likelihood value and the third likelihood value. The target object determination apparatus according to claim 1, wherein the target object represented by the displayed image determines whether the target object is mounted. On the computer,
For an object represented by an image obtained by imaging with an imaging means for imaging the object, an image obtained by imaging with the imaging means, an image of the object with the attached object, and the attached object A first likelihood value calculating step of calculating a first likelihood value indicating the likelihood of an object using a machine learning system that has learned to react to an image of an object that is not mounted;
A machine learning system in which an object represented by an image captured by the image capturing unit is learned to react to an image captured by the image capturing unit and an image of the object mounted with the mounted object. A second likelihood value calculating step for calculating a second likelihood value indicating the likelihood of the object wearing the wearing object using
For the object represented by the image captured by the imaging unit, the machine learned to react to the image captured by the imaging unit and the image of the object not mounted with the mounted object A third likelihood value calculating step of calculating a third likelihood value indicating the likelihood of an object not wearing the wearing object using a learning system;
Based on the calculated first likelihood value, second likelihood value, and third likelihood value, is the object represented by the image captured by the imaging means wearing the wearing object? A determination step for determining whether or not;
A program for running On the computer,
Based on an image obtained by imaging with an imaging means for imaging the object, a predetermined region that is estimated to be present when the object is mounted is assumed when the object is mounted. An extraction step to extract;
For the object represented by the image captured by the imaging means, the image of the extracted area, the image of the predetermined area in the image of the object to which the wearing object is attached, and the wearing object First likelihood value calculation for calculating a first likelihood value indicating the likelihood of an object using a machine learning system that has learned to react to an image of the predetermined region in an image of the object that is not attached Steps,
The object represented by the image captured by the imaging unit is responsive to the image of the extracted area and the image of the predetermined area in the image of the object to which the wearing object is attached. A second likelihood value calculating step of calculating a second likelihood value indicating the likelihood of the object wearing the wearing object using the learned machine learning system;
The object represented by the image captured by the imaging means reacts to the image of the extracted area and the image of the predetermined area in the image of the object not wearing the attachment. A third likelihood value calculating step of calculating a third likelihood value indicating the likelihood of an object not wearing the attachment using the machine learning system learned as described above,
Based on the calculated first likelihood value, second likelihood value, and third likelihood value, is the object represented by the image captured by the imaging means wearing the wearing object? A determination step for determining whether or not;
A program for running

Images