JP5133677B2 - Monitoring system - Google Patents

Description

  The present invention relates to a remote monitoring system, and in particular, determines the state by image recognition of an image obtained by capturing the expression of a baby, an infant or an elderly person from a camera, and makes an equivalent determination even if the observer does not always see the state of the infant. The present invention relates to a monitoring system for sequentially providing information to a monitor.

Conventionally, various techniques for monitoring an object to be monitored from a place where a monitor is remote have been disclosed. In such a remote monitoring system, it is most basic that a monitoring camera is installed near the monitoring target and the monitoring target is monitored by a monitor installed at a location away from the monitoring person. In -129590, when surrounding sounds are collected and the sound pressure and frequency are determined to be abnormal, the monitoring camera is turned and the image information of the point where the abnormal sound is generated is transmitted to the monitor monitored by the monitor. An error is reported to the monitoring person by displaying it.

In Japanese Patent Laid-Open No. 55-132165, a sound collecting device installed near a monitoring target and a loudspeaker installed near the monitoring person can listen to sounds around the monitoring target via a telephone extension. ing.

In addition, the abnormality notification system disclosed in Japanese Patent Application Laid-Open No. 10-11474 provides a threshold value for an abnormal value by a sensor such as sound, temperature, microphone, body temperature, etc., and compares it with a set value to monitor the abnormality. It was a system to notify the person.

As a similar conventional technique related to the present invention, there is Japanese Unexamined Patent Publication No. 2001-92963 as a face matching technique. However, since the present invention is not a matching but an analysis of facial expressions, it is slightly different. Japanese Patent Laid-Open No. 2001-43345 is a facial expression recognition technique. This does not differ greatly in terms of expression and emotion, such as laughter, anger, joy, and sadness. Use time-series data such as transitions. As for face detection, obtaining a front image is related to recognition accuracy, and as a technique therefor, JP 2006-259900 A and JP 2004-46399 A can be cited.

Japanese Patent Application Laid-Open No. 11-99140, which is a detection device for detecting a sleeping state or higher, detects abnormalities in facial movement using moving image processing and pattern matching and reports them.
JP-A-1-129590 JP 55-132165 A Japanese Patent Laid-Open No. 10-11474 JP 2001-92963 A JP 2001-43345 A JP 2006-259900 A JP 2004-46399 A JP-A-11-99140 P. Viola and M. Jones: Rapid object detection using a boosted cascade of simple features, Proc. On CVPR, Vol. 1.Pp 511-520 2001 Paul Ekman, Wallace V. Friesen, Kudo Riki: “Introduction to facial expression analysis”, Seishin Shobo, 1987

In the above-described prior art, efforts are made to improve the accuracy of abnormality monitoring by individual use or combined use of sensors such as sound, temperature, microphone, body temperature, etc., but in normal conditions, the temperature and body temperature suddenly increase. It is hard to think about whether it will change, and what actually functions depends heavily on the sound information.
In addition, depending on sound information, the influence of ambient information and TV sound is large, and it is difficult to improve accuracy. It can also be assumed that there is no sound.
As described above, since the accuracy of the abnormality report state cannot be set very finely, there are many misreports, and the burden on the monitor is still increased.

Regarding the prior art related to face authentication, this system recognizes facial expressions, but does not look at the transition state but performs sequential processing. In addition, as a means for obtaining a front image when the face cannot be detected from the camera due to the orientation of the face of the person being monitored , this technique is based on the movement of the sound or object. There is a place to draw attention. In addition, the system that detects abnormalities in the sleeping state detects the movement and time of the person being monitored from the moving image, and determines whether the movement is abnormal based on this, but it is determined under limited conditions. Therefore, it can be said that it is insufficient to determine the state of the infant.

However, if the state of the person being monitored can be determined by determining the state based on facial expressions using image recognition, high-accuracy information can be obtained by reporting the image only when there is an abnormality in the expression. Can be sent.

An object of the present invention is to provide a monitoring system that determines a facial expression of a monitoring target person with high accuracy by image recognition and learning, and reports only when there is an abnormality in the monitoring target person.

Monitoring system of the present invention, a mobile terminal monitored person carries, is composed of a communicable abnormality notification device to another in the installed the mobile terminal and the wireless near monitored, the abnormality notification device, monitoring Face detecting means for detecting the target face, face state judging means for judging the face state from the detected face, sound collecting means for collecting surrounding sounds, and means for judging abnormality of surrounding sounds from the detected sounds with the door, and a control means for transmitting an abnormality notification signal when it is determined that determines based on abnormality prestored predetermined value of the abnormality or ambient sound of the face state is abnormal to the portable terminal, wherein When the abnormality reporting device detects a face, if the face cannot be detected from the camera due to the orientation of the face of the person being monitored, the movement caused by the movement of the cradle connected to the abnormality reporting device or the mother from the speaker Call Registration of such or, by the motion of the interrogation sound and things from the mobile terminal, by drawing the attention of the person to be monitored, to suited the face of the person to be monitored in the direction of the camera. Further, the reaction of that time, with the means for determining the status of the person to be monitored, wherein the portable terminal, a display unit and an audio unit, abnormality notification on the display means based on the abnormality notification signal received from the abnormality notification device characterized in that and a control means for generating a warning sound from the display and the audio means.

Further, the mobile terminal periodically acquires a history of notifications from the abnormality notification device with video and sound at settable intervals, and is displayed in a table or graph as the state of the monitoring target person to indicate the health state of the monitoring target person, It is preferable to provide means capable of grasping at least one of the meal state and the excretion state.

Further, it is preferable that the mobile terminal includes a means for checking the previous history at the time of reporting from the abnormality reporting device or at any time so as to be able to learn the detected state and to improve the accuracy of the state determination.

The face detection means includes means for controlling with a single-axis or two-axis motor, and detects the position and direction of the face of the person to be monitored, and includes a face tracking means for control so that it is always captured in the camera. And good.

Further, the face detection means comprises means for determining that the face detection state is at least one of a “presence state”, a “absence state”, a “futon cover state”, and a “lie down state”, It is preferable to provide means for reporting an abnormality when the monitoring subject is in an undesirable state.

In addition, if the face state determining means, by the pre-Symbol face detection means of the "presence state", "vigilance" in the face, "sleep state", "crying state", "mood is good state", "Normal The state of the monitoring subject is provided with means for estimating the state of the monitoring subject along with the information on the movement of the face by estimating the basic state such as the state of ”,“ the state of laughing ”,“ the state of vomit ” It is preferable to provide a means for reporting an abnormality when is unfavorable.

Further, the portable terminal includes means for grasping the state of the person being monitored with a regular daily rhythm, statistically displaying the management of sleep time such as a nap, and urging attention when there is a shift. good.

In addition, the abnormality reporting device may include means for acquiring a front image by drawing the attention of the monitoring subject when the face of the monitoring subject is in a blind spot from the camera during face detection.

In addition, the abnormality reporting device may detect the breathing of the monitoring target person during sleep based on a moving part on the screen.

In the present invention, when the face cannot be detected from the camera due to the face orientation of the person to be monitored, the movement caused by the operation of the gargle connected to the anomaly reporting device, the mother calling from the speaker, etc. Or, it is equipped with means to draw the attention of the monitoring subject by the calling sound from the mobile terminal or the movement of the object, so that the face of the monitoring subject is directed toward the camera. The portable terminal is configured to display an abnormality report on the display unit based on an abnormality report signal received from the abnormality report device and generate a warning sound from the voice unit. by a control means for, monitored person, automatically determines the face information which is information to be monitored, the monitoring person without watching the person to be monitored, and gaze and And giving the same sense of security, there is an effect to reduce the labor and burden to watch. In addition, since the learning function is provided, it is possible to reduce the state estimation error due to the difference between individuals of the monitoring target each time it is used, and it is possible to improve the accuracy.

Hereinafter, an embodiment of the present invention will be specifically described with reference to the drawings.

1 and 2 are block diagrams showing an embodiment of a monitoring system according to the present invention. FIG. 1 shows an abnormality notification device installed near an infant or an elderly person who is a monitoring target. FIG. 2 shows a portable terminal device that a supervisor keeps at hand.

The abnormality notification device in FIG. 1 is installed so that the camera 5 faces the person to be monitored. The shooting range 35 is adjusted according to the movement state of the monitoring subject. For example, if the newborn has little movement, the face may occupy half the screen. The input from the camera 5 is converted into a predetermined format by a digital signal by the video input unit 4 and is taken into the memory 3. Here, the CPU 2 determines whether or not there is a face in the image.

The face detection method is performed by a face detection method using a typical Adaboost method as in Reference 1. Face detection is performed, and in the initial state, the face is always placed and detected.
After the detection is completed, the position of the face is followed on the screen, and the camera control table is adjusted from the CPU 2 through the camera controller 6 to control the posture of the camera 5 so that it always becomes the center of the screen. Alternatively, the motor control may be omitted, and a warning may be issued if a face cannot be detected from the screen. Here, when the direction of the face of the person to be monitored is the same as the direction of the camera, the face cannot be detected from the camera.

In this case, it is determined whether the face is not detected based on the position where the detection is lost or whether the face is out of the field of view of the camera or the face direction is the same in the camera field of view. In the former case, the camera controller 6 may be controlled by driving the camera controller 6 to operate the camera in the direction in which the previous face was detected from the camera center, or it may be recognized that the camera is out of the field of view. Just be there. These flows are shown in FIG. Each time an image is input 23, face detection 24 is performed as shown in FIG. 4A, and if a face is detected within the camera field of view or within a specified face detection range (face detection? 25), the face center is detected. A coordinate 36 is obtained 26. At this time, since the face is detected, the face is present on the screen, and the “exist” state 27 is confirmed to determine that the face is in a normal state 28. If face detection is not possible, the history of face center coordinates such as the previous time and the previous time is judged, the face movement vector amount is calculated 29, the movement amount is large, and the current position has moved out of the detection range. Is estimated as “absent 30”. This corresponds to (c) of FIG. When the movement amount is small, the motion 31 around the face is detected in the motion detection area 44, and the optical flow by the moving image processing in the motion detection area is detected as shown in FIG. c) 41-42-43) is determined to be in the “futon cover 32” state, and if there is little movement, it is determined in 38-39-40) in FIG. In the case of “absent”, “futon-covered”, and “turned over” states, it is determined that the state is abnormal 34, and if a state where face detection cannot be performed continues for a certain period of time, a time is set in advance, ”Report“ futon cover ”and“ turn over ”.

In the latter case, by registering the movements of gargles connected to the error reporting device, calling the mother from the speakers, or calling the monitor person's attention by the movement of the calling sound or things from the mobile terminal, , Ru skein toward the face of the person to be monitored in the direction of the camera. Further, the state of the monitoring subject is determined based on the reaction at that time.

When the face detection means can detect the face and determines that the state is “existing”, the state is further determined according to the flow shown in FIG. When the face position is detected, the state of each organ is checked within the face range. Mouth detection is performed and the condition around the mouth is checked to check whether there is any abnormality around the mouth. If it is different from the normal state, it is determined that the foreign object is detected, and it is determined that the state is “discharging”. In the case of “Vomiting state”, it is determined as an abnormal state, and the terminal is notified.
When detecting each organ, the feature point 64 of FIG. 8 is searched, the positional relationship is calculated from the feature vector 65, and the facial expression and state are analyzed. Further, as shown in FIG. 7, the ratio between the vertical length 58 and the horizontal length 59 of the eyes, the vertical length 62 of the mouth, the ratio of the horizontal length 61, the ratio of the vertical length 62 and the horizontal length 63 of the mouth and eye rectangles, and the positional relationship. You may see.

When there is nothing around the mouth, the open / closed state of the eye is examined from the feature amount of the eye part in FIG. It may be detected by the aspect ratio of the eye feature vector. Here, when it comes out that the eyes are closed, it is determined as “sleep state”, otherwise it is determined as “wake state” and it is determined whether or not it is a smile. Other “crying”, “feeling good”, “normal”
Judgment is made from the relationship between the vectors of feature points in FIG. This takes in advance data from a large number of unspecified subjects and correlates the relationship between the feature quantity and the “state”. When in use, the state is learned by the supervisor. This data is fed back to improve the accuracy of face condition detection.

On the terminal side, the data history so far can be displayed and confirmed, for example, in units of one day as shown in FIG. It may be possible to check on a weekly or monthly basis. In the example of FIG. 9, the state display items 66 are “absent”, “woke up”, “sleeping”, “futon cover”, “vomit”, and “collapse”, but this can be arbitrarily set. The event items are also “milk” and “diapers”, but these can be arbitrarily set. The state can be displayed in a graph 68, and the event can also be displayed 69 with time and number of times. In this way, if milk or diaper replacement information is entered, it can be used as a guideline for the next time. It can be entered as 71 in the input mode of the input yes terminal. If the status display is wrong, a learning input 70 can be made.

In the sleeping state, it can be confirmed whether there is any abnormality by looking at the breathing of the subject. In the case of an infant as shown in FIG. 10, since abdominal breathing is the main, after detecting the position of the face, it is possible to see the breath by examining the movement 72 in the vicinity of the chest abdomen at a certain distance. it can. Whether or not the person is breathing may be determined by simply determining whether or not the movement is moving by moving image processing.

In addition to face detection, the movement and face orientation of the monitoring subject are detected by moving image analysis to determine whether the infant who is the monitoring subject is in an active state or a resting state. In the sleep state, there is no particular problem. However, since attention such as an injury is necessary in the active state, it is notified that the active state has been entered.

At the same time, an audio signal is detected, and a notification is given when the surrounding sound changes abruptly.

Note that the mobile terminal of this embodiment may use a device having an existing communication means such as a mobile phone. In the embodiment, although described as a monitoring system for infants, it can be used as a monitoring system for sick people, care recipients, etc. It is also possible to inspect the state of the employee's condition. In this case, it may be a system that checks the fatigue level etc. and encourages a break in a certain time, or the driver's condition while driving You can check the status of the workers in the factory, you can check the automatic status of watching TV, and it can be used for various situations such as checking human fatigue and status.

  INDUSTRIAL APPLICABILITY The present invention can be used in a monitoring system that can easily check the state of an infant or the like from a remote location and can notify a monitor of an abnormal state as necessary.

Component parts of an abnormality notification device according to an embodiment Component parts of abnormal reception terminal according to one embodiment Flow of face detection state in one embodiment Face position history vector of one embodiment Motion detection area around the face in one embodiment Flow of face detection state in one embodiment Features of facial organs according to one embodiment Feature amount of facial organ in one embodiment Display example of information terminal according to one embodiment Chest motion detection range of one embodiment

Explanation of symbols

1 Abnormality reporting device 2 CPU
3 Memory 4 Video input 5 Camera 6 Camera controller 7 Camera control stand 8 Audio input 9 Microphone 10 Transceiver 11 Antenna 12 Abnormality notification device 13 CPU
14 Memory 15 Video input 16 Camera 17 Camera controller 18 Camera control table 19 Audio input 20 Microphone 21 Transceiver 22 Antenna 23 Image input 24 Face detection 25 Face detection?
26 Face center coordinate acquisition 27 Existence 28 Normal state determination 29 Face coordinate history movement vector large?
30 Absence 31 Movement around the face?
32 Duvet cover 33 Roll over 34 Abnormal state determination 35 Face detection area 36 Face center coordinates 37 Detection face area 38 n-1st detection face center position history 39 nth detection face center position history 40 n + 1th detection face center position history 41 n-1st detected face center position history 42 nth detected face center position history 43 n + 1th detected face center position history 44 motion detection area 45 presence 46 mouth detection 47 What is around the mouth?
48 Foreign object detection 49 Exhalation state 50 Abnormal state judgment 51 Eye open state 52 Sleep state 53 Smile?
54 Laughing State 55 Other State 56 Feature Learning 57 Normal State Judgment 58 Eye Vertical Length 59 Eye Horizontal Length 60 Mouth Vertical Length 61 Mouth Horizontal Length 62 Rectangle Vertical Length Created by Eye and Mouth 63 Horizontal length of eyes and mouth 64 Feature point 65 Feature vector 66 State display item 67 Event display item 68 State graph 69 Event graph 70 State learning input 71 Event input 72 Chest movement

Claims (9)

A mobile terminal monitored person carries, is composed of a communicable abnormality notification device to another in the installed said portable terminal wirelessly near monitored,
The abnormality reporting device includes a face detection unit that detects a face to be monitored, a face state determination unit that determines a face state from the detected face, a sound collection unit that collects surrounding sounds, and a detected sound to Means for determining an abnormality of the sound, and an abnormality notification signal is transmitted to the portable terminal when the abnormality of the face state or the abnormality of the surrounding sound is determined based on a predetermined value stored in advance and it is determined that there is an abnormality. Control means,
When the anomaly reporting device detects a face, if the face cannot be detected from the camera due to the orientation of the face of the monitored person, the monitoring person's face is drawn in the direction of the camera. With means to
It said portable terminal, monitoring system comprising a display means and audio means, and control means for generating a warning sound from the display and the audio means abnormality notification on the display means based on the abnormality notification signal received from the abnormality notification device . When the face of the person being monitored cannot be detected from the camera, means for drawing the attention of the person to be watched and directing the face of the person being watched toward the camera is a registered sound. The monitoring system according to claim 1, further comprising : The monitoring system according to claim 2, wherein the registered sound is a call sound . When the face cannot be detected from the camera due to the face direction of the person to be monitored, the means for drawing the attention of the person to be watched and directing the face of the person to be watched toward the camera is a movement of the object. The monitoring system according to any one of claims 1 to 3 . Response when the means for drawing the attention of the monitoring subject and directing the monitoring subject's face in the direction of the camera when the face cannot be detected from the camera due to the orientation of the monitoring subject's face The monitoring system according to any one of claims 1 to 4 , wherein the state of the person to be monitored is determined by : The mobile terminal periodically acquires a history of notification from the abnormality notification device in video and sound at a settable interval, and is displayed in a table or graph as the state of the monitoring target person, and the health state, meal state of the monitoring target person , comprising means that can be grasped at least one state of the excretion state monitoring system according to any one of claims 1-5. It said portable terminal, when notification of the abnormality notification device, or confirms the previous history at any time, provided with means to improve the accuracy of state determination so as to perform the learning of the detection state, claim 1 The monitoring system according to any one of -6 . The face detection means includes face state determination means for determining whether the face detection state is at least one of “existing state”, “absent state”, “futon-covered state”, and “collapsed sleeping state”. The monitoring system according to any one of claims 1 to 7 . When the face state judging means is “existing state” by the face detecting means, the “wake state”, “sleep state”, “crying state”, “good state”, “normal state” of the face The monitoring according to claim 8 , further comprising means for estimating a state of the person to be monitored together with information on facial movement by estimating a basic state such as “smiling state” and “screaming state”. system.

Images