JP6944901B2 - Biometric information detection device and biometric information detection method - Google Patents

Description

The present invention relates to a technique for detecting biological information.

As a method for acquiring biological information, there is a technology that can detect in real time without contact using microwaves or a camera. In particular, pulse detection using a camera has been miniaturized in recent years, and has been installed in mobile terminals including smartphones and has become widespread.

As a technique for detecting a pulse by imaging, there is a method of detecting a pulse by tracing a G signal in particular among RGB signals in a facial image (Non-Patent Document 1). Further, in order to separate noise by using RGB signals, there is a method of using independent component analysis (Non-Patent Document 2), and there is a method of specifying a pulse signal from the spectral distribution of a time series signal (Patent Document 1). ).

Japanese Unexamined Patent Publication No. 2012-239661

Verkruysse, Wim, Lars O. et al. Svaasand, J. Mol. Stuart Nelson, "Remote plethysmographic imaging ambient light.", Optics express 16.26 (2008): 21434-21445. Ming-Zher Poh, Daniel J. et al. McDuff, Rosalind W. et al. Picard, "Non-contact, automated cardiac plus measurement measurement video imaging and blend source separation.", Optics Express 18.10 (2010): 107.10 (2010).

With the conventional technique, it is possible to stably detect the pulse when the constant light hits the face. However, when the external light changes in the imaging environment, each spectrum is affected and erroneous detection occurs. The applicant has previously proposed a method that is resistant to changes in the brightness of external light due to the environment (Japanese patent application, Japanese Patent Application No. 2017-210828).

A general mobile camera or Web camera has an automatic correction function that automatically performs backlight correction (ABLC), exposure correction (AE), white balance correction (AWB), and the like according to changes in the environment. Therefore, the color and the brightness change in the imaging depending on the environment. This change in the image in the time axis direction has an adverse effect on the detection of the change in complexion, and the pulse wave may be erroneously detected or cannot be measured.

It is also possible to stop the automatic correction function and make adjustments manually. However, depending on the scene, the subject often does not appear correctly, and it is difficult to adjust a plurality of parameters at the same time even when adjusting manually.

Further, in order to detect the pulse wave, it is necessary to adjust the image so that the change in complexion can be detected. Therefore, there are those that put the face in the adjustment range of the camera, and those that automatically detect and adjust the face. The latter is convenient for correction, but it is often difficult to use it together because the cost of the CPU is high when the pulse wave is detected by a mobile device or the like.

Therefore, an object of the present invention is to provide a pulse detection method suitable for an imaging environment without having an adverse effect due to automatic correction by a camera.

A preferred aspect of the present invention is an automatic correction unit that corrects an image captured by a camera, a detection region signal generation unit that detects a skin color region from the image, and a temporal wavelength change of the skin color region. The wavelength fluctuation detection unit that generates the wavelength difference data signal, the pulse detection unit that estimates the pulse based on the wavelength difference data signal, and the automatic correction unit stops the correction when the pulse estimation is started. It is a biological information detection device including a correction control unit.

Another preferable aspect of the present invention is an imaging process for imaging a measurement target, an automatic correction process for correcting an image obtained by the imaging process, and a detection region signal generation process for detecting a skin color region from an image. The wavelength fluctuation detection process that detects the temporal color change of the skin color region and generates the wavelength difference data signal, the pulse detection process that estimates the pulse of the measurement target based on the wavelength difference data signal, and the pulse estimation are started. This is a biometric information detection method that performs a correction control process that stops the automatic correction process when the automatic correction process is performed.

According to the present invention, it is possible to provide a pulse detection method suitable for an imaging environment without having an adverse effect due to automatic correction by a camera.

The block diagram which shows the example of the block diagram of the block diagram of the biological information detection apparatus according to the Example of this invention. The block diagram explaining an example of the detection area signal generation part of the biological information detection apparatus in Example 1. FIG. The conceptual diagram which shows the designated range of HSV color space and partial color space. The conceptual diagram which shows an example of dynamic face detection and complexion change detection by Example 1. FIG. FIG. 6 is a conceptual diagram showing an example of a fixed frame for face detection and detection of a change in complexion according to the first embodiment. The block diagram explaining an example of the wavelength fluctuation detection part of the biological information detection apparatus in Example 1. FIG. FIG. 3 is a block diagram illustrating an example of a pulse detection unit of the biological information detection device according to the first embodiment. The flow chart which shows the flow of the process of Example 1. FIG. FIG. 3 is a block diagram illustrating an example of wavelength fluctuation detection to which a function of suppressing erroneous detection due to external light of the biological information detection device according to the second embodiment is added. FIG. 3 is a block diagram illustrating an example of a wavelength fluctuation detection unit including a noise removal unit for two regions of the biological information detection device according to the third embodiment. FIG. 3 is a block diagram illustrating an example of a wavelength fluctuation detection unit including a noise removal unit for a low frequency 2 region of the biological information detection device according to the third embodiment. FIG. 6 is a conceptual diagram illustrating an example of two regions of the noise removing unit in the third embodiment. FIG. 3 is a block diagram illustrating an example of a wavelength fluctuation detection unit including a noise removal unit for a plurality of frequency regions of the biological information detection device according to the fourth embodiment. FIG. 6 is a conceptual diagram illustrating an example of a plurality of regions of a noise removing unit in the fourth embodiment. FIG. 5 is a block diagram illustrating an example of a wavelength fluctuation detection unit including a noise removal unit for a plurality of frequency regions of each divided space of the biological information detection device according to the fifth embodiment. The conceptual diagram explaining the division of the space of the noise removing part and an example of a plurality of regions in Example 5.

Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not necessarily limited to these embodiments. In each drawing for explaining the embodiment, the same members may be designated by the same reference numerals, and the repeated description thereof may be omitted.

When there are a plurality of elements having the same or similar functions (for example, modified examples), they may be described by adding different subscripts to the same code. However, if it is not necessary to distinguish between a plurality of elements, the subscript may be omitted for explanation.

The notations such as "first", "second", and "third" in the present specification and the like are attached to identify the components, and do not necessarily limit the number, order, or contents thereof. is not it. Further, the numbers for identifying the components are used for each context, and the numbers used in one context do not always indicate the same composition in the other contexts. Further, it does not prevent the component identified by a certain number from having the function of the component identified by another number.

The position, size, shape, range, etc. of each configuration shown in the drawings and the like may not represent the actual position, size, shape, range, etc. in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings and the like.

The publications, patents and patent applications cited herein form part of the description herein.

Components represented in the singular form herein shall include the plural form unless explicitly stated in the context.

In one typical example of this embodiment, the facial image is corrected until the complexion change can be detected by the automatic correction of the camera, and after that, the automatic correction function is stopped until the complexion change cannot be detected. In a more specific device configuration, with respect to the video signal captured by the camera, a detection region signal generation unit that sets the skin color region within the face range detected by face detection as the detection target region for pulse detection, and the generated detection A pulse detection unit that estimates the pulse based on the frame-to-frame difference between the detected video area obtained from the area signal and the video signal and the detected video area one frame before, and an automatic correction function that corrects the image according to the subject and environment. It is provided with a correction control unit that automatically corrects the complexion until the change in complexion can be detected and stops the automatic correction until the change cannot be detected.

In this embodiment, an example of a biological information detection device having a function of detecting a pulse from a facial image using a camera, performing automatic correction until it becomes possible to detect a change in complexion, and stopping the automatic correction until it cannot be detected. explain.

FIG. 1 is an example of a configuration diagram of a biological information detection device according to this embodiment. The biological information detection device in this embodiment includes a camera 200, a video input unit 300, a video processing unit 400, a face detection unit 103, a correction control unit 106, and a data display unit 110. The camera 200 may be, for example, a camera using a general solid-state image sensor. The data display unit 110 may be, for example, a general liquid crystal display.

The camera 200, the image input unit 300, the image processing unit 400, the face detection unit 103, and the correction control unit 106 other than the optical system can be configured by executing software on a microcomputer (not shown) to perform control. Alternatively, the same functions as those configured by software can be realized by hardware such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit). It also has a storage device such as a semiconductor memory for storing and executing software. The video data storage unit 303 and the wavelength data storage unit 403 are configured by a storage device.

The camera 200 includes an automatic correction unit 205 that digitizes the electric signal 202 obtained by photoelectrically converting the light that reaches the CCD 201 through the lens by the AD converter 203 and corrects the obtained RAW data signal 204 according to the environment. To be equipped with.

The automatic correction unit 205 performs automatic correction of the camera, which is generally performed in existing products. In the automatic correction of the camera, for example, when an image is taken in a dark room where light is backlit from a window, the exposure, backlight, and white balance are adjusted on a partial screen or a full screen. By correcting the color and brightness of the image, the visibility of the image is improved, but the color and brightness of the face also change.

Returning to FIG. 1, the video input unit 300 includes a video acquisition unit 301 that inputs the imaging data signal 101 acquired from the camera 200 and outputs the RGB signal 302 of the video. Further, a video data storage unit 303 is provided which temporarily stores the RGB signal 302 as an input signal and outputs the delayed RGB data signal 102.

The image processing unit 400 includes a detection region signal generation unit 420, a wavelength data storage unit 403, a wavelength fluctuation detection unit 440, and a pulse detection unit 480.

The detection area signal generation unit 420 uses the delayed RGB data signal 102 and the face area signal 104 as input signals, and outputs the level signal 401 and the wavelength data signal 402. The detection area signal generation unit 420 extracts a flesh-colored area from the face area of the subject. The detection region signal generation unit 420 will be described in detail with reference to FIG. In this embodiment, the face region is a region determined to be a human face by the face detection unit 103, but other parts such as hands and arms may be used to measure the pulse. It may also be an exposed area of the skin of an animal other than a human.

The wavelength data storage unit 403 stores the wavelength data signal 402 and outputs the delayed wavelength data signal 404.

The wavelength fluctuation detection unit 440 uses the level signal 401, the wavelength data signal 402, and the delay wavelength data signal 404 as input signals, acquires the inter-frame difference of the wavelength, and outputs the wavelength difference data signal 105. The wavelength fluctuation detection unit 440 detects an average color change in the skin color region of the face region of the subject.

The pulse detection unit 480 uses the wavelength difference data signal 105 as an input signal, detects the signal pulse, and outputs the pulse signal 109.

FIG. 2 is a diagram illustrating an example of the detection region signal generation unit 420. The detection area signal generation unit 420 performs image processing for each pixel. The detection area signal generation unit 420 includes a spatial filter 421, an HSV conversion unit 424, and a skin color area detection unit 427.

The spatial filter 421 generally weights the brightness of the central pixel and its peripheral pixels, and sets the value obtained by the product-sum calculation or the non-linear calculation as the value of the central pixel. Specifically, a delayed RGB data signal 102 having a line delay equivalent to the tap of the convolution kernel is input, and for example, a value obtained by performing a convolution operation with the kernel centering on a pixel of interest in an image is a smoothed RGB signal 422. It becomes. The kernel value is a coefficient of weighted average, and the total value may be 1.0. For example, smoothing may be an average value distribution or a Gaussian distribution. The smoothing process by the spatial filter 421 can reduce the influence of noise in the image. For example, since the human face is not stationary during imaging, the previous and next frame images do not match on a pixel-by-pixel basis. For this reason, pores and moles also become a kind of noise, so it is effective to spatially smooth (average) the image in order to detect the pulse by the color change in the wavelength difference calculation unit 441 or later.

The HSV conversion unit 424 receives an unpacked signal 423 obtained by decomposing the RGB signal 422 smoothed by the spatial filter 421 into R, G, and B signals as an input, and receives an H signal (hue), that is, a wavelength data signal 402 and an S signal (saturation signal). ) 425, converted to V signal (brightness signal) 426.

The skin color region detection unit 427 receives a wavelength data signal (hue signal) 402, an S signal (saturation signal) 425, a V signal (brightness signal) 426 and a face region signal 104, and indicates the skin color region in the face region signal. The level signal 401 is output.

The face region signal 104 identifies whether or not each pixel in the frame is included in the face region. The detection region signal generation unit 420 of this embodiment extracts the wavelength signal (H signal) of the skin color region from the face region and detects the pulse from the change of the wavelength signal.

FIG. 3 is a diagram for explaining the concept of the HSV signal obtained by converting the RGB signal by the HSV conversion unit 424. FIG. 3 shows a designated range of the HSV color space and the partial color space. The HSV color space is represented by cylindrical coordinates. The vertical axis represents the brightness of the color by Value (V), that is, the brightness, and the radial axis represents the intensity of the color by Saturation (S), that is, the saturation. The angle of rotation is Hue (H), that is, hue. Hue is independent of intensity and intensity, and is considered to correspond to the wavelength component of reflected light, considering that the imaging captures the reflection of light. Similarly, lightness can be thought of as indicating the intensity of a particular wavelength.

The flesh color region detection unit 427 specifies a flesh color region using a partial color space in this HSV color space as shown in 500 in FIG. 3, and when the HSV value of each pixel is included in the flesh color region, the level If the signal 401 does not include "1", "0" may be output. Further, the skin color region detection unit 427 outputs "0" as the level signal 401 for the pixel whose face region signal 104 is "0" without determining the HSV value of the pixel. That is, the skin color region detection unit 427 detects the skin color region only in the face region cut out by the face detection unit 103 in the image of the entire frame.

In order to set the partial color space in the HSV color space shown in FIG. 3, the range may be specified for each of hue (H), saturation (S), and lightness (V). The hue (H) can be specified in the range of 0 to 360 degrees, and the lightness can be specified in the range of 0% to 100% again. For example, 0 degree = 360 degrees of hue is red, 120 degrees is green, 240 degrees is blue, and the section specified by color 1 and color 2 may be the applicable range, and the saturation is 0% for light color and 100% for 100%. As for the dark color and the lightness, 0% is a dark color and 100% is a light color, and the range may be specified in the same manner.

As described above, the skin color region detection unit 427 can appropriately determine whether or not the pixel is a human skin portion according to the individual skin color and the lighting condition. In this specification, a region composed of pixels whose skin color region detection unit 427 determines that the skin color region is a human skin portion and whose level signal 401 is “1” is referred to as a skin color region. Since the skin color region can be arbitrarily defined by specifying the range described above, it is called "skin color", but it is not necessarily limited to a specific color.

Returning to FIG. 1, the face detection unit 103 uses the face detection changeover switch 107 for turning on / off the face detection function and the delayed RGB data signal 102 as input signals, performs face detection, and uses the face area signal 104 as an output signal. The face detection may be a method of dynamically detecting a face such as a known Viola-Jones method, or a method of detecting a face within a fixed frame. The face detection unit 103 cuts out the face part, outputs "1" to the face area signal 104 when each pixel in the frame is included in the cut out face part, and outputs "1" to the face area signal 104 when the face part is not included. “0” is output to the signal 104.

In the method of dynamically detecting a face, for example, the face area of the subject is detected by pattern recognition, and the movement of the detected face area is followed within the frame. In the method of detecting a face in a fixed frame, the face is detected by putting the face in the fixed frame and detecting, for example, the skin color.

In this embodiment, by detecting and following the face region, the pulse can be detected based on the image change of the skin-colored portion of the face region. For tracking the face area, a well-known technique that is also applied to existing products may be applied. For example, the face area is extracted by pattern recognition, and then the area where the frame difference is small from the face area is extracted and followed.

The moving image captured by the camera is temporarily stored in the video data storage unit 303 for each frame, and then the frame difference between the wavelength data signal (hue signal) 402 and the delayed wavelength data signal (hue signal) 404 is taken. .. When the movement of the face is large, the difference area is also small. When performing face detection dynamically, for example, comparison may be performed for each rectangular range. Further, in the example of the present technology, the frame difference has been described in terms of the hue component after color conversion, but it may be performed by the RGB signal 302 and the delayed RGB data signal 102.

Returning to FIG. 1, the correction control unit 106 uses the wavelength difference data signal 105 as an input signal, and the face detection changeover switch 107 and the automatic correction unit switch after a predetermined time (for example, several seconds) after the difference data flows in the main input signal. 108 is turned off, and when the difference data is 0, the face detection changeover switch 107 and the automatic correction unit switch 108 are turned on.

By this process, when the wavelength difference data signal 105 is detected by the wavelength fluctuation detection unit 440, the automatic correction of the image by the automatic correction unit 205 is stopped. As a result, the change in color and brightness of the imaging image due to the automatic correction is eliminated, and the influence on the detection of the pulse wave is suppressed.

Further, when the wavelength difference data signal 105 is detected, the face detection by the face detection unit 103 is stopped. By stopping the automatic correction of the image by the automatic correction unit 205, it is possible to prevent the face detection from malfunctioning. In addition, resources such as the CPU can be concentrated on pulse detection. However, when the processing capacity of the image processing unit is high and the face detection accuracy is high, it is not necessary to switch off the face detection unit. While the face detection by the face detection unit 103 is stopped, the face area signal 104 outputs "1" in all the pixels.

FIG. 4 is a diagram showing an example of dynamic face detection and detection of complexion change. Reference numeral 4001 is a diagram showing a state in which automatic correction such as exposure, backlight, and white balance is performed before the subject 4007 enters the camera. Reference numeral 4002 is a diagram showing a state when the subject 4007 enters the camera frame and face detection starts to be performed. The detected face is tracked as face area 4006. 4003 is a diagram showing a state in which the subject is backlit while moving. 4004 is a diagram showing that backlight correction has been performed and image adjustment has been performed. FIG. 4005 is a diagram showing a state in which a complexion change can be detected after a preset time elapses from face detection or an image change due to image adjustment has settled down, that is, a wavelength difference data signal 105 has flowed and pulse detection has been started. Is. After that, the correction control unit 106 turns off the correction function of the automatic correction unit 205.

FIG. 5 is a diagram showing an example of a fixed frame for face detection and detection of a change in complexion. Reference numeral 5001 is a diagram showing a state in which automatic correction such as exposure, backlight, and white balance is performed before the subject 5007 enters the camera. Reference numeral 5002 is a diagram showing a state when a subject enters the fixed frame 5006 of the camera frame and face detection is started. Reference numeral 5003 is a diagram showing a state in which the subject is backlit while moving. Reference numeral 5004 is a diagram showing that backlight correction has been performed and image adjustment has been performed. FIG. 5005 is a diagram showing a state in which a change in complexion can be detected after a preset time elapses from face detection or an image change due to image adjustment has settled down, that is, a wavelength difference data signal 105 has flowed and pulse detection has been started. Is. After that, the correction control unit 106 turns off the correction function of the automatic correction unit 205.

FIG. 6 is a diagram illustrating an example of a wavelength fluctuation detection unit of the biological information detection device according to the first embodiment. The wavelength fluctuation detection unit 440a of FIG. 6 describes the details of an example of the wavelength fluctuation detection unit 440 in FIG. 1. The wavelength fluctuation detection unit 440a includes a wavelength difference calculation unit 441, a skin color area calculation unit 443a, a wavelength difference integration unit 444, and an average wavelength difference calculation unit 447a.

The wavelength difference calculation unit 441 inputs a level signal 401 indicating a skin color region, a wavelength data signal 402, and a delay wavelength data signal 404. When the signal of the pixel in the skin color region is input (that is, when “1” is input as the level signal 401), the wavelength difference calculation unit 441 starts from the input wavelength data signal 402 and the delay wavelength data signal 404. The calculated wavelength difference data signal 442 of the pixels in the skin color region is output, and when the signal of the pixels outside the skin color region is input, a “0” value is output as the wavelength difference data signal 442.

Further, the skin color area calculation unit 443a inputs the level signal 401 indicating the skin color region, counts the number of pixels in the skin color region for each frame, and outputs the skin color area signal 445.

The wavelength difference integrating unit 444 receives the wavelength difference data signal 442 of the skin color region pixel as an input, integrates the wavelength difference for each frame, and outputs the integrated wavelength difference data signal 446.

The average wavelength difference calculation unit 447a receives the skin color area signal 445 and the integrated wavelength difference data signal 446 as inputs, and divides the integrated wavelength difference data and the skin color area to average the wavelength difference for all the pixels in the skin color region within one frame. The data signal 105 is output.

As is clear from the above, when the input pixel signal does not include the skin color region, all the wavelength difference data signals 442 become “0” values, and the output of the wavelength difference integration unit 444 and the average wavelength difference calculation unit 447a The output of is also 0.

FIG. 7 is a diagram illustrating details of an example of the pulse detection unit 480 of the biological information detection device according to the first embodiment. The pulse detection unit 480 performs video processing for each frame. The pulse detection unit 480 includes a difference data storage unit 481, a smoothing filter 483, a smoothing data storage unit 485, a tilt detection unit 487, a code data storage unit 489, and an extreme value detection unit 491.

The difference data storage unit 481 receives the wavelength difference data signal 105 as an input and outputs the delayed wavelength difference data signal 482. Further, the smoothing filter 483 receives the wavelength difference data signal 105 and the delayed wavelength difference data signal 482 as inputs, and outputs the wavelength difference data signal 484 smoothed from the wavelength data for a plurality of frames on the continuous time axis. The smoothing filter 483 is configured to block frequency components well above, for example, the expected human pulse frequency. With this configuration, noise in the detection signal is removed.

The smoothed data storage unit 485 receives the smoothed wavelength difference data signal 484 as an input, holds the wavelength difference data for a plurality of frames, and outputs the smoothed delayed wavelength difference data signal 486. Further, the inclination detection unit 487 obtains the difference between two consecutive frame data, and outputs a code data signal 488 for obtaining the inclination code. With this configuration, the change (that is, the slope) of the smoothed wavelength difference data is obtained. Instead of the difference of two consecutive frames, the slope of the difference is calculated by comparing the average of the wavelength difference data of multiple consecutive frames with the average of the wavelength difference data of multiple consecutive frames before it. You may.

The code data storage unit 489 receives the code data signal 488 as an input, holds code data for a plurality of frames, and outputs a delay code data signal 490. Further, the extreme value detection unit 491 receives the code data signal 488 and the delay code data signal 490 as inputs, and maximizes the frame in which the sign change of the inclination changes from the positive value to the negative value, and sets the frame in which the sign change from the negative value to the positive value. The extreme value is obtained by setting the minimum value, and for example, the maximum value (or the minimum value) is output as the pulse signal 109.

By smoothing the difference data signal by the smoothing filter 483 as described above, erroneous detection of the pulse due to minute fluctuations in the difference data signal due to noise or the like is prevented. The inclination detection unit 487 detects the change (inclination) of the difference data between adjacent frames, and the extremum detection unit 491 detects the maximum value or the minimum value of the difference data based on the result, thereby accurately performing the pulse signal. Can be generated. When the inclination detection unit 487 obtains the difference between the average frames of a plurality of frames in the vicinity in succession, erroneous detection of the pulse is prevented as in the above smoothing.

In the above embodiment, the function of the correction control unit 106 enables the automatic correction by the camera to be turned off while the wavelength difference data signal 105 is being detected. Therefore, since there is no change in image quality due to automatic correction, the pulse can be detected stably. Here, turning off the automatic correction means that the backlight correction (ABLC), the exposure correction (AE), the white balance correction (AWB), and the like due to changes in the environment are not performed thereafter. After turning off the automatic correction, the correction parameter immediately before turning it off may be kept fixed as it is.

The fact that the wavelength difference data signal 105 is detected means that the skin color region for which the pulse is to be detected can be detected in the frame. Therefore, as long as the wavelength difference data signal 105 is detected, the pulse can be detected. On the other hand, when the wavelength difference data signal 105 cannot be detected, the correction control unit 106 is turned on to restart the normal automatic correction, and the detection of the skin color region is started. The case where the wavelength difference data signal 105 cannot be detected can be determined, for example, by the fact that the wavelength difference data signal 105 cannot be detected for a predetermined time (for example, 5 seconds).

Further, in this embodiment, the function of the face detection unit 103 is also turned on / off in synchronization with the on / off of the correction control unit 106. As this merit, by turning off the correction control, it is possible to prevent the face detection from malfunctioning. However, if the performance of the face detection unit 103 is high, the face detection may remain on.

While the face detection unit 103 is in the off state, all the face area signals 104 are set to "1", so that the detection area signal generation unit 420 also detects the skin color included in the background of the subject as the skin color area. .. Since the flesh-colored region is not a flesh-colored region that fluctuates according to the beat of the subject, it becomes noise for the pulse signal 109. However, assuming that the ratio of the face area in the frame is large in a normal usage environment, it is considered that this is not a substantial problem.

FIG. 8 is a flowchart showing a processing flow of the biological information detection device of the first embodiment. The process is started, for example, when the power is turned on or when a command for instructing the start of the biological information detection process is input.

In the process S801, the automatic correction of the automatic correction unit 205 and the face detection process of the face detection unit 103 are turned on. In this state, exposure, white balance, and the like are controlled by automatic correction, and the face region is detected and tracked.

In the process S802, when the face detection unit 103 succeeds in detecting the face, tracking is performed, and the detection area signal generation unit 420 detects the skin color area. As a result, the wavelength fluctuation detection unit 440 can detect the color change (wavelength fluctuation) in the skin color region of the face region. The color change in the skin color region is output as a wavelength difference data signal 105.

In the process S803, it is determined that the change in complexion can be detected. For this purpose, for example, it is determined whether or not a change in the skin color time difference signal is detected after face detection. In a preferred example, in order to determine whether or not the wavelength difference data signal 105 is stably detected, for example, when the wavelength difference data signal 105 is detected for 10 seconds, it is assumed that the wavelength difference data signal 105 is stably detected.

If the wavelength difference data signal 105 is not stably detected, the processing S802 and the processing S803 are continued while the automatic correction of the automatic correction unit 205 and the face detection processing of the face detection unit 103 are turned on.

When the wavelength difference data signal 105 is stably detected, the correction control unit 106 turns off the automatic correction of the automatic correction unit 205 and the face detection process of the face detection unit 103 in the process S804. In the automatic correction, the correction parameter immediately before turning off shall be maintained. Even if these processes are turned off, as long as the wavelength difference data signal 105 is stably detected, it can be considered that the skin color region of the face region to be measured is detected, and the pulse detection. S805 becomes possible. As described above, the face detection process of the face detection unit 103 does not have to be turned off.

In the process S806, it is monitored whether or not the wavelength difference data signal 105 is stably detected. The criterion for determination can be changed from that of process S803. For example, when the wavelength difference data signal 105 is interrupted for 1 second, it may be determined that the signal is not stably detected.

If stable detection becomes impossible after the wavelength difference data signal 105 is stably detected, the processing S801 determines the automatic correction of the automatic correction unit 205, and (the face detection processing in the processing S804). (When is turned off), the face detection process of the face detection unit 103 is turned on. Then, the face area and the skin color area are detected again.

According to the above configuration, it is possible to provide a pulse detection method suitable for the imaging environment without having an adverse effect due to automatic correction by the camera.

In the configurations of FIGS. 1 and 8, the correction control unit 106 is controlled based on the wavelength difference data signal 105 from the wavelength fluctuation detection unit 440. The following can be considered as signals and determination criteria used for controlling the correction control unit 106.

In the determination of the process S803, the automatic correction is turned off in the following cases. (1) A signal exceeding a predetermined intensity can be obtained as the wavelength difference data signal 105. (2) A signal exceeding a predetermined area can be obtained with the skin color area signal 445 of the face area. (3) With the pulse signal 109, a signal having a predetermined frequency (for example, in the case of a human heart rate, the normal heart rate is 50 to 100 per minute, but it can be set arbitrarily) can be obtained. (4) With the pulse signal 109, a signal having a predetermined frequency and a signal exceeding a predetermined amplitude can be obtained. The accuracy can be improved by combining a plurality of conditions (1) to (4) above. Further, it may be added to the condition that the above condition is continuously satisfied for a predetermined time.

In the determination of the process S806, the automatic correction is turned on in the following cases. (1) The wavelength difference data signal 105 has a predetermined intensity or less. (2) The skin color area signal 445 shows a value equal to or less than a predetermined area. (3) The pulse signal 109 cannot obtain a signal having a predetermined frequency (for example, 50 to 100 per minute for a human heart rate). (4) The pulse signal 109 cannot obtain a signal exceeding a predetermined amplitude. The accuracy can be improved by combining a plurality of conditions (1) to (4) above. Further, it may be added to the condition that the above condition is continuously satisfied for a predetermined time.

FIG. 9 is a diagram illustrating an example of wavelength fluctuation detection to which a function of suppressing erroneous detection due to external light of the biological information detection device is added. In the wavelength fluctuation detection unit of the second embodiment, the wavelength fluctuation detection unit 440 of FIG. 1 is replaced with the wavelength fluctuation detection unit 440a of FIG. 6 to 440b of FIG. The same configuration as in FIG. 6 is designated by the same reference numerals, description thereof will be omitted, and different parts will be described.

First, a skin color area calculation unit 443b that receives a level signal 401 indicating a skin color region is provided. Here, information indicating the brightness level of the signal is added to the level signal 401 indicating the skin color region. The skin color area calculation unit 443b includes a skin color area calculation unit 443b that counts the number of pixels other than the “0” value of the skin color region, that is, the level signal 401 for each frame, and outputs the skin color area signal 445 and the brightness level signal 448. ..

Further, the area data storage unit 450 is provided with an area data storage unit 450 that inputs the skin color area signal 445 and the brightness level signal 448 and outputs the delayed skin color area signal 451 and the delayed brightness level signal 449.

Further, it includes an integrated data storage unit 456 that receives the wavelength difference data signal 105 as an input, holds data between a plurality of frames, and outputs a delayed integrated wavelength difference data signal 457.

Further, the skin color area signal 445, the brightness level difference signal 452 between frames, the skin color area difference signal 453, the integrated wavelength difference data signal 446, and the delayed integrated wavelength difference data signal 457 are input to divide the integrated wavelength difference data and the skin color area. It also includes an average wavelength difference calculation unit 447b that outputs an average wavelength difference data signal 105 between frames.

Here, it is assumed that a sudden change occurs in the external light received by the subject to be imaged. In such a case, it is considered that the inter-frame brightness level difference signal 452 changes more than the inter-frame skin color area difference signal 453. Alternatively, it is conceivable that the skin color area difference signal 4534 between frames suddenly increases.

Therefore, in this embodiment, in addition to the skin color area signal 445 and the integrated wavelength difference data signal 446, the average wavelength difference calculation unit 447b is provided with an inter-frame brightness level difference signal 452, an inter-frame skin color area difference signal 453, and a brightness level difference. It is assumed that the threshold value 455 and the skin color area difference threshold value 454 are input. Here, the brightness level difference threshold value 455 and the skin color area difference threshold value 454 are both preset constant values.

In the average wavelength difference calculation unit 447b, when a sudden change in external light occurs, the brightness level difference signal 452 becomes larger than the brightness level difference threshold value 455. In this case, the average wavelength difference calculation unit 447b may use the delayed integrated wavelength difference data signal 457 from the past frame (for example, one frame before) instead of the integrated wavelength difference data signal 446 from the current frame. Alternatively, the average value of the delayed integrated wavelength difference data signal 457 and the integrated wavelength difference data signal 446 may be used.

Similarly, when the change in the detected skin color region is large, that is, when the skin color area difference signal 453 is larger than the brightness level difference threshold 455, the delay integrated wavelength difference data signal 457 or the delayed integrated wavelength difference data signal 457 is used. An average value with the integrated wavelength difference data signal 446 can be used.

Due to the configuration of the wavelength fluctuation detection unit 440b, it is possible to suppress a sudden change in the wavelength difference data signal 105 even if the brightness or area of the skin color region suddenly changes due to a sudden change in external light, so that the pulse signal 109 It is possible to suppress sudden fluctuations in.

Here, on the data display unit 110, the maximum value of the pulse signal is regarded as the R wave of the heartbeat, the maximum value interval, that is, the R wave interval is calculated for a plurality of frames, and frequency analysis is performed. A peak of component (LF) is seen. Generally, LF / HF is called a stress index and can be used for detecting a stress state.

According to the above configuration, it is possible to provide a pulse detection method suitable for the imaging environment without the harmful effects of automatic correction by the camera.

In the first embodiment, an example of a biological information detection device having a function of detecting a pulse from a facial image using a camera, performing automatic correction until it becomes possible to detect a change in complexion, and stopping the automatic correction until it cannot be detected. explained. In this embodiment, a biometric information detection device having a configuration in which noise is suitably removed at the same time as face detection to improve pulse wave detection accuracy in Example 1 will be described. The configuration of this embodiment is the same as that of FIG. 1 of Example 1 except for the wavelength fluctuation detection unit 440.

FIG. 10 is a diagram illustrating an example of a wavelength fluctuation detection unit including a noise removal unit for two regions of the biological information detection device according to the third embodiment. Since the wavelength fluctuation detection unit 440c is a modification of the wavelength fluctuation detection unit 440 in FIG. 1, the difference from the wavelength fluctuation detection unit 440a in FIG. 6 will be mainly described.

Similar to the wavelength fluctuation detection unit 440a (FIG. 6), the wavelength fluctuation detection unit 440c includes a wavelength difference calculation unit 441, a skin color area calculation unit 443a, a wavelength difference integration unit 444, and an average wavelength difference calculation unit 447a.

In addition to the above configuration, the wavelength fluctuation detection unit 440c includes a signal inversion circuit 461 that inverts the level signal 401 indicating the skin color region. In the signal inversion circuit 461, the level signal “0” indicating the outside of the skin color region is inverted to “1”.

Further, it is provided with a skin color outer area calculation unit 463a that receives an inverted level signal 462 outside the skin color region as an input, counts the number of pixels outside the skin color region for each frame, and outputs a skin color outer area signal 464.

The wavelength difference calculation unit 441b inputs the inverted signal of the level signal 401 indicating the skin color region, the wavelength data signal 402, and the delayed wavelength data signal 404. The inversion signal of the level signal 401 indicating the skin color region indicates an region other than the “skin color of the face region”, but when the function of the face detection unit 103 is turned off, the skin color regardless of whether it is the face region or not. It becomes a level signal indicating an area other than. When the signal of the pixel outside the skin color region is input (that is, when “0” is input as the level signal 401), the wavelength difference calculation unit 441b starts with the input wavelength data signal 402 and the delay wavelength data signal 404. The calculated wavelength difference data signal 442b of the pixels outside the skin color region is output, and when the signal of the pixels inside the skin color region is input, a “0” value is output as the wavelength difference data signal 442b.

The wavelength difference integrating unit 444b receives the wavelength difference data signal 442b of the pixels outside the skin color region as input, integrates the wavelength difference for each frame, and outputs the integrated wavelength difference data signal 446b of the pixels outside the skin color region.

Further, the skin color outer area signal 464 and the integrated wavelength difference data signal 446b are input, and the wavelength difference data signal 466a outside the skin color region before noise processing averaged between frames is obtained by dividing the integrated wavelength difference data and the skin color outer area. The average wavelength difference calculation unit 465a to be output is provided.

Further, the wavelength difference data signal 499 before noise processing and the wavelength difference data signal 466a outside the skin color region before noise processing are used as input signals, and noise is removed by using a two-variable independent component analysis method to reduce noise. A noise removing unit 467 that outputs a difference data signal 105 is provided.

As is well known, independent component analysis is a method of separating observation signals measured at a plurality of different observation points into a plurality of original signals by evaluating the independence between the signals (see, for example, Non-Patent Document 2). The wavelength difference data signal 466a outside the skin color region, which is the output of the average wavelength difference calculation unit 465a, includes wavelength fluctuations outside the skin color region, and this component acts as noise in the pulse signal. Therefore, by separating this component from the wavelength difference data signal 499, a more accurate pulse signal can be obtained.

FIG. 11 is a block diagram of a wavelength fluctuation detection unit 440d in which a frequency decomposition unit 468a is added to the wavelength fluctuation detection unit 440c of FIG. The frequency decomposition unit 468a receives the integrated wavelength difference data signal 446b as an input, performs a low-pass filter process for passing through a low frequency region, and outputs a low frequency integrated wavelength difference data signal 469a. Factors that change the color and brightness of facial images include changes in external light and changes in surrounding conditions (for example, shadows caused by the passage of other people by the side). These are slow changes compared to the pulse frequency. Therefore, by monitoring the low-frequency components of the image in this way, it becomes easy to acquire noise due to changes in color and brightness due to external light.

FIG. 12 is a diagram illustrating an example of two regions of the noise removing unit in the third embodiment. The area 1201 (white part) in the figure shows the skin color area after face detection, and the area 1202 (gray part) is a background area outside the skin color area. The difference in configuration between FIGS. 10 and 11 is whether or not region 1202 is a low frequency region among the observation variables used in the two independent component analyzes.

FIG. 13 is a diagram illustrating an example of a wavelength fluctuation detection unit including a noise removal unit for a plurality of frequency regions of the biological information detection device according to the fourth embodiment. Since the wavelength fluctuation detection unit 440e is considered to be a modification of the wavelength fluctuation detection unit 440d in FIG. 11, the difference from the wavelength fluctuation detection unit 440d in FIG. 11 will be mainly described.

The wavelength fluctuation detection unit 440e has the frequency decomposition unit 468a of the wavelength fluctuation detection unit 440d extended to a plurality of bands to the frequency decomposition unit 468b, and the noise removal unit 467a to the noise removal unit 467b corresponding to the independent component analysis method of a plurality of components. It is a modified version. As a result, the average wavelength difference calculation unit 465b calculates the average wavelength difference for each frequency domain decomposed by the frequency decomposition unit 468b, and the noise removal unit 467b performs noise removal processing.

FIG. 14 is a diagram illustrating an example of a plurality of regions of the noise removing unit in the fourth embodiment. Area 1201 in the figure indicates a skin color area after face detection. The other areas 1402, 1403, and 1404 are background areas. For example, the region 1402 is an empty portion in the image, and is a low frequency region as an image signal. Further, the region 1403 is a lawn portion, and the region 1404 is a human torso portion, which is a medium to high frequency region as an image signal. In this case, the accuracy of the pulse signal can be improved by the independent component analysis using the four components divided in the frequency domain.

FIG. 15 is a diagram illustrating an example of a wavelength fluctuation detection unit including a noise removal unit 467b for a plurality of frequency regions of each divided space of the biological information detection device according to the fifth embodiment. The wavelength fluctuation detection unit 440f describes the wavelength fluctuation detection unit 440 in FIG. 1, and is a region for performing region division by inputting an inverted level signal 462 outside the skin color region to the wavelength fluctuation detection unit 440e. The division portion 470 is added.

The different parts will be described as compared with the configuration of FIG. In the embodiment of FIG. 15, the image region is divided by the region division portion 470. The wavelength difference calculation unit 441c calculates the wavelength difference for each divided region, and the wavelength difference integration unit 444c integrates the wavelength difference for each region. The frequency decomposition unit 468c decomposes the frequency for each divided region. In addition, the skin color outer area calculation unit 463c calculates the area for each divided region. The average wavelength difference calculation unit 465b calculates the average wavelength difference for each divided region and each frequency region.

FIG. 16 is a diagram illustrating an example of division of the space of the noise removing portion and a plurality of regions in the fifth embodiment. The accuracy of the pulse signal can be improved by performing an independent component analysis by using a plurality of components obtained as the product of the number of divided spaces and the number of frequency domains. In the example of FIG. 16, for the regions other than the skin color, 72 components can be obtained in the case of the number of spaces 4 × 6 = 24 and the number of frequency regions 3 (regions 1402, 1403, 1404).

According to the above configuration, a pulse detection method suitable for the imaging environment is provided without the harmful effects of automatic correction by the camera, and the pulse is obtained by using the signal obtained by face detection and frequency decomposition used in this method. The detection accuracy of is also improved.

200 camera, 300 video input unit, 400 video processing unit, 103 face detection unit, 106 correction control unit, 205 automatic correction unit

Claims (14)

An automatic correction unit that corrects the image captured by the camera,
A detection area signal generation unit that detects a skin color area from the image,
A wavelength fluctuation detection unit that detects a change in wavelength over time in the skin color region and generates a wavelength difference data signal.
A pulse detection unit that estimates the pulse based on the wavelength difference data signal,
When the pulse estimation is started, the correction control unit that stops the correction by the automatic correction unit and the correction control unit.
A range detection unit that detects and tracks a predetermined range in the image is provided.
The detection region signal generation unit detects a skin color region from the predetermined range.
Biometric information detector. The automatic correction unit corrects at least one of the exposure, backlight, and white balance of the image.
The biological information detection device according to claim 1. The correction control unit fixes the correction parameter by the automatic correction unit when the correction by the automatic correction unit is stopped.
The biometric information detection device according to claim 1 or 2. The correction control unit is configured to stop the correction by the automatic correcting section, that abolish stop the detection and tracking of the predetermined range by the range detector,
The biometric information detection device according to any one of claims 1 to 3. The predetermined range is a human face portion, and the skin color region is an exposed region of the facial skin.
The biometric information detection device according to any one of claims 1 to 4. When the wavelength difference data signal includes a signal exceeding a predetermined intensity, the correction control unit determines that the pulse estimation has started.
The biometric information detection device according to any one of claims 1 to 5. The correction control unit determines that the pulse estimation has started when the skin color region detected by the detection region signal generation unit exceeds a predetermined area.
The biometric information detection device according to any one of claims 1 to 5. The correction control unit determines that the pulse estimation has started when a signal having a predetermined frequency is obtained from the output of the pulse detection unit.
The biometric information detection device according to any one of claims 1 to 5. The correction control unit determines that the pulse estimation has started when a signal having a predetermined frequency and a signal exceeding a predetermined amplitude are obtained from the output of the pulse detection unit.
The biometric information detection device according to any one of claims 1 to 5. The correction control unit stops the correction by the automatic correction unit when the pulse estimation is started, and then restarts the correction by the automatic correction unit when the pulse cannot be estimated. ,
The biometric information detection device according to any one of claims 1 to 9. In addition, it is equipped with a noise removal unit.
The noise removing unit inputs the wavelength difference data signal and the background wavelength difference data signal that detects a change in wavelength in a region other than the skin color region where the wavelength difference data signal is obtained, and performs independent component analysis. Removes noise by
The biometric information detection device according to any one of claims 1 to 10. A frequency decomposition unit that divides the background wavelength difference data signal into a plurality of frequency domains is provided, and the divided background wavelength difference data signal is used as an input of the noise removal unit.
The biological information detection device according to claim 11. The background wavelength difference data signal is generated for each of a plurality of regions in which the image is divided, and the background wavelength difference data signal generated for each of the plurality of regions is used as an input of the noise removing unit.
The biological information detection device according to claim 11. Imaging processing to image the measurement target and
Automatic correction processing that corrects the image obtained by the imaging process, and
A detection area signal generation process for detecting a skin color area from the image, and
Wavelength fluctuation detection processing that detects changes in color over time in the skin color region and generates a wavelength difference data signal, and
Pulse detection processing that estimates the pulse of the measurement target based on the wavelength difference data signal, and
A correction control process for stopping the automatic correction process when the pulse estimation is started, and a correction control process for stopping the automatic correction process.
A range detection process for detecting and tracking a predetermined range in the image is performed.
In the detection region signal generation process, the skin color region is detected from the predetermined range.
Biometric information detection method.

Images