JP5189536B2 - Monitoring device - Google Patents

Description

  The present invention relates to a monitoring device, and more particularly to a monitoring device that monitors both images and sound.

  Monitoring cameras are often installed in stores or various facilities for security reasons. Many of these surveillance camera devices have specifications that are always in operation, and by continuously capturing images, if an abnormality occurs, the recorded video at the time of the abnormality can be confirmed later. it can. The image captured in this way can be a powerful clue for knowing the situation when an abnormality occurs.

  2. Description of the Related Art Conventionally, there has been proposed a surveillance camera device that captures an image, detects abnormal sound, and can capture an image of a sound source that emits the abnormal sound. For example, there has been proposed a surveillance camera that can take an image centering on the occurrence position only when an abnormal sound occurs (see Patent Document 1). The surveillance camera described in Patent Document 1 determines whether or not the external sound input to the sound input unit is an abnormal sound, detects the generation position of the external sound, and determines that the external sound is an abnormal sound. In such a case, the camera body is driven so as to photograph the position where the external sound is generated.

  There has also been proposed a surveillance camera device capable of determining the situation around the device main body from sound detected by a sound sensor (see, for example, Patent Document 2). The surveillance camera device described in Patent Document 2 detects a component characterizing a predetermined sound, which is present in a signal output from a sound sensor installed in the device. By comparing the component characterizing the predetermined sound detected in this way with a preset condition, the type of abnormality for the predetermined sound is determined. In this way, the monitoring camera device described in Patent Document 2 can also direct the monitoring camera device in the direction in which the sound determined to be abnormal sound is generated, and can also be a sound generated outside the imaging area of the monitoring camera. Even if it exists, the surrounding situation can be judged from the sound.

JP 2006-94251 A JP 2002-123878 A

  By using the monitoring camera device as described above, it is possible to record and confirm an image of a location (sound source) where an abnormal sound has occurred. Furthermore, according to such a monitoring camera device, it is possible to greatly save the consumption and power consumption of the recording medium by performing imaging with the camera only when abnormal sound is detected.

  However, the surveillance camera devices described in Patent Literature 1 and Patent Literature 2 described above change the orientation of the camera device after detecting abnormal sounds for sounds generated outside the imaging area of the surveillance camera. doing. For this reason, when the abnormal sound is a short instantaneous sound, it is possible that the image at the moment when the abnormal sound is detected cannot be taken even if the orientation of the camera device is adjusted after detecting the abnormal sound. In order to cope with such a problem, it is necessary to take measures such as widening the range of the imaging area of the camera in advance or reducing the blind spot by installing a large number of cameras.

  Further, in the above-described monitoring camera device, when an abnormal sound is generated, even if an image of the position where the sound is generated can be captured, the position of the sound source cannot be visually recognized only from the image. Cases are also conceivable. For example, if an image in which something is physically destroyed can be captured at the position where the abnormal sound is generated, the cause of the abnormal sound can be easily identified by checking the image. However, when an image having no appearance change (no abnormality) is captured, the position where the abnormal sound is generated cannot be determined from the image, and the sound source is visually recognized. It is not possible.

  For example, if a rupture occurs in a box but the box is not destroyed, or if a sound is generated due to a chemical reaction, even if the image at the moment when the abnormal sound is generated can be confirmed, It is also conceivable that the position where the abnormal sound has occurred cannot be specified. Alternatively, even when an acoustically transparent visual obstacle or the like is present in front of the location where the abnormal sound is generated, the position of the sound source cannot be specified from the change on the image captured by the camera. As a prominent example, for example, a fire may have occurred in the vicinity of the monitoring camera device, so that only smoke is captured in the captured image.

  Furthermore, a sound source that emits an abnormal sound is not always present at a stationary position. For example, when the sound source moves at a high speed or when sound is continuously generated at a plurality of different locations, it is considered difficult to move the direction of the camera device to track the sound source. Even in such a case, measures such as setting the range of the imaging area of the camera in advance to a wide angle are possible. However, as described above, if there is no change in the appearance of the sound source, check the captured image. Even so, it is impossible to know which part of the image is the source of the sound.

  Therefore, an object of the present invention made in view of such circumstances is to output a temporal trajectory of the sound source so that it can be visually recognized even when the sound source generating the abnormal sound moves. It is to provide a monitoring device.

  Another object of the present invention is to output the direction in which the sound source exists so that it can be visually recognized even when the sound source that generates the abnormal sound is located outside the imaging region that can be imaged by the camera. An object of the present invention is to provide a monitoring device capable of performing the above.

  Furthermore, the other object of this invention is to provide the monitoring apparatus which can raise the precision which detects abnormal sound.

The invention according to claim 1 that achieves the above object is as follows:
An imaging means for capturing an image;
A microphone array having a plurality of omnidirectional microphones arranged around the imaging means and receiving sound from a sound source;
From the arrival time difference or phase difference of the sound collected by the microphone array, the direction of the sound from the sound source with respect to the monitoring device is calculated, and the imaging means is based on the calculated direction of the sound from the sound source. Sound source position calculating means for calculating the position of the sound source in the captured image;
A display indicating the position of the sound source calculated by the sound source position calculating unit is output in combination with an image captured by the imaging unit in a display mode corresponding to the passage of time, and the calculated position of the sound source is the imaging When it is outside the predetermined imaging area in the image captured by the means, a display indicating that the position of the sound source is outside the predetermined imaging area is displayed on the edge of the predetermined imaging area corresponding to the position of the sound source. Combination means for outputting in combination with the image ;
It is characterized by providing.

The invention according to claim 2 is the monitoring device according to claim 1,
The sound source position calculating means includes
When the first peak of the sound pressure of the sound collected by the microphone array is detected, the position of the sound source is not calculated for a predetermined period, and the sound source is based on the sound pressure waveform near the first peak. Is calculated.

The invention according to claim 3 is the monitoring apparatus according to claim 1,
When the sound level collected by the microphone array exceeds a threshold value updated based on a temporal average of the sound levels collected for a predetermined period of time in the microphone array, the sound is an abnormal sound. Further comprising an abnormal sound determination means for determining,
The sound source position calculating means includes
The position of the sound source is calculated only when the sound collected by the microphone array is determined to be an abnormal sound by the abnormal sound determination means.

  The monitoring device according to the present invention calculates the position of the sound source in the image captured by the camera based on the direction of the sound from the sound source calculated based on the arrival time difference or phase difference of the sound collected by the microphone array. At the same time, the monitoring apparatus according to the present invention outputs a display indicating the position of the sound source in combination with the image in a display mode corresponding to the passage of time. Accordingly, when the monitoring device according to the present invention detects abnormal sound, it can indicate the position where the abnormal sound has occurred in the image captured by the camera, and even if the position where the abnormal sound occurs moves, The temporal trajectory of the movement can be shown visually recognizable.

  Further, according to the present invention, when the sound source that generates abnormal sound is located outside the area that can be imaged by the camera, the direction outside the imaging area is displayed in a display mode according to the passage of time on the edge of the imaging area. Can show. Therefore, the direction of the sound source can be known even when the sound source that generates the abnormal sound is outside the imaging region of the camera.

  Furthermore, according to the present invention, when determining the sound collected by the microphone array as an abnormal sound, the threshold value for determining the abnormal sound is updated as time passes. According to the present invention, when the first peak of the abnormal sound is detected, the position of the sound source is calculated based on only the first peak without calculating the position of the sound source for a predetermined period. Therefore, when detecting an abnormal sound, it is possible to suppress an adverse effect caused by a reflected wave that arrives after the direct wave.

It is a block diagram which shows schematic structure of the monitoring apparatus by embodiment of this invention. It is a figure for demonstrating calculation of the arrival direction of the sound by the level difference between unidirectional microphones. It is a figure for demonstrating calculation of the arrival direction of the sound by the arrival time difference of a sound. It is a figure which shows the example of arrangement | positioning of a some omnidirectional microphone. It is a figure which shows the example of the image acquired by the monitoring apparatus by this invention. It is a flowchart explaining operation | movement of embodiment of the monitoring apparatus by this invention. It is a figure which shows the example of the image acquired by the monitoring apparatus by this invention. It is a figure explaining specification of the direction outside an imaging region. It is a flowchart explaining the operation | movement including the combination image output process of FIG. 6, and the sound time passage display process. It is a figure which shows the example which displayed the time history of the moving sound source by time progress display processing. It is a figure explaining the example which applied the time passage display process to the process which shows the direction outside an imaging region. It is a figure which shows the power spectrum and envelope profile of the breaking sound of glass. It is a time chart explaining the method of reducing the influence by reflected sound.

    An embodiment of a monitoring device according to the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing an embodiment of a monitoring device 1 according to the present invention. The monitoring apparatus 1 shown in FIG. 1 includes a camera 10, a microphone array 20, an acoustic processing unit 30, an abnormal sound determination unit 40, a sound source position calculation unit 50, an image combination unit 60, an alarm processing unit 70, A network processing unit 80.

  The camera 10 constitutes an imaging unit that captures an image. The output of the image captured by the camera 10 is displayed on the monitor 100 as an external display unit via the image combination unit 60. At this time, the image captured by the camera 10 is output by the image combination unit 60 in combination with information on the sound source position, which will be described later, and information on the abnormality of the sound.

  The microphone array 20 has a plurality of omnidirectional microphones arranged around the optical axis of the lens of the camera 10. That is, the microphone array 20 is configured by arranging a plurality of omnidirectional microphones in the horizontal direction (x axis) and the vertical direction (y axis) of an area that can be captured and displayed by the angle of view of a lens attached to the camera 10. Composed. The arrangement of a plurality of omnidirectional microphones in the microphone array 20 will be described later.

  The acoustic processing unit 30 performs reverberation and noise removal of the sound collected by the microphone array 20 and level correction. That is, the acoustic processing unit 30 eliminates the influence of room reverberation and ambient environmental noise from the sound collected by the microphone array 20, and when the sound is attenuated by moving away from the microphone array 20, Processing such as increasing the level of the received signal is performed. By doing so, it is possible to easily and accurately perform subsequent processing, for example, determination of the abnormality of the sound in the abnormal sound determination unit 40, and calculation of the direction and position of the sound source by the sound source position calculation unit 50. become.

  The abnormal sound determination unit 40 constitutes an abnormal sound determination unit that determines whether or not the sound subjected to various processes by the acoustic processing unit 30 is an abnormal sound based on a predetermined reference. For example, the abnormal sound determination unit 40 determines sound anomaly by analyzing an envelope profile, frequency analysis, sound level, sound time course, and the like. When it is determined that the sound is an abnormal sound, the abnormal sound determination unit 40 also ranks the abnormal sound according to a predetermined condition in order to visualize the rank of the abnormality of the abnormal sound. This ranking can be performed based on the weighting by the analysis result when the abnormal sound determination unit 40 determines abnormality.

  The sound source position calculation unit 50 calculates the direction of the sound from the sound source with respect to the monitoring device 1 from the arrival time difference or phase difference of the sound collected by the microphone array 20. At the same time, the sound source position calculation unit 50 calculates the position of the sound source in the image captured by the camera 10 based on the calculated sound direction from the sound source. Therefore, in the present embodiment, the sound source position calculation unit 50 constitutes a sound source position calculation unit. The “sound source position” calculated by the sound source position calculation unit 50 does not mean the position in the space where the sound source actually exists, but means the pixel position on the image captured by the camera 10. . The sound source position calculation unit 50 uses the same coordinates as the area imaged by the camera 10 to visualize the direction of arrival of abnormal sound (ie, the position of the sound source), and the horizontal azimuth angle of the incoming abnormal sound. (Φ) and the vertical elevation angle (θ) are converted into x-axis and y-axis coordinates to calculate the position of the abnormal sound source.

  The image combination unit 60 constitutes combination means for outputting a display indicating the position of the sound source calculated by the sound source position calculation unit 50 in combination with an image captured by the camera 10. The image combination unit 60 uses the x-axis and y-axis coordinates of the image display area that can be displayed by the camera 10 and the point where the x- and y-axis coordinates of the position of the abnormal sound source coincide with each other as a sound source. Is displayed on the monitor 100 with a mark or the like. Alternatively, the image combination unit 60 can change, color, and shade the luminance value of the area corresponding to the arrival direction of the abnormal sound (that is, the position of the sound source) by dividing the display screen into multiple parts (36 parts, 64 parts). It is also possible to display it on the monitor 100 by, for example.

  Furthermore, the image combination unit 60 continuously updates the sound source position of the abnormal sound calculated by the sound source position calculation unit 50 and the abnormality rank determined by the abnormal sound determination unit 40 at short-term intervals. The latest display is performed on the monitor 100. Accordingly, the real-time display of the abnormal sound source position and the rank of the abnormal sound is enabled, and the abnormal sound moving state and the plurality of sound sources are displayed on the monitor 100 by holding the processed abnormal sound display for a predetermined period. Can be displayed. Furthermore, when the moving state of the abnormal sound and the plurality of sound sources are displayed on the monitor 100, the display indicating the position of the sound source is performed in a display mode corresponding to the passage of time. Thereby, the movement state of the abnormal sound and the sound generation by a plurality of sound sources can be represented together with the time history in the image at an arbitrary time point. Note that a display mode corresponding to the passage of time of the display indicating the position of the sound source will be described later.

  In addition, when the sound source position of the abnormal sound calculated by the sound source position calculation unit 50 is outside a predetermined imaging region in the image captured by the camera 10, for example, it protrudes from the display screen range due to the angle of view of the lens of the camera 10. There may be cases where In this case, the image combination unit 60 displays a colored mark such as a bar shape or a wave shape on the upper, lower, right, or left portion of the screen end of the monitor 100, and an abnormal sound arrives as an approximate position of the sound source. Indicates the direction to do. For example, when the sound source position of the abnormal sound calculated by the sound source position calculation unit 50 is outside the upper area of the display screen, a predetermined mark or the like is displayed on the upper edge of the screen of the monitor 100 to indicate the direction of the sound source. Indicates that it is in the upward direction outside the screen. Furthermore, when the sound source position of the abnormal sound calculated by the sound source position calculation unit 50 is outside the imaging region other than the top, bottom, left, and right in the image captured by the camera 10, that is, in a diagonal direction, a mark that indicates the direction Etc. are displayed. Details of the display in the case of outside the imaging area described above will be described later.

  Further, the image combination unit 60 can also change the mark display of the center position of the sound source based on the ranking of the sound abnormality by the abnormal sound determination unit 40. In this case, the image combination unit 60 changes the luminance value, color, shape, etc. of the mark at the center position of the sound source, or further displays the luminance value when the display screen is divided into multiple areas. Ranking can be expressed by changing it higher or changing the color to a more red color.

  The alarm processing unit 70 outputs an alarm when the abnormality rank of the sound exceeds a predetermined level, and notifies the monitor of real-time abnormal sound occurrence information. Further, the alarm processing unit 70 notifies the information of the direction of the sound source position of abnormal sound to another pan / tilt / zoom (PTZ) camera (not shown) or the like, thereby zooming the sound source direction with another PTZ camera. It is possible to perform linked confirmation operations such as displaying.

  The network processing unit 80 distributes the image data output from the image combination unit 60 and the alarm output from the alarm processing unit 70 to a mobile terminal or the like via the network 200.

  Next, calculation of the direction of arrival of sound by the microphone array 20 according to the present embodiment due to the level difference between the microphones will be further described.

  FIG. 2 is a diagram showing a microphone array including a general unidirectional microphone group. Using such a microphone array composed of the unidirectional microphone group 21, it is possible to calculate the sound arrival direction due to the level difference between the microphones. In FIG. 2, the unidirectional microphone group 21 that calculates the direction of the sound coming from the sound source 500 is arranged in a recess of the ceiling 300. In addition, the dome type camera 11 is disposed on the flat portion of the ceiling 300 in the vicinity of the unidirectional microphone group 21. In this case, when sound is generated from the sound source 500, the direction of the sound source 500 can be specified based on the maximum sound level received by one unidirectional microphone 22.

  In this way, the direction of the sound source can be calculated using a plurality of microphones using the level difference between the microphones. However, in this case, since it is necessary to use a large number of unidirectional microphones, the device configuration becomes complicated and the equipment cost increases. In FIG. 2, seven unidirectional microphones are used in order to correspond to seven directions. The azimuth resolution is basically determined by the number of microphones and the sharpness of directivity. Here, a configuration is shown in which a one-dimensional direction due to a level difference is calculated by a unidirectional microphone group 21 in which a plurality of unidirectional microphones are arranged in a line. However, the image that can be picked up by the dome-type camera 11 is two-dimensional, and in order to specify the position of the sound source on a two-dimensional plane, one more axis is required, and the number of necessary channels is doubled. The number of microphones to be installed also increases. In addition, such a unidirectional microphone group 21 is difficult to reduce in size because of the complexity of the configuration, and it is difficult to mount or incorporate it in the dome camera 11 itself. Furthermore, since the basic angular resolution depends on the number of microphones, it is necessary to increase the number of microphones in order to improve the resolution. When a microphone array is configured using unidirectional microphones as described above, a large number of microphones are generally required.

  FIG. 3 is a diagram for explaining the calculation of the sound arrival direction based on the sound arrival time difference using a small number of omnidirectional microphones. In FIG. 3, a dome-type camera 12 including a microphone array composed of omnidirectional microphones 25 and 26 that calculate the direction of arrival of a sound source 510 is disposed on a flat portion of a ceiling 310. As shown in FIG. 3, when calculating the direction of the sound source 510, when using the sound arrival time difference resulting from the distance difference L between the sound source 510 and the microphones 25 and 26, the microphones 25 and 26 are not used. Directional ones can be used. Therefore, in this case, it is sufficient to use a small number of non-directional microphones constituting the microphone array, and two microphones are sufficient if a one-dimensional direction is calculated. Further, when calculating a two-dimensional direction, only three microphones are required, so that the configuration of the apparatus can be simplified. Thus, when calculating the direction of the sound source by the difference in arrival time of sound using an omnidirectional microphone, a plurality of microphones are mounted on a housing including a camera body, or a dome is used as shown in FIG. It can be installed in the dome of a type camera.

  FIG. 4 is a diagram showing an example of a microphone array in which omnidirectional microphones are arranged. FIG. 4 shows an example different from the case where a small omnidirectional microphone, such as the arrangement of the microphone array in FIG. FIG. 4 shows an example in which a small omnidirectional microphone is placed around the lens of a box camera (such as a fixed surveillance camera).

  FIG. 4A shows a case in which two microphones 28 arranged at equal intervals on the x axis about the optical axis 17 of the lens 15 of the camera 10 are used. Thereby, the position of the sound source can be detected one-dimensionally on the x-axis. Note that the reference number of the microphone 28 is attached to only one microphone in each drawing. FIG. 4B shows a case in which two microphones 28 arranged at equal intervals on the y-axis with the optical axis 17 of the lens 15 of the camera 10 as the center are used. Thereby, the position of the sound source can be detected one-dimensionally on the y-axis. FIG. 4C shows a case in which four microphones 28 arranged at equal intervals on the top and bottom on the y-axis and on the left and right on the x-axis with the optical axis 17 of the lens 15 of the camera 10 as the center are shown. Thereby, the position of the sound source can be detected two-dimensionally on the xy plane. FIG. 4D shows a case in which three microphones 28 are used in which the distances from the y-axis and the x-axis are arranged at equal intervals around the optical axis 17 of the lens 15 of the camera 10. . This also allows the position of the sound source to be detected two-dimensionally on the xy plane. Therefore, in order to specify the position of the sound source two-dimensionally on the xy plane, if there are three microphones, a microphone array can be configured.

  Next, the combination image according to the present embodiment will be described. In the present embodiment, when the microphone array 20 detects abnormal sound, the image combination unit 60 outputs an image obtained by superimposing the display of the position of the abnormal sound source on the image captured by the camera 10. In this case, an image obtained by superimposing the display of the position of the abnormal sound source on the image captured by the camera 10 is hereinafter referred to as a “combination image”.

  In generating the combination image, first, the sound source position calculation unit 50 calculates the direction of the sound from the sound source with respect to the monitoring device 1 by measuring the arrival time difference or phase difference of the sound received by each microphone of the microphone array 20. To do. With this calculation, a horizontal azimuth angle (φ) and a vertical elevation angle (θ) with respect to the microphone array 20 are obtained from the sound source of the incoming abnormal sound. Next, the sound source position calculation unit 50 converts the direction of the sound from the sound source obtained in this way into coordinates on the x-axis and the y-axis in the same coordinate system as the area of the image captured by the camera 10. When the direction of the sound from the sound source is converted into coordinates on the x-axis and the y-axis, the image combination unit 60, an image captured by the camera 10, and an image that displays the position of the sound source at the calculated coordinates are displayed. Are combined and output. At this time, it is preferable to output an image such as a small disk-shaped mark in combination with the center position of the sound source as a display indicating the position of the sound source.

  FIGS. 5A and 5B show an example in which a combination image output by the monitoring apparatus 1 according to the present invention is output and displayed on the monitor 100. FIG. FIG. 5A shows the camera 10 with the microphone array 20 attached, taken in a state of being arranged horizontally with respect to the floor, output to the monitor 100 and displayed. FIG. 5B shows an image captured with the camera 10 with the microphone array 20 attached on the ceiling being output and displayed on the monitor 100. In both FIGS. 5A and 5B, a combined image in which the position of the sound source is displayed with a small disk-shaped mark corresponding to the hand of the subject who is the sound source is clapping is output. Has been. Both FIG. 5A and FIG. 5B show that information about the position of the sound source can be accurately combined with an image captured by the camera 10.

  Next, the combination image output process according to the present embodiment will be described. This process determines whether or not the sound is an abnormal sound from the time when a plurality of microphones constituting the microphone array 20 receive the sound, and combines the position of the sound source of the abnormal sound with the image of the camera 10. Processing until output. That is, the combined image output process is a process that represents the operation of monitoring the image and sound by the monitoring device 1 including the operation of outputting the above-described combined image.

  FIG. 6 is a flowchart for explaining combination image output processing according to the embodiment of the monitoring device 1 of the present invention. In FIG. 6, the combined image output process according to the present embodiment starts from the time when some sound is generated and received by the microphones constituting the microphone array 20. When the microphone array 20 collects the generated sound, the acoustic processing unit 30 removes reverberation and noise from the collected audio signal, and further performs level correction as necessary (step S11).

  Next, the abnormal sound determination unit 40 analyzes the abnormality of the sound acoustically processed by the acoustic processing unit 30 (step S12). At this time, the abnormal sound determination unit 40 extracts an abnormal sound from the collected sound signal based on the sound level, the sound envelope profile, the analysis based on the frequency, and the like, if necessary, by correlation with statistical data. However, the abnormality of the sound is determined by adding the determination from the passage of time depending on the situation. In step S12, when the abnormal sound determination unit 40 determines that the sound is abnormal, if it is determined that the sound is not an abnormal sound, the combined image is not output for a sound that is not an abnormal sound, and the combined image output process ends. . This is performed when it is determined that the sound collected by the microphone array 20 is not an abnormal sound but a part of the environmental sound. In this case, since the display indicating the position of the sound source is not included in the combined image, the image captured by the camera 10 is output as it is from the monitoring device 1.

  If it is determined in step S12 that the sound is an abnormal sound, the abnormal sound determination unit 40 determines, for example, from the sound level, the sound envelope profile, the frequency analysis, the number of sound occurrences, the duration, and the like. Is scored for each item, and the anomaly is ranked by the total score (step S13). Thereby, the abnormal sound determination unit 40 ranks the rank of the abnormality of the sound into, for example, “abnormality = high / medium / low”.

  If the rank of the abnormality of sound is specified in step S13, the sound source position calculation unit 50 next calculates the sound source position (step S14). In step S14, as described above, the angle from the sound source is calculated by measuring the time difference due to the distance between the microphones of the sound that reaches the microphones of the microphone array 20, and the x direction is calculated based on the calculated direction. The position of the sound source in the axial direction and the y-axis direction is calculated.

  When the position of the sound source is calculated in step S14, the image combination unit 60 determines whether or not the coordinates of the calculated sound source position are within the imaging region of the camera 10 (step S15). In step S15, when the coordinates of the sound source position are within the predetermined imaging area of the camera 10, that is, when the coordinates of the sound source position are within the range of the angle of view of the camera 10, they can be directly displayed as an image. Thus, a combination image is output (step S16). That is, in step S16, the image combination unit 60 combines (superimposes) an image captured by the camera 10 and an image displaying the position of the sound source at the calculated coordinates.

  Here, if necessary, images can be combined including information on the rank of abnormality. That is, the image combination unit 60 may perform processing such as changing the color of the mark representing the sound source displayed in the combined image based on the rank of the abnormality of the sound specified by the abnormal sound determination unit 40 in step S13. it can. For example, a mark indicating a sound source of a sound determined by the abnormal sound determination unit 40 as “abnormality = medium” is output in yellow, and a mark indicating a sound source of a sound determined as “abnormality = high” is output in red. Etc. can be performed.

  In this way, after the combined image is output by the image combining unit 60 in step S16, the image can be displayed on the monitor 100 or the like thereafter. Therefore, if the supervisor looks at this image, the position of the sound source of the sound determined as an abnormal sound and the rank of the abnormality of the sound can be easily recognized visually.

  Thereafter, it is determined whether or not the rank of the abnormality of the sound specified in step S13 is equal to or higher than a predetermined rank (step S17). In step S17, when it is determined that the rank of the sound abnormality is not equal to or higher than the predetermined rank, the process is terminated. Thus, for example, when a sound that is abnormal but relatively low in danger is detected, such as a sound determined to be “abnormality = low”, it is avoided that notification is made to the supervisor one by one. be able to.

  On the other hand, if it is determined in step S17 that the rank of the sound abnormality is equal to or higher than the predetermined rank, the network processing unit 80 and / or the alarm processing unit 70 outputs alarm information to the outside based on a predetermined setting. (Step S18). In step S18, the alarm processing unit 70 outputs alarm information based on the rank of the abnormality of the sound specified by the abnormal sound determination unit 40. In addition, the network processing unit 80 that has received the notification from the alarm processing unit 70 distributes the alarm information to a remote monitoring location, a mobile phone, etc. via the network 200, so that the monitor is informed of the abnormal sound. You can also be notified. At this time, if the monitor side wants the information on the sound source position and the information on the abnormality rank, the information can be notified to the monitor via the network 200.

  If a separate PTZ camera is installed, use this alarm information to check the image of the sound source by pointing the PTZ camera in the direction of abnormal sound and zooming in further. Can do. Furthermore, by sending live sound acquired by the microphone array 20 to the monitoring room in real time via the network 200, the abnormal sound can be confirmed by a human ear.

  By performing processing in this way, it is possible to prevent the sound identified as having low anomaly from being notified to the outside one by one, and when a highly abnormal sound that needs to be notified to the outside is detected, It can be surely notified to the outside.

  FIG. 7A shows a screen display example in which the sound source position is displayed on the screen with small disk-like marks as an example of an image output to the monitor 100 by the above-described combination image output processing. In FIG. 7A, the sound sources A and B that can be directly displayed on the image within the range of the angle of view of the camera 10 are displayed as disk-shaped marks. In this case, the sound sources A and B are color-coded according to the rank of abnormality. With such a display, the supervisor can easily recognize that the sound determined to be abnormal has occurred at the positions A and B when the image is captured. Further, the rank of the abnormality of the sound generated at each of the positions A and B can be easily recognized by the colors of the marks shown at the positions A and B.

  FIG. 7B shows another example of an image output to the monitor 100 by the combination image output process described above, and the screen is divided into 64 parts, so that the position of the sound source and the abnormality of the sound are divided into the divided small areas. An example of screen display showing levels is shown. As shown in FIG. 7B, the screen-divided small areas corresponding to the sound sources A and B existing within the range of the angle of view of the camera 10 may be displayed in different colors according to the abnormality rank. . Even with such a display, the supervisor can easily recognize that the sound determined to be abnormal has occurred at the positions A and B when the image is captured.

  In step S15 of FIG. 6, if the coordinates of the sound source position are not within the imaging area of the camera 10, that is, if the image of the sound source position cannot be directly displayed within the range of the angle of view of the camera 10, the image combination unit 60 In order to indicate the approximate position of the sound source, a process for specifying the direction outside the imaging region is performed (step S19). Specifically, when the position of the sound source protrudes from the range of the angle of view of the camera 10, an edge in the range of the angle of view of the camera 10 corresponding to the protruding direction is specified.

  FIG. 8 is a diagram illustrating the specification of the direction outside the imaging region. In FIG. 8, the range of the region that can be collected by the microphone array 20 is such that the azimuth angle in the x-axis direction is −180 degrees to 180 degrees and the elevation angle in the y-axis direction is −180 degrees to 180 degrees. . Of these ranges, the range of the field of view that can be captured by the camera 10 is such that the azimuth angle in the x-axis direction is −φm to φm and the elevation angle in the y-axis direction is −θm to θm. Let e be the field of view angle to be expressed. This region e corresponds to the limit that can be displayed on the display unit of the monitor 100.

  In this case, if an abnormal sound is generated in the area e (Yes in step S15), the mark of the position of the sound source can be displayed as it is in the area of the image captured by the camera 10. However, if the position of the sound source is in the areas b, d, f, and h outside the area of the image captured by the camera 10 (No in step S15), the area corresponding to each area as the approximate position of the sound source Specify the edge of e. When the sound source is located in the areas a, c, g, and i (No in step S15), as the approximate position of the sound source, two edges corresponding to each area of the four corners of the area e are included. Identify the intersection.

  Thus, when the direction outside the imaging region is specified in step S19, then in step S16, the image combination unit 60 combines the image captured by the camera 10 and the image displaying the direction of the sound source. Output. That is, when the position of the sound source is in the area b of FIG. 8, for example, a long and narrow mark is displayed on the upper edge of the area e, and when the position of the sound source is in the area f, the right edge of the area e is displayed. A long and narrow mark is displayed.

  For example, in FIGS. 7A and 7B, an elongated mark C indicating the direction of the sound source is displayed on the right edge of the angle of view in the captured image. This means that an abnormal sound source exists in the region f. That is, since the position of the sound source C is outside the region of the image captured by the camera 10 (out of the angle of view), it cannot be displayed on the monitor 100, but by performing a display different from the normal mark indicating the sound source. This shows that the position of the abnormal sound source is on the right side outside the image area. In this case as well, the marks can be displayed in different colors according to the rank of the abnormality of the sound generated by the sound source C.

  In addition, when the positions of the sound sources are in the areas a, c, g, and i shown in FIG. 8, L-shaped so that the two edges meet at the corners of the area e corresponding to each area. Display the shape mark. Thereby, it can be shown that the position of the sound source outside the region of the image captured by the camera 10 is an oblique region other than the regions b, d, f, and h.

  Next, the sound time-lapse display process according to the present embodiment will be described. This process displays the time history of the sound source position, so that even if the sound source moves or multiple sound sources continuously generate sound, the supervisor can check the situation at a glance. This is a process for displaying.

  FIG. 9 is a flowchart for explaining the operation of the monitoring device 1 including the combined image output process described in FIG. 6 and the sound time lapse display process. If the microphone array 20 detects a sound in step S31, in step S32, after performing the combination image output process demonstrated in FIG. 6, it will transfer to step S33. If the microphone array 20 does not collect sound in step S31, the process proceeds to step S33 as it is.

  Next, in step S33, the image combination unit 60 determines whether or not a predetermined time has elapsed. If the predetermined time has not elapsed, the image combination unit 60 returns to step S31 and continues the processing. . On the other hand, if the predetermined time has elapsed in step S33, the image combination unit 60 performs the time elapsed display process (step S34), and then returns to step S31 to continue the process. The predetermined time determined in step S33 is monitored every 0.5 seconds, every 1 second, or every 3 seconds, for example, to change (update) the display mode indicating the position of the sound source in units of that time. The time which seems to be suitable is set in advance according to the target (assumed sound source).

  In performing the operation shown in FIG. 9, once the position of the sound source calculated by the sound source position calculation unit 50 is input to the image combination unit 60, the image combination unit 60 receives the position of the sound source sequentially from the camera 10. The images are combined so that they remain displayed on the captured image. That is, for example, even if the sound stops for a moment, the mark indicating the position of the sound source is displayed for a moment, but does not disappear immediately, but remains displayed. However, when such a display is performed, when the sound source moves, the screen displayed on the monitor 100 is filled with the history of the mark image indicating the position of the sound source, and the image captured by the camera 10 is captured. It may be difficult to see.

  Therefore, as shown in FIG. 10A, the image combination unit 60 changes the image of the mark for displaying the position of the sound source in a display mode corresponding to the passage of time. The disc-shaped mark shown in FIG. 10A indicates that when a certain sound source emits a sound that stops instantaneously, the first output as a mark image indicating the sound source position is the mark a. ing. FIG. 10A shows a state in which the display changes in the order of marks b, c, d, and e as the predetermined time elapses even after the sound of the sound source stops. As shown in FIG. 10A, it is preferable that the mark image indicating the sound source is made smaller with the passage of time and is completely erased after a certain amount of time has passed.

  By performing such a display, when the mark indicating the position of the sound source is relatively large, the monitor can easily recognize that the sound is being generated at the nearest time. On the other hand, if the mark indicating the position of the sound source is relatively small, it can be easily recognized by the supervisor that some time has elapsed since the sound was generated at that position. In addition, even if an object that can be a sound source cannot be confirmed from the image at the mark indicating the position of the sound source, it can indicate that the sound source existed at that position in the near past. The movement history can also be shown so that the observer can easily confirm.

  FIG. 10B is a diagram illustrating an example in which the history of the moving sound source is displayed by the time passage display process. FIG. 10B shows an example in which a walking person is captured by the camera 10 and is output to the monitor 100 in combination with an image displaying the position of the sound source. For example, it is assumed that this walking person makes a loud noise and walks in the direction of the monitoring device 1. In this case, the sound made by the walking person is a sound that is continuously generated with substantially the same magnitude, but since the sound source is moving, the mark of the sound source position shown in FIG. In the order of b, c, d, and e, a history of sounds moving from the latest to the past is shown. With such a display, the supervisor can quickly and easily confirm the past sound history to a certain degree by simply looking at an image at a certain time.

  FIG. 11 is a diagram for explaining an example in which the above-described time-lapse display process for displaying a sound history is applied to a process indicating a direction outside an area of an image captured by the camera 10. In each of the elongated marks a, b, c, d, and e shown on the upper edge of FIG. 11A, the position of the sound source of the sound collected by the microphone array 20 exists in the range b shown in FIG. It shows what to do. If this sound is a sound that stops instantaneously, the mark a is displayed immediately after the sound source emits the sound, and thereafter, every time a predetermined time elapses, the mark b, c, d, e, etc. The history of the past sound is displayed by changing to small display in order. In each of the elongated marks f, g, h, i, and j shown on the left edge of FIG. 11A, the position of the sound source collected by the microphone array 20 exists in the range d shown in FIG. It shows what to do. Also in this case, if the sound is a sound that stops instantaneously, the mark f is displayed immediately after the sound source emits the sound, and thereafter, every time a predetermined time elapses, the marks g, h, i, j In this way, the display is changed to smaller ones in order.

  Each of the elongated marks a, b, c, d, and e shown at the edge of the upper left corner of FIG. 11B indicates the position of the sound source collected by the microphone array 20, as shown in FIG. In other words, it is present in the range a. When this sound is a sound that stops in a moment, the mark a is displayed immediately after the sound source emits the sound, and thereafter, every time a predetermined time elapses, like the marks b, c, d, e, Change to small display in order. By performing the display as described above, even if the position of the sound source that emits the abnormal sound is outside the region of the image, the direction of the sound source and the history of the sound can be shown.

  Next, various measures that can be taken to improve the accuracy of detecting abnormal sounds by the acoustic processing unit 30 and the abnormal sound determination unit 40 of the present embodiment will be described.

  First, in the monitoring device 1 according to the present invention, a normal line profile of sound used by the abnormal sound determination unit 40 to determine abnormal sound will be described together with a power spectrum for comparison. Conventionally, in order to determine an abnormal sound, there are mainly two determinations: (1) determination as to whether the sound level exceeds a threshold; and (2) the power spectrum meets a preset criterion. Whether or not is being determined.

  According to the determination of (1) above, any sound is determined to be an abnormal sound as long as it is a loud sound exceeding the threshold. The determination in (2) is intended to identify the type of sound using the power spectrum. Compared with the determination in (1), there is much information as a clue to determine the abnormality of the sound. It is considered that the accuracy of the remarkably improves.

  However, since the determination in (2) does not use information in the time direction, it is impossible to determine a sound having a similar tone color but having a different duration. For example, it is considered that there are many cases where it should be determined that motor sounds and wind noises associated with the operation of an air conditioner are not abnormal sounds. However, when a similar sound is generated for a moment or the level is suddenly changed, it may be determined that the sound is not an air conditioner operation sound but an abnormal sound.

  The envelope profile used in the monitoring device 1 according to the present invention is sound level time variation information, and anomaly judgment is made by taking into account the time variation information that is missing in the judgments of (1) and (2). To improve the accuracy. For example, the power spectrum of a sample of glass breaking sound is as shown in FIG. In FIG. 12A, the spectrum has a fairly wide band, and it is considered that the sound hardly occurs normally. However, in such a power spectrum, time information is missing because the frequency is represented on the horizontal axis. Therefore, it cannot be determined whether the detected sound is an impact sound or a white noise continuous sound emitted by an air nozzle or the like.

  On the other hand, the envelope profile of the sample of the glass breaking sound is as shown in FIG. In FIG. 12B, the glass breaking sound is settled after showing a sharp peak that rises suddenly several times, and can be determined to be some kind of impact sound. Therefore, the clues for determining the abnormality can be increased by comparing with the preset determination criteria. Thus, according to the present embodiment, various abnormal sounds can be determined by determining sound anomalies using an envelope profile, frequency analysis, sound level, or sound time course. Can do.

  In addition, by removing the effects of room reverberation and ambient noise from the incoming sound, and by optimizing the level of the incoming sound, it becomes easier to recognize abnormal sounds, and the direction and direction of sound arrival. The accuracy of determination can be improved.

  Hereinafter, a method for reducing the influence of sound reflected from a wall or an obstacle on the calculation of the arrival direction of the sound will be described. FIG. 13 is a diagram for explaining a method for reducing the influence of reflected sound, and is shown in a time chart with time on the horizontal axis. A time chart A in FIG. 13 shows the sound pressure level of the sound collected by one of the microphones constituting the microphone array 20 and corrected by the acoustic processing unit 30 when there is reflected sound. In addition, time charts B, C, and D respectively show a sound pressure level peak detection section, a hold-off section that waits for sound acquisition, and a time-out section for arrival direction processing.

  The time chart of FIG. 13 will be described. As shown in the time chart A, the sound pressure level collected by the microphone varies with the direct sound or reflected sound from the sound source as time passes. At this time, the abnormal sound determination unit 40 determines that an abnormal sound different from the environmental sound is detected when the sound pressure level exceeds a predetermined threshold Th, and the peak detection signal rises as shown in the time chart B. . Thereafter, when the direct sound from the sound source arrives and the first peak P1 is observed, the peak detection signal of the time chart B falls.

  As shown in the time chart A, after the direct sound from the sound source arrives and the first peak P1 is generated, the second peak P2 is generated by the reflected sound. At this time, it is not preferable to calculate the arrival direction of the sound when the second peak P2 that is the reflected sound is detected, because an error is caused in the calculation process of the sound source position. Therefore, when the peak detection signal falls after the detection of the first peak P1 in the time chart B, as shown in the time chart C, the hold-off signal rises. Thereafter, the hold-off signal falls when the sound pressure level falls below a predetermined threshold Th.

  As described above, a section from when the sound pressure level exceeds the first peak P1 to below the predetermined threshold Th is defined as a prohibition section (hold-off section) T1, and the sound source position calculation unit 50 includes Even if the peak of the sound pressure level is observed, the calculation process of the sound source position is not performed. That is, when the sound source position calculation unit 50 detects the first peak of the sound pressure of the sound collected by the microphone array 20, the sound pressure near the first peak is not calculated for a predetermined period of time. Based on the waveform, the position of the sound source is calculated. By doing so, it is possible to prevent an error from being generated in the calculation of the sound source position by taking into account the influence of the reflected sound.

  In the time chart A, it is preferable to cancel the capture prohibition section T1 as soon as it falls below a predetermined threshold Th and capture the next incoming sound (third peak P3). Therefore, an upper limit value (timeout period) T2 of a section in which hold-off is performed is provided, and if the sound pressure level does not fall below the threshold value Th even after the time-out period T2, a time-out occurs and sound capturing is resumed. . That is, when the peak detection signal falls in the time chart B, the timeout processing signal rises as in the time chart D, and the timeout processing signal falls when the timeout section T2 has elapsed. If the sound pressure level exceeds a predetermined threshold Th when the timeout period T2 falls, the peak detection signal rises as shown in the time chart B, assuming that the third peak P3 is detected. The timeout period T2 can be set to 1.5 seconds at the maximum, for example.

  Note that a microphone (not shown) different from the microphone array 20 may be separately provided as a microphone for measuring the sound pressure level. At this time, it is preferable to perform level correction on the sound collected by the microphone for measuring the sound pressure level.

  As described above, in the abnormal sound determination unit 40, the level of sound collected by at least one of the microphones constituting the microphone array 20 or the microphone provided separately from the microphone array 20 exceeds a predetermined threshold Th. Among sounds, a sound that has not been collected prior to a sound exceeding a predetermined threshold Th within a predetermined time (timeout section T2 or hold-off section T1) is selected and output to the sound source position calculation unit 50. In this way, the influence of the reflected sound can be removed, and the sound source position can be calculated with higher accuracy using only the direct sound.

  Further, the abnormal sound determination unit 40 can appropriately change the above-described predetermined threshold Th, which determines the sound collected by the microphone array 20 as an abnormal sound, according to the ambient environmental sound level. When the monitoring device 1 detects abnormal sound, if the predetermined threshold Th is a constant value in places where the person's coming and going fluctuates irregularly or in places where the ambient noise level varies greatly between day and night, the environmental sound is regarded as abnormal sound. Inconvenience occurs in that it is determined or, on the contrary, abnormal sound is not determined as abnormal sound. Accordingly, the abnormal sound determination unit 40 changes the value of the predetermined threshold Th in accordance with the change in the environmental sound.

Specifically, the abnormal sound determination unit 40 obtains the maximum value (maximum sound level) for each time frame of the sound level corrected by the sound processing unit 30. The time frame is, for example, 1/30 seconds. And the abnormal sound determination part 40 calculates the average value of the maximum sound level over several frames, and makes the calculated value the predetermined threshold value Th. That is, ₁ the maximum sound level of the first frame _a, the 2 maximum sound level th frame the maximum sound level of a _2, n th frame and a _n, the average value of the maximum sound level over n frames a _ave Can be represented by the following formula (1).
By using a _ave obtained in this way, the threshold value can be adaptively changed in response to the surrounding environmental sound, and the sound source position can be calculated with higher accuracy. That is, in this case, the abnormal sound determination unit 40 sets a threshold value at which the sound level collected by the microphone array 20 is updated based on the temporal average of the sound levels collected by the microphone array 20 for a predetermined period. If so, it is determined that the sound is abnormal. The sound source position calculation unit 50 calculates the position of the sound source only when the sound collected by the microphone array 20 is determined as an abnormal sound by the abnormal sound determination unit 40 in this way. .

  In Expression (1), by changing the number of frames n used for averaging, the threshold value can be changed in accordance with the temporal fluctuation of the environmental sound. For example, when the environmental sound varies greatly, the average number of frames n may be reduced. On the contrary, when the environmental sound fluctuation is small, the average number of frames n may be increased.

  As described above, according to the present invention, the sound source position of the abnormal sound and the rank of the sound abnormality are displayed on the screen as a mark display or multi-divided area display, so that the supervisor can hear the sound. Even without recognizing the abnormality, it is possible to know the degree of abnormality and the position of the sound source of the abnormal sound simply by looking at the screen. Furthermore, according to the present invention, even when the sound source moves, the supervisor can immediately and visually know the history of the moving sound.

  In addition, humans cannot distinguish between multiple different sounds (abnormal sounds), but by visualizing them on a monitor, they can instantly recognize the occurrence of multiple sounds and their ranks of abnormalities and make appropriate decisions. it can. Also, by performing processing in real time, the abnormal state can be recognized in real time. Furthermore, by looking at the image after the abnormal sound is generated, it is possible to confirm at a glance the movement of the sound source that caused the abnormal sound and the order of the sounds generated continuously from the plurality of sound sources. Therefore, when the monitor wants to know the history of sound generation, the image output from the monitoring device 1 is continuously recorded on a recording medium and the like is read back to the past image before and after the occurrence of abnormal sound. There is no need to continue to check the image.

  Also, if necessary, specific abnormal sounds can be directly monitored from a monitoring room or the like. Further, the position of the abnormal sound and the degree of the abnormal sound can be easily visually confirmed at a distance using a portable terminal or the like.

  When the camera 10 to which the microphone array 20 is attached is arranged at a position perpendicular to the floor, there is an advantage that the range of sound to be captured is widened, and the camera to which the microphone array 20 is attached is arranged on the ceiling. Has the advantage that the sound can be captured in any direction.

  According to this embodiment, the arrival direction of the sound collected by the microphone array is calculated using the arrival time difference or the phase difference of the sound. As a result, even if a small number (minimum of three) omnidirectional microphones are used, information on the position of the sound source and the abnormality of the sound can be detected with high accuracy. As described above, since the information about the position of the sound source can be calculated with high accuracy, the calculated position of the sound source and the actual direction of arrival of the sound can be matched with high accuracy. Can be accurately combined with the captured image. In addition, since a small number of omnidirectional microphones are used, the apparatus can be configured without increasing the size, and further, the cost is not increased.

  Further, even when the calculated sound source position is outside the imaging area of the camera 10, an L-shape is formed at each of the top, bottom, left, and right edges of the display area corresponding to the sound source position, or at the four corners of the display area corresponding to the sound source position The direction of the sound source can be displayed by displaying a shape mark or the like. With such a display, even if an abnormal sound is generated outside the imaging area of the camera 10, the supervisor can recognize the direction in which the abnormal sound comes.

  According to the present embodiment, even when the lighting is turned off due to a power failure or the like, and the smoke is filled with fire and the video monitoring by the surveillance camera cannot function, the abnormal sound can be detected, so that the abnormal state is known. Can do. Also, even within the range of images displayed by a normal camera, abnormal sounds that are generated outside the door or wall, and abnormal sounds that are generated outside the range of view angle of the camera (area that is not reflected in the camera) Since it is detected and displayed, the position where the abnormality has occurred is easily identified, and an immediate response is possible.

  In addition, this invention is not limited to the said embodiment, Many change and deformation | transformation are possible. For example, by pre-calibrating the camera and correcting the lens wide-angle distortion (barrel distortion etc.) to Cartesian coordinates, errors in the combination of the sound source position information and the acquired image are minimized. can do. Also, the arrangement of the plurality of microphones is not limited to the configuration in which the microphones are arranged at equal intervals in the vertical and horizontal directions around the optical axis of the camera lens. For example, a configuration in which a plurality of microphones are installed so as to form a triangular shape with the distance from the apex centered on the optical axis of the lens at equal intervals, or microphones are arranged so as to form other polygonal shapes. You can also. Further, the microphone may be arranged on the circumference of the radius r around the optical axis of the lens.

DESCRIPTION OF SYMBOLS 1 Monitoring apparatus 10 Cameras 11 and 12 Dome type camera 15 Lens 17 Optical axis 20 Microphone array 21 Unidirectional microphone group 22 Unidirectional microphones 25, 26, and 28 Omnidirectional microphone 30 Acoustic processing unit 40 Abnormal sound determination unit 50 Sound source position calculation unit 60 Image combination unit 70 Alarm processing unit 80 Network processing unit 100 Monitor 200 Network 300, 310 Ceiling 500, 510 Sound source

Claims (3)

An imaging means for capturing an image;
A microphone array having a plurality of omnidirectional microphones arranged around the imaging means and receiving sound from a sound source;
From the arrival time difference or phase difference of the sound collected by the microphone array, the direction of the sound from the sound source with respect to the monitoring device is calculated, and the imaging means is based on the calculated direction of the sound from the sound source. Sound source position calculating means for calculating the position of the sound source in the captured image;
A display indicating the position of the sound source calculated by the sound source position calculating unit is output in combination with an image captured by the imaging unit in a display mode corresponding to the passage of time, and the calculated position of the sound source is the imaging When it is outside the predetermined imaging area in the image captured by the means, a display indicating that the position of the sound source is outside the predetermined imaging area is displayed on the edge of the predetermined imaging area corresponding to the position of the sound source. Combination means for outputting in combination with the image ;
A monitoring device comprising: The sound source position calculating means includes
When the first peak of the sound pressure of the sound collected by the microphone array is detected, the position of the sound source is not calculated for a predetermined period, and the sound source is based on the sound pressure waveform near the first peak. To calculate the position of
The monitoring apparatus according to claim 1. When the sound level collected by the microphone array exceeds a threshold value updated based on a temporal average of the sound levels collected for a predetermined period of time in the microphone array, the sound is an abnormal sound. Further comprising an abnormal sound determination means for determining,
The sound source position calculating means includes
Only when the sound collected by the microphone array is determined to be an abnormal sound by the abnormal sound determination means, the position of the sound source is calculated.
The monitoring apparatus according to claim 1.