JP6785437B2 - Human behavior monitoring device and human behavior monitoring system - Google Patents

Description

The present invention relates to a human behavior monitoring device and a human behavior monitoring system that generate monitoring images from images taken by a plurality of cameras installed in a facility and display them on a display device.

On the premises of a railway station, if there is a person who takes dangerous actions, it will hinder the safe and smooth operation of the train, so promptly identify the person who needs attention who is likely to take such dangerous actions. It is necessary to discover it and take appropriate measures at an early stage.

As a technique for quickly finding such a person requiring attention, a technique for determining a suspicious behavior by recognizing the behavior of the person from an image taken by a camera and finding a suspicious person has been conventionally known (patented). Reference 1). In addition, a person is detected from the image taken by the camera, the number of appearances appearing in the photographed image is acquired for each person, and when the number of appearances is large, it is determined that there is a high possibility of wandering, and the wandering person is detected. The technique is known (see Patent Document 2).

Japanese Unexamined Patent Publication No. 2010-272958 International Publication No. 2014/132841

However, with these conventional techniques, in an environment where a large number of people come and go, such as in a station yard, in some cases a large number of people needing attention are found, and it is appropriate to determine which person should be focused on. There was a problem that it was not possible to make a judgment and the observer could not efficiently perform the monitoring work of monitoring the person staying in the facility in real time.

Therefore, according to the present invention, even in an environment where many people come and go, the person behavior that allows the observer to appropriately pay attention to the person requiring attention and efficiently perform the monitoring work of monitoring the person staying in the facility in real time. The main purpose is to provide a monitoring device and a human behavior monitoring system.

The person behavior monitoring device of the present invention is a person behavior monitoring device that generates a monitoring image based on images taken by a plurality of cameras installed in the facility and displays it on a display device, and is described for each of the plurality of cameras. It includes a processor that acquires a captured image, acquires tracking information by analyzing the captured image, and generates a surveillance image including an image of a prowling person, and the processor is detected from the captured image for each of the cameras. The tracking process of the person is performed, the tracking information including the long-term stay flag corresponding to the person being tracked is registered in the first tracking table, and among the persons registered in the first tracking table, the said The tracking information corresponding to the person who has completed the tracking process is registered in the second tracking table, and the person being tracked registered in the first tracking table is the person whose long-stay flag is true for the first time. In the case of, the first standard image indicating that the person is a prowling person candidate is superimposed on the captured image in correspondence with the person, and the long time of the tracking person registered in the first tracking table. When the stay flag becomes true and the person being tracked is matched with the person whose long-term stay flag registered in the second tracking table truly corresponds to the person and the person is determined to be the same person, the person is determined to be the same person. The configuration is such that the first fixed image is changed to a second fixed image indicating that the person is a prowling person, and the second fixed image is superimposed on the captured image to generate a monitoring image.

Further, the person behavior monitoring system of the present invention includes a plurality of cameras installed in the facility, a server device that acquires images taken by the plurality of cameras via a network and generates a surveillance image, and outputs from the server device. A person behavior monitoring system including a display device for displaying the monitored image, wherein the server device acquires the captured image for each of a plurality of the cameras and analyzes the captured image for tracking information. Is provided with a processor that generates a surveillance image including an image of a prowling person, and the processor performs tracking processing of a person detected from the captured image for each camera and corresponds to the person being tracked. The tracking information including the long-term stay flag is registered in the first tracking table, and among the persons registered in the first tracking table, the tracking information corresponding to the person who has completed the tracking process is second. If the person being tracked registered in the tracking table of the above and registered in the first tracking table is the person whose long-stay flag is true for the first time, the person is associated with the person and is a prowling person candidate. The first standard image indicating that there is is superimposed on the captured image, the long-term stay flag of the tracking person registered in the first tracking table becomes true, and the tracking person and the second When a person is collated with a person whose long-term stay flag registered in the tracking table is truly applicable and the person is determined to be the same person, the first standard image is displayed as a prowling person for that person. The configuration is changed to a second standard image indicating that the image is present, and the second standard image is superimposed on the captured image to generate a monitoring image.

According to the present invention, the first standard image and the second standard image that make it possible to identify the degree of stay of a person based on the tracking information registered in the first and second tracking tables are superimposed on the captured image. By doing so, the observer can pay attention to the prowling person, recognizing the high need for attention. As a result, even in an environment where many people come and go, it is possible to efficiently perform monitoring work for monitoring the prowling people staying in the facility in real time.

Overall configuration diagram of the person behavior monitoring system according to this embodiment Block diagram showing a schematic configuration of server 2 Explanatory drawing showing the monitoring image displayed on the display device 3. Explanatory drawing showing an example of the tracking situation of a person Explanatory diagram showing the registered contents of the real-time tracking table and the tracking completion table managed by the server 2. Explanatory diagram showing the registered contents of the prowling person table managed by the server 2. A flow chart showing a procedure of processing performed by the processor 12 of the server 2. Flow chart showing the processing procedure of long-term stay determination (ST105) Flow chart showing the processing procedure of prowl detection (ST108)

The first invention made to solve the above problems is a person behavior monitoring device that generates a monitoring image based on images taken by a plurality of cameras installed in a facility and displays it on a display device. The camera includes a processor that acquires the captured image for each of the cameras, acquires tracking information by analyzing the captured image, and generates a surveillance image including an image relating to a prowling person. The processor is provided for each camera. The tracking process of the person detected from the captured image is performed, the tracking information including the long-term stay flag corresponding to the person being tracked is registered in the first tracking table, and the person is registered in the first tracking table. Among the persons who have completed the tracking process, the tracking information corresponding to the person who has completed the tracking process is registered in the second tracking table, and the person being tracked registered in the first tracking table is the first person to stay for a long time. When is a true person, a first fixed image indicating that the person is a prowling person candidate is superimposed on the photographed image in correspondence with the person, and the tracking is being registered in the first tracking table. The long-stay flag of the person becomes true, and the person being tracked and the person registered in the second tracking table to which the long-stay flag truly corresponds are collated with the same person. If it is determined, the first fixed image is changed to a second fixed image indicating that the person is a prowling person, and the monitoring image obtained by superimposing the second fixed image on the captured image is used. The configuration is to be generated.

According to this, based on the tracking information registered in the first and second tracking tables, the first standard image and the second standard image that can identify the degree of stay of the person are superimposed on the captured image. This allows the observer to pay attention to the prowling person, recognizing the high need for attention. As a result, even in an environment where many people come and go, it is possible to efficiently perform monitoring work for monitoring the prowling people staying in the facility in real time.

Further, in the second invention, in the processor, the same person as the person whose long-stay flag is true by the tracking process of the captured image by the first camera is photographed by the second camera. When the long-term stay flag becomes true by the tracking process of the image, the first standard image is changed to the second standard image.

According to this, when the long-term stay flag of the same person becomes true in the images taken by different cameras, the observer's attention is drawn by changing the first standard image to the second standard image. can do.

Further, in the third invention, the first standard image and the second standard image are frame images surrounding at least a part of a person area in the captured image, and the colors, shades, and lines in the frame image. At least one of the display attributes of the seed and the line thickness shall be changed according to the degree of stay.

According to this, the degree of stay of the person can be clearly displayed on the surveillance image.

Further, in the fourth invention, the first standard image and the second standard image are configured to consist of at least one of a figure, a character, and a symbol displayed inside or near a person area in the captured image. And.

According to this, the degree of stay of the person can be clearly displayed on the surveillance image.

Further, according to the fifth invention, when the processor has a plurality of tracking information for which the long-term stay flag is true for the same person, the alarm device has a person requiring attention. The configuration is such that an alarm operation is performed to notify the observer.

According to this, it is possible to quickly inform the observer that there is a person requiring attention.

The sixth invention is a plurality of cameras installed in the facility, a server device that acquires images taken by the plurality of cameras via a network and generates a surveillance image, and the above-mentioned output from the server device. A person behavior monitoring system including a display device for displaying a surveillance image, wherein the server device acquires the captured image for each of a plurality of the cameras and acquires tracking information by analyzing the captured image. A processor that generates a surveillance image including an image of a prowling person is provided, and the processor performs tracking processing of a person detected from the captured image for each camera for a long time corresponding to the person being tracked. The tracking information including the stay flag is registered in the first tracking table, and among the persons registered in the first tracking table, the tracking information corresponding to the person who has completed the tracking process is stored in the second tracking table. If the person being tracked registered in the first tracking table is the person whose long-term stay flag is true for the first time, it is indicated that the person is a prowling person candidate in correspondence with that person. The first standard image shown is superimposed on the captured image, the long-term stay flag of the tracking person registered in the first tracking table becomes true, and the tracking person and the second tracking table become true. When a person is collated with a person whose long-term stay flag registered in the above is truly applicable and the person is determined to be the same person, it is determined that the person is a person who wanders around the first standard image. The configuration is changed to the second standard image shown, and the monitoring image is generated by superimposing the second standard image on the captured image.

According to this, as in the first invention, it is possible to efficiently perform the monitoring work of monitoring the prowling person staying in the facility in real time even in an environment where many people come and go.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is an overall configuration diagram of a human behavior monitoring system according to the present embodiment.

This person behavior monitoring system monitors the behavior of a person staying in the facility, and includes a camera 1, a server 2 (person behavior monitoring device), a display device 3, and an alarm device 4. ..

A plurality of cameras 1 are installed in the facility, and each of the plurality of monitoring areas set in the facility is photographed. The camera 1 is communicably connected to the server 2 via a network.

The server 2 is installed at an appropriate place inside or outside the facility. The server 2 is composed of an information processing device such as a PC, and a display device 3 is connected to the server 2. In this server 2, the photographed image is acquired from the camera 1, the information about the state of the person in the surveillance area is acquired by the analysis of the photographed image, the surveillance image including the information is generated, and the surveillance image is generated. It is displayed on the display device 3 and viewed by the observer. In the present embodiment, a prowling person (person requiring attention) is detected by analyzing the captured image, and a monitoring image in which a standard image showing the prowling person is superimposed on the captured image is generated and displayed on the display device 3.

The alarm device 4 is connected to the server 2 and performs an alarm operation such as lighting a lamp or outputting a voice in response to an instruction from the server 2. In the present embodiment, when the prowling person is detected, the alarm device 4 is made to perform an alarm operation to notify the observer that the prowling person exists.

Next, the schematic configuration of the server 2 will be described. FIG. 2 is a block diagram showing a schematic configuration of the server 2.

The server 2 includes a communication unit 11, a processor 12, a storage unit 13, and an output unit 14.

The communication unit 11 communicates with the camera 1 via the network. In the present embodiment, the captured image transmitted from the camera 1 is received. The storage unit 13 stores a captured image of the camera 1, a program executed by the processor 12, and the like. The output unit 14 outputs screen information to the display device 3 and outputs an alarm operation instruction to the alarm device 4 based on the control of the processor 12.

The processor 12 executes the program stored in the storage unit 13. In the present embodiment, a person is tracked by the image captured by the camera 1, and a long-term stay determination is performed to determine whether or not the person is a long-term stayer who has continuously stayed in the shooting area of one camera 1 for a predetermined time or longer. Based on the result of this long-term stay determination, a prowling person (a person requiring attention who needs a high degree of attention) is detected. Then, a monitoring image in which a standard image showing the prowling person is superimposed on the captured image is generated and displayed on the display device 3, and an alarm operation for notifying the observer of the existence of the prowling person is performed on the alarm device 4. Let me do it.

Next, the monitoring image displayed on the display device 3 will be described. FIG. 3 is an explanatory diagram showing a monitoring image displayed on the display device 3.

On the premises of the station, there are people who stay for a relatively long time, such as people waiting for the arrival of the train, people taking a break, and people meeting with another person. Such a person does not hang around very much and stays within the shooting area of one camera 1. On the other hand, a person who takes a dangerous action tends to stay in the shooting area of one camera 1 while move around in a relatively wide range and walk across the shooting areas of a plurality of cameras 1.

Therefore, in the present embodiment, a person is tracked by the captured image 31 of the camera 1, and it is determined whether or not the person is a long-term stayer who stays in the shooting area of one camera 1 for a long time. A person who is determined to be a time resident multiple times for the same person, detects that person as a prowling person (a person who needs attention with a high degree of caution), and represents the prowling person. A monitoring image in which the rectangular shape 32 (standard image, frame image) is superimposed on the position corresponding to the corresponding person in the captured image 31 is generated and displayed on the display device 3.

In addition, if a person who may be determined to be a prowling person is made to pay attention to the observer before being confirmed as a prowling person, it becomes possible to immediately respond to the prowling person at the confirmed timing. Therefore, in the present embodiment, if it is first determined to be a long-term resident, that is, if the determination to be a long-term resident is made once, a prowling candidate person (a person requiring attention with a low degree of caution). A person rectangle 32 (standard image) representing a prowling candidate person is generated as a monitoring image superimposed on a position corresponding to the corresponding person in the captured image 31 and displayed on the display device 3.

Further, in the present embodiment, the wandering candidate person and the wandering person are identified by the difference in the display mode of the person rectangle 32. In the example shown in FIG. 3, it is possible to identify by the difference in the line type of the frame line of the person rectangle 32. That is, the person rectangle 32 is displayed as a dotted line for the prowling candidate person, and the person rectangle 32 is displayed as a solid line for the prowling person. Further, in the example shown in FIG. 3, a person ID assigned to each person is displayed in the vicinity of the person rectangle 32.

When there are a plurality of people who display the person rectangle 32 in one captured image 31, that is, a prowl candidate person or a prowl person, different colors (for example, red, orange, blue, green, etc.) are used for each person. ) To display the person rectangle 32.

Here, FIG. 3A is a surveillance image of the first camera 1 (CamA) for photographing the ticket gate. In this surveillance image, two people (Id1, Id2) are determined to be long-term residents because the staying time is long, and the person rectangle 32 is displayed, but because they are two people and a candidate for wandering. , A dotted person rectangle 32 representing a prowling candidate person is displayed.

FIG. 3B is a surveillance image of the second camera 1 (CamB) for photographing the home, and is taken after the surveillance image of FIG. 3A. In this monitoring image, two people (Id11, Id22) are determined to be long-term residents because the staying time is long, and the person rectangle 32 is displayed, but one person (Id11) is shown in the figure. Since it is the same as the person determined to be a long-term resident at the timing of the monitoring image of 3 (A), the person becomes a prowling person, and a solid line person rectangle 32 representing the prowling person is displayed. Further, since the other person (Id22) has not been determined to be a long-term resident in the past, he / she becomes a prowling candidate person, and a dotted person rectangle 32 representing the prowling candidate person is displayed.

In this embodiment, a person rectangle 32 (frame image) surrounding the whole body of the person is displayed as a standard image showing the distinction between the prowling candidate person and the prowling person (degree of stay of the person), and the line type of the frame line. (Dotted line and solid line) are used to distinguish between a prowling candidate and a prowling person, but the display attribute for distinguishing between a prowling candidate and a prowling person is not limited to the border line type.

For example, the color of the border may be used to distinguish between the prowling candidate and the prowling person. For example, the prowling candidate may have a green border, and the prowling person may have a red border. In addition, the prowling candidate person and the prowling person may be distinguished by the shade of the border. For example, the prowling candidate may have a light border and the prowling person may have a dark border. In addition, the thickness of the border may be used to distinguish between the prowling candidate and the prowling person. For example, the prowling candidate may have a thin border, and the prowling person may have a thick border.

Further, the person rectangle 32 (frame image) may be any one that surrounds all or part of the person area, and may be, for example, one that surrounds the upper body or face of the person.

Furthermore, the standard image showing the distinction between the prowling candidate person and the prowling person (degree of stay of the person) is not limited to the frame image surrounding the person area, but consists of at least one of a figure, a character, and a symbol. , May be displayed inside or near the person area.

For example, as a standard image, a figure or symbol having an appropriate shape such as a circle is displayed in the center of the person area, and a figure (bloom or arrow) indicating the person area is displayed in the vicinity of the person area. The display mode of the figure may be changed between the prowling person and the prowling candidate person. Further, the characters "prowling candidate" and "prowling" may be displayed in the vicinity of the person area. In addition, symbols representing the prowling person and the prowling candidate person may be displayed inside or near the person area.

Next, the long-term stay determination and the wandering detection performed by the processor 12 of the server 2 will be described. FIG. 4 is an explanatory diagram showing an example of a person's tracking situation.

In the present embodiment, a long-term stay determination is performed to determine whether or not the person detected from the image captured by the camera 1 is a long-term stayer. In this long-term stay determination, the time during which a person is being tracked by the image captured by one camera 1, that is, the elapsed time from the tracking start time to the current time is measured. Then, this measured time is set as the staying time during which the person stays in the shooting area of one camera 1, and when the staying time exceeds a predetermined threshold value (for example, 20 seconds), the person is used for a long time. Judges as a resident and displays the person rectangle of the prowling candidate person.

Further, in the present embodiment, when it is determined to be a long-term stayer by the long-term stay determination, a person is between a person who was determined to be a long-term stayer in the past and a person who was determined to be a long-term stayer this time. When collation (person reidentification processing) is performed and a person determined to be the same person is found, that person is determined to be a prowling person, and the person rectangle of the prowling person is displayed.

In the example shown in FIG. 4, a person (Id1) first appears in the captured image of the second camera 1 (CamB), and this person is determined to be a long-term resident because the staying time is long, and is a prowl candidate. The person rectangle of the person is displayed. Next, a person (Id2) appears in the captured image of the first camera 1 (CamA), and this person is not determined to be a long-term resident because the staying time is short, so that the person rectangle is not displayed.

Next, a person (Id11) appears in the captured image of the first camera 1 (CamA), and this person is determined to be a long-term resident because the staying time is long. At this time, since there is a person (Id1) who has been determined to be a long-term resident in the past, a person collation is performed with this person. If the same person is determined by this person collation, the person rectangle of the prowling person is displayed.

Further, a person (Id44) appears in the captured image of the first camera 1 (CamA), and this person is determined to be a long-term resident because the staying time is long. At this time, a person collation is performed with a person (Id1) who has been determined to be a long-term resident in the past, but since they are not the same person, a person rectangle of a prowling candidate person is displayed.

Further, a person (Id222) appears in the captured image of the second camera 1 (CamB), and this person is determined to be a long-term resident because the staying time is long. At this time, since there are three persons (Id1, Id11, Id44) who have been determined to be long-term residents in the past, person matching is performed with these persons, but since they are not the same person, they wander. The person rectangle of the candidate person is displayed.

In the present embodiment, the person is collated with the person who was determined to be a long-term resident in the past at the timing when it is determined to be a long-term resident, but the person is detected from the captured image. At the same timing, a person may be collated with a person who has been determined to be a long-term resident in the past. In this way, as soon as a prowling candidate person or a person determined to be a prowling person in the past is detected in the captured image, the person is made to display a person rectangle corresponding to the prowling candidate person and the prowling person. Can be done.

Further, in this way, the person who was determined to be a prowling candidate in the past has a longer staying time in the shooting area of the current camera 1, and when it is determined to be a long-term staying person, the prowling candidate is to the prowling person. It is changed to, and it is displayed by switching from the person rectangle of the prowling candidate person to the person rectangle of the prowling person.

Next, the real-time tracking table and the tracking completion table managed by the server 2 will be described. FIG. 5 is an explanatory diagram showing the registered contents of the real-time tracking table and the tracking completion table.

The real-time tracking table (real-time tracking information) shown in FIG. 5A registers information about the person being tracked, and the camera ID (Cam), person ID (Id), and tracking are stored in the real-time tracking table. Information on each item of the start time (Begin Time), the latest stay time (Exist Time), and the long stay flag (Long Stay Flag) is registered.

In the present embodiment, the processor 12 of the server 2 performs each process of person detection and person tracking for each of the frames (captured images) sequentially transmitted from the camera 1, and the camera ID (Cam) acquired by this process. ), Person ID (Id), tracking start time (Begin Time), and latest stay time (Exist Time) are registered in the real-time tracking table.

Here, the camera ID (Cam) is identification information given to each camera 1. The person ID (Id) is identification information given to each person detected from the captured image. The tracking start time (Begin Time) is the time when a person first appears in the captured image and the tracking is started, that is, the shooting time of the frame in which the person is first detected. The latest stay time (Exist Time) is the current time being tracked, that is, the shooting time of the current frame in which a person is detected. The long stay flag (Long Stay Flag) indicates whether or not the person is a long-term stayer (True / False).

Further, in the processor 12 of the server 2, the difference between the tracking start time (Begin Time) and the latest stay time (Exist Time) of the real-time tracking table is calculated as the stay time, and whether or not the detected person is a long-term stayer. A long-stay determination is performed to determine whether or not, and the determination result is registered in the real-time tracking table as a long-stay flag (Long Stay Flag). That is, the long stay flag is rewritten from "False" to "True" for the person determined to be a long-term stayer.

The tracking completion table (fixed tracking information) shown in FIG. 5B registers information about a person who has completed tracking, and the camera ID (Cam), person ID (Id), and the person ID (Id) are registered in this tracking completion table. Information on each item of the tracking start time (Begin Time), the tracking end time (Exist Time), and the long stay flag (Long Stay Flag) is registered.

Here, the item that was the latest stay time (Exist Time) in the real-time tracking table shown in FIG. 5A is that the target person leaves the shooting area of the camera 1 and the tracking is completed. It is determined as the tracking end time, that is, the last time detected in the captured image. Others are the same as the real-time tracking table.

Next, the prowling person table managed by the server 2 will be described. FIG. 6 is an explanatory diagram showing the registered contents of the prowling person table.

This prowling person table (prowling person information) registers information about a person who is determined to be a prowling person, and the camera ID (Current Cam) currently being tracked and the person currently being tracked are registered in this prowling person table. Information on each item of ID (Current Id), camera ID (Previous Cam) when previously tracked, person ID (Previous Id) when previously tracked, and prowling person degree (Walk Around Score) is registered. There is.

In the present embodiment, in the processor 12 of the server 2, when the long stay flag (Long Stay Flag) becomes "True" in the real-time tracking table (see FIG. 5 (A)), the person, that is, the long time Prowl detection is performed to determine whether the resident is a prowl person.

In this prowl detection, a person whose long stay flag (Long Stay Flag) is set to "True" in the tracking completion table (see FIG. 5 (B)), that is, a person who has been determined to be a long-term stayer in the past. , The same person among the person whose Long Stay Flag is set to "True" in the real-time tracking table (see FIG. 5 (A)), that is, the person who is determined to be a long-term stayer this time. Performs person verification (person reidentification processing) to determine whether or not.

In this person collation, the similarity between two people is calculated as the walking around score, and when the walking around score exceeds a predetermined threshold value (for example, 0.80), it is determined to be the same person. To do. Then, when a person determined to be the same person is found, that person is determined to be a prowling person, and information about that person is registered in the prowling person table.

Next, the procedure of the processing performed by the processor 12 of the server 2 will be described. FIG. 7 is a flow chart showing a procedure of processing performed by the processor 12 of the server 2. FIG. 8 is a flow chart showing a processing procedure for long-term stay determination (ST105). FIG. 9 is a flow chart showing a processing procedure for prowl detection (ST108).

The processor 12 of the server 2 first acquires a captured image (frame) transmitted from the camera 1 and received by the communication unit 11, and adds a camera ID and a frame number (ST101). Then, a person is detected on the captured image (ST102). In the camera 1, a shooting image (frame) is transmitted with a shooting time added.

In this person detection, a person area in the captured image is specified, a person rectangle is set on the photographed image so as to surround the person area, and information on the position and size of the person rectangle, that is, the upper left of the person rectangle. Get the coordinates (x, y), width (w), and height (h) of the vertices of. A known image recognition technique may be used for this person detection. A person ID is assigned to the person rectangle.

Next, it is determined whether or not a person is detected from the captured image (ST103). Here, if no person is detected, the process proceeds to the next captured image (frame) process. On the other hand, if a person is detected, then the person is tracked (ST104).

In this person tracking, the process of associating the person detected in the photographed image (frame) this time with the person detected in the photographed image (frame) in the past is performed. A known image recognition technique may be used for this person tracking.

At this time, three tracking states of new tracking (new), tracking completion (complete), and tracking (track) are determined. In the case of new tracking (new), the information of the corresponding person is registered in the real-time tracking table (see FIG. 5 (A)), and in the case of tracking completion (complete), the tracking completion table (FIG. 5 (B)). ), Register the information of the corresponding person, and if tracking is in progress (track), update the latest stay time (Exist Time) of the real-time tracking table.

Next, a long-term stay determination is performed (ST105). In this long-term stay determination, it is determined whether or not the person being tracked by the captured image is a long-term stayer who has continuously stayed in the shooting area of the camera 1 that captures the captured image for a predetermined time or longer. ..

FIG. 8 shows a detailed procedure for determining the long-term stay. Here, when the tracking state (s) is being tracked (track) and the staying time T exceeds the threshold value TH (for example, 20 seconds) (Yes in ST203), the person stays for a long time. The long stay flag of the corresponding person in the real-time tracking table (see FIG. 5A) is set to “True” (ST204). Here, n is the serial number of the person rectangle, N is the total number of the person rectangles, and each person detected by the current person detection is determined in order.

The stay time T is the elapsed time from the tracking start time to the current time, and is the difference between the tracking start time (Begin Time) and the latest stay time (Exist Time) in the real-time tracking table (see FIG. 5 (A)). Is calculated as. In addition, a person whose tracking state (s) is new tracking (new) or tracking completed (complete) is excluded from the target of long-term stay determination.

When the long-term stay determination (ST105) is performed in this way, then, as shown in FIG. 7, the prowling candidate person is displayed (ST106). Here, a person who is determined to be a long-term resident, that is, a person whose long stay flag (Long Stay Flag) of the real-time tracking table (see FIG. 5 (A)) is "True" is set as a prowl candidate. indicate. That is, the person rectangle (dotted line) of the prowling candidate person is displayed on the corresponding person in the captured image (see FIG. 3).

Next, whether or not there is a person who has been determined to be a long-term stayer in the past, that is, a person whose long stay flag (Long Stay Flag) is "True" in the tracking completion table (see FIG. 5B). (ST107). Here, if there is no person determined to be a long-term resident in the past (No in ST107), the process proceeds to the next captured image (frame). On the other hand, if there is a person who has been determined to be a long-term resident in the past (Yes in ST107), prowl detection is performed (ST108).

In this prowl detection (ST108), a person is collated (person reidentification processing) between a person who has been determined to be a long-term resident in the past and a person who has been determined to be a long-term resident this time, and is determined to be the same person. If the person is found, the person is determined to be a prowling person, and the prowling person table (see FIG. 6) is updated.

The processing procedure for this prowl detection is shown in FIG. 9 in detail. Here, k and K are persons who have been determined to be long-term residents in the past, that is, the long-stay flag (Long Stay Flag) is set to "True" in the tracking completion table (see FIG. 5B). The serial numbers and total numbers of the persons, m and M, are the persons determined to be long-term residents this time, that is, the long stay flag (Long Stay Flag) is "in the real-time tracking table (see FIG. 5 (A))". It is the serial number and total number of people who are "True", and one-to-one person matching is performed in order between the person who was determined to be a long-term resident in the past and the person who was determined to be a long-term resident this time. ing.

Here, in the present embodiment, the person collation is performed for the person determined to be a long-term resident within the predetermined collation target period retroactive from the present to the past. The length of the collation target period may be set to, for example, about several tens of minutes. In addition, it is preferable that the user can appropriately set the length of the collation target period.

Moreover, a known method may be used for person collation. Specifically, for two people to be collated, a person image obtained by cutting out a person rectangular area from a photographed image is acquired, and a color feature amount is extracted from the two person images. For example, a feature amount vector is acquired by generating a histogram of the HSV color space from each person image. Next, the similarity between the two person images is calculated by comparing the color features. Then, the combination having the highest degree of similarity is selected, and when the degree of similarity is higher than a predetermined threshold value, the persons in the two person images are determined to be the same person.

It should be noted that the person determined to be a long-term resident in the past and the person determined to be a long-term resident this time do not always appear in different images taken by the camera 1. That is, when all the places of the facility are not photographed by the camera 1, after disappearing from the image captured by the camera 1, they appear in the image captured by the same camera 1 without appearing in the image captured by another camera 1. In this case, the images taken by the same camera 1 may be continuously determined to be long-term residents.

When the prowl detection (ST108) is performed in this way, then, as shown in FIG. 7, the prowl person is displayed (ST109). Here, a person rectangle for a prowling person is displayed on a photographed image as a person detected as a prowling person by prowling detection. At this time, for the prowling person, the long stay flag is set to "True" in the real-time tracking table (see FIG. 5 (A)), and the same camera ID and person ID as in the real-time tracking table are combined. It is registered in the prowling person table (see FIG. 6). Further, while the long stay flag is set to "True" in the real-time tracking table, that is, while a person is detected, the display of the person rectangle for the prowling person is continued.

As described above, embodiments have been described as an example of the techniques disclosed in this application. However, the technique in the present disclosure is not limited to this, and can be applied to embodiments in which changes, replacements, additions, omissions, etc. have been made. It is also possible to combine the components described in the above embodiments into a new embodiment.

For example, in the above-described embodiment, an example of a railway station has been described, but the present invention is not limited to such a railway station, and can be widely applied to various facilities.

Further, in the above-described embodiment, the monitoring screen is displayed on the display device connected to the server so that the observer can view it. By installing the server and the display device in the monitoring room, the monitoring screen of the display device is displayed. However, if a mobile device such as a smartphone or tablet terminal can be connected to a server via a wireless communication network such as a wireless LAN, the monitor screen can be displayed anywhere in the facility. Can be viewed. Further, if a user terminal such as a mobile device or a PC is configured to be able to connect to a server via a wide area communication network such as the Internet, the monitoring screen can be viewed at any place outside the facility.

Further, in the above-described embodiment, when the determination result that the same person is a long-term resident is obtained once, a standard image (personal rectangle) of the prowling candidate person is displayed, and the length is long for the same person. When the judgment result of being a time resident is obtained twice, the standard image of the prowling person is displayed, but when the judgment result of being a long-time resident is obtained three times or more for the same person. May further change the display mode of the standard image according to the number of determinations.

In addition, after displaying the standard image (person rectangle) of the prowling candidate person, the standard image (person rectangle) of the prowling person is not immediately displayed in the judgment result of the next long-term resident, and the long-time resident is judged. When the results are continuous (twice or more), a standard image of the prowling person may be displayed.

The person behavior monitoring device and the person behavior monitoring system according to the present invention are monitoring operations in which the observer appropriately pays attention to the person requiring attention and monitors the person staying in the facility in real time even in an environment where many people come and go. It has the effect of being able to efficiently perform the above, and is useful as a person behavior monitoring device and a person behavior monitoring system that generate a monitoring image from images taken by a plurality of cameras installed in the facility and display it on a display device. ..

1 camera 2 server (personal behavior monitoring device, server device)
3 Display device 4 Alarm device 11 Communication unit 12 Processor 13 Storage unit 14 Output unit

Claims (6)

A human behavior monitoring device that generates surveillance images based on images taken by multiple cameras installed in the facility and displays them on a display device.
It is provided with a processor that acquires the captured image for each of the plurality of cameras, acquires tracking information by analyzing the captured image, and generates a surveillance image including an image relating to a prowling person.
The processor
For each camera, the tracking process of the person detected from the captured image is performed, and the tracking information including the long-term stay flag corresponding to the person being tracked is registered in the first tracking table.
Among the persons registered in the first tracking table, the tracking information corresponding to the person who has completed the tracking process is registered in the second tracking table.
When the person being tracked registered in the first tracking table is the person whose long-stay flag is true for the first time, the first person is associated with the person and indicates that the person is a prowling person candidate. The standard image is superimposed on the captured image,
The long-stay flag of the tracking person registered in the first tracking table becomes true, and the long-stay flag registered in the tracking person and the second tracking table truly corresponds. When a person is collated with a person and determined to be the same person, the first standard image is changed to a second standard image indicating that the person is a prowling person, and this second standard image is changed. A human behavior monitoring device characterized by generating a monitoring image in which a standard image is superimposed on the captured image. The processor
The same person whose long-stay flag is true by the tracking process of the captured image by the first camera is the same person as the person whose long-stay flag is true by the tracking process of the captured image by the second camera. The person behavior monitoring device according to claim 1, wherein when is true, the first standard image is changed to the second standard image. The first standard image and the second standard image are frame images that surround at least a part of a person area in the captured image, and at least the color, shading, line type, and line thickness in the frame image. The person behavior monitoring device according to claim 1 or 2, wherein any of the display attributes is changed. Claim 1 or the second standard image is characterized in that the first standard image and the second standard image consist of at least one of a figure, a character, and a symbol displayed inside or in the vicinity of a person area in the captured image. The person behavior monitoring device according to any one of claims 2. When the same person has a plurality of tracking information for which the long-term stay flag is true, the processor performs an alarm operation to notify the observer that the prowling person exists in the alarm device. The person behavior monitoring device according to any one of claims 1 to 4, wherein the person behavior monitoring device is characterized. With multiple cameras installed in the facility
A server device that acquires images taken by these multiple cameras via a network and generates surveillance images,
A display device that displays the monitoring image output from this server device, and
It is a person behavior monitoring system equipped with
The server device
It is provided with a processor that acquires the captured image for each of the plurality of cameras, acquires tracking information by analyzing the captured image, and generates a surveillance image including an image relating to a prowling person.
The processor
For each camera, the tracking process of the person detected from the captured image is performed, and the tracking information including the long-term stay flag corresponding to the person being tracked is registered in the first tracking table.
Among the persons registered in the first tracking table, the tracking information corresponding to the person who has completed the tracking process is registered in the second tracking table.
When the person being tracked registered in the first tracking table is the person whose long-stay flag is true for the first time, the first person is associated with the person and indicates that the person is a prowling person candidate. The standard image is superimposed on the captured image,
The long-stay flag of the tracking person registered in the first tracking table becomes true, and the long-stay flag registered in the tracking person and the second tracking table truly corresponds. When a person is collated with a person and determined to be the same person, the first standard image is changed to a second standard image indicating that the person is a prowling person, and this second standard image is changed. A human behavior monitoring system characterized by generating a monitoring image in which a standard image is superimposed on the captured image.

Images