JP6617085B2 - Object situation estimation system, object situation estimation apparatus, object situation estimation method, and object situation estimation program - Google Patents

Description

  The present invention relates to an object situation estimation system, an object situation estimation apparatus, an object situation estimation method, and an object situation estimation program that detect a specific detection target object from an image captured by a camera and estimate the situation.

  Install a camera at an intersection, etc., detect a specific detection target object such as a person (pedestrian) from an image (captured image) taken with the camera, and estimate its situation (for example, across a road, etc.) An object situation estimation system is known. Further, it is known that machine learning technology is promising as a technology for detecting a person from a captured image.

Japanese Patent Laying-Open No. 2015-90679

  The installation position of the camera is different in each place, and the appearance (photographing direction, size, etc.) of the person to be detected reflected in the photographed image in each camera is also different. Therefore, when using a machine learning technique for detecting a person, it is necessary to prepare a dictionary corresponding to the installation position of each camera.

  However, it is difficult to prepare a dictionary for machine learning technology for each camera from the viewpoint of cost and man-hour. Machine learning for updating a dictionary has already been proposed, but it is still not easy to prepare learning data necessary for updating the dictionary.

  The present invention has been made in view of the above problems, and an object situation estimation system, an object situation estimation apparatus, an object situation estimation method, and an object situation capable of automatically generating learning data necessary for updating a dictionary. The purpose is to provide an estimation program.

  An object situation estimation system according to an aspect of the present invention uses a camera that shoots and generates a photographed image and a dictionary to detect a specific detection target object from the photographed image, and to detect the situation of the detection target object. Learning that generates learning data by determining an object detection unit that performs estimation, a movement information calculation unit that calculates movement information of a moving object in the captured image, and a situation of the moving object based on the movement information It has a configuration including a data generation unit and an additional learning unit that updates the dictionary based on the learning data.

  With this configuration, in a system that uses a dictionary to detect a specific detection target object from a captured image and estimates the status of the detection target object, the learning data necessary to update the dictionary is automatically generated from the captured image. Can be generated.

  In the object state estimation system, the state of the detection target object and the moving object may be a direction of the detection target object and the moving object.

  With this configuration, it is possible to determine the orientation information necessary for the learning data from the movement information.

  The object situation estimation system may further include an object candidate region extraction unit that extracts an object candidate region from the captured image, and the object detection unit includes a part of the captured image that includes the object candidate region. The detection target object may be detected from the region.

  With this configuration, the detection target object is detected in a partial region including the object candidate region, so that the processing load of the object detection can be reduced and the processing speed can be improved.

  The object situation estimation system may further include an object candidate area extraction unit that extracts an object candidate area from the captured image, and the movement information calculation unit uses the object candidate area as the movement information of the moving object. The movement information may be calculated.

  With this configuration, since the movement information is calculated using the object candidate region as a moving object, learning data can be generated as negative example data for a moving object that is not detected as a detection target object by the object detection unit.

  In the object situation estimation system, the camera may continuously shoot and generate a plurality of frames of the captured image, and the object candidate area extraction unit may be based on a difference image of the captured images of a plurality of frames. The object candidate area may be extracted.

  With this configuration, it is possible to reduce the processing load for extracting the object candidate region and improve the processing speed.

  In the object situation estimation system, the learning data generation unit includes a determination unit that determines whether or not the moving object is the detection target object, and the moving object that is determined to be the detection target object Learning data may be generated.

  With this configuration, learning data can be generated as positive example data for a moving object that corresponds to a detection target object.

  In the object state estimation system, the determination unit may determine whether the moving object is the detection target object based on the detection result.

  With this configuration, it is possible to determine whether the moving object is a detection target object using the detection result of the object detection unit.

  An object situation estimation apparatus according to an aspect of the present invention is an object situation estimation apparatus that is communicably connected to a camera that shoots and generates a photographed image, and that obtains the photographed image from the camera. An object detection unit that detects a specific detection target object from the captured image and estimates the state of the detection target object, a movement information calculation unit that calculates movement information of a moving object in the captured image, and the movement It has the structure provided with the learning data generation part which produces | generates learning data by determining the condition of the said moving object based on information.

  With this configuration, a device that detects a specific detection target object from a captured image using a dictionary and estimates the state of the detection target object automatically acquires learning data necessary for updating the dictionary from the captured image. Can be generated.

  An object situation estimation method according to an aspect of the present invention includes a step of shooting to generate a shot image, a dictionary is used to detect a specific detection target object from the shot image, and a status of the detection target object A step of estimating, a step of calculating movement information of the moving object in the captured image, a step of generating learning data by determining a situation of the moving object based on the movement information, and the learning data And updating the dictionary based on the above.

  With this configuration, a specific detection target object is detected from the captured image using a dictionary, the situation of the detection target object is estimated, and learning data necessary to update the dictionary can be automatically generated from the captured image. .

  The object situation estimation program according to one embodiment of the present invention uses a dictionary of a computer of an object situation estimation apparatus that is communicably connected to a camera that shoots and generates a photographed image and acquires the photographed image from the camera. An object detection unit that detects a specific detection target object from the photographed image and estimates the state of the detection target object, a movement information calculation unit that calculates movement information of a moving object in the photographed image, and the movement information By determining the state of the moving object based on the above, the learning data generating unit that generates learning data and the additional learning unit that updates the dictionary based on the learning data are caused to function.

  With this configuration, a specific detection target object is detected from the captured image using a dictionary, the situation of the detection target object is estimated, and learning data necessary to update the dictionary can be automatically generated from the captured image. .

  According to the present invention, in a system for detecting a specific detection target object from a photographed image using a dictionary and estimating the situation of the detection target object, learning data necessary for updating the dictionary is obtained from the photographed image by itself. Can be generated automatically.

The block diagram which shows the structure of the object condition estimation system of embodiment of this invention The figure which shows the process which produces | generates the background difference image of embodiment of this invention The figure which shows the process which produces | generates the difference image between frames of embodiment of this invention. The figure explaining the process in the object detection part of embodiment of this invention The figure which shows the process of the output determination part of embodiment of this invention The figure which shows the process of the motion vector calculation part of embodiment of this invention The flowchart of the person presence determination process in the person presence determination part of embodiment of this invention Flow chart of person orientation determination processing in the person orientation determination unit of the embodiment of the present invention Flow chart of person similarity determination processing in the person similarity determination unit of the embodiment of the present invention

  Embodiments of the present invention will be described below with reference to the drawings. The embodiment described below shows an example when the present invention is implemented, and the present invention is not limited to the specific configuration described below. In carrying out the present invention, a specific configuration according to the embodiment may be adopted as appropriate.

  FIG. 1 is a block diagram showing a configuration of an object situation estimation system according to an embodiment of the present invention. The object situation estimation system 1 includes a camera 10 and an object situation estimation apparatus 20. The camera 10 is installed on the street, and specifically, for example, is installed so as to face an intersection from a relatively high place. The camera 10 continuously shoots at a predetermined frame rate to generate a shot image. The camera 10 sequentially transmits captured images obtained continuously to the object state estimation apparatus 20 at a predetermined transmission rate.

  The object state estimation device 20 acquires a captured image generated by the camera 10 and detects a specific detection target object (in this embodiment, a walking person (pedestrian)) reflected in the captured image. Estimate the situation. In the present embodiment, the situation of the person estimated by the object situation estimation apparatus 20 is the direction of the person, and based on this, whether the person is waiting for a signal, crossing the road, or You can determine whether you have crossed the road.

  The camera 10 and the object situation estimation apparatus 20 are communicably connected to each other so that the camera 10 transmits a captured image to the object situation estimation apparatus 20 and the object situation estimation apparatus 20 can receive the photographed image. The camera 10 and the object situation estimation device 20 may be connected by wire or may be connected wirelessly. Further, a communication network such as the Internet may be interposed between the camera 10 and the object state estimation device 20. That is, the object state estimation device 20 may be provided at a position geographically separated from the camera 10.

  The object situation estimation apparatus 20 is configured by a computer including a storage device, a ROM, a RAM, a CPU, and the like. The CPU reads and executes an object situation estimation program stored in the storage apparatus, thereby realizing the following functions. The The object situation estimation device 20 includes an object candidate area extraction unit 21, an object detection unit 22, a movement information generation unit 23, a learning data generation unit 24, an additional learning unit 25, a past image storage unit 31, and a background image. A storage unit 32, a learning data storage unit 33, and a person dictionary storage unit 34 are provided, and an object situation estimation method is executed using these.

  In the present embodiment, the time transition is discretely shown, the current time point is represented by a symbol t, the time point one step before the present time is represented by a symbol t-1, and the time point two steps before the current time is represented by a symbol t. -2. Since one step corresponds to one frame, the current latest frame is represented by a symbol t, the previous frame is represented by a symbol t-1, and the second previous frame is represented by a symbol t-2.

  The captured image transmitted from the camera 10 to the object situation estimation apparatus 20 is stored in the past image storage unit 31 and input to the object candidate region extraction unit 21. The past image storage unit 31 stores a predetermined number of frames of captured images of past frames. The object candidate area extraction unit 21 extracts an area where an object is considered to exist (hereinafter referred to as “object candidate area”) from the captured image. In particular, the object candidate area extraction unit 21 of the present embodiment extracts an area where a moving object exists as an object candidate area.

  The object detection unit 22 detects a specific detection target object (person) from the object candidate region extracted from the object candidate region extraction unit 21. When there are a plurality of persons in the captured image, the object detection unit 22 detects each of them. In addition, the object detection unit 22 estimates the situation of each detected person, and outputs the presence / absence of the person (person detection result) and the situation estimation result.

  The movement information calculation unit 23 is based on the captured image read from the past image storage unit 31 and the object candidate area input from the object candidate area extraction unit 21, and the movement information (not limited to a person) of movement information (not limited to a person). Specifically, the motion vector of each moving object is calculated. The movement information calculation unit 23 outputs the calculated movement information to the learning data generation unit 24.

  The learning data generation unit 24 determines the direction of the moving object based on the movement information of each moving object input from the movement information calculation unit 23, and generates learning data for each direction. The generated learning data is stored in the additional learning data storage unit 33 as additional learning data.

  The additional learning unit 25 performs additional learning using the learning data for additional learning stored in the additional learning data storage unit 33 and updates the person dictionary stored in the person dictionary storage unit 34. A new person dictionary is generated.

  In the object situation estimation apparatus 20, the object candidate area extraction unit 21, the object detection unit 22, the past image storage unit 31, the background image storage unit 32, and the person dictionary storage unit 34 detect a person from a captured image, and the situation The object candidate region 21, the movement information calculation unit 23, the learning data generation unit 24, the additional learning unit 25, the past image storage unit 31, and the additional learning data storage unit 33 perform additional learning. It is the structure for going and updating a person dictionary. Hereinafter, each function of the object situation estimation apparatus 20 will be described in more detail.

  First, a configuration for detecting a person from a captured image and estimating the situation will be described. The past image storage unit 31 stores a captured image of a past frame (hereinafter also referred to as “past frame image”) used for calculating the inter-frame difference image FDt. The past frame image is a captured image taken by the camera 10 at a past time point, and in the present embodiment, the past image storage unit 31 stores the past frame image It-1 in the last two steps (the past two frames). , It-2 is stored. The past frame image stored in the past image storage unit 31 is sequentially updated every time a newest photographed image at the present time (hereinafter also referred to as “current frame image”) is newly received from the camera 10.

  The background image storage unit 32 stores a captured image (past frame image) used for calculation of the background difference image BDt as a background image BG. The background image storage unit 32 reads and stores captured images received from the camera 10 and stored in the past image storage unit 31. The background image storage unit 32 updates the background image at a predetermined time interval (for example, one hour interval). As the background image, it is desirable to adopt a captured image that does not include a moving object but includes only a stationary object.

  The object candidate area extraction unit 21 includes a background difference calculation unit 211 and an inter-frame difference calculation unit 212. The background difference calculation unit 211 generates a background difference image BDt using the current frame image It received from the camera 10 and the background image BG stored in the background image storage unit 32.

  FIG. 2 is a diagram illustrating processing for generating the background difference image BDt. The background difference calculation unit 211 acquires the current frame image It from the camera 10, reads the background image BG from the background image storage unit 32, and excludes the information of the background image BG from the current frame image It as shown in FIG. As a result, the background difference image BDt is calculated.

  Since the background image BG includes information related to a stationary object in the current frame image It, the background difference image BDt excludes information related to the stationary object included in the current frame image It and extracts a moving object from the current frame image It. The resulting image. In the example of FIG. 2, the background difference image BDt is an image obtained by extracting one vehicle M1 temporarily stopped on the road and two vehicles M2 and M3 traveling on the road from the current frame image It. It has become.

  The interframe difference calculation unit 212 uses the current frame image It received from the camera 10 and the past frame images It-1 and It-2 stored in the past image storage unit 31 to calculate the interframe difference image FDt. calculate.

  FIG. 3 is a diagram illustrating processing for generating the inter-frame difference image FDt. The inter-frame difference calculation unit 212 acquires the current frame image It from the camera 10 and reads the past frame images It-1 and It-2 from the past image storage unit 31. The past frame images It-1 and It-2 are photographed images one frame before and two frames before the current time t. The interframe difference calculation unit 212 calculates an interframe difference image FDt from the current frame image It, the past frame image It-1, and the past frame image It-2.

  As shown in FIG. 3, the inter-frame difference calculation unit 212 first calculates the difference image D1 between the current frame image It and the past frame image It-1, and the past frame image It-1 and the past frame image It-2. The difference image D2 is calculated. Next, the interframe difference calculation unit 212 calculates a common part (AND) of the difference image D1 and the difference image D2 as an interframe difference image FDt. As described above, the inter-frame difference image FDt is obtained by extracting difference information from the time-series three-frame images, and the amount of change between frames can be stably extracted.

  The inter-frame difference image FDt is an image obtained by extracting a short-term feature (for example, an object moving instantaneously between frames) from the current frame image It, and reflects the long-term feature of the current frame image It. This is different from the above-described background difference image BDt. For example, when the inter-frame difference image FDt in FIG. 3 is compared with the background difference image BDt in FIG. 2, the automobile M1 temporarily stopped in the current frame image It can be extracted and visually recognized in the background difference image BDt. The intermediate difference image FDt is not extracted and cannot be visually recognized. In both the background difference image BDt and the inter-frame difference image FDt, an object that exists in the current frame image is extracted instead of the past frame image. Also called “moving object”.

  The inter-frame difference calculation unit 212 further stores the current frame image It in the past image storage unit 31 as the past frame image It-1 and the current past frame image It-1 as the past frame image It-2. The past image storage unit 31 is updated.

  Note that the past frame images used by the inter-frame difference calculation unit 212 to calculate the inter-frame difference image FDt are not limited to the two most recent frames (past frame images It-1 and It-2) of the present embodiment, but are the most recent. 3 frames or more, or a plurality of past frame images having a predetermined number of frames opened can be used.

  The object candidate area extraction unit 21 includes a moving object extracted from the background difference image BDt (automobiles M1 to M3 in the example of FIG. 2) and a moving object extracted from the inter-frame difference image FDt (in the example of FIG. 3, A frame surrounding the automobiles M2 and M3) is generated, and the frame is output as an object candidate region to the object detection unit 22 and the movement information calculation unit 23. This frame is a rectangular frame that includes all the extracted moving objects. The size (vertical and horizontal width) of the frame is appropriately determined depending on the size of the extracted moving object. The object candidate area extraction unit 21 outputs the position (coordinates) of the frame reference point (for example, the upper left corner) in the captured image and the vertical and horizontal widths of the frame as the object candidate area information.

  The object detection unit 22 detects a specific detection target object (a person in the present embodiment) based on the current frame image It and information on the object candidate area, and information (situation information) about the state of the object. ) And output. The object detection unit 22 includes an ELM (Extreme Learning Machine) 221 and an output determination unit 222.

  FIG. 4 is a diagram for explaining processing in the object detection unit 22. The object detection unit 22 sets a rectangular detection target area DA that includes the object candidate area CA and is wider than the object candidate area CA in the current frame image It. Further, the object detection unit 22 slides a rectangular cut area CLA (m × n pixels) smaller than the detection target area DA in the detection target area DA, while regarding the cut image CIa in the cut area CLA. The ELM 221 performs person detection and situation estimation.

  The ELM 221 inputs each pixel of the cut-out image CI to the input layer 2211, and outputs information on the presence / absence of the person and the situation from the output layer 2213 through the intermediate layer 2212. The weights between layers in the ELM 221 are stored in the person dictionary storage unit 34 as a person dictionary. The output of the ELM 221, that is, the presence / absence of a person and status information is represented by a probability (0 to 1.0). The ELM 221 of the present embodiment outputs each probability of the person's direction (forward, backward, right, or left) as the person status information. Note that, by the above processing, a plurality of cut images CIa are obtained for one object candidate area CA, and the output of the ELM 221 is obtained for each cut image CIa.

  Based on the probabilities output from the ELM 221, the output determination unit 222 determines the presence and status of a person and outputs the result. FIG. 5 is a diagram illustrating processing of the output determination unit 222. The output determining unit 222 first determines whether or not the person presence / absence probability value is 0.5 or more (step S51). If the person presence / absence probability value is less than 0.5 (NO in step S51), it is determined that the moving object is not a person (step S52).

  If the person presence / absence probability value is 0.5 or more (YES in step S51), the output determination unit 222 determines that the moving object is a person (step S53), and further determines the forward probability, backward probability, and rightward probability. , And the direction having the maximum probability value among the leftward probabilities is selected (step S54). Then, it is determined whether or not the probability value in the selected direction is 0.5 or more (step S55).

  If the probability value of the selected direction is 0.5 or more (YES in step S55), the output determination unit 222 determines the direction as the detected person's direction (step S56). If the probability value in the selected direction is less than 0.5 (NO in step S55), the output determination unit 222 means that the probability value is less than 0.5 in any direction. It is determined that the direction is unknown (step S57).

  As described above, a plurality of ELM outputs can be obtained for one object candidate area CA, but the output determination unit 222 performs the above processing for all ELM outputs, and any one of the clipped images CIa is regarded as having a person. Then, it is determined that there is a person in the object candidate area CA, and the direction with the highest probability among the determined directions is estimated as the direction of the person, and the detection result of the person (indicating the presence or absence of the person), And the estimation result of the direction of the person (indicating the direction of the person in front, back, left or right) is output.

  Through the above processing, a person detection result and an inorganic estimation result can be obtained by detecting a person from a photographed image and estimating its orientation. The object situation estimation apparatus 20 can detect a person and estimate the direction of the captured image obtained by the camera 10 in real time by performing the above process every time a captured image is acquired from the camera 10. it can.

  In the following, a configuration for updating the person dictionary by learning so as to improve the accuracy of the person detection and the estimation of the situation (orientation) will be described. In this configuration, learning data for updating the person dictionary is generated from the shooting data of the camera 10. This learning data is data configured by adding information on whether or not the image data is a person and the orientation (forward, backward, leftward, rightward) in the case of a person.

  As illustrated in FIG. 1, the movement information calculation unit 23 includes a motion vector calculation unit 231. FIG. 6 is a diagram illustrating processing of the motion vector calculation unit 231. The motion vector calculation unit 231 acquires the cut image CIbt obtained by cutting out the object candidate region CAt of the current frame image It from the object region extraction unit 21 and reads the past frame image It-1 from the past image storage unit 31. This cut-out image CIb is an image of a moving object (not necessarily a person). The motion vector calculation unit 231 performs region matching by pattern matching while sliding the clipped image CIbt within the past frame image It-1 (step S61).

  Further, the motion vector calculation unit 231 determines the object detection area CAt in the current frame image It from the position of the area in the past frame image It-1 determined to be the area corresponding to the clipped image CIbt as a result of the area matching. A motion vector (optical flow) V toward the position (the position of the cut image CIbt) is calculated (step S62). This motion vector V represents the amount and direction of movement of the object detection area CAt from the past frame image It-1 in the current frame image It by the length and direction of the vector.

  The motion vector calculation unit 231 calculates the motion vector for each object candidate region by performing the above process on all the object candidate regions extracted from the current frame image It by the object candidate region extraction unit 21. The movement information calculation unit 23 outputs the motion vector V to the learning data generation unit 24 in association with the cut image CIbt.

  The learning data calculation unit 24 includes a person presence / absence determination unit 241, a person orientation determination unit 242, and a person similarity determination unit 243. The learning data calculation unit 24 acquires the determination result from the object detection unit 22, and acquires the cut image CIb and its motion vector V from the movement information calculation unit 23.

  The person presence / absence determination unit 241 determines whether the clipped image CIb should be positive example data or negative example data. FIG. 7 is a flowchart of person presence / absence determination processing in the person presence / absence determination unit 241. The person presence / absence determination unit 241 determines whether or not the object detection unit 22 has determined that the moving object of the clipped image CIb has a person (step S71).

  When it is determined that the person is not a person (NO in step S71), the person presence / absence determination unit 241 determines that the movement amount of the motion vector V is based on a predetermined threshold (for example, 10 km / h) set as the speed at which the person runs. It is determined whether it is larger (step S72). If the moving amount is faster than the running speed of the person (YES in step S72), since it is clear that the moving object is some object other than the person, the person presence / absence determination unit 241 uses the cut image CIb as a negative example. It adds as data (step S73). This negative example data is an image showing a moving object that is not a person (for example, a traveling vehicle or the like).

  If the movement amount is equal to or less than the running speed of the person (NO in step S72), it is determined whether or not the movement amount is substantially 0 (within an error range of 0) (step S74). Specifically, for example, it is determined whether or not the moving amount is 0.1 km / h or less. If the movement amount is not within the error range of 0, the person presence / absence determination unit 241 sets this as an unidentified moving object and does not add data (step S76). If the movement amount is within an error range of 0 (NO in step S74), and if the stay time exceeds a predetermined threshold T [second] (YES in step S75), the presence / absence determination of the person The unit 241 determines that the object is an object other than the person who has stopped after moving, and adds the cut image CIb to the additional learning data storage unit 33 as negative example data (step S73).

  On the other hand, if the movement amount is within the error range of 0 (YES in step S74), but the stay time is equal to or less than the threshold T [seconds] (NO in step S75), this is not a person but a person walking It is determined that the object is an unidentified moving object that moves at a speed similar to the speed, or some noise, and no data is added (step S76).

  On the other hand, if it is determined that there is a person in the determination result in the object detection unit 22 (YES in step S71), the person presence / absence determination unit 241 sets the movement amount of the motion vector V as the speed at which the person runs. It is determined whether or not it is greater than a predetermined threshold (for example, 10 km / h) (step S77). If the amount of movement is faster than the running speed of the person (YES in step S77), the person presence / absence determination unit 241 determines that the person is not a pedestrian, such as a person riding a two-wheeled vehicle. The output image CIb is added to the additional learning data storage unit 33 as negative example data (step S78).

  If the movement amount is equal to or less than the speed of the person running (YES in step S77) and the movement amount is outside the error range of 0 (NO in step S79), the moving object determined to be this person is Since the person is moving at a speed approximately equal to the walking speed of the person, the person presence / absence determination unit 241 sets the cut image CIb as a positive example data candidate (step S80). Positive example data is an image of a walking person (pedestrian).

  On the other hand, if the movement amount is equal to or less than the speed at which the person runs (NO in step S77) and the movement amount is within the error range of 0 (YES in step S79), the stay time is a predetermined threshold T [seconds]. ] Is determined (step S81). If the staying time is longer than the threshold T (YES in step S81), the person presence / absence determination unit 241 determines that the object is a fixed object other than a person, and uses this cut-out image CIb as negative example data for additional learning data. It adds to the memory | storage part 33 (step S78).

  If the amount of movement is outside the error range of 0 (the person is moving at a walking speed) (YES in step S79), but the stay time is less than or equal to the threshold T (NO in step S81), the presence / absence determination of the person The unit 241 sets this as an unidentified moving object and does not add data (step S82).

  FIG. 8 is a flowchart of person orientation determination processing in the person orientation determination unit 242. The person orientation determination unit 242 determines the orientation of the person in the cut-out image CIb determined as the positive example data candidate in the person presence / absence determination process. First, the person orientation determination unit 242 classifies the direction of the motion vector (motion direction) of the positive example data candidate into one of four classes of front, back, right, and left, and gives a label indicating the direction. (Step S91).

  Next, the person direction determination unit 242 compares the direction detected by the object detection unit 22 (the direction of the person indicated by the situation information) with the direction of the motion vector calculated by the motion vector calculation unit 231 and compares them. Are determined to match (step S92). If they match, the person orientation determination unit 242 sets the matching orientation as the person orientation and sets the cut image CIb as a positive example data candidate (step S93).

  If the direction estimated by the object detection unit 22 (the direction of the person indicated by the situation information) does not match the direction of the motion vector calculated by the motion vector calculation unit 231 (NO in step S92), the object detection unit After changing the direction estimated in 22 to the direction of the motion vector (step S94), the cut image CIb is set as a positive example data candidate (step S93). As a modification, when the direction estimated by the object detection unit 22 and the direction of the motion vector do not match, the cut image CIb is excluded from the positive example data candidates, and the cut image CIb The learning data may not be generated.

  The problem of overfitting can be avoided by combining the cut-out images CIb of the same person in the plurality of cut-out images CIb determined as the positive example data candidates by the person orientation determination unit 242. Therefore, the person similarity determination unit 243 determines the degree of similarity between the plurality of cut images CIb determined as the positive example data candidates, and stores the cut images CIb of the same person redundantly in the additional learning data storage unit 22. Do not let it.

  FIG. 9 is a flowchart of person similarity determination processing in the person similarity determination unit 243. The person similarity determination unit 243 obtains the degree of similarity between the extracted image CIb1 of a certain positive example data candidate and the extracted image CIb2 of another positive example data candidate by normalized cross-correlation matching (NCC). If the degree of similarity is high, one of those cut-out images ICb is deleted.

  Specifically, as shown in FIG. 9, the person similarity determination unit 243 first calculates the similarity by NCC for the positive example data candidate CIb1 and the positive example data candidate CIb2 (step S101), and the calculated similarity It is determined whether or not the degree is a predetermined threshold value (0.5 or more in the present embodiment) (step S102).

  When the degree of similarity is higher than the threshold (YES in step S102), the person similarity determination unit 243 deletes the positive example data candidate with the lower probability of the presence or absence of a person (step S103). If the similarity is lower than the threshold (NO in step S102), the person similarity determination unit 243 leaves both positive example data candidates (step S104).

  The person similarity determination unit 243 performs the above processing for all the positive example data candidates, and as the data of the person who is walking the extracted image of the remaining positive example data candidates, along with the direction information, as the positive example data The data is output to the additional learning data storage unit 33. The additional learning data storage unit 33 stores the cut image output from the learning data generation unit 24 together with information on the orientation of the person. The additional learning data storage unit 33 also stores negative example data output from the learning data generation unit 24.

  The additional learning unit 25 includes a person dictionary generation unit 251. When a certain amount of learning data (positive example data and negative example data) is accumulated in the additional learning data storage unit 33, the additional learning unit 25 reads the learning data from the additional learning data storage unit 33. The timing at which the additional learning unit 25 performs additional learning may be determined by the number of generated learning data (for example, additional learning may be performed every time 1000 learning data is added), or the additional learning unit 25. May perform additional learning periodically (for example, every week) at predetermined time intervals. The person dictionary creation unit 251 performs additional learning using the additional learning data read from the additional learning data storage unit 33, and generates an updated person dictionary. In the present embodiment, the additional learning unit 25 performs machine learning as learning using additional learning data. When the additional learning unit 25 generates the person dictionary updated by the person dictionary creation unit 251, the additional learning unit 25 updates the person dictionary stored in the person dictionary storage unit 34 using the person dictionary.

  As described above, the object situation estimation apparatus 20 according to the present embodiment has a function of detecting the presence or absence of a person based on a captured image obtained from the camera 10 and estimating the situation (orientation), and the same camera 10. A function of updating a person dictionary used for detecting a person and estimating a situation based on the photographed image is provided. Since the captured image used to update the person dictionary is captured by the camera 10 under the same conditions (position and angle) as the captured image that is the object detection target, the additional learning data is stored in the camera 10. This is learning data suitable for detecting a search target object and estimating a situation for a captured image. Therefore, by repeating the learning, the accuracy of detection of the search target object and estimation of the situation (orientation) is improved.

  Such an object situation estimation system 1 performs additional learning after the camera 10 is installed at a desired position. Therefore, immediately after the camera 10 is installed, formal operation, that is, various controls based on the result of detection of the search target object and the estimation of the situation (orientation) may be started. The fixed period until the additional learning is performed may be a preparation period for updating the person dictionary, and the formal operation may be started after the preparation period ends. In any case, by operating, the person dictionary is updated, and the accuracy of detection of the search target object and estimation of the situation is improved.

  As described above, according to the object situation estimation system 1 of the present embodiment, even a captured image in which the object detection unit 22 cannot detect a specific detection target object or cannot estimate the situation is included. Learning data can be generated by estimating the situation of the object that is considered to be a specific detection target object from the movement information. Therefore, by performing additional learning using this learning data, it is possible to detect a specific detection target object from a captured image in which the detection of the specific detection target object or the estimation of the situation could not be performed. We can expect to be able to estimate.

  In the object situation estimation apparatus 20 described above, the function of generating the learning data does not need to be performed in real time, and the captured images of a certain number of frames are stored, and those captured images are used to capture the captured images. It may be executed at a later time than when the image is obtained.

  Further, in the object state estimation device 20 described above, in order to detect a person who is a moving object as a specific detection target object, the object candidate region is detected using a difference image before detection and estimation by machine learning in the object detection unit 22. However, if the specific detection target object is not limited to the moving object, the object candidate region extraction unit 21 may be omitted and the specific detection target object may be detected from the entire captured image. In this case, the motion vector calculation unit 231 uses the detection target object detection result and the situation estimation result in the object detection unit 22.

  The object situation estimation system 1 estimates the direction of the detection target object as the state of the detection target object, and the learning data generation unit 24 determines the direction in the learning data based on the direction of the motion vector. The state of the detection target object is not limited to the direction. For example, the state of the detection target object may be a movement state of whether the person who is the detection target object is running, walking, or stopped. In this case, the motion calculated by the motion vector calculation unit 231 The situation (running, walking, stopping) added in the learning data can be determined by the magnitude (speed) of the vector.

  The present invention uses a dictionary to detect a specific detection target object from a photographed image, and automatically estimates learning data necessary for updating the dictionary from the photographed image in a system that estimates the state of the detection target object. It is useful as an object situation estimation system that detects a specific detection target object from an image photographed by a camera and estimates its situation.

DESCRIPTION OF SYMBOLS 1 Object condition estimation system 10 Camera 20 Object condition estimation apparatus 21 Object candidate area extraction part 211 Background difference calculation part 212 Inter-frame difference calculation part 22 Object detection part 221 ELM
222 output determination unit 23 movement information calculation unit 231 motion vector calculation unit 24 learning data generation unit 241 person presence determination unit 242 person orientation determination unit 243 person similarity determination unit 25 additional learning unit 251 person dictionary generation unit 31 past image storage unit 32 background Image storage unit 33 Additional learning data storage unit 34 Person dictionary storage unit

Claims (12)

A camera that shoots and generates a shot image;
Using a dictionary, an object detection unit that detects a specific detection target object from the captured image and estimates the state of the detection target object;
A movement information calculation unit for calculating movement information of a moving object in the captured image;
Based on the movement information, learning data is generated by adding information on the presence / absence of the detection target object to the image data including the moving object, and information on the status of the detection target object when the detection target object exists. A learning data generation unit to
An additional learning unit that updates the dictionary based on the learning data;
An object situation estimation system.   2. The object state estimation system according to claim 1, wherein the learning data generation unit generates, as the learning data, positive example data in which information on the presence / absence of the detection target object is information that the detection target object is present.   3. The object state estimation system according to claim 1, wherein the learning data generation unit generates, as the learning data, negative example data in which information on the presence / absence of the detection target object is information indicating that there is no detection target object. The object state estimation system according to claim 1, wherein the states of the detection target object and the moving object are directions of the detection target object and the moving object. An object candidate area extracting unit that extracts an object candidate area from the captured image;
The object detection unit detects the detection target object from a partial region including the object candidate region in the captured image;
The object situation estimation system according to claim 4 . An object candidate area extracting unit that extracts an object candidate area from the captured image;
The movement information calculation unit calculates movement information of the object candidate area as movement information of the moving object.
The object situation estimation system according to claim 4 . The camera continuously shoots to generate a plurality of frames of the captured image,
The object situation estimation system according to claim 5 or 6 , wherein the object candidate area extraction unit extracts the object candidate area based on a difference image of the captured images of a plurality of frames. The learning data generation unit includes a determination unit that determines whether or not the moving object is the detection target object, and generates learning data for the moving object determined to be the detection target object. 1. The object state estimation system according to 1. The object state estimation system according to claim 8 , wherein the determination unit determines whether the moving object is the detection target object based on the detection result. An object situation estimation device that is communicably connected to a camera that shoots and generates a captured image, and that acquires the captured image from the camera,
Using a dictionary, an object detection unit that detects a specific detection target object from the captured image and estimates the state of the detection target object;
A movement information calculation unit for calculating movement information of a moving object in the captured image;
Based on the movement information, learning data is generated by adding information on the presence / absence of the detection target object to the image data including the moving object, and information on the status of the detection target object when the detection target object exists. A learning data generation unit to
An additional learning unit that updates the dictionary based on the learning data;
An object state estimation device comprising: Shooting and generating a shot image;
Using a dictionary, detecting a specific detection target object from the captured image, and estimating the state of the detection target object;
Calculating movement information of a moving object in the captured image;
Based on the movement information, learning data is generated by adding information on the presence / absence of the detection target object to the image data including the moving object, and information on the status of the detection target object when the detection target object exists. And steps to
Updating the dictionary based on the learning data;
An object state estimation method comprising: A computer of an object state estimation device that is communicably connected to a camera that takes a photograph and generates a photographed image and acquires the photographed image from the camera,
Using a dictionary, an object detection unit that detects a specific detection target object from the captured image and estimates the state of the detection target object;
A movement information calculation unit for calculating movement information of a moving object in the captured image;
Based on the movement information, learning data is generated by adding information on the presence / absence of the detection target object to the image data including the moving object, and information on the status of the detection target object when the detection target object exists. A learning data generation unit, and an additional learning unit that updates the dictionary based on the learning data,
Object situation estimation program to function as.

Images