JP6898587B2 - Object tracking methods, object tracking programs, and object tracking systems - Google Patents

Description

The present invention relates to object tracking methods, object tracking programs, and object tracking systems.

Object tracking is a technique for identifying and tracking an object moving in a plurality of images (frames) arranged in a time series.

As a conventional object tracking technology, for example, there is a technology for detecting a preceding vehicle. In this technique, among the measurement points detected by the millimeter-wave radar, the measurement points that are close to each other and have the same relative velocity are grouped to generate a measurement point group. From the generated measurement point groups, the measurement point groups that satisfy the conditions of the preceding vehicle are extracted, and the measurement point groups extracted more than a predetermined number of times according to the passage of time are detected as the preceding vehicle (Patent Document 1). ..

Japanese Unexamined Patent Publication No. 2005-173806

However, in the prior art, at the stage of determining whether or not the measurement point detected at the present time is the same as the measurement point of the measurement point group recognized as the preceding vehicle, all the measurement point groups and the current time point are determined. We are investigating the degree of relevance to the measurement points in. Therefore, the load of calculating the correspondence between the preceding vehicle (object) recognized in the past time and the current measurement point is large.

Therefore, an object of the present invention is to provide an object tracking method capable of reducing the computational load of object tracking. Another object of the present invention is to provide an object tracking program that can reduce the computational load of object tracking. Furthermore, another object of the present invention is to provide an object tracking system capable of reducing the computational load of object tracking.

The above objectives are achieved by the following means.

(1) An object that is detected in a moving image in which a plurality of frames are arranged in chronological order and whose elapsed time since the detection continues for a predetermined time or longer is the first. The step (a) of classifying an object as a first-class object and classifying an object less than a predetermined time as a second-class object.
The step (b) of investigating the correspondence between the unrecognized object and the first-class object, using the object detected in the current frame as an unrecognized object,
In the step (b), when it is determined that the unrecognized object and the first-class object correspond to each other, the step (c) of associating the unrecognized object with the first-class object is determined.
When it is determined in the step (b) that the unrecognized object and the first-class object do not correspond to each other, the step (d) of examining the correspondence between the unrecognized object and the second-class object is performed.
In the step (d), when it is determined that the unrecognized object and the second-class object correspond to each other, the step (e) of associating the unrecognized object with the second-class object is performed.
An object tracking method that has.

(2) In the step (e), if the elapsed time from the detection of the second-class object associated with the unrecognized object as an object has passed the predetermined time or more, the second-class object is used. The object tracking method according to (1) above, which comprises the step (e1) of classifying the object into the first type object.

(3) In the step (a), the type 2 object is associated with the elapsed time since it was detected as an object.
The object according to (1) or (2) above, wherein in the step (d), the second-class object examines the correspondence with the unrecognized object in order from the one having the longest elapsed time since it was detected as an object. Tracking method.

(4) In the step (a), the first-class object is associated with the elapsed time since it was detected as an object.
In the step (b), any one of the above (1) to (3), wherein the first-class object examines the correspondence with the unrecognized object in order from the one having the longest elapsed time since it was detected as an object. The object tracking method described in 1.

(5) In the step (a), the first-class object that does not correspond to the unrecognized object in the step (b) is classified as a third-class object.
In the step (d), when it is determined that the unrecognized object and the second-class object do not correspond to each other, the step (f) of examining the correspondence between the unrecognized object and the third-class object is performed.
In the step (f), when it is determined that the unrecognized object and the third-class object correspond to each other, the unrecognized object and the third-class object are associated with each other, and the third-class object is associated with the third-class object. At the stage (g) of reclassifying an object into the first-class object,
In the step (f), when it is determined that the unrecognized object and the third-class object do not correspond to each other, the unrecognized object is classified into the second-class object (h).
The object tracking method according to any one of (1) to (4) above.

(6) In the step (d), if it is determined that the unrecognized object does not correspond to any of the second-class objects, the unrecognized object is classified into the second-class object (d1). The object tracking method according to any one of (1) to (4) above.

(7) The object tracking method according to any one of (1) to (6) above, wherein the predetermined time is 0.2 seconds or more and less than 1 second.

(8) The object tracking method according to any one of (1) to (7) above, wherein the time is counted in units of the frame.

(9) The object tracking method according to any one of (1) to (8) above, which has a step (i) of displaying the first-class object together with information indicating that it is a moving object.

(10) An object tracking program for causing a computer to execute the object tracking method according to any one of (1) to (9) above.

(11) An image acquisition sensor that acquires frames in chronological order,
A computer that executes the program described in (10) above,
Has an object tracking system.

According to the present invention, it is investigated whether or not the first-class object that has been continuously detected up to the previous frame for a predetermined time or longer corresponds to the unrecognized object, and if it corresponds to the first-class object, it is not yet. It was decided to associate the perceptible object with the first-class object and investigate the correspondence with the second-class object whose elapsed time from detection is shorter than that of the first-class object only when it does not correspond. As a result, if the first-class object and the unrecognized object are associated with each other, it is not necessary to investigate the second-class object. Therefore, the calculation load can be reduced accordingly.

It is explanatory drawing for demonstrating the whole structure of the object tracking system. It is a block diagram for demonstrating the structure of the object tracking apparatus. It is a list diagram which shows an example of the list which classified the object. It is a flowchart for demonstrating the processing procedure of the object tracking method of this embodiment. It is a flowchart for demonstrating the processing procedure of the object tracking method of this embodiment. It is a flowchart for demonstrating the processing procedure of the object tracking method of this embodiment. It is explanatory drawing explaining the example of the moving body, the 1st moving body candidate, and the 2nd moving body candidate recognized up to the previous frame. It is a list diagram which shows each list at the time of FIG. It is explanatory drawing explaining the example which detected the unrecognized object in the present frame. It is explanatory drawing which superposed the front frame shown in FIG. 7 and the present frame shown in FIG. It is explanatory drawing explaining the state which the moving body M1 and the unrecognized object Ob1 corresponded. As a comparative example, it is explanatory drawing explaining the example which examines the unrecognized object detected in the present frame and all the objects recognized up to the previous frame by brute force. It is explanatory drawing explaining the state which the unrecognized object Ob2 and the first moving body candidate C1 corresponded. It is a list diagram which shows each list at the time when the processing has progressed to the end. It is explanatory drawing explaining the current frame at the end of processing. It is a figure which shows the example of the 3D distance image obtained by the laser radar. It is sectional drawing explaining the structure of a laser radar. It is explanatory drawing explaining the state which scans in the monitoring space of the monitoring apparatus MD by the laser spot light SB (shown by hatching) which emits with the rotation of a mirror unit MU.

Hereinafter, embodiments of the present invention will be described with reference to the attached drawings. The present invention is not limited to the following embodiments. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicate description will be omitted. In addition, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may differ from the actual ratios. Further, in the following description, among the frames arranged in chronological order, the current frame is referred to as the current frame, the frame past the current frame is referred to as the past frame, and the frame immediately before the current frame is referred to as the previous frame. ..

(Object tracking system)
FIG. 1 is an explanatory diagram for explaining the overall configuration of the object tracking system. FIG. 2 is a block diagram for explaining the configuration of the object tracking device.

The object tracking system includes a laser radar 101 that acquires frames arranged in chronological order, and an object tracking device 102 that tracks moving objects in frames arranged in chronological order in chronological order. Frames arranged in chronological order are played back (displayed) in chronological order to form a moving image.

The laser radar 101 is an image acquisition sensor. The details of the laser radar 101 will be described later, but the distance from the reflected light to the object is measured in real time by scanning the laser. Therefore, the distance and size (height, width, etc.) from the installation position of the laser radar 101 to the unrecognized object can be obtained. Further, this makes it possible to obtain an image called a distance image. By arranging (that is, continuously displaying) these distance images in chronological order as one scan and one frame, a moving image is obtained. The distance image (frame) from the laser radar 101 is transmitted to the object tracking device 102.

The object tracking device 102 is a computer, and may be a general personal computer or a dedicated computer. As shown in FIG. 2, the object tracking device 102 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, and an HDD (Hard Disk Drive) 14.

The RAM 12 is a storage unit used by the CPU 11 as a work area, and temporarily stores, for example, programs and data required for processing.

The ROM 13 stores a basic program necessary for starting the object tracking device 102 and performing basic operations as a computer.

The CPU 11 reads a program according to the processing content from the HDD 14 and expands it into the RAM 12, and executes the expanded program. Here, the moving image is mainly composed based on the frame obtained from the laser radar 101, and the object tracking in the moving image is performed.

The HDD 14 stores programs and data required for each process. A non-volatile semiconductor memory (for example, a flash memory) may be used instead of the HDD.

Further, the object tracking device 102 includes an input device 15 such as a mouse and a keyboard, a display 16 for displaying an image (moving image), and a network interface 17 for connecting an external device.

The network interface 17 is, for example, Ethernet (registered trademark) or the like, and is used for communication with the laser radar 101. The network interface 17 is also connected to a LAN (Local Area Network) or the like, and is also used for communication with a server (described later).

The object tracking device 102 is achieved by such a computer executing a processing procedure of an object tracking method described later. The object tracking device 102 may acquire only the measured values from the laser radar 101 to form a distance image and a moving image thereof.

The object tracking device 102 also has, for example, a server 103 connected by a network. The server 103 receives and stores a moving image composed of frames arranged in time series from the object tracking device 102. Further, the server 103 stores other moving images and provides them in response to reading from the object tracking device 102. The other moving images are not limited to the moving images acquired by the laser radar 101, but include moving images of a normal movie camera and the like.

(Object tracking method)
The object tracking method of the present embodiment will be described.

In the present embodiment, for object tracking, objects other than backgrounds and stationary objects in the frame acquired by the laser radar 101 are classified as follows.

(1) "Moving body" (type 1 object): An object that is continuously recognized for a predetermined first predetermined time or longer.

(2) "First moving object candidate" (type 2 object): An object that is continuously recognized for a second predetermined time or more and less than a first predetermined time.

(3) "Second moving object candidate" (type 2 object): An object that is continuously recognized for a third predetermined time or more and a second predetermined time or less.

(4) "Unrecognized object": An object detected in the current frame (current time).

The length of each predetermined time is 1st predetermined time> 2nd predetermined time> 3rd predetermined time.

The time can be measured by counting the system clock of the computer constituting the object tracking device 102. The first predetermined time is, for example, preferably 0.2 seconds or more and less than 1 second, and more preferably 0.3 seconds or more and less than 0.5 seconds. By setting such a time, it is possible to prevent noise that appears and disappears for a very short time, and flowers and trees swaying in the wind from being recognized as moving objects. Of course, the time set as the first predetermined time is arbitrary, and may be set according to, for example, the width and depth of the measurement range by the sensor, the monitoring target, and the like.

Here, the distance image acquired by the laser radar 101, that is, the frame is, for example, 10 frames per second. In this case, the time of one frame is 1/10 second. Therefore, in the present embodiment, instead of directly counting the system clock, the time is counted in frame units. That is, it counts the number of frames instead of time.

Therefore, the predetermined time is also redefined in frame units as follows. Here, the case is 10 frames per second.

(1) "Moving object": An object that has been continuously recognized in the immediately preceding 3 frames or more of the past frames.

(2) "First moving object candidate": An object that is continuously recognized in the immediately preceding two frames of the past frames.

(3) "Second moving object candidate": An object recognized only in the immediately preceding frame among the past frames.

(4) "Unrecognized object": An object currently detected in the frame.

The moving body, the first moving body candidate, and the second moving body candidate defined as described above are classified as a list and stored in the RAM 12. A storage device other than RAM may be used, but a storage device that can read and write at high speed is preferable so that at least object tracking processing can be performed.

The present invention is not limited to such a number of frames and time. If the number of frames per second changes, the time of one frame also changes. For example, in the case of 5 frames per second used in surveillance cameras and the like, 1 frame is 1/5 second. On the contrary, it is 1/30 second for a moving image of 30 frames per second.

FIG. 3 is a list diagram showing an example of a list in which objects are classified.

As shown in FIG. 3, the moving body, the first moving body candidate, and the second moving body candidate are classified as a list. That is, the moving body is classified into a moving body list (3 frames or more continuous), the 1st moving body candidate is classified into the 1st moving body candidate list (2 frames continuous), and the 2nd moving body candidate is classified into the 2nd moving body candidate list (only 1 frame).

Further, in the present embodiment, a lost moving body list is provided. The lost moving object list is a list in case the moving object disappears temporarily. That is, the moving object may disappear from the frame, which is the measurement range of the sensor, but may be temporarily invisible in the frame. Temporarily invisible means, for example, when a moving object is hidden behind a stationary object when viewed from the sensor installation position, or when there are multiple moving objects and one moving object is hidden behind another moving object. And so on. In such a case, an unrecognized object is not detected in the current frame, or no correspondence is made with any of the moving objects recognized in the past frame. Therefore, we decided to set up a list of lost moving objects and register moving objects that do not have such a correspondence as lost moving objects (type 3 objects). Then, when an unrecognized object is detected, if there is no correspondence with the moving object or moving object candidate, by examining the correspondence with the lost moving object in this lost moving object list, it is associated with whether or not it is the same object as the previous moving object. be able to. The lost moving body registered in this lost moving body list may be continuously stored until the system is stopped, or may be deleted after a certain period of time or a predetermined time has passed. For example, it may be deleted after a few seconds to a few minutes or a few hours, or it may be deleted at midnight every day.

The lost moving object list does not have to be provided. In that case, moving objects that do not correspond to the unrecognized object need only be deleted from the moving object list. Even in that case, if the same object as the object deleted from the moving object list reappears, it will be a newly appearing object, but object tracking will be performed.

These lists are stored in the RAM 12. In FIG. 3, each list is grouped together, but the present invention is not limited to this, and for example, each list may be stored independently.

Objects classified in each list are managed by, for example, assigning an identification number (ID). In FIG. 3, the moving objects are M1, M2, M3, ..., The moving object candidates are C1, C2, C3, ... (No distinction between the first and second), and the unrecognized objects are Ob1, Ob2, Ob3, .... Here, for convenience of explanation, the moving body M, the moving body candidate C, the unrecognized object Ob, etc. are shown separately, but in the actual processing, the moving body, the moving body candidate, the ID and the tag given to the unrecognized object, etc. are shown. , These may be serial numbers without distinction, or may be numbers or codes based on time, and are not limited.
The processing procedure of the object tracking method will be described.

4 to 6 are flowcharts for explaining the processing procedure of the object tracking method of the present embodiment.

This processing procedure is carried out by the object tracking device 102. In the following description, the unrecognized object is an object that is currently detected in the frame and is not classified into any of them, as already defined.

First, the measurement data, that is, the current frame is acquired from the laser radar 101 (S1). In this S1, it is assumed that the moving body, the first moving body candidate, and the second moving body candidate recognized in the past frame are listed by the processing procedure described here.

Subsequently, an unrecognized object is detected from the acquired current frame (S2). In the detection of an unrecognized object, for example, the difference between a predetermined reference frame and the current frame is taken, and an object existing only in the current frame is set as an unrecognized object. The reference frame may be, for example, one of the past frames (for example, the previous frame), or a frame with only a background without moving objects may be prepared (a frame without moving objects visually found and selected by the user). Such). It should be noted that such a reference frame is stored in the HDD 14, expanded in the RAM 12 during operation, and used for comparison with the current frame.

In this S2, if there are a plurality of unrecognized objects, all of them are detected. An ID, a label, or the like required for processing is assigned to the detected unrecognized object (for example, unrecognized objects Ob1, Ob2, Ob3, ..., Etc.).

Further, in this S2, by clustering the detected objects, it may be selected whether or not they are unrecognized objects that can be moving objects. For example, objects in the measurement data of the laser radar 101 are clustered to a constant size (number of pixels). If there are multiple objects, cluster each object. Then, if it is equal to or larger than a predetermined size threshold value, it is regarded as an unrecognized object. The size threshold value may be arbitrarily determined depending on the monitoring location, the monitoring target, and the like. For example, if a person is to be tracked and monitored, the minimum value of the size of a normal person may be used as the size threshold value for clustering. On the contrary, if all objects are to be tracked, the size threshold value may be a small value, for example, about 1 to several pixels, or the size threshold value may not be set.

Then, as a result of S2 (or after returning from the step described later), if there is no unrecognized object (S3: NO), the process proceeds to S40 (FIG. 6), while if there is an unrecognized object (S3: YES). , Subsequently, one of the unrecognized objects is selected (S4). This is because when a plurality of unrecognized objects are detected in S2, the unrecognized objects are examined one by one. The order of selection is not particularly limited, and for example, the order of the IDs assigned in S2 may be used, or the coordinate system may be set in the frame and the order may be set from the one closest to the origin (the one with the smaller coordinate value).

Then, it is confirmed whether or not there is an unsupported moving object in the moving object list (S5). An unsupported moving object is a moving object that is not associated with an unrecognized object in this processing procedure. Here, if there is no unsupported moving object in the moving object list (S5: NO), the process proceeds to S8. On the other hand, if there is an unsupported moving object in the moving object list (S5: YES), then the unsupported moving object in the moving object list (hereinafter simply referred to as the moving object) and the unrecognized object selected in S4. Check the correspondence of (S6).

The process of investigating the correspondence between the moving object and the unrecognized object is performed by examining the degree of relevance between them. The degree of relevance will be described.

The degree of relevance can be examined using, for example, distance, moving speed, distance from the sensor, size, number of pixels, color, and the like as indexes.

When the image acquisition sensor is the laser radar 101, the relevance of the distance is the value of the distance between the moving object in the front frame and the unrecognized object among the moving objects. In the case of the laser radar 101, the distance between the moving object in the previous frame and the unrecognized object can be known from the measurement data in the three-dimensional position of the object. The position of the moving object and the position of the unrecognized object can be found as coordinate values. Therefore, the distance between them is obtained as the difference between the coordinate values. In the case of a movie camera, the distance between the moving object in the previous frame and the unrecognized object in the frame is compared, and the value is used as the relevance. In the case of a movie camera image, a two-dimensional coordinate system can be set in the frame, and the difference in coordinate values can be obtained in the same manner as described above. In either case, the closer the distance (the smaller the value), the higher the degree of relevance. In the following description, the "position" is also a coordinate value.

The moving speed is determined by comparing the moving speed of the moving object with the moving speed of the unrecognized object and using the value as the relevance. The moving speed of the moving body is obtained by dividing this by the time (time of one frame) because, for example, the difference between the position of the moving body in the frame before the previous frame and the position of the moving body in the previous frame is the moving distance of the moving body. The moving speed of an unrecognized object is the time (1 frame) of the difference between the position of the front frame of the moving object and the position of the unrecognized object, assuming that the moving object whose correspondence is being investigated is the same object as the unrecognized object. Divide by time) to find. The closer the movement speed is, the higher the relevance is.

The distance from the sensor is determined by comparing the distance from the laser radar 101 to the moving object in the front frame and the distance from the laser radar 101 to the unrecognized object, and using that value as the degree of relevance. The closer the distance values are, the higher the relevance.

For the size, the size of the moving object in the front frame measured by the laser radar 101 and the size of the unrecognized object are compared, and the value is used as the degree of relevance. The closer the size value is, the higher the relevance.

The number of pixels is determined by comparing the number of pixels occupied by the moving object in the previous frame taken by the movie camera and the unrecognized object, and using that value as the degree of relevance. The closer the number of pixels, the higher the degree of relevance.

For the color, the color of the moving object in the previous frame taken by the color movie camera is compared with the color of the unrecognized object, and the value is used as the degree of relevance. For the colors, for example, the color difference based on the color information such as the Lab value may be compared, or the difference in the RGB gradation values may be simply compared. The smaller the color difference, the higher the relevance. Further, the number of colors (the number of pixels) having the largest number, the average value of colors, and the like may be used.

These indicators may be used alone or in combination of two or more. When a plurality of indexes are used in combination, for example, in the case of a color movie camera (two-dimensional image), the degree of relevance is examined using the distance and the color. In this case, the distance and the color are weighted to check the degree of relevance. For example, "distance x 0.6 + color difference x 0.4". The weighting constants used here are empirical values and values obtained through experiments. In addition, when a movie camera is used, various combinations such as distance and number of pixels, moving speed and color, number of pixels and color, etc. are combined, and each is weighted by experience value or experimental value to examine the degree of relevance.

Further, in the case of the laser radar 101, for example, the degree of relevance is examined using the distance and the size. Further, in the case of the laser radar 101, the distance from the sensor, the size, and the like may be combined. Further, the moving speed and size may be used. By examining the degree of relevance using multiple indexes, a more accurate degree of relevance can be obtained.

Here, when there is one moving object and one unrecognized object in the previous frame, it may be determined that they are related without checking the degree of relevance. This is because if there is one moving object and one unrecognized object, it is highly probable that the moving object in the previous frame was detected as an unrecognized object in the current frame. However, there is a possibility that the unrecognized object is noise or suddenly appears. Therefore, it is preferable to check the degree of relevance even if there is one moving object and one unrecognized object in the front frame. Then, when the degree of relevance is equal to or higher than a predetermined threshold value, they are associated with each other as being the same. In this case, the predetermined threshold value differs for each index, but it is preferable to obtain it in advance as an empirical value or an experimental value.

On the other hand, when there are two or more moving objects in the moving object list, the degree of relevance between them is examined and a combination having a high degree of relevance is associated.

For example, if there are two moving objects M1 and M2 in the moving object list at the stage of coming to S6, the degree of relevance between the moving body M1 and the unrecognized object Ob1 and the degree of relevance between the moving body M2 and the unrecognized object Ob1 are examined and more related. The combination with the higher degree is associated.

When there are a plurality of moving objects, which moving object should be checked first is, for example, the degree of relevance between the old moving body and the unrecognized object in chronological order. When checking the relevance between an old moving object and an unrecognized object in chronological order, the time (time) when the moving object was detected as an object or the elapsed time since it was detected is added as a tag in the moving object list. I will do it. The time (time) to be given is, for example, real time. The elapsed time may be the time itself, but it may also be the number of frames since it was detected. Note that "detected as an object" is the time when it is detected as an object (that is, an unrecognized object) in the current frame.

Further, for example, a coordinate system may be set in the frame (three-dimensional coordinate system in the frame of the laser radar 101, two-dimensional coordinate system in the frame of the movie camera), and the one closest to the origin may be examined first. Further, the examination is not limited to these, and may be examined in any order.

Return to the flowchart and continue the explanation. As a result of examining the correspondence between the moving object and the unrecognized object in S6, if it is determined that they correspond (S7: YES), the moving object and the unrecognized object are associated with each other, and the moving object is associated in the moving object list. It records what has been done with a tag or the like and updates it (S30). After that, the process returns to S3 and the process is continued.

On the other hand, if it is determined in S7 that the moving object and the unrecognized object do not correspond (S7: NO), then it is confirmed whether or not there is the first moving object candidate in the first moving object candidate list (S8). If there is no first moving object candidate (S8: NO), the process proceeds to S11 (see FIG. 5). On the other hand, if there is a first moving object candidate in the first moving object candidate list (S8: YES), the correspondence between the first moving object candidate and the unrecognized object selected in S4 is examined (S9). Here, as in S6, the degree of relevance between the first moving object candidate and the unrecognized object is obtained, and it is examined whether or not they correspond. The method of checking the relevance is omitted because the moving body in S6 is only changed to the first moving body candidate.

As a result of S9, if it is determined that the first moving object candidate and the unrecognized object correspond to each other (S10: YES), the associated first moving object candidate is changed to a moving object and moved to the moving object list (S31). ). That is, at this stage, the first moving object candidate that has been recognized for two consecutive frames is recognized as the same as the unrecognized object, so that it is recognized for three consecutive frames. Therefore, the first moving body candidate is changed to a moving body. After that, the process returns to S3 and the process is continued.

On the other hand, if it is determined as a result of S9 that the first moving object candidate and the unrecognized object do not correspond (S10: NO), then (see FIG. 5 below), the second moving object candidate is in the second moving object candidate list. It is confirmed whether or not (S11), and if there is no second moving object candidate (S11: NO), the process proceeds to S15. On the other hand, if there is a second moving object candidate in the second moving object candidate list (S11: YES), the correspondence between the second moving object candidate and the unrecognized object selected in S4 is examined (S12). Here, as in S6, the degree of relevance between the second moving object candidate and the unrecognized object is obtained and checked whether they correspond to each other. As for the method of checking the degree of relevance, the explanation is omitted because the moving body in S6 is only changed to the second moving body candidate.

As a result of S12, if it is determined that the second moving object candidate and the unrecognized object correspond to each other (S13: YES), the second moving object candidate is changed to the first moving object candidate and moved to the first moving object candidate list. (S32). That is, at this stage, the second moving object candidate, which was recognized only for one frame, was recognized as the same as the unrecognized object, so that it was recognized for two frames in a row. Therefore, the second moving body candidate is changed to the first moving body candidate.

On the other hand, if it is determined as a result of S12 that the second moving object candidate and the unrecognized object do not correspond (S13: NO), then it is confirmed whether or not there is a lost moving object in the lost moving object list (S15). If there is no lost moving object in the lost moving object list (S15: NO), the unrecognized object selected in S4 is included in the second moving object candidate list as the second moving object candidate (S33). On the other hand, if there is a lost moving object in the lost moving object list (S15: YES), the correspondence between the lost moving object and the unrecognized object is examined (S16). Here, as in S6, the degree of relevance between the lost moving object and the unrecognized object is obtained and checked to see if they correspond to each other. Since the method of checking the degree of relevance is the same as in S6, the description thereof will be omitted.

As a result of S16, if it is determined that the lost moving object and the unrecognized object correspond to each other (S17: YES), the lost moving object moves from the lost moving object list to the moving object list as a moving object (S18).

If it is determined as a result of S16 that the moving object and the unrecognized object do not correspond to each other (S17: NO), the unrecognized object selected in S4 is included in the second moving object candidate list as the second moving object candidate (S33). After that, the process returns to S3 and the process is continued.

A case where there is no unrecognized object in S3 (S3: NO) will be described (see FIG. 6). When there is no unrecognized object, the correspondence between the unrecognized object and the moving object, the first and second moving object candidates, and the lost moving object is determined by the case where the unrecognized object is not detected in the current frame in the first place and the processing already described. This is the case when the unrecognized object disappears after the investigation.

When there is no unrecognized object in S3 (S3: NO), it is subsequently confirmed whether or not there is an unrecognized moving object in the moving object list (S40).

Here, if there is an unsupported moving body in the moving body list (S40: YES), all the unsupported moving bodies in the moving body list are moved to the lost moving body list as lost moving bodies (S41). On the other hand, if there is no unsupported moving object in the moving object list (S40: NO), the process proceeds to S42 as it is.

Subsequently, the first moving body candidate and the second moving body candidate in the first moving body candidate list and the second moving body candidate list are deleted (S42), and the process ends. After the processing is completed, the process returns to S1 and the next processing is continued.

In the processing procedure described above, when there are a plurality of unrecognized objects, the unrecognized objects to be checked for correspondence are selected one by one (processing in S4). Instead of this, for multiple unrecognized objects, at the same time (as the same step even if there is a time difference), the correspondence relationship is examined and the processing according to each correspondence, that is, moving body, moving body candidate, lost moving body, etc. Or the processing that does not correspond to any of the above may be executed in parallel.

Next, the object tracking method will be described with reference to examples of a moving object, a first moving object candidate, a second moving object candidate, and an unrecognized object.

FIG. 7 is an explanatory diagram illustrating an example of a moving body, a first moving body candidate, and a second moving body candidate recognized up to the previous frame. The illustrated state indicates the previous frame at the time when the current frame is acquired in S1 of the above processing procedure. At this time, the moving body M1, the first moving body candidate C1, and the second moving body candidate C2 are recognized. FIG. 8 is a list diagram showing each list in the case of FIG. 7. The moving body M1, the first moving body candidate C1, and the second moving body candidate C2 are classified and registered, respectively. There is no lost motion in the lost motion list here.

FIG. 9 is an explanatory diagram illustrating an example in which an unrecognized object is detected in the current frame. In the illustrated state, the unrecognized objects Ob1 and Ob2 are detected in S2 in the current frame acquired in S1.

FIG. 10 is an explanatory view in which the front frame shown in FIG. 7 and the current frame shown in FIG. 9 are superimposed. In the actual processing, such superposition (image composition) of the previous frame and the current frame is not necessary.

In this example, in S6, the correspondence between the moving object M1 and the unrecognized object Ob1 will be investigated.

As a result of the processing of S6, it is assumed here that the moving body M1 and the unrecognized object Ob1 correspond to each other. FIG. 11 is an explanatory diagram illustrating a state in which the moving body M1 and the unrecognized object Ob1 correspond to each other. In FIG. 11, the moving body M1 and the unrecognized object Ob1 correspond to each other (Ob1 = C2). In the figure, the arrow indicates that the moving body M1 has moved to the unrecognized object Ob1.

At this stage, the unrecognized object Ob1 is associated with the moving body M1. Therefore, it is not necessary to investigate the correspondence with the first moving body candidate C1 and the second moving body candidate C2 existing in the previous frame. Therefore, the calculation load for tracking the object is reduced accordingly.

In this regard, in the conventional method, the unrecognized object currently detected in the frame and all the objects recognized up to the previous frame (in this example, the moving body M1, the first moving body candidate C1, and the second moving body candidate C2). Investigate the correspondence with. FIG. 12 is an explanatory diagram illustrating an example in which an unrecognized object detected in the current frame and all the objects recognized up to the previous frame are brute-forced as a comparative example.

As shown in FIG. 12, in the comparative example by the conventional method, all the correspondences between the unrecognized object Ob1 and the moving body M1, the first moving body candidate C1, and the second moving body candidate C2 are investigated. As a result, when the second moving body candidate C2 is closer to the unrecognized object Ob1 than the moving body M1, the nearby second moving body candidate C2 is erroneously recognized as the same as the moving body from the previous frame. , It may be determined that the unrecognized object Ob1 and the second moving object candidate C2 correspond to each other (Ob1 = C2). In this case, the second moving object candidate C2 has moved to the unrecognized object Ob1 as shown by the arrow in the figure. In addition, it takes a lot of time to check the correspondence between the unrecognized object and all the objects in the past frame. In the explanation of this comparative example, for the sake of clarity, the objects are classified into the moving body M1, the first moving body candidate C1, the second moving body candidate C2, and the like, but the object recognized in the past frame in the conventional method. There is no classification for.

On the other hand, in the present embodiment, since the correspondence with the moving body recognized continuously for a predetermined time (first predetermined time) or more in the past frame is investigated first, there is no erroneous recognition as in the comparative example. Further, in the present embodiment, since the correspondence with the object already recognized as a moving object from the past frame (that is, the object classified as a moving object here) is investigated first, there is a possibility that the unrecognized object corresponds. It gets higher. Then, it is not necessary to check the correspondence with other objects. In particular, since it is not necessary to check the correspondence with an object that appears only for a short time such as one frame or two frames, the calculation load is reduced and the processing time is shortened accordingly.

The explanation will be continued by returning to the case of FIG. After the correspondence result of the unrecognized object Ob1 is obtained, the correspondence of the unrecognized object Ob2 is continuously investigated. At this stage, since the moving body M1 has already been associated, the process proceeds to S8. In S8, the correspondence between the unrecognized object Ob2 and the first moving object candidate C1 will be investigated. As a result, here, it is assumed that the unrecognized object Ob2 and the first moving object candidate C1 correspond to each other. FIG. 13 is an explanatory diagram illustrating a state in which the unrecognized object Ob2 and the first moving object candidate C1 correspond to each other. In FIG. 13, the first moving object candidate C1 that has been recognized for two consecutive frames is further recognized here as the same as the unrecognized object Ob2, so that it is recognized for three consecutive frames. As a result, the first moving body candidate C1 becomes the moving body M2. Further, as shown by the arrow, the first moving body candidate C1 has moved to the moving body M2.

After that, when the processing proceeds further, since there is no corresponding object in the second moving object candidate C2, it is secondly deleted from the moving object candidate list. At this point, the second moving object candidate C2 is, for example, noise.

FIG. 14 is a list diagram showing each list at the time when the processing has progressed to the end. As shown in the figure, the moving objects M1 and M2 are registered in the moving object list, and the first moving object candidate list and the second moving object candidate list are not registered. The lost motion list is not originally registered.

FIG. 15 is an explanatory diagram illustrating a current frame at the end of processing. As shown in FIG. 15, at the end of processing, the moving objects M1 and M2 are recognized in the current frame.

(Laser radar 101)
Next, the laser radar 101 will be described.

The laser radar 101 is also called a lidar (LiDAR (Light Distance And Ringing)), and it launches a laser and measures the time until it is reflected by an object and returns (TOF (Time of Flight)). Find the distance. By performing this while scanning within a certain range, a three-dimensional distance image can be obtained. FIG. 16 is a diagram showing an example of a three-dimensional distance image obtained by the laser radar 101. Each of these distance images becomes a frame, and a moving image is formed by being continuous in chronological order.

In the distance image, as shown in the figure, a three-dimensional image in which the distance from the installation position of the laser radar 101 is known can be obtained. Here, a plurality of moving objects M1, M2, M3 and the like are shown. In addition, stationary objects St1, St2, St3, etc. are also shown. These are the ones in which the measurement results by the laser radar 101 are drawn and imaged. Then, in this image example, information indicating that the object is a moving object is superimposed and displayed on the object recognized as the moving object. Here, the object (moving body) is surrounded by a rectangular parallelepiped frame (frame of the white line in the figure) as information indicating that it is a moving body. By doing so, when the user looks at the screen (moving image), it becomes easy to understand the object recognized as a moving object. It should be noted that such a border may not be provided, or may be added or removed by a command from the user. Further, the information indicating that the object is a moving object is not limited to the frame line, and may be any mark such as a triangle or an arrow, or even character information.

FIG. 17 is a cross-sectional view illustrating the configuration of the laser radar 101.

The laser radar 101 shown in FIG. 17 is an example of being used as a monitoring device MD. Although the monitoring device MD is shown in a state of being attached to the inclined wall surface WL, the shape and length of the components may differ from the actual ones. Further, in FIG. 17, it is assumed that the monitoring device MD is installed with the top and bottom turned upside down.

The monitoring device MD is, for example, a pulse type semiconductor laser LD that emits a laser beam, a collimating lens CL that converts divergent light from the semiconductor laser LD into parallel light, and a laser beam paralleled by the collimating lens CL. A mirror unit MU that scans and projects light toward the surveillance space by a rotating mirror surface and reflects the reflected light from an unrecognized object, and a lens that collects the reflected light from the unrecognized object reflected by the mirror unit MU. The LS, the photodiode PD that receives the light collected by the lens LS, and the processing circuit (processing unit) PROC that obtains the distance information according to the time difference between the emission timing of the semiconductor laser LD and the reception timing of the photodiode PD. , A motor MT for rotationally driving the mirror unit MU, and a housing CS for accommodating these. The photodiode PD has a plurality of pixels arranged in the Y direction.

The semiconductor laser LD and the collimating lens CL form an exit LPS, the lens LS and the photodiode PD form a light receiving unit RPS, the mirror unit MU constitutes a scanning unit, and these form a light emitting / receiving unit. ..

It is preferable that the optical axes of the emitting unit LPS and the light receiving unit RPS are orthogonal to the rotation axis RO of the mirror unit MU.

The box-shaped housing CS fixed to the wall surface WL or the like has an upper wall CSa, a lower wall CSb facing the upper wall CSa, and a side wall CSc connecting the upper wall CSa and the lower wall CSb. An opening CSd is formed in a part of the side wall CSc, and a transparent plate TR is attached to the opening CSd.

The mirror unit MU has a shape in which two quadrangular pyramids are joined in opposite directions and integrated, that is, four pairs (but not limited to four pairs) of mirror surfaces MR1 and MR2 tilted in a pair and facing each other. ) Have. The mirror surfaces MR1 and MR2 are preferably formed by depositing a reflective film on the surface of a resin material (for example, PC (Polycarbonate)) in the shape of a mirror unit.

The mirror unit MU is connected to the shaft MTa of the motor MT fixed to the housing CS and is rotationally driven. Here, the axis (rotational axis) of the axis MTa extends in the Y direction inclined with respect to the vertical direction, and the ZX plane formed by the Z direction orthogonal to the Y direction and the X direction is inclined with respect to the horizontal plane. However, the axis of the axis MTa may be aligned in the vertical direction.

The principle of detecting an unrecognized object of the monitoring device MD will be described. In FIG. 17, the divergent light emitted intermittently in a pulse shape from the semiconductor laser LD is converted into a parallel light beam by the collimated lens CL, is incident on the first mirror surface MR1 of the rotating mirror unit MU, and is reflected here. Further, after being reflected by the second mirror surface MR2, it is scanned and projected as laser spot light having a vertically long rectangular cross section, for example, through the transparent plate TR and directed toward the external monitoring space. The direction in which the emitted laser spot light is reflected by an unrecognized object and returned as reflected light is called the light emitting / receiving direction. The laser spot luminous flux traveling in the same light emitting / receiving direction is detected by the same pixel.

FIG. 18 is an explanatory diagram illustrating a state in which the laser spot light SB (shown by hatching) emitted in response to the rotation of the mirror unit MU scans in the monitoring space of the monitoring device MD.

Here, the crossing angles are different in the combination of the first mirror surface MR1 and the second mirror surface MR2 of the mirror unit MU.

The laser light is sequentially reflected by the rotating first mirror surface MR1 and the second mirror surface MR2. First, the laser beam reflected by the first pair of first mirror surface MR1 and second mirror surface MR2 horizontally moves from left to right in the uppermost region Ln1 of the monitoring space according to the rotation of the mirror unit MU. Is scanned.

Next, the laser light reflected by the second pair of the first mirror surface MR1 and the second mirror surface MR2 horizontally covers the second region Ln2 from the top of the monitoring space from the left in accordance with the rotation of the mirror unit MU. Scanned to the right.

Next, the laser light reflected by the third pair of the first mirror surface MR1 and the second mirror surface MR2 horizontally covers the third region Ln3 from the top of the monitoring space from the left in accordance with the rotation of the mirror unit MU. Scanned to the right.

Next, the laser light reflected by the 4th pair of the first mirror surface MR1 and the second mirror surface MR2 horizontally moves from left to right in the lowermost region Ln4 of the monitoring space according to the rotation of the mirror unit MU. Is scanned.

This completes one scan of the entire monitoring space that can be monitored by the monitoring device MD. In this way, when the laser spot luminous flux is two-dimensionally scanned without a gap (when the trajectories of the scanned laser spot luminous flux are adjacent to each other (for example, region Ln1 and region Ln2), the adjacent trajectories are in contact with each other without a gap. , Including the case where some of them overlap), and when the monitoring device MD is set, a distance image that is easy for the user to intuitively grasp the space can be obtained, which is preferable.

The images obtained by scanning the regions Ln1 to Ln4 are combined to obtain one frame FL. Then, if the first pair of first mirror surface MR1 and second mirror surface MR2 return after one rotation of the mirror unit MU, the area from the top region Ln1 to the bottom region Ln4 of the monitoring space is restored again. The scanning is repeated to obtain the next frame FL.

With reference to FIG. 17, a part of the laser light reflected by the unrecognized object among the light beams scanned and projected passes through the transparent plate TR again and is transmitted through the transparent plate TR again to the second mirror surface MR2 of the mirror unit MU in the housing CS. Is incident on the light, is reflected here, is further reflected by the first mirror surface MR1, is condensed by the lens LS, and is detected pixel by pixel on the light receiving surface of the photodiode PD.

Further, the processing circuit (not shown) obtains the distance information according to the time difference between the emission timing of the semiconductor laser LD and the light reception timing of the photodiode PD. As a result, unrecognized objects are detected in the entire area in the monitoring space, and as a distance image (also called a measurement point marker group) having distance information (three-dimensional measurement value) for each pixel (measurement point). A frame FL (see FIG. 18) can be obtained. The shape of the distance image is the same as the shape of the spot luminous flux SB actually scanned. The distance image is sent from the processing circuit to the object tracking device 102 to perform object tracking. Further, the distance image is sent to the server 103 and stored as needed.

According to the embodiment described above, the following effects are obtained.

(1) In the present embodiment, an object that has been continuously detected for a predetermined time or longer up to the previous frame is classified as a type 1 object, and an object for less than a predetermined time is classified as a type 2 object. In the present embodiment, the first-class object is referred to as a moving object, and the second-class object is referred to as a moving object candidate. Then, it is examined whether the unrecognized object detected in the current frame corresponds to the moving object first. When an unrecognized object corresponds to a moving object, it is recognized as a moving object by associating the unrecognized object with the moving object, and if the unrecognized object does not correspond to the moving object, the correspondence with the moving object candidate is investigated. As a result, if the moving object and the unrecognized object are associated with each other, it is not necessary to investigate the moving object candidate. Therefore, the calculation load can be reduced and the processing time can be shortened accordingly. In addition, noise such as 1 frame or 2 frames that appears only for a very short time or shaking of trees is evaluated later as a moving object candidate. Therefore, such noise and the like are not mixed in the entire object tracking data.

(2) A moving object candidate associated with an unrecognized object is classified as a moving object if the elapsed time from being detected as an object has passed a predetermined time or more. As a result, when the same object is recognized over time, the moving object candidates can be classified into moving objects and the object can be tracked.

(3) The moving body candidates are classified into a plurality of stages, such as the first moving body candidate and the second moving body candidate, in order from the one having the longest elapsed time since being detected as an object (the one having the largest number of frames). Then, it was decided to investigate the correspondence with the unrecognized object in order from the one with the longest elapsed time (the one with the largest number of frames). As a result, the amount of calculation can be reduced and the influence of noise can be reduced more finely.

(4) Moving objects are also classified into multiple stages in order from the one with the longest elapsed time since being detected as an object (the one with the largest number of frames), and the correspondence with unrecognized objects in order from the one with the longest elapsed time. I decided to investigate. This makes it possible to track objects that are recognized as moving objects for a longer period of time.

(5) Further, in the present embodiment, a moving object that does not correspond to an unrecognized object is classified as a type 3 object. Again, in this embodiment, the third-class object is referred to as a lost moving object. Then, when the unrecognized object corresponds to the lost moving object, it is reclassified as a moving object. As a result, an object temporarily hidden in the frame can be recognized as the same moving object when it reappears.

(6) When it is determined that the unrecognized object does not correspond to any of the moving object candidates, the unrecognized object is classified as a moving object candidate as if it newly appeared in the current frame. As a result, an object that appears only in the current frame can be classified as a moving object candidate, and if the same object is not recognized with the passage of time, it can be excluded without being a moving object.

(7) By setting the predetermined time to 0.2 seconds or more and less than 1 second, it is possible to prevent the noise that appears and disappears for a very short time, and the flowers and trees that live swaying in the wind from being recognized as moving objects.

(8) It was decided to count the time in frame units. This makes it possible to know the elapsed time by counting the frames. Further, since the recognition of the object in the frame is performed in frame units, it is possible to count the object as a time interval for recognizing the object in a well-cut time.

(9) It was decided to display the recognized moving object together with the information indicating that it is a moving object. This makes it easier for the user to see the recognized moving object when looking at the screen.

Although the embodiments to which the present invention is applied have been described above, the present invention is not limited to these embodiments.

As the image acquisition sensor, an ordinary movie camera can be used instead of the laser radar 101. When a movie camera is used, three-dimensional distance information cannot be obtained, but it is possible to recognize how long (the number of frames) an object continues to exist in a frame. Therefore, it is possible to track an object from a moving image (frames arranged in chronological order) using a movie camera in the same manner as in the above-described embodiment (except for three-dimensional measurement).

Further, in the embodiment, the moving body candidate (type 2 object) has two stages of the first moving body candidate and the second moving body candidate. That is, it is associated with two elapsed times (number of frames) after being detected as an object. However, this may be one step in association with one elapsed time, or may be n steps (nth moving body candidate (where n is a natural number)) in association with a plurality of elapsed times.

Further, the moving body may be managed in multiple stages according to the elapsed time (number of frames). For example, for each elapsed time (number of frames) since it was detected as an object, the first to nth moving objects (where n is a natural number) are set. By doing so, it is possible to investigate the correspondence with the unrecognized object in order from the moving object having the longer recognized time (the larger number of frames) in the past frame.

In addition, the present invention can be modified in various ways based on the configurations described in the claims, and these are also within the scope of the present invention.

11 CPU,
12 RAM,
14 HDD,
101 laser radar,
102 Object tracking device.

Claims (11)

A first-class object is an object that is detected in a moving image in which multiple frames are lined up in chronological order, and the elapsed time since it was detected continues for a predetermined time or longer. In the step (a), in which an object having less than a predetermined time is classified as a second-class object.
The step (b) of investigating the correspondence between the unrecognized object and the first-class object, using the object detected in the current frame as an unrecognized object,
In the step (b), when it is determined that the unrecognized object and the first-class object correspond to each other, the step (c) of associating the unrecognized object with the first-class object is determined.
When it is determined in the step (b) that the unrecognized object and the first-class object do not correspond to each other, the step (d) of examining the correspondence between the unrecognized object and the second-class object is performed.
In the step (d), when it is determined that the unrecognized object and the second-class object correspond to each other, the step (e) of associating the unrecognized object with the second-class object is performed.
An object tracking method that has. In the step (e), if the elapsed time from the detection of the second-class object associated with the unrecognized object as an object has passed the predetermined time or more, the second-class object is referred to as the first type object. The object tracking method according to claim 1, further comprising a step (e1) of classifying into a seed object. In the step (a), the type 2 object is associated with the elapsed time since it was detected as an object.
The object tracking method according to claim 1 or 2, wherein in the step (d), the second-class object examines the correspondence with the unrecognized object in order from the one having the longest elapsed time since it was detected as an object. In the step (a), the first-class object is associated with the elapsed time since it was detected as an object.
The first-class object according to any one of claims 1 to 3, wherein in the step (b), the correspondence with the unrecognized object is examined in order from the one having the longest elapsed time since the first-class object was detected as an object. Object tracking method. In the step (a), the first-class object that does not correspond to the unrecognized object in the step (b) is classified as a third-class object.
In the step (d), when it is determined that the unrecognized object and the second-class object do not correspond to each other, the step (f) of examining the correspondence between the unrecognized object and the third-class object is performed.
In the step (f), when it is determined that the unrecognized object and the third-class object correspond to each other, the unrecognized object and the third-class object are associated with each other, and the third-class object is associated with the third-class object. At the stage (g) of reclassifying an object into the first-class object,
In the step (f), when it is determined that the unrecognized object and the third-class object do not correspond to each other, the unrecognized object is classified into the second-class object (h).
The object tracking method according to any one of claims 1 to 4. In the step (d), if it is determined that the unrecognized object does not correspond to any of the second-class objects, the unrecognized object is classified into the second-class object (d1). , The object tracking method according to any one of claims 1 to 4. The object tracking method according to any one of claims 1 to 6, wherein the predetermined time is 0.2 seconds or more and less than 1 second. The object tracking method according to any one of claims 1 to 7, wherein the time is counted in units of the frames. The object tracking method according to any one of claims 1 to 8, further comprising a step (i) of displaying the first-class object together with information indicating that it is a moving object. An object tracking program for causing a computer to execute the object tracking method according to any one of claims 1 to 9. An image acquisition sensor that acquires frames in chronological order, and
A computer that executes the program according to claim 10
Has an object tracking system.

Images