JP7509040B2 - Apparatus and method for tracking objects in roadway surveillance video - Google Patents

Description

The present invention relates to the field of video surveillance technology.

Video surveillance systems are widely applied in daily life, for example, a camera head can monitor a road and detect various types of traffic events. In order to detect an event, the video surveillance configuration information, such as a region of interest (ROI), road width, road direction, etc., can generally be pre-set.

In traffic surveillance videos, the size of detected objects (also called targets, e.g., vehicles) decreases from near to far, and obstacles and other objects may block the detected targets. Scale changes and blocking are one of the problems in traditional object tracking.

Traffic analysis based on artificial intelligence (AI) technology is a hot research field, which can detect traffic events, measure vehicle speed, and calculate traffic volume. Almost all of these applications are based on some basic modules, such as object detection, multi-object tracking, event judgment, etc. It is one of the most important parts for tracking traffic objects.

The inventors discovered that although state-of-the-art tracking methods based on deep learning can solve some of the problems of scale change and blocking, they are computationally intensive and require pre-training on the application scenario.

When it comes to real-time road monitoring systems, there are multiple modules that need to work together, including scenario understanding, object detection, tracking, etc. Therefore, the processing time of each module needs to be balanced. Usually, the detection module takes the longest time. Therefore, the time required for the tracking module needs to be short. Tracking methods based on deep learning are difficult to apply to real-time road monitoring systems.

In view of at least one of the above problems, embodiments of the present invention provide an apparatus and method for object tracking in road surveillance videos, which can solve the problems of scale changes and/or blocking in real-time road surveillance videos.

According to a first aspect of an embodiment of the present invention, there is provided an object tracking apparatus for road surveillance video, comprising:
an object detection unit for performing object detection on a real-time monitoring video of a road area of interest and obtaining detection information of one or more objects;
a tracking matching unit for matching the detection information of the one or more objects with tracking information for object tracking;
a tracking update unit for updating the tracking information based on a matching result; and a road update unit for updating road information of the region of interest based on the detection information and/or the tracking information.

According to a second aspect of an embodiment of the present invention, there is provided a method for tracking objects in road surveillance videos, comprising:
Performing object detection on the real-time surveillance video of the road area of interest, and obtaining detection information of one or more objects;
Matching the one or more object detection information with tracking information for object tracking;
updating the tracking information based on the matching result; and updating road information of the area of interest based on the detection information and/or the tracking information.

One of the beneficial effects of the embodiment of the present invention is that by matching detection information of one or more objects with tracking information for object tracking, and updating the tracking information based on the matching result, and updating road information of the region of interest based on the detection information and/or tracking information, rapid object tracking can be achieved, and problems of scale change and/or blocking in real-time road surveillance video can be solved.

FIG. 2 illustrates a method for tracking objects in road surveillance video in an embodiment of the present invention. FIG. 13 is a diagram showing an example of updating a roadway width in an embodiment of the present invention. FIG. 11 is a diagram showing an example of parameters for tracking matching in an embodiment of the present invention. FIG. 11 is a diagram showing a processing method for certain tracking information that does not match detection information in an embodiment of the present invention. FIG. 4 is a diagram showing an example of a current frame in an embodiment of the present invention. FIG. 2 is a diagram showing an example of an input image for performing template matching in an embodiment of the present invention. FIG. 4 is a diagram showing an example of a template image for performing template matching in the embodiment of the present invention. FIG. 2 illustrates an object tracking device for road surveillance video in an embodiment of the present invention. 1 is a diagram showing an electronic device according to an embodiment of the present invention;

Below, a preferred embodiment for carrying out the present invention will be described in detail with reference to the attached drawings.

In the embodiment of the present invention, a traffic event and road monitoring are taken as an example, and a video image including multiple frames can be captured by a camera head, which may be a PTZ camera head used in intelligent transportation video monitoring, which can continuously capture images of a road monitoring area. However, the present invention is not limited thereto.

<Example of the first aspect>
An embodiment of the present invention provides a method for tracking objects in road surveillance videos. Figure 1 is a diagram showing the method for tracking objects in road surveillance videos in an embodiment of the present invention. As shown in Figure 1, the method includes the following steps:

101: Perform object detection on a real-time surveillance video of a road region of interest (ROI) to obtain detection information of one or more objects;
102: Matching detection information of one or more objects with tracking information for object tracking;
103: Updating the tracking information based on the matching result; and 104: Updating road information of the area of interest based on the detection information and/or tracking information.

As shown in FIG. 1, optionally, the method may further include the following steps:

105: Determine whether the system is in an initialization phase, and if so, update road information for the area of interest based on the detection information (104); if not, match the detection information of the one or more objects with tracking information for object tracking (102).

Note that the above-mentioned FIG. 1 is merely for illustrative purposes of an embodiment of the present invention, and the present invention is not limited thereto. For example, the execution order between each operation (step) may be appropriately adjusted, or some operations (steps) may be increased or decreased. In other words, a person skilled in the art may make appropriate changes to the above-mentioned content based on the description in FIG. 1.

In some embodiments, object detection and tracking can be performed based on the video sequence of the camera head and a configuration file. The configuration file may include an ROI region, and the road information may include road width and road direction, which may be represented as a matrix.

For example, two matrices can be used to store the lane width and lane direction values (travel (heading) angles) for each Y position in each ROI, and these two matrices are denoted as lane_width_mat and lane_dir_mat, respectively. For example, if there are two ROIs and the image height is 360 (pixels), the size of these two matrices is 2*360.

In some embodiments, an object detection method, model, or detector can be selected based on the computing power of the hardware configuration, for example, objects can be detected by machine learning or deep learning models. In the initial stage, the detection range can be adjusted according to the ROI region, and the object detection module can output detection information (including rectangle, type, and attributes) of each detected object to the tracking module. Hereinafter, the detection information detected by the detection module is referred to as blob information.

For example, the YOLO V3 model after pruning can be used as a detector, and the model can be trained using surveillance camera head samples. The model can detect seven types of objects, namely, cars, trucks, buses, vans, motorcycles, bicycles, and people. For details on how to perform object detection, please refer to the related art.

In some embodiments, the trackers can be used to store tracking information for tracking objects. For example, each vehicle tracker has one buffer in the container, and the goal of tracking is to update the container of tracker buffers and also remove trackers that have left the ROI region from the container.

For example, one storage space can be allocated for each newly detected vehicle. A newly detected vehicle indicates that the vehicle does not match the blob of any buffer. For example, the data structure of the buffer is as shown in Table 1, where we allocate the rectangle of the detected vehicle for "track_rect" in the new buffer. Other information in the buffer is set to 0 and can be updated in the subsequent steps.

In an embodiment of the present invention, road information can be used to solve the problems of scale change and blocking. For example, the road width value can be used to adjust the distance threshold for tracking. When a single fixed threshold is used for track matching, the tracking accuracy may decrease when the vehicle enters a distant or nearby area. When the tracker and the detected blob do not match, the road width value can be further used to adjust the scale before template matching.

In some embodiments, the width of the rectangle in the detection information is multiplied by a predetermined scale value to update the width of the road within a predetermined height range corresponding to the detection information, and/or the direction of the road within a predetermined height range corresponding to the detection information is updated by calculating the angle of the vector between multiple frames based on the midpoint of the base of the detection information.

For example, in the initialization phase, the roadway width update is after the object detection module, and after the initialization phase, the roadway width update is after the tracking module. The roadway width can refer to the width of the "car" rectangle. The actual roadway width is usually 3.5m to 4m, and the car width is 1.6m to 1.8m. Therefore, the roadway width can be 1.5 times the car width (i.e., the predetermined scale value). Note that the predetermined scale value here is not limited to 1.5 and may be other values.

Figure 2 shows an example of roadway width updating in an embodiment of the present invention, illustrating a case where the roadway width value is adaptively updated based on detection information. The road monitoring area includes ROI 1 and ROI 2, and the distance between line 201 and line 202 along the roadway is the adaptive roadway width for each Y position.

As shown in FIG. 2, for example, when a car blob 203 is detected, its bottom Y coordinate is y and its width is w in ROI 1, the lane width value between y-10 and y+10 in the first row of lane_width_mat is updated to 1.5*w. The 10 pixel shift is to ensure that the update can cover the ROI area from top to bottom after a certain period of time.

Also, for example, roadway direction updates are after the tracking module and can be used to remove false matches in the tracking module. When an object is tracked for a period of time, the driving angle (0°-360°) is stable. Taking the midpoint of the base as the reference point, the angle between the two trackers can be calculated. The duration of a frame is t (e.g., 5).

For example, for an object, if five frames are tracked and the reference point of the current i-th frame is (x _i , y _i ) and the reference point of the (i-5)th frame is (x _i-5 , y _i-5 ), the angle of the vector between these two points is updated to lane_dir_mat. Similar to updating the roadway width, the area above and below the current Y position (e.g., the area between y-10 and y+10) can also be updated.

In some embodiments, determining whether a certain detection information and a certain tracking information can be matched can be performed. Specifically, it can be determined that a certain detection information and a certain tracking information can be matched when at least one or any combination of the following conditions is satisfied:
an object corresponding to the detection information is located in the same region of interest as an object corresponding to the tracking information;
The distance between the midpoint of the base of the detection information and the midpoint of the base of the tracking information is smaller than a value obtained by multiplying the width of the roadway corresponding to the tracking information by a first predetermined coefficient;
The difference between the area of the rectangle of the detection information and the area of the rectangle of the tracking information is smaller than a value obtained by multiplying the relatively smaller area of the two areas by a second predetermined coefficient; and the angle between the matching vector direction of the detection information and the roadway driving direction (the driving direction of the vehicle on the roadway) corresponding to the tracking information is smaller than a predetermined angle.

For example, to monitor real-time road video, this means that we can process dozens of frames within one second, so the moving distance of an object cannot be too long. A comparison can be made between each detected blob and the buffer tracker based on the tracker's own distance, area and history information.

For example, if the detected blob is blob[i], the buffer tracker is tracker[j], the value of the lane width of tracker[j] is lane_wj, and the driving angle of tracker[j] is anglej, the matching condition is:
1) corresponding objects are located in the same ROI;
2) The distance between the midpoints of the two bases is less than 0.5*lane_wj;
3) the difference in area between the two rectangles is less than 0.3 times the smaller area of the two rectangles; and 4) the angle between the roadway driving direction of the tracker [j] and the matching vector is less than 45°.

Note that the above matching methods and conditions are merely specific examples of the present invention, and the present invention is not limited to these.

Figure 3 is a diagram showing an example of parameters for tracking matching in an embodiment of the present invention. As shown in Figure 3, the dotted line below tracker[j] is the historical trajectory, and the roadway driving direction below tracker[j] is vector direction 301, which is the direction angle value in lane_dir_mat of the Y coordinate of the midpoint of the base of tracker[j]; and the angle between blob[i] and the midpoint of the base of tracker[j] (also called the matching vector direction, i.e., the direction of vector 302) can be calculated. The angle of the above condition 4) is the angle between vector 301 (roadway driving direction) and vector 302 (matching vector direction), and the angle is less than 45°.

In some embodiments, if a certain tracking information corresponds to one or more detection information, the most matching detection information is selected to update the tracking information; if a certain detection information corresponds to one or more tracking information, the detection information is used to update the most matching tracking information; if a certain detection information does not match the tracking information, the detection information is a new object.

For example, after matching each detected blob with a tracker, we can obtain a matrix named match_mat. If we detect N blobs in the current frame and have M trackers, the size of the matching matrix is N*M. The tracking container is updated based on the following cases:

である場合、最も一致した検出ブラッブを選択して追跡子［ｊ］を更新し、ｊ番目の列における他のマッチングフラグが０に設定される。 For a given tracker [j], if there are one or more detection blobs that match it, i.e.,

If so, select the best matching detection blob and update the tracker[j], and the other matching flags in the jth column are set to 0.

である場合、最も一致した追跡子を選択し、ｂｌｏｂ［ｉ］を用いてそれを更新し、ｉ番目の行における他のマッチングフラグが０に設定される。 If one or more tracers match a detected blob[i], that is,

If so, select the best matching tracker, update it with blob[i], and set the other matching flag in the i-th row to 0.

である場合、検出した該ブラッブを、ＲＯＩに入った新しいオブジェクトと見なし、その後、それに対して初期化を行い、また、それを追跡コンテナにプッシュすることができる。 If there is no tracer that matches the detected blob[i], i.e.,

If so, then we consider the detected blob as a new object that has entered the ROI, and we can then perform initialization on it and push it into the tracking container.

In some embodiments, if a tracking information does not match the detection information, prediction information is constructed for the tracking information or the tracking information is removed.

である場合、２つの可能性を含み、１つは、該オブジェクトがＲＯＩを離れており、もう１つは、検出モジュールが一時的に該オブジェクトを見逃している。後者の場合、本発明の実施例は、テンプレートマッチング方法により矩形を見つけ、又は、履歴情報に基づいて予測操作を行うことができる。 For example, if no blob[j] matching the tracker is found, i.e.,

If , there are two possibilities: the object has left the ROI, and the detection module misses the object temporarily. In the latter case, the embodiment of the present invention can find the rectangle by template matching method, or perform predictive operation based on historical information.

Figure 4 illustrates a processing method for certain tracking information that does not match the detection information in an embodiment of the present invention. As shown in Figure 4, for certain tracking information that does not match the detection information, the processing method includes the following steps:

401: Determine whether the number of template matching frames (matching-cnt) is less than a first threshold (threshold 1). If yes, execute 402; if no, execute 405.

For example, the number of template matching frames (matching-cnt) may be frame_cnt in Table 1 minus predict_cnt and vehicle_cnt.

402: Construct an input image using the current frame, and construct a template image using the previous frame (e.g., the previous frame).

In some implementations, a rectangle may be found in the current frame to create an input image for template matching, where a Kalman filter can be used to predict the displacement vector (dx, dy) between frames.

For example, the midpoint of the bottom edge of the rectangle is taken as the reference point, and the displacement of the reference point between frames is (dx, dy). To initialize the Kalman filter, there may be four state value dimensions, namely dx, dy, dx', and dy'. The transition matrix A is defined as follows:

During the update process of the tracking module, the Kalman filter updates the prediction state based on the measurement results. When the prediction function is called with the measurement matrix of the previous frame, it returns the prediction result of the current frame, for example, the position of the prediction reference point is predict_point.

In some embodiments, a corresponding first road width is obtained by calculating a predicted point of the current frame based on a Kalman filter; a corresponding second road width is obtained based on the midpoint of the base of the predicted information; a road width ratio is calculated based on the first road width and the second road width; the input image is obtained from the current frame based on the road width ratio, and/or the template image is obtained from the previous frame based on the road width ratio.

Figure 5 shows an example of a current frame in an embodiment of the present invention. For example, first, determine the range of the input image in the current frame. "predict_point" is the midpoint of the prediction base from the Kalman filter prediction, as shown at 501 in Figure 5. track_rect is the rectangle of the tracking information in the tracker, as shown at 502 in Figure 5.

If the Y coordinate of track_rect is y1, the Y coordinate of predict_point is y2, and the corresponding lane width values are lane_width_mat[y1] and lane_width_mat[y2], respectively, then the lane width ratio is lane_width_mat[y2]/lane_width_mat[y1].

Then, the midpoint of the bottom side of "track_rect" is moved to "predict_point", and the size of the box of track_rect can be adjusted according to the ratio of the roadway width mentioned above, and the new rectangle after the ratio adjustment is called "predict_rect", as shown in box 503 in FIG. 5. Then, the range of this rectangle is expanded by 0.1 times to reduce the error, thereby forming a matching range, as shown in box 504 in FIG. 5, for example.

Finally, the input image for template matching can be obtained by cutting the image within the matching range in the current frame.

Figure 6 shows an example of an input image for performing template matching in an embodiment of the present invention.

In some embodiments, a template image for template matching may be constructed based on the previous frame. Since the frame-to-frame variation in the size of a car may be large, the template image also needs to be adjusted based on the roadway width ratio.

Figure 7 shows an example of a template image for performing template matching in an embodiment of the present invention. As shown in Figure 7, the image on the left is a small image in the previous frame, and the image on the right is an image after size adjustment, for example multiplied by the roadway width ratio described above, and the adjusted image is used as the template image.

As shown in FIG. 4, the processing method may further include the following steps:

403: Perform template matching between the input image and the template image.

In some embodiments, the matching range is WxH and the blob template is wxh. In this case, the grayscale image of each blob can be used as the template. The matching function slides in the background image in the manner of WxH, compares the overlapping color blocks of wxh, and stores the comparison results in mat. The size of the mat is (W-w+1)x(H-h+1).

For example, template matching can be performed by the normalized correlation coefficient method, as shown in the following formula: where I represents the image (the buffered background image in the matching range), T represents the template (the grayscale image in the blob area), and R represents the result matrix. A normalization process is performed on the image and the template before the similarity calculation, which can avoid errors caused by changes in illumination. The value of R ranges from -1 to 1. 1 indicates that the image is the same as the template, -1 indicates the opposite (i.e., the two are different), and 0 indicates that there is no linear relationship between them.

Note that although only template matching has been described above as an example, the present invention is not limited to this. For specific details of template matching, please refer to related art.

As shown in FIG. 4, the processing method may further include the following steps:

404: Determine whether the matching degree of the template matching is greater than a second threshold (threshold 2); if yes, execute 405; if no, execute 406;
405: Updating the tracking information based on the input image as the predicted information.

In some embodiments, after performing the template matching between the above template image and the input image, a result matrix R and corresponding matching rectangles can be obtained. The maximum value in R is the final matching degree match_degree (between -1 and 1). The corresponding rectangle of match_degree is the matching rectangle match_rect.

In some embodiments, a matching degree threshold may be set to determine whether the matching result is valid. For example, the threshold may be 0.7, and if the matching degree is greater than 0.7, it indicates that the template matching is valid, and in this case, proceed to 405 for updating tracking information; otherwise, it means that the template matching has failed, and proceed to 406 for prediction operation.

In an embodiment of the present invention, if there is no tracking information that matches the detection information and the template matching process also fails, the object still cannot be determined as a distant target because the object may be temporarily blocked by an obstacle or other object. Therefore, a prediction operation may be used to continue tracking for several frames.

As shown in FIG. 4, the processing method may further include the following steps:

406: Determine whether the predicted frame count (predict-cnt) is less than a third threshold (threshold 3) and whether the total frame count (frame-cnt) is less than a fourth threshold (threshold 4). If yes, execute 407; if no, execute 411.

407: A predicted image is constructed as the prediction information using the current frame.

For example, the input image described above may be the predicted image, but the present invention is not limited to this.

As shown in FIG. 4, the processing method may further include the following steps:

408: Determine whether the predicted image is valid, and if the predicted image is valid, execute 409; if the predicted image is invalid, execute 410;
409: Updating the tracking information based on the predicted image;
410: Remove the tracking information.

In some embodiments, the prediction operation cannot be performed forever, and predict_cnt stores the number of consecutive predicted frames. If predict_cnt is higher than a threshold, the prediction rectangle is not adopted and a delete operation is performed on this tracker. After updating the prediction rectangle to track_rect, add 1 to predict_cnt. Alternatively, if track_rect is updated according to the detected blob or template matching result, set predict_cnt to 0.

Note that the above-mentioned FIG. 4 merely illustrates an example embodiment of the present invention, and the present invention is not limited thereto. For example, the execution order between each operation (step) may be appropriately adjusted, or some operations (steps) may be added or removed. In other words, a person skilled in the art may make appropriate changes to the above content based on the description of FIG. 4.

In some embodiments, the tracker is updated based on the results of a blob obtained by track matching, or a blob obtained by template matching, or a blob obtained by a prediction operation. track_rect is updated with a rectangle of one of the three types of blobs, frame_cnt is incremented by 1, and the midpoint of the base of this new rectangle is taken as the measurement for updating kalman_filter. If the rectangle is from track matching, vehicle_cnt is incremented by 1, and if the rectangle is from a prediction operation, predict_cnt is incremented by 1.

In some embodiments, if subsequent modules require the trajectory information, a container may be added to the tracker structure to store the trajectory points, which are then used to determine traffic events. In a traffic analysis system, the tracking results can be used for traffic target counting, congestion detection, wrong direction detection, lane change detection, illegal parking detection, etc.

Although only each step or process according to the present invention has been described above, the present invention is not limited thereto. The object tracking method may further include other steps or processes, and the specific contents of these steps or processes may be referred to in the prior art. In addition, although the above formulas have been used as examples to explain the embodiments of the present invention, the present invention is not limited to these formulas, and these formulas may be modified as appropriate, all of which fall within the technical scope of the embodiments of the present invention.

Furthermore, the above-mentioned embodiments are merely illustrative examples of the present invention, and the present invention is not limited thereto. Appropriate modifications may be made based on the above-mentioned embodiments. For example, the above-mentioned embodiments may be used alone, or one or more of the above-mentioned embodiments may be used in combination.

As can be seen from the above embodiments, by matching detection information of one or more objects with tracking information for object tracking, updating the tracking information based on the matching result, and updating road information of the region of interest based on the detection information and/or tracking information, fast object tracking can be achieved and problems of scale change and/or blocking in real-time road surveillance video can be solved.

<Example of the second aspect>
An embodiment of the present invention provides an object tracking apparatus for road surveillance video, and the same contents as those of the embodiment of the first aspect are omitted here.

Figure 8 illustrates an object tracking device for road surveillance video in an embodiment of the present invention. As shown in Figure 8, the object tracking device for road surveillance video 800 includes:

Object detection unit 801: performs object detection on the real-time surveillance video of the road area of interest, and obtains detection information of one or more objects;
A tracking matching unit 802: matches the detection information of the one or more objects with tracking information for object tracking;
A tracking update unit 803: updates the tracking information based on the matching result; and a road update unit (road information update unit) 804: updates the road information of the area of interest based on the detection information and/or the tracking information.

In some embodiments, as shown in FIG. 8, the road surveillance video object tracking device 800 may further include:

Initialization determination unit 805: determines whether it is in the initialization phase, and if it is, the road update unit 804 updates the road information of the area of interest based on the detection information, and if it is not in the initialization phase, the tracking matching unit 802 matches the detection information of the one or more objects with tracking information for object tracking.

In some embodiments, the road update unit 804 includes:
It is used to update the road width within a predetermined height range corresponding to the detection information by multiplying the width of the rectangle of the detection information by a predetermined scale value; and/or to calculate the angle of a vector between multiple frames based on the midpoint of the base of the detection information, and update the road direction within a predetermined height range corresponding to the detection information.

In some embodiments, the tracking matching unit 802 determines that a given detection information and a given tracking information can be matched if at least one or any combination of the following conditions is satisfied:
an object corresponding to the detection information and an object corresponding to the tracking information are located in the same region of interest;
The distance between the midpoint of the base of the detection information and the midpoint of the base of the tracking information is smaller than a value obtained by multiplying the road width corresponding to the tracking information by a first predetermined coefficient;
The condition is that the difference between the area of the rectangle of the detection information and the area of the rectangle of the tracking information is smaller than a value obtained by multiplying the smaller of the two areas by a second predetermined coefficient; and the angle between the matching vector direction of the detection information and the roadway driving direction corresponding to the tracking information is smaller than a predetermined angle.

In some embodiments, the tracking update unit 803 is used to, when a piece of tracking information corresponds to one or more pieces of detection information, select the most matching detection information to update the tracking information; when a piece of detection information corresponds to one or more pieces of tracking information, use the detection information to update the most matching tracking information.

The tracking update unit 803 is further used to: make the detection information a new object when there is no matching tracking information with the detection information; construct prediction information for the tracking information when there is no matching detection information with the tracking information, or delete the tracking information.

In some embodiments, the tracking update unit 803 includes:
If the number of template matching frames is less than a first threshold, construct an input image using the current frame, and construct a template image using the previous frame;
Perform template matching between the input image and the template image; and when the matching degree of the template matching is greater than a second threshold, it is used to update the tracking information based on the input image as the prediction information.

In some embodiments, the tracking update unit 803 further comprises:
Calculate a predicted point of the current frame based on the Kalman filter, and obtain the corresponding first roadway width;
Obtain a corresponding second roadway width based on the midpoint of the base of the predicted information;
Calculate a roadway width ratio based on the first roadway width and the second roadway width; and used to obtain the input image from the current frame based on the roadway width ratio and/or obtain the template image from the previous frame based on the roadway width ratio.

In some embodiments, the tracking update unit 803 further comprises:
When the number of template matching frames is equal to or greater than the first threshold, or the matching degree of the template matching is equal to or less than the second threshold, determine whether the number of predicted frames is smaller than a third threshold and whether the total number of frames is smaller than a fourth threshold;
When the number of predicted frames is smaller than the third threshold and the total number of frames is smaller than the fourth threshold, using the current frame to construct a predicted image as the prediction information; and when the predicted image is valid, it is used to update the tracking information based on the predicted image.

In some embodiments, the tracking update unit 803 is further configured to delete the tracking information when the predicted frame number is greater than or equal to the third threshold, or the total frame number is greater than or equal to the fourth threshold, or the predicted image is invalid.

For convenience, FIG. 8 shows only the connection relationships or signal directions between each component or module, but those skilled in the art should understand that various related technologies, such as connection by a bus, may also be adopted. In addition, each of the above-mentioned components or modules may be realized by hardware, such as a processor, a memory, etc., but the embodiment of the present invention is not limited thereto.

The above-mentioned embodiments are merely illustrative of the embodiments of the present invention, and the present invention is not limited thereto. Appropriate modifications may be made based on the above-mentioned embodiments. For example, the above-mentioned embodiments may be used alone, or one or more of the above-mentioned embodiments may be used in combination.

As can be seen from the above embodiments, by matching detection information of one or more objects with tracking information for object tracking, and then updating the tracking information based on the matching result, and updating road information of the region of interest based on the detection information and/or tracking information, rapid object tracking can be achieved and problems of scale change and/or blocking in real-time road surveillance video can be solved.

<Example of the third aspect>
An embodiment of the present invention provides an electronic device, which includes the road surveillance video object tracking device according to the embodiment of the second aspect, the contents of which are incorporated herein by reference. The electronic device may be, for example, a computer, a server, a workstation, a laptop, a smartphone, etc., but the embodiment of the present invention is not limited thereto.

Figure 9 is a diagram showing an electronic device in an embodiment of the present invention. As shown in Figure 9, the electronic device 900 may include a processor (e.g., a central processing unit CPU) 910 and a memory 920, and the memory 920 is connected to the central processing unit 910. The memory 920 can store various data, and can also store a program 921 for information processing, and can execute the program 921 under the control of the processor 910.

In some embodiments, the functionality of the road surveillance video object tracking device 800 is integrated into the processor 910. The processor 910 is configured to implement the road surveillance video object tracking method described in the embodiments of the first aspect.

In some embodiments, the road surveillance video object tracking device 800 may be located separately from the processor 910, for example, the road surveillance video object tracking device 800 may be configured as a chip connected to the processor 910, and the functions of the road surveillance video object tracking device 800 may be realized under the control of the processor 910.

In some embodiments, the processor 910 may be configured to perform the following controls: perform object detection on real-time surveillance video of a road area of interest to obtain detection information of one or more objects; match the detection information of the one or more objects with tracking information for object tracking; update the tracking information based on the matching result; and update road information of the area of interest based on the detection information and/or the tracking information.

In some embodiments, the processor 910 may be further configured to perform the following controls: determine whether it is in an initialization phase; if it is, update road information for the region of interest based on the detection information; if it is not in the initialization phase, match the detection information of the one or more objects with tracking information for object tracking.

In some embodiments, the processor 910 may be further configured to perform the following controls: multiplying the width of the rectangle of the detection information by a predetermined scale value to update the road width within a predetermined height range corresponding to the detection information; and/or calculating the angle of the vector between multiple frames based on the midpoint of the base of the detection information to update the road direction within a predetermined height range corresponding to the detection information.

In some embodiments, the processor 910 may be further configured to perform the following control: determine that a given detection information and a given tracking information may match if at least one or any combination of the following conditions is satisfied:
an object corresponding to the detection information and an object corresponding to the tracking information are located in the same region of interest;
The distance between the midpoint of the base of the detection information and the midpoint of the base of the tracking information is smaller than a value obtained by multiplying the road width corresponding to the tracking information by a first predetermined coefficient;
The difference between the area of the rectangle of the detection information and the area of the rectangle of the tracking information is smaller than a value obtained by multiplying the smaller of the two areas by a second predetermined coefficient; and the angle between the matching vector direction of the detection information and the roadway driving direction corresponding to the tracking information is smaller than a predetermined angle.

In some embodiments, the processor 910 may be further configured to perform the following controls: if a certain tracking information corresponds to one or more detection information, select the most matching detection information to update the tracking information; if a certain detection information corresponds to one or more tracking information, use the detection information to update the most matching tracking information; and if there is no tracking information that matches the certain detection information, make the detection information a new object; if there is no detection information that matches the certain tracking information, construct prediction information for the tracking information or delete the tracking information.

In some embodiments, the processor 910 may be further configured to perform the following control: if the number of template matching frames is less than a first threshold, construct an input image using the current frame and construct a template image using the previous frame; perform template matching between the input image and the template image; and if the matching degree of the template matching is greater than a second threshold, update the tracking information using the input image as the prediction information.

In some embodiments, the processor 910 may be further configured to perform the following controls: calculate a predicted point of the current frame based on a Kalman filter and obtain a corresponding first road width; obtain a corresponding second road width based on the midpoint of the base of the predicted information; calculate a road width ratio based on the first road width and the second road width; and obtain the input image from the current frame based on the road width ratio and/or obtain the template image from the previous frame based on the road width ratio.

In some embodiments, the processor 910 may be further configured to perform the following control: if the number of template matching frames is equal to or greater than a first threshold, or the matching degree of the template matching is equal to or less than the second threshold, determine whether the number of predicted frames is smaller than a third threshold and whether the total number of frames is smaller than a fourth threshold; if the number of predicted frames is smaller than the third threshold and the total number of frames is smaller than the fourth threshold, use the current frame to construct a predicted image as the prediction information; and if the predicted image is valid, update the tracking information based on the predicted image.

In some embodiments, the processor 910 may be further configured to perform the following control: delete the tracking information if the number of predicted frames is greater than or equal to the third threshold, or the total number of frames is greater than or equal to the fourth threshold, or the predicted image is invalid.

As shown in FIG. 9, the electronic device 900 may further include an input/output (I/O) device 930, a display 940, etc., of which the functions of these components are similar to those of the prior art, and therefore detailed description thereof will be omitted here. Note that the electronic device 900 does not need to include all of the components shown in FIG. 9. The electronic device 900 may also include components not shown in FIG. 9, but reference can be made to related art regarding this.

An embodiment of the present invention further provides a computer-readable program, which, when executed in an electronic device, causes a computer to execute the method for tracking objects in road surveillance video described in the embodiment of the first aspect in the electronic device.

An embodiment of the present invention further provides a storage medium having a computer-readable program recorded thereon, in which the computer-readable program causes a computer to execute the method for tracking objects in road surveillance video described in the embodiment of the first aspect in an electronic device.

The apparatus and methods described with reference to the embodiments of the present invention can be realized by hardware, software modules executed by a processor, or a combination of both. For example, one or more functions in the functional block diagram and/or one or more combinations of functions in the functional block diagram may correspond to each software module in a computer program or each hardware module. Furthermore, each of these software modules may correspond to each step shown in the diagram illustrating the method. These hardware modules can be realized by solidifying these software modules using, for example, an FPGA (field-programmable gate array).

In addition, the devices, methods, etc. according to the embodiments of the present invention may be realized by software, by hardware, or by a combination of hardware and software. The present invention also relates to such computer-readable programs, i.e., the programs, when executed by logic components, can cause the logic components to realize the above-mentioned devices or components, or cause the logic components to realize the above-mentioned methods or steps thereof. Furthermore, the present invention also relates to storage media, such as hard disks, magnetic disks, optical disks, DVDs, flash memories, etc., that store the above-mentioned programs.

In addition, the following notes are provided regarding the above examples.

(Appendix 1)
1. A method for tracking objects in road surveillance videos, comprising:
Performing object detection on the real-time surveillance video of the road area of interest, and obtaining detection information of one or more objects;
Matching the one or more object detection information with tracking information for object tracking;
updating the tracking information based on the matching result; and updating road information for the area of interest based on the detection information and/or the tracking information.

(Appendix 2)
2. The method according to claim 1, comprising:
The method further comprises:
determining whether it is in an initialization phase;
A method comprising: if the initialization phase is in progress, updating road information for the area of interest based on the detection information; and if the initialization phase is not in progress, matching the detection information of the one or more objects with tracking information for object tracking.

(Appendix 3)
3. The method according to claim 1 or 2,
A method for updating a road width within a predetermined height range corresponding to the detection information by multiplying the rectangular width of the detection information by a predetermined scale value; and/or calculating an angle of a vector between multiple frames based on the midpoint of the base of the detection information, and updating a road direction within a predetermined height range corresponding to the detection information.

(Appendix 4)
4. The method according to any one of claims 1 to 3, comprising:
Determine that a certain detection information and a certain tracking information can be matched if at least one or any combination of the following conditions is satisfied:
an object corresponding to the detection information and an object corresponding to the tracking information are located in the same region of interest;
The distance between the midpoint of the base of the detection information and the midpoint of the base of the tracking information is smaller than a value obtained by multiplying the width of the roadway corresponding to the tracking information by a first predetermined coefficient;
a difference between an area of the rectangle of the detection information and an area of the rectangle of the tracking information is smaller than the smaller of the two areas multiplied by a second predetermined coefficient; and an angle between a matching vector direction of the detection information and a roadway driving direction corresponding to the tracking information is smaller than a predetermined angle.

(Appendix 5)
5. The method according to any one of claims 1 to 4, comprising:
When a certain tracking information corresponds to one or more detection information, select the most matching detection information and update the tracking information, and when a certain detection information corresponds to one or more tracking information, use the detection information to update the most matching tracking information; and when there is no tracking information that matches a certain detection information, make the detection information a new object, and when there is no detection information that matches a certain tracking information, construct prediction information for the tracking information or delete the tracking information.

(Appendix 6)
6. The method according to claim 5, comprising:
The method further comprises:
If the number of template matching frames is less than a first threshold, construct an input image using the current frame, and construct a template image using the previous frame;
performing template matching between the input image and the template image; and updating the tracking information based on the input image as the predicted information when a matching degree of the template matching is greater than a second threshold.

(Appendix 7)
7. The method according to claim 6, comprising:
The method further comprises:
Calculate the predicted point of the current frame based on the Kalman filter, and obtain the corresponding first roadway width;
Obtain a corresponding second roadway width based on the midpoint of the base of the predicted information;
calculating a roadway width ratio based on the first roadway width and the second roadway width; and obtaining the input image from the current frame based on the roadway width ratio and/or obtaining the template image from the previous frame based on the roadway width ratio.

(Appendix 8)
8. The method according to claim 6 or 7,
The method further comprises:
If the number of template matching frames is equal to or greater than the first threshold, or the matching degree of the template matching is equal to or less than the second threshold, determine whether the number of predicted frames is less than a third threshold and whether the total number of frames is less than a fourth threshold;
if the number of predicted frames is less than the third threshold and the total number of frames is less than the fourth threshold, using a current frame to construct a predicted image as the prediction information; and if the predicted image is valid, updating the tracking information based on the predicted image.

(Appendix 9)
9. The method of claim 8, further comprising:
The method further comprises:
removing the tracking information if the predicted number of frames is greater than or equal to the third threshold, or if the total number of frames is greater than or equal to the fourth threshold, or if the predicted image is invalid.

(Appendix 10)
1. A road surveillance video object tracking device, comprising:
an object detection unit for performing object detection on a real-time monitoring video of a road area of interest and obtaining detection information of one or more objects;
a tracking matching unit for matching the detection information of the one or more objects with tracking information for object tracking;
a tracking update unit for updating the tracking information based on a matching result; and a road update unit for updating road information of the region of interest based on the detection information and/or the tracking information.

(Appendix 11)
11. The apparatus of claim 10, further comprising:
The apparatus further comprises:
An initialization determination unit for determining whether the device is in an initialization stage;
When in the initialization phase, the road update unit updates road information of the area of interest based on the detection information, and when not in the initialization phase, the tracking matching unit matches the detection information of the one or more objects with tracking information for object tracking.

(Appendix 12)
12. The apparatus of claim 10 or 11,
The road update unit includes:
The device multiplies the width of the rectangle of the detection information by a predetermined scale value to update the road width within a predetermined height range corresponding to the detection information; and/or calculates the angle of a vector between multiple frames based on the midpoint of the base of the detection information to update the road direction within a predetermined height range corresponding to the detection information.

(Appendix 13)
13. The apparatus of any one of claims 10 to 12, further comprising:
The tracking matching unit determines that a certain detection information can be matched with a certain tracking information if at least one or any combination of the following conditions is satisfied:
an object corresponding to the detection information and an object corresponding to the tracking information are located in the same region of interest;
The distance between the midpoint of the base of the detection information and the midpoint of the base of the tracking information is smaller than a value obtained by multiplying the width of the roadway corresponding to the tracking information by a first predetermined coefficient;
a difference between an area of the rectangle of the detection information and an area of the rectangle of the tracking information is smaller than the smaller of the two areas multiplied by a second predetermined coefficient; and an angle between a matching vector direction of the detection information and a roadway driving direction corresponding to the tracking information is smaller than a predetermined angle.

(Appendix 14)
14. The apparatus of any one of claims 10 to 13, further comprising:
The tracking update unit is
When a certain tracking information corresponds to one or more detection information, the most matching detection information is selected to update the tracking information, and when a certain detection information corresponds to one or more tracking information, the detection information is used to update the most matching tracking information; and when there is no tracking information that matches a certain detection information, the detection information is treated as a new object, and when there is no detection information that matches a certain tracking information, the device constructs prediction information for the tracking information, or deletes the tracking information.

(Appendix 15)
15. The apparatus of claim 14, further comprising:
The tracking update unit is
If the number of template matching frames is less than a first threshold, construct an input image using the current frame, and construct a template image using the previous frame;
performing template matching between the input image and the template image; and updating the tracking information based on the input image as the predicted information when a matching degree of the template matching is greater than a second threshold.

(Appendix 16)
16. The apparatus of claim 15, further comprising:
The tracking update unit further comprises:
Calculate a predicted point of the current frame based on the Kalman filter, and obtain the corresponding first roadway width;
Obtain a corresponding second roadway width based on the midpoint of the base of the predicted information;
calculating a roadway width ratio based on the first roadway width and the second roadway width; and obtaining the input image from the current frame based on the roadway width ratio and/or obtaining the template image from the previous frame based on the roadway width ratio.

(Appendix 17)
17. The apparatus of claim 15 or 16,
The tracking update unit further comprises:
If the number of template matching frames is equal to or greater than the first threshold, or the matching degree of the template matching is equal to or less than the second threshold, determine whether the number of predicted frames is less than a third threshold and whether the total number of frames is less than a fourth threshold;
If the number of predicted frames is less than the third threshold and the total number of frames is less than the fourth threshold, using a current frame to construct a predicted image as the prediction information; and if the predicted image is valid, updating the tracking information based on the predicted image.

(Appendix 18)
18. The apparatus of claim 17, further comprising:
The tracking update unit further comprises:
The apparatus deletes the tracking information if the predicted number of frames is greater than or equal to the third threshold, or if the total number of frames is greater than or equal to the fourth threshold, or if the predicted image is invalid.

(Appendix 19)
An electronic device,
A memory and a processor are included,
An electronic device, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the method for tracking objects in road surveillance videos described in any one of Supplementary Notes 1 to 9.

The above describes a preferred embodiment of the present invention, but the present invention is not limited to this embodiment, and any modification to the present invention falls within the technical scope of the present invention as long as it does not deviate from the spirit of the present invention.

Claims (9)

1. An apparatus for tracking an object in a road surveillance video, comprising:
an object detection unit for performing object detection on a real-time surveillance video of a road area of interest and obtaining detection information of one or more objects;
an initialization determination unit for determining whether the device is in the initialization phase;
a tracking matching unit for matching the detection information of the one or more objects with tracking information for object tracking if not in the initialization stage ;
a tracking update unit for updating the tracking information based on a matching result; and
A road updating unit for updating road information of said area of interest based on said detection information and/or said tracking information when in said initialization phase . 2. The apparatus of claim 1,
The road update unit includes:
The device multiplies the width of the rectangle of the detection information by a predetermined scale value to update the road width within a predetermined height range corresponding to the detection information; and/or calculates the angle of a vector between multiple frames based on the midpoint of the base of the detection information to update the road direction within a predetermined height range corresponding to the detection information. 2. The apparatus of claim 1,
The tracking matching unit includes:
Determine that a certain detection information and a certain tracking information can be matched if at least one or any combination of the following conditions is satisfied:
an object corresponding to the detection information and an object corresponding to the tracking information are located in the same region of interest;
The distance between the midpoint of the base of the detection information and the midpoint of the base of the tracking information is smaller than a value obtained by multiplying the width of the roadway corresponding to the tracking information by a first predetermined coefficient;
The device has the following conditions: a difference between an area of the rectangle of the detection information and an area of the rectangle of the tracking information is smaller than the smaller of the two areas multiplied by a second predetermined coefficient; and an angle between a matching vector direction of the detection information and a roadway driving direction corresponding to the tracking information is smaller than a predetermined angle. 2. The apparatus of claim 1,
The tracking update unit is
When a certain tracking information corresponds to one or more detection information, select the most matching detection information and update the tracking information, and when a certain detection information corresponds to one or more tracking information, use the detection information to update the most matching tracking information; and when there is no tracking information that matches a certain detection information, make the detection information a new object, and when there is no detection information that matches a certain tracking information, construct prediction information for the tracking information, or delete the tracking information. 5. The apparatus of claim 4 ,
The tracking update unit is
If the number of template matching frames is less than a first threshold, construct an input image using the current frame, and construct a template image using the previous frame;
performing template matching between the input image and the template image; and updating the tracking information based on the input image as the predicted information when a matching degree of the template matching is greater than a second threshold. 6. The apparatus of claim 5 ,
The tracking update unit further comprises:
Calculate a predicted point of the current frame based on the Kalman filter, and obtain the corresponding first roadway width;
Obtain a corresponding second roadway width based on the midpoint of the base of the predicted information;
calculating a roadway width ratio based on the first roadway width and the second roadway width; and obtaining the input image from the current frame based on the roadway width ratio and/or obtaining the template image from the previous frame based on the roadway width ratio. 6. The apparatus of claim 5 ,
The tracking update unit further comprises:
If the number of template matching frames is equal to or greater than the first threshold, or the matching degree of the template matching is equal to or less than the second threshold, determine whether the number of predicted frames is less than a third threshold and whether the total number of frames is less than a fourth threshold;
If the number of predicted frames is less than the third threshold and the total number of frames is less than the fourth threshold, constructing a predicted image as the prediction information using a current frame; and if the predicted image is valid, updating the tracking information based on the predicted image. 8. The apparatus of claim 7 ,
The tracking update unit further comprises:
The apparatus deletes the tracking information if the predicted number of frames is greater than or equal to the third threshold, or if the total number of frames is greater than or equal to the fourth threshold, or if the predicted image is invalid. 1. A method for tracking an object in a road surveillance video, comprising:
Performing object detection on a real-time surveillance video of a road area of interest, and obtaining detection information of one or more objects;
Determine if we are in the initialization phase;
If not in the initialization stage, matching the detection information of the one or more objects with tracking information for object tracking;
updating the tracking information based on the matching results; and
If in said initialization phase, the method comprises updating road information for said area of interest based on said detection information and/or said tracking information.

Images