CN107918765A - A kind of Moving target detection and tracing system and its method - Google Patents

Abstract

The invention discloses a moving target detection and tracking system and method thereof, including a moving target tracking system, based on the image containing the target in the current video image frame and the coordinates in the current video image frame, predicting the target contained in the next video image frame coordinate of. It also includes: the camera video acquisition module realizes the real-time acquisition of each frame of the camera picture, as the real-time image data input of the moving target detection module and the moving target tracking module; the moving target detection module realizes the detection and recognition of the moving target, and the The original coordinate information is transmitted to the moving target tracking module; the moving target tracking module realizes the tracking of the moving target, and sends the real-time coordinates of the tracking target to the control module to realize real-time tracking. By reasonably selecting the algorithm to realize the module, and reasonably allocating the amount of calculation on the embedded system including logic hardware and general-purpose processor, efficient and accurate real-time tracking can be realized, and the power consumption requirement of the mobile device can be met.

Description

A moving target detection and tracking system and method thereof

technical field

The invention belongs to the technical field of computer vision, and in particular relates to a moving target detection and tracking system and a method thereof, in particular to a moving target tracking system realized by computer vision.

Background technique

Object detection and tracking is an important research direction in academia and industry in recent years. Especially in the application of vehicle assisted driving system, because of its important application value, it has become the most active research topic in the field of computer vision and automatic driving. In addition, object detection and tracking has a large available space and potential significance in the fields of security, transportation and games.

Since the target detection process is affected by specific individual differences (for example, movement with significant changes in posture and clothing in movement detection), and the background where the detection target is located is also continuously changing (for example, tracking fast running movement), this is The robustness requirements of detection and tracking algorithms are very high. Second, target detection/tracking systems usually need to react instantly to target actions, which requires consideration of system size and power consumption.

Although after years of research and accumulation, the visual target detection and tracking algorithm has made great progress, but in the existing technology, there is still a lack of a robust and real-time detection algorithm that can meet the needs of small Target detection/tracking system with high performance and low power consumption requirements. Therefore, how object detection and tracking systems can be widely used in mobile terminals such as mobile devices, automobiles, robots and mobile phones is still a research hotspot in the industry.

Contents of the invention

Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a moving target detection and tracking system and method thereof, using an algorithm that can be efficiently implemented on logical hardware to construct at least a part of the moving target detection and tracking module, Therefore, while satisfying the robustness of the algorithm, the computing efficiency can be improved, and the power consumption of the system can be further reduced, laying the foundation for the miniaturization application of the system.

Technical scheme: in order to achieve the above object, the technical scheme adopted in the present invention is:

A moving target detection and tracking system, comprising a sequentially connected camera acquisition module, a moving target detection module, a moving target tracking module and a control module, the camera acquisition module is also connected with the moving target tracking module; wherein the moving target detection These two parts of the module and the moving target tracking module are realized by logic hardware, and the logic hardware is a GPU; first, the camera acquisition module obtains each frame of the camera, and then the frame of the picture is passed to the target detection module, and the target detection module passes The previous network based on GPU training detects the target object in this frame of pictures, and sends the detection result to the moving target tracking module. The moving target tracking module determines the tracking target according to the detection result, and sends the tracking target information to the control module. Module controls tracing.

Further, the moving object detection module includes a moving object detection algorithm module, and the detection algorithm uses a deep learning-based object detection algorithm SSD to detect images including moving objects in the current video image frame.

Further, the mobile detection of the moving object detection module needs to be based on the deep learning object detection algorithm SSD in the early stage, and the relevant moving object data set is trained under the caffe framework, and the trained network model is directly used in the moving detection module. Accelerate training through logical hardware GPU. The logical hardware GPU accelerates the training, which improves the training speed and improves the accuracy in real-time detection.

Further, the neural network training on the target data of the target detection algorithm SSD module based on deep learning is realized by logic hardware.

Further, the moving target tracking module includes a Fourier transform unit, an inverse Fourier transform unit, a dot product unit and a dot division unit, wherein at least the Fourier transform unit, the inverse Fourier transform unit and the dot product The unit is implemented by logical hardware.

Further, the moving object tracking module adopts a kernelized correlation filter, that is, the KCF algorithm for object tracking; the moving object tracking module is based on the moving object in the current video image frame and the coordinates of the moving object in the current video image frame to track the object , to obtain moving target information, including the position and size of the moving target.

Further, the KCF algorithm constitutes a tracking algorithm module, an image feature extraction module is set before the tracking algorithm module, and the image feature extraction module extracts image features related to the target from the partial map, and inputs these image features into the tracking algorithm module .

Further, the camera acquisition module completes video acquisition based on opencv function library related programs, and performs preprocessing on the image including noise reduction, and sends each frame of image to the moving target detection at a speed of 25 frames per second module.

Further, the camera acquisition module is realized by an embedded system of a general-purpose processor, and the moving target detection module and the moving target tracking module are all realized by an embedded system and logic hardware of a general-purpose processor, and the embedded system is NVIDIA Jerson TX2 development system; the communication between each module is realized in the form of nodes through the robot operating system ROS, each module is a node, and the sending and receiving of information is completed by sending and subscribing to messages.

A method for a moving target detection and tracking system, based on the image containing the target in the current video image frame and the coordinates in the current video image frame, predicting the coordinates of the target in the next video image frame, specifically comprising the following steps:

1) The camera acquisition module realizes the real-time acquisition of each frame picture of the camera, as the real-time image data input of the moving object detection module and the moving object tracking module;

2) The moving target detection module realizes the detection and recognition of the moving target, and transmits the original coordinate information of the moving target to be tracked to the moving target tracking module, including the position and size of the moving target;

3) The moving target tracking module realizes the tracking of the moving target, and sends the real-time coordinates of the tracking target to the control module to realize real-time tracking;

4) The control module controls the motion posture of the mobile device, tracks the tracking target in real time and ensures the smooth operation of the mobile device.

Beneficial effects: the moving target detection and tracking system and method thereof provided by the present invention can make full use of the characteristics of logical hardware parallelism and low power consumption by using logic hardware, especially GPU to realize at least part of the target detection and tracking functions, Fast detection and accurate tracking of targets can be achieved, and the size of the tracking system can be further reduced.

Description of drawings

Fig. 1 shows a schematic diagram of a target tracking system according to an embodiment of the present invention.

Fig. 2 shows a schematic diagram of a moving object detection module according to an embodiment of the present invention.

Fig. 3 shows a schematic composition diagram of a moving object tracking module according to an embodiment of the present invention.

Fig. 4 shows a schematic diagram of a control module according to an embodiment of the present invention.

Fig. 5 shows a schematic diagram of the operation of the target tracking system according to an embodiment of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings.

An important direction of human efforts is to replace manpower with machines, and visual target detection and tracking is an important part of machine intelligence. In the application of visual target detection and tracking, the problem of insufficient real-time performance of the system caused by a large amount of data processing is the key factor restricting its development. In order to solve this problem, research institutions at home and abroad mainly come from two aspects: one is to use high-performance processors, and the other is to propose new visual processing algorithms, and a considerable number of papers have been published on this. Facing the worldwide visual technology research in full swing, the inventors of the present invention found a unique way to realize at least part of the functions of target detection and tracking by software using off-the-shelf logic hardware processors. Therefore, by using the characteristics of logic hardware parallel operation and low power consumption, a real-time detection and tracking system that can meet practical applications is realized, and the power consumption of the system is kept at a relatively low level and can be miniaturized.

Fig. 1 shows a schematic diagram of a target tracking system according to an embodiment of the present invention. The target tracking system can track one or more targets in the video in real time or near real time by processing and calculating each video image frame in the input video image stream.

The target tracking system includes a target tracking module. As shown in Figure 1, at least a part of the target tracking module is implemented on logic hardware, and the target tracking module is based on the local map of the target in the current video image frame and the local map in the current video image frame. coordinates, predict the coordinates of the local map containing the target in the next video image frame.

Since the image-based target detection algorithm requires a high degree of computing parallelism, the use of a general-purpose processor (CPU) in the existing technology center will bring great performance, power consumption and cost costs, although the continuous development of processor performance Today, it is still difficult to meet the requirements of mobile products for small computing resources, low power consumption, and low cost only by the target detection implemented on the processor. In contrast, the present invention uses a logic device (GPU) that is suitable for parallel computing to realize at least a part of the target tracking function, which can greatly reduce the power consumption of the system, making the target tracking system of the present invention especially suitable for some available Mobile devices. In one embodiment, the input and output on the logic hardware are both fixed-point numbers, so as to meet the inherent operating requirements of the logic hardware while greatly reducing the amount of calculation and correspondingly improving the calculation efficiency.

Fig. 2 shows a schematic diagram of a moving object detection module according to an embodiment of the present invention. The moving target detection system detects moving targets based on each frame of image transmitted by the camera through the previously trained SSD network based on the deep learning caffe architecture, then outputs the original coordinate information of the detected target, and then transmits it to the target tracking module.

Fig. 3 shows a schematic diagram of a moving object tracking module according to an embodiment of the present invention. The mobile tracking system calculates the new coordinates of the moving target through the original target coordinate information output by the moving target detection module and the KCF tracking algorithm, and then outputs it to the control module.

FIG. 4 shows a motion control module of a mobile device according to an embodiment of the present invention. The module uses the new coordinates of the target output by the target tracking module, calculates the target position and Kalman filter algorithm, and then controls the motion posture to track the moving target in real time.

Specifically, the target detection module can be used to detect a local image containing a target in the current video image frame, that is, to realize the detection/location function of the region of interest movement as described above. Detection of the motion local map includes determining the coordinates of the motion in the video image frame.

In addition, the image input module can intercept a partial image from the current video image frame according to the moving coordinates in the current video image frame, and input the partial image to the target tracking module. In a system where the target detection module and the image input module exist at the same time, the image input module can be used to send the local map located/detected by the target detection module to the target tracking module. Although only the moving target detection module is shown in FIG. 2 , it should be understood that only the target detection module, only the image input module or both may correspond to different embodiments of the present invention, And all of them can achieve at least part of the beneficial effects of improving computing efficiency and reducing power consumption of the present invention.

Usually, the target detection module detects the moving partial image containing the target at the beginning of the detection, and the image input module sends the partial image to the target tracking module. Subsequently, the target detection module can remain idle, and the target tracking module can predict the coordinates of the next local map containing the same target by itself and continue to perform subsequent target tracking.

However, in some cases, the target detection module is still required to detect the moving partial image. For example, when receiving an input command from the user, or when the system determines that the target is lost. In addition, some systems also specify timing re-inspection, that is, when a predetermined period of time has elapsed since the last input of the current video image frame to the object detection module, the object detection module re-detects the partial image.

Preferably, in order to reduce the dimensionality of data that the tracking algorithm module needs to calculate, the target tracking module can also set an image feature extraction module before the tracking algorithm module. The image feature extraction module extracts image features related to the target from the partial image, and inputs these image features into the tracking algorithm module. The features that reflect the essential attributes of the pattern can be obtained through image feature extraction, so it can greatly facilitate the follow-up tracking algorithm to track specific targets.

The module configuration example of the target tracking system according to the present invention has been described above with reference to FIGS. 1-4 . It should be understood that those skilled in the art may make different permutations and combinations of various features in the above embodiments to meet the needs of practical applications. These different combinations are within the scope covered by the appended claims of the present invention.

As follows, we will describe the principle of the present invention in conjunction with specific algorithm applications, especially to flexibly divide software and hardware tasks according to specific applications, thereby realizing a real-time, accurate, and compact target detection system with low power consumption as much as possible.

As mentioned above, in some embodiments of the present invention, the target tracking system may include a target detection module, configured to detect a partial image including the target from the current image video frame. In the prior art, the division of local graphs is generally divided into four methods: motion-based, distance-based, image feature-based and camera parameter-based. Among them, the image feature-based method refers to obtaining a local map by detecting image features related to the target, and is more and more widely used because of its wide application range and robust results.

Artificial Neural Networks (ANN), also referred to as neural networks (NNs), is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. In recent years, neural networks have developed rapidly and are widely used in many fields, including image recognition, speech recognition, natural language processing, apocalypse forecasting, gene expression, content push and so on. It has been proven in academia and industry that after training with massive data, neural network algorithms can achieve high target detection accuracy. For the technical solution of the present invention, it is a good choice to use a deep learning SSD algorithm based on a convolutional neural network algorithm to perform target detection based on image features. Therefore, in one embodiment, the target detection module of the present invention adopts a deep learning SSD algorithm to detect a partial image including the target in the current video image frame. In addition, since SSD performs distributed parallel computing, implementing object detection functions through logical hardware, especially GPU, has natural computing advantages, and compared with software execution, it can achieve lower power consumption.

SSD is essentially a classifier that extracts pixel values and/or feature values from an input image, and outputs a judgment of whether an object is a specific target. In many cases, what is given is the probability value that the object is a specific target. In the present invention, a series of positive and negative sample training can be used to train the target detection module. After the training, the SSD can process the input image frame (unknown sample) and determine the moving part containing the target with a certain probability. picture.

In one embodiment, the image feature extraction module adopts a Histogram of Oriented Gradients (HOG) algorithm for image feature extraction. In specific applications, such as in motion detection and tracking, HOG can be calculated in key areas such as the moving head and limbs, thus effectively reducing the vector dimension to greatly improve the detection rate while the detection rate is basically unchanged. speed.

The image features extracted by the image feature extraction module can then be input into the tracking algorithm module implemented by the Kernel Correlation Filter (KCF) algorithm, thereby realizing real-time tracking of the target. All of the image feature extraction module and at least a part of the tracking algorithm module are realized by logical hardware. Preferably, the tracking algorithm module may include a Fourier transform unit, an inverse Fourier transform unit, a dot product unit and a dot division unit. Among them, the Fourier transform unit, the inverse Fourier transform unit and the dot product unit, which are especially suitable for logic hardware realization, are realized by logic hardware. In one embodiment, the erasing unit can be implemented by software. In another embodiment, if the logical hardware resources are sufficient, the above-mentioned deletion unit may also be implemented on logical hardware.

In one embodiment, the object tracking system of the present invention can be implemented on an embedded system including a general-purpose processor and logical hardware. Among them, the logic hardware is preferably GPU. For example, the technical solution of the present invention can be realized on the Jerson TX2 embedded system of Nvidia including GPU and ARM core.

In specific applications, functions such as the image input module that are more suitable for software implementation can be implemented by a general-purpose processor through software programming. For example, a general-purpose processor may include a software-implemented controller module. The controller can be used to control the operation of the entire system, such as input and output of video images, module call control, etc. Therefore, the controller module can realize the above image input and/or target loss judging functions.

The object detection function implemented by SSD is preferably implemented by logic hardware. The functions of the target tracking module are preferably at least partially implemented by logic hardware. In the case where the target tracking module includes an image feature extraction module (for example, HOG algorithm) and a tracking algorithm module (KCF), for example, all of the image feature extraction module and a part of the tracking algorithm module can be placed on the logic hardware according to the usage of the logic hardware realized.

Therefore, through the creative selection of algorithms (i.e., using SSD for object detection, HOG for feature extraction, and KCF for tracking of object coordinates), and flexibly and appropriately distribute operations on hardware and software (e.g., will be suitable for parallel The calculation algorithm is at least partially implemented on the logic hardware, while functions that are more suitable for software implementation, such as control, are implemented by the processor), which can achieve real-time accurate detection and tracking of targets with higher efficiency. Due to the higher detection and tracking efficiency, in one embodiment, the target tracking system of the present invention is also suitable for tracking multiple targets. For example, for a system with a processing capability of 200 fps, the target tracking system can realize, for example, simultaneous tracking of 10 targets with a precision of 20 fps according to the frame rate of the input video image. In addition, it needs to be clarified that the target tracking module of the present invention can predict the local map coordinates of the next frame according to the local map of the current frame and its coordinates. The "next frame" here refers to the "next" frame coming into the system for processing for that particular object, not the next frame captured by the image capture source or simply the next frame fed into the system frame.

The principles of the present invention are described above in conjunction with the accompanying drawings and specific algorithms. Image-based object detection algorithms require a high degree of computational parallelism. In the existing technical solutions, the neural network algorithm is usually used to implement it in a general-purpose processor (CPU). However, this will bring great performance, power consumption and cost costs, especially unable to adapt to the small amount of calculation and performance of mobile products. The requirement of low power consumption and low cost brings great difficulties to the tracking and monitoring of mobile products. Image-based target tracking algorithms and detection algorithms have a large amount of computation overall. Under the conditions of performance, cost, computing resources and power consumption, it is difficult to achieve a high-quality target detection/ tracking system.

Based on the Jerson TX2 embedded system, this application proposes a flexible division of software and hardware computing tasks to achieve efficient and flexible control logic and computing performance, based on deep learning algorithm (SSD) and tracking algorithm (KCF), complete target detection and tracking system. On mobile products, satisfactory performance can be achieved, and the cost and power consumption can be kept at a low level. In addition, since the technical solution of the present application can obtain high performance with a small amount of computing resources, it is also applicable to non-mobile terminal products (such as servers), so that higher tracking performance can be achieved with more computing resources.

Example

A specific application according to the present invention will be described below with reference to FIG. 5 . Fig. 5 shows a schematic diagram of the operation of the target tracking system according to an embodiment of the present invention. The system shown in Fig. 5 can realize high-performance real-time target detection and tracking with very little hardware resource consumption. Specifically, the system can be a software-hardware collaborative system for real-time target detection and tracking of a GPU-based embedded system, wherein both input and output on the GPU can be fixed-point, so as to further simplify calculation and improve calculation efficiency. The system includes: a target detection module, which is used to globally locate the position and size of the target, and reposition the target when the system starts, the user inputs, and the time expires. In this example, a deep learning network framework (SSD) is used to implement the target detection module.

The system also includes: a target tracking module, which is used to track the target in real time, after the target is detected (that is, a partial image including the target is detected), the target is tracked in real time. In this example, the Histogram of Oriented Gradient (HOG) algorithm is used to extract features from the image, and the Kernel Correlation Filter algorithm (KCF) is used to predict the position and size of the target in real time based on the image features.

The control module is used to control the operation of the entire system, such as input and output of video images, module call control, and so on. For example, the image input module mentioned above may be a part of the control module.

The specific operation steps of the example system will now be described in conjunction with Figure 4:

Step 1: The system starts to run, calls the SSD, locates the input full-image target, and obtains the position and size of the target in the video screen image (ie, detection of the moving partial image).

Step 2: The moving target detection module sends the calculated original target coordinates to the KCF calculation module. If the SSD occupies resources on the GPU, the KCF calculation task can be divided to the microprocessor side. The entire calculation process of KCF is controlled by the microprocessor.

Step 3: The position and size of the target calculated by KCF are sent to the controller, and the controller marks the output video image.

Step 4: After each KCF calculation is completed, the controller will compare the confidence probability attached to the calculation with the preset threshold value. If it is less than the threshold value, it will be considered that the tracking is lost, and the system will start from step 1. If the confidence probability is greater than the threshold, it is considered normal, and the system will start to run from step 2.

This system can dynamically correct the tracking object, and when the tracking is lost, it can re-call the algorithm of the whole image target detection to reposition the target. In addition, this system also solves the computing resource allocation problem of the target tracking system. According to the size of GPU resources, it can flexibly divide the software and hardware tasks for the realization of the algorithm. It can map a large-scale system to the hardware for realization according to the existing hardware resources, and obtain as close as possible to the existing hardware structure. Extreme value performance.

The object tracking system according to the present invention has been described in detail above with reference to the accompanying drawings.

The above is only a preferred embodiment of the present invention, it should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, some improvements and modifications can also be made, and these improvements and modifications are also possible. It should be regarded as the protection scope of the present invention.

Claims (10)

1. a kind of Moving target detection and tracing system, it is characterised in that：Including sequentially connected camera acquisition module, movement Module of target detection, movable object tracking module and control module, the camera acquisition module also with movable object tracking Module connects；Wherein, this two parts of the Moving target detection module and movable object tracking module are real by logic hardware Existing, the logic hardware is GPU；Camera acquisition module obtains each frame picture of camera first, then by the frame figure Piece passes to module of target detection, and module of target detection based on the GPU networks trained by detecting the mesh in this frame picture before Object is marked, and testing result is sent to movable object tracking module, movable object tracking module determines to chase after according to testing result Track target, and tracking target information is sent to control module, controlled and followed the trail of by control module.

2. Moving target detection according to claim 1 and tracing system, it is characterised in that：The Moving target detection mould Block includes Moving target detection algoritic module, and the detection algorithm is examined using the algorithm of target detection SSD based on deep learning Surveying current video image frame includes the image of mobile target.

3. Moving target detection according to claim 2 and tracing system, it is characterised in that：The Moving target detection mould Need the mobile detection early period of block to be based on deep learning algorithm of target detection SSD, the related mobile target of training under caffe frameworks Data set, is directly used in mobile detection module by trained network model, is accelerated in the training process by logic hardware GPU Training.Logic hardware GPU accelerates training, improves training speed, and in real-time detection, accuracy is also improved.

4. Moving target detection according to claim 2 and tracing system, it is characterised in that：It is described based on deep learning The neural metwork training to target data of algorithm of target detection SSD modules is realized by logic hardware.

5. Moving target detection according to claim 1 and tracing system, it is characterised in that：The mobile target tracking mould Block includes Fourier transform unit, inverse Fourier transform unit, dot product unit and point and removes unit, and wherein at least described Fourier becomes Change in unit, inverse Fourier transform unit and dot product unit and realized by logic hardware.

6. Moving target detection according to claim 1 and tracing system, it is characterised in that：The movable object tracking mould Block takes core correlation filter i.e. KCF algorithms to carry out target tracking；The mobile target tracking module is based on current video The coordinate tracking target of mobile target and mobile target in current video image frame in picture frame, obtains mobile target letter Breath, includes the positions and dimensions of mobile target.

7. Moving target detection according to claim 6 and tracing system, it is characterised in that：The tracking of KCF Algorithm constitutions is calculated Method module, sets image characteristics extraction module before the tracing algorithm module, and described image characteristic extracting module is from part Extraction and the relevant characteristics of image of target in figure, and by these characteristics of image inputting, tracing algoritic modules.

8. Moving target detection according to claim 1 and tracing system, it is characterised in that：The camera acquisition module By completing the collection of video based on opencv function libraries relative program, and the pretreatment including noise reduction is carried out to image, Each two field picture is sent to by Moving target detection module with the speed of 25 frames/second.

9. Moving target detection according to claim 1 and tracing system, it is characterised in that：The camera acquisition module By the Implementation of Embedded System of general processor, the Moving target detection module, mobile target tracking module are by logical Realized with the embedded system and logic hardware of processor, the embedded system is the tall and handsome Jerson TX2 exploitations system reached System；Communication between modules realizes that each module is exactly one by robot operating system ROS in the form of node A node, the transmission of information and receiving are completed by sending message and subscribing to message.

10. a kind of method of Moving target detection and tracing system, it is characterised in that：Based on including mesh in current video image frame Coordinate in target image and current video image frame, predicts the coordinate that the target is included in next video image frame, tool Body comprises the following steps：

1) camera acquisition module realizes each frame picture of real-time acquisition camera, as Moving target detection module and movement The realtime image data input of target tracking module；

2) Moving target detection module realizes that the detection to mobile target identifies, and by the original coordinates of mobile target to be tracked Information is transferred to movable object tracking module, includes the positions and dimensions of mobile target；

3) movable object tracking module realizes the tracking to mobile target, and tracking target real-time coordinates are sent to control module Realize real-time tracing；

4) control module realizes the athletic posture of control mobile device, and tracking target is followed the trail of in real time and ensures mobile dress Put even running.