KR102189502B1 - Object tracking apparatus and object tracking method - Google Patents

Abstract

An object tracking device and an object tracking method for tracking an object using a forward image, a depth map, and a road profile are provided.
An object tracking apparatus according to an embodiment of the disclosed invention includes: a road profile acquisition unit that acquires a road profile from at least one of a front image and a depth map according to a predetermined frame rate; An image processing unit configured to detect a boundary between an object and a road existing in the region of interest of the front image using the depth map, map the road profile to the detected boundary, and measure 3D information of the object every frame; And an object tracking unit that predicts 3D information of the object in a next frame using a predetermined velocity model, compensates the predicted 3D information according to the 3D information measured in the next frame, and tracks the object. ; It may include.

Description

Object tracking device and object tracking method {OBJECT TRACKING APPARATUS AND OBJECT TRACKING METHOD}

The present invention relates to an object tracking device and an object tracking method that detects an object in front and tracks the detected object.

Since the Advanced Driver Assistance System (ADAS) or Autonomous Driving System (ADS) controls the vehicle according to the surrounding environment of the vehicle being driven, it is important to detect the surrounding environment information of the vehicle.

The surrounding environment information of the vehicle may include presence information, speed information, distance information, and information other than adjacent vehicles such as pedestrians, traffic signals, driving roads, and the like of front, rear, and/or side vehicles.

Since many driver assistance systems or autonomous driving systems operate by reflecting the surrounding environment in real time, an object tracking device that tracks an object such as a vehicle or a person existing in front of the vehicle is essential to continuously recognize the surrounding environment. I can.

In general, a 3D object tracking apparatus may continuously estimate 3D information of an object by simultaneously acquiring image information and depth information using a combination configuration of a stereo camera or a camera-depth measurement sensor (radar, lidar). Most of these object tracking devices estimate the state of an object by processing depth information based on a measurement value (2D box) acquired from an image-based object detector.

However, the above-described method is effective for estimating the state of an object in a 2D image, but when applied to a 3D object tracking, the accuracy of object tracking may be degraded. Specifically, the accuracy of the 3D position measurement of the object may be degraded due to the depth map error and the limitation of acquiring 3D information from the sensor. Difficulty in measurement may occur.

An aspect of the disclosed invention provides an object tracking device and an object tracking method for tracking an object using a forward image, a depth map, and a road profile.

An object tracking apparatus according to an embodiment of the disclosed invention includes: a road profile acquisition unit that acquires a road profile from at least one of a front image and a depth map according to a predetermined frame rate; An image processing unit configured to detect a boundary between an object and a road existing in the region of interest of the front image using the depth map, map the road profile to the detected boundary, and measure 3D information of the object every frame; And an object tracking unit that predicts 3D information of the object in a next frame using a predetermined velocity model, compensates the predicted 3D information according to the 3D information measured in the next frame, and tracks the object. ; It may include.

In addition, the image processing unit may include a region of interest setting unit configured to set the region of interest by detecting a position of the object on the front image; A boundary line detector configured to detect a boundary between the object and the road using the depth map and gradient information of the front image within the set region of interest; And mapping the road profile to the detected boundary to measure the 3D information of the object.

In addition, the 3D information measuring unit may determine the height of the object from the detected boundary using the object information identified from the front image.

In addition, the image processing unit may measure the 3D information including the 3D location of the object and the size of the object using depth information of the boundary to which the road profile is mapped.

In addition, the object tracking unit may include a 3D information prediction unit for predicting 3D information of the object in the next frame by applying the predetermined velocity model to the measured 3D information of the object; A correlation checking unit that checks a degree of association between the predicted 3D information and a plurality of 3D information measured in the next frame, and allocates the measured 3D information with the highest correlation to the object; And a 3D information update unit for compensating the predicted 3D information according to the 3D information allocated to the object. It may include.

In addition, the object tracking unit may include: an initialization unit that allocates a newly created object to 3D information not allocated to the object, or deletes an object to which the measured 3D information is not allocated for a predetermined frame or more; It may further include.

An object tracking method according to an embodiment of the disclosed invention includes: acquiring a road profile from at least one of a front image and a depth map according to a predetermined frame rate; Detecting a boundary between an object existing in the ROI and a road in the front image using the depth map; Measuring 3D information of the object every frame by mapping the road profile to the detected boundary; And tracking the object by predicting 3D information of the object in a next frame using a predetermined velocity model, and compensating the predicted 3D information according to the 3D information measured in the next frame. It may include.

In addition, detecting the boundary between the object and the road may include setting the region of interest by detecting the position of the object on the front image; And detecting a boundary between the object and the road using the depth map and gradient information of the front image within the set region of interest. It may include.

In addition, in the measuring of the 3D information of the object, the height of the object may be determined from the detected boundary using the object information identified from the front image.

In addition, in the measuring of the 3D information of the object, the 3D information including the 3D location of the object and the size of the object may be measured using depth information of a boundary to which the road profile is mapped. .

Further, the step of tracking the object may include predicting 3D information of the object in the next frame by applying the predetermined velocity model to the measured 3D information of the object; Checking a degree of association between the predicted 3D information and a plurality of 3D information measured in the next frame, and allocating the measured 3D information with the highest association degree to the object; And compensating the predicted 3D information according to the 3D information allocated to the object. It may include.

In addition, allocating a newly created object to 3D information not allocated to the object; And deleting an object to which the measured 3D information is not allocated more than a predetermined frame. It may further include.

According to the object tracking device and the object tracking method according to an aspect of the disclosed invention, since an object is tracked using a road profile, it is possible to easily obtain 3D information of an object. In addition, since it is possible to improve the accuracy of tracking a distant object, it can be applied to a high speed driving environment such as a highway. Furthermore, by using the road profile as constraint information, it is possible to robustly estimate the 3D information of the object, and as a result, the utility in driver assistance systems such as collision avoidance and collision warning can be improved.

1 is a control block diagram of an object tracking apparatus according to an embodiment of the disclosed invention.
2 is a diagram illustrating a front image acquired by a front image acquisition unit according to an embodiment of the disclosed invention.
3 is a diagram illustrating a depth map acquired by a depth map acquisition unit according to an embodiment of the disclosed invention.
4 is a diagram illustrating a road profile acquired by a road profile obtaining unit according to an embodiment of the disclosed invention.
5 is a detailed control block diagram of an image processing unit according to an embodiment of the disclosed invention.
6 is a diagram for describing an object detection step by an image processing unit according to an embodiment of the disclosed invention.
7 is a detailed control block diagram of an object tracking unit according to an embodiment of the disclosed invention.
8 is a flowchart of an object tracking method according to an embodiment of the disclosed invention.

The embodiments described in the present specification and the configurations shown in the drawings are only preferred examples of the disclosed invention, and there may be various modified examples that may replace the embodiments and drawings of the present specification at the time of filing of the present application.

The terms used herein are used to describe embodiments and are not intended to limit and/or limit the disclosed invention.

For example, in the present specification, a singular expression may include a plurality of expressions unless the context clearly indicates otherwise.

*In addition, terms such as "include" or "have" are intended to express the existence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and one or more other features It does not exclude the possibility of additional presence or addition of, numbers, steps, actions, components, parts, or combinations thereof.

In addition, terms including ordinal numbers such as "first" and "second" are used to distinguish one component from other components, and do not limit the one component.

In addition, terms such as "~ unit", "~ group", "~ block", "~ member", and "~ module" may mean a unit that processes at least one function or operation. For example, the terms may refer to at least one hardware such as field-programmable gate array (FPGA)/application specific integrated circuit (ASIC), at least one software stored in a memory, or at least one process processed by a processor. have.

Hereinafter, exemplary embodiments of the disclosed invention will be described in detail with reference to the accompanying drawings. The same reference numbers or reference numerals shown in the accompanying drawings may indicate parts or components that substantially perform the same function.

1 is a control block diagram of an object tracking apparatus according to an embodiment of the disclosed invention, FIG. 2 shows a front image acquired by a front image acquisition unit according to an embodiment of the disclosed invention, and FIG. 3 is A depth map obtained by a depth map acquisition unit according to an embodiment is shown, and FIG. 4 is a diagram showing a road profile acquired by a road profile acquisition unit according to an embodiment of the disclosed invention.

Referring to FIG. 1, an object tracking apparatus 1 according to an embodiment of the disclosed invention includes: an acquisition unit 100 for acquiring a front image, a depth map, and a road profile; An image processing unit 200 that detects an object using a front image, a depth map, and a road profile acquired by the acquisition unit 100; And an object tracking unit 300 for tracking the detected object. It may include.

The acquisition unit 100 may acquire a front image, a depth map, and a road profile used to detect an object. To this end, the acquisition unit 100 according to an embodiment includes a front image acquisition unit 110 for acquiring a front image; A depth map acquisition unit 120 for acquiring a depth map for the front side; And a road profile obtaining unit 130 for obtaining a road profile. It may include.

The front image acquisition unit 110 may acquire a front image. Here, the front image may mean an image including a plurality of objects existing in the traveling direction of the vehicle. Referring to FIG. 2, the front image may include a plurality of objects, such as a plurality of preceding vehicles traveling on a driving road.

In order to acquire the front image, the front image acquisition unit 110 may be implemented as a camera including an image sensor such as a Complementary Metal-Oxide Semiconductor (CMOS) or a Charge Coupled Device (CCD). The image sensor of the camera may acquire a front image by converting visible light detected from the front into an electrical signal. The front image acquisition unit 110 implemented as a camera may be installed inside or outside the vehicle to face the front of the vehicle.

Also, the front image acquisition unit 110 may continuously acquire a front image according to a predetermined frame rate. That is, the front image acquisition unit 110 may periodically acquire the front image every frame.

In addition, the front image acquisition unit 110 implemented as a camera may be a monocular camera, or alternatively, may be a stereo camera. If the front image acquisition unit 110 is implemented as a stereo camera, the stereo image acquired by the front image acquisition unit 110 may be used by the depth map acquisition unit 120 to be described later to acquire a depth map.

The depth map acquisition unit 120 may be implemented in various ways within the technical idea of acquiring a depth map for the front side. For example, the depth map acquisition unit 120 may acquire a depth map by matching the stereo image acquired by the front image acquisition unit 110 implemented as a stereo camera. In contrast, the depth map acquisition unit 120 includes a depth measurement sensor such as a radar or a lidar, and a depth map using the front image acquired by the front image acquisition unit 110 and a sensing value acquired by the depth measurement sensor. You can also get 3 illustrates a depth map acquired by using the front image of FIG. 2.

The depth map acquisition unit 120 may acquire a depth map according to a predetermined frame rate by synchronizing with the front image acquisition unit 110. That is, the depth map acquisition unit 120 may periodically acquire the depth map every frame.

The road profile acquisition unit 130 may acquire a road profile for a driving road ahead. Here, the road profile is structure information of a road surface, and may include 3D information of a road, and may be expressed as a function.

The road profile acquisition unit 130 may be variously implemented within the technical idea of acquiring a road profile. For example, the road profile acquisition unit 130 may acquire a road profile for a driving road ahead by using the depth map acquired by the depth map acquisition unit 120.

The road profile acquisition unit 130 may acquire a road profile according to a predetermined frame rate by synchronizing with the front image acquisition unit 110 and the depth map acquisition unit 120. That is, the road profile acquisition unit 130 may periodically acquire a road profile every frame.

Referring back to FIG. 1, the image processing unit 200 may detect an object using a front image, a depth map, and a road profile acquired by the acquisition unit 100. Hereinafter, the operation of the image processing unit 200 will be described in detail with reference to FIGS. 5 and 6.

5 is a detailed control block diagram of an image processing unit according to an embodiment of the disclosed invention, and FIG. 6 is a diagram illustrating an object detection step by the image processing unit according to an embodiment of the disclosed invention.

Referring to FIG. 5, the image processing unit 200 according to an embodiment of the disclosed invention includes: an ROI setting unit 210 for setting an ROI on a front image; A boundary line detector 220 that detects a boundary between an object and a road within the region of interest; And a 3D information measuring unit 230 for measuring 3D information of the object from the detected boundary. It may include.

The ROI setting unit 210 may detect the position of the object on the front image. That is, the ROI setting unit 210 may operate as an image-based object detector. For example, the region of interest setting unit 210 may detect the position of the object using a background estimation algorithm, and, unlike this, may detect the position of the object using a learning algorithm using a feature point.

As a result of such image-based object detection, the ROI setting unit 210 may detect a 2D box-shaped rectangular area as the location of the object. The region of interest setting unit 210 may set the detected rectangular region as the region of interest.

Referring to FIG. 6, it can be seen that the region of interest setting unit 210 detects a preceding vehicle as an object, detects two yellow square regions, and sets these as regions of interest.

The boundary line detector 220 may detect the boundary between the object and the road within the set region of interest. To this end, the boundary line detector 220 may use the depth map and gradient information of the front image.

Referring to FIG. 6, the boundary line detector 220 may detect an outer boundary of a preceding vehicle within each ROI on a depth map as a red solid line. In this case, the boundary line detection unit 220 may use the gradient of the front image together.

The 3D information measuring unit 230 may measure 3D information of an object by mapping a road profile to the detected boundary. Specifically, the 3D information measuring unit 230 may obtain the same result by mapping a road profile of, to the boundary of the preceding vehicle detected in FIG. 6.

In particular, the 3D information measuring unit 230 may measure 3D information of the object by mapping a road profile on the boundary between the road and the object. Specifically, the 3D information measuring unit 230 may measure 3D information including a 3D location of an object and a size of the object using depth information of the mapped boundary.

In this case, the 3D information measuring unit 230 may determine the height of the object from the boundary between the object and the road, using information on the object identified from the front image. Referring to FIG. 6, 3D information of the preceding vehicle extending from the boundary between the preceding vehicle and the road at the bottom by the height determined from the front image is confirmed.

Referring back to FIG. 1, the object tracking unit 300 may track the detected object in the next frame. Hereinafter, the operation of the object tracking unit 300 will be described in detail with reference to FIG. 7.

7 is a detailed control block diagram of an object tracking unit according to an embodiment of the disclosed invention.

Referring to FIG. 7, the object tracking unit 300 according to an embodiment of the disclosed invention includes a 3D information prediction unit 310 for predicting 3D information of an object in a next frame; A correlation checking unit 320 for allocating one of the 3D information measured in the next frame with the highest correlation with the predicted 3D information to a corresponding object; A 3D information update unit 330 for compensating the predicted 3D information according to the allocated 3D information; And an initialization unit 340 that allocates a new object to unallocated 3D information, or deletes an object to which 3D information measured more than a predetermined frame is not allocated. It may include.

The 3D information predictor 310 may predict 3D information of an object in a next frame using the measured 3D information. Specifically, the 3D information prediction unit 310 may predict 3D information in a next frame by using a predetermined dynamic model on the measured 3D information. Here, the dynamics model is a model representing the motion of an object and may include a constant velocity model and a constant accelerator model.

The association degree check unit 320 may check a degree of association between a plurality of 3D information measured in a next frame and predicted 3D information of a specific object. Also, the correlation check unit 320 may allocate the measured 3D information having the highest correlation to a specific object.

The 3D information update unit 330 may compare the allocated 3D information with the predicted 3D information and compensate 3D information of a specific object by using the allocated 3D information. That is, the 3D information updater 330 may update the position and speed of a specific object in the current frame predicted in the last frame to a value measured in the current frame.

In this process, the 3D information update unit 330 may adopt various probability-based filtering methods such as a Kalman filter and an Alpha-Beta-Gamma filter.

Finally, the initialization unit 340 may allocate a new object to the measured 3D information to which the object has not been allocated. As a result, the steps of predicting 3D information, checking association, and updating 3D information can be applied from the next frame to the new object.

Also, the initialization unit 340 may delete an object to which 3D information measured more than a predetermined frame is not allocated. Through this, the object tracking unit 300 may stop tracking the corresponding object.

Referring back to FIG. 1, the object detected and tracked in this way may be provided to the user by the display unit D. 1 illustrates a case where the object tracking device is provided independently from the display unit D, but the display unit D may be provided as a component of the object tracking device.

8 is a flowchart of an object tracking method according to an embodiment of the disclosed invention.

First, by the acquisition unit 100, a road profile may be obtained from at least one of a front image and a depth image according to a predetermined frame rate. (S100) Here, the road profile is structure information of a road surface, and the road profile is It may contain three-dimensional information and may be expressed as a function. For example, the road profile may include distance information from the vehicle, for example, the front image acquisition unit 110 to the road surface as 3D information of the road.

A method of obtaining a road profile may be implemented in various ways. For example, the acquisition unit 100 may acquire a road profile for a driving road ahead using the depth map.

When the road profile is acquired, the image processing unit 200 may detect the boundary between an object existing in the ROI and the road among the front images using the depth map (S110). In this case, the ROI is an image-based object. It may be set by a detection method, and the depth map and gradient information of the front image may be used to detect the boundary between the object and the road.

Then, the image processing unit 200 maps the road profile to the detected boundary to measure 3D information of the object every frame (S120). In particular, the 3D information measurement unit 230 includes a road and a road profile. By mapping road profiles on the boundary between objects, it is possible to measure 3D information of objects. Specifically, the 3D information measuring unit 230 may measure 3D information including the 3D position, speed, and size of the object using depth information of the mapped boundary.

When the 3D information is measured, the object tracking unit 300 may predict the 3D information of the object in the next frame using a predetermined dynamics model. (S130) Here, the dynamics model represents the movement of the object. Can mean model. Specifically, the object tracking unit 300 may predict 3D information of the object in the next frame according to the constant velocity model or the constant acceleration model.

Finally, the object tracking unit 300 compensates for the predicted 3D information according to the 3D information measured in the next frame to track the object (S140).

As described above, according to the object tracking apparatus 1 and the object tracking method according to an aspect of the disclosed invention, since the object is tracked using a road profile, it is possible to easily obtain 3D information of the object. In addition, since it is possible to improve the accuracy of tracking a distant object, it can be applied to a high speed driving environment such as a highway. Furthermore, by using the road profile as constraint information, it is possible to robustly estimate the 3D information of the object, and as a result, the utility in driver assistance systems such as collision avoidance and collision warning can be improved.

The above detailed description should not be construed as restrictive in all respects and should be considered as illustrative. The scope of the disclosed invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the disclosed invention are included in the scope of the disclosed invention.

1: object tracking device
100: acquisition unit
110: front image acquisition unit
120: depth map acquisition unit
130: road profile acquisition unit
200: image processing unit
210: region of interest setting unit
220: boundary line detection unit
230: 3D information measuring unit
300: object tracking unit
310: 3D information prediction unit
320: correlation check unit
330: 3D information update unit
340: initial unit
400: control unit
500: drive unit
510: gate driver
520: data driver
530: backlight driver

Claims (8)

An acquisition unit that acquires an image used to detect an object;
An image processing unit that detects an object using the image acquired by the acquisition unit; And
The object is tracked by predicting 3D information of the object in the next frame using a dynamics model representing the motion of the object, and compensating the predicted 3D information according to the 3D information of the object measured in the next frame It is configured to include an object tracking unit,
The acquisition unit,
A front image acquisition unit for obtaining a front image;
A depth map acquisition unit that acquires a depth map for the front side;
A road profile acquisition unit that acquires a road profile about a structure of a road surface including three-dimensional information of the road from at least one of a front image and a depth map according to a predetermined frame rate,
The image processing unit,
A region of interest setting unit that detects a position of an object in the front image;
A boundary line detector configured to detect a boundary between an object and a road using a depth map and a gradient of a front image within the region of interest set by the region of interest setting unit;
And a 3D information measuring unit configured to measure 3D information of the object based on 3D information of the mapped road every frame by mapping the road profile to the detected boundary,
The 3D information measuring unit determines the height of the object from the boundary between the object and the road by using information on the object identified from the front image. delete The method of claim 1,
The image processing unit,
An object tracking device that measures the 3D information including the 3D position of the object and the size of the object using depth information of the boundary to which the road profile is mapped. The method of claim 1,
The object tracking unit,
A 3D information predictor configured to predict 3D information of the object in the next frame by applying the predetermined velocity model to the measured 3D information of the object;
A correlation checking unit that checks a degree of association between the predicted 3D information and a plurality of 3D information measured in the next frame, and allocates the measured 3D information with the highest correlation to the object; And
A 3D information update unit for compensating the predicted 3D information according to the 3D information allocated to the object; Object tracking device comprising a. Obtaining a front image by a front image acquisition unit;
Obtaining a depth map for the front side by the depth map acquisition unit;
Acquiring a road profile about a structure of a road surface including 3D information of the road from at least one of a front image and a depth map according to a frame rate predetermined by the road profile obtaining unit;
Detecting the position of the object in the front image by the region of interest setting unit;
Detecting a boundary between an object and a road by using a depth map and a gradient of a front image within the region of interest set by the region of interest setting unit by a boundary line detection unit;
Mapping the road profile to the detected boundary by a 3D information measuring unit and measuring 3D information of the object based on 3D information of the mapped road every frame;
The object tracking unit predicts the 3D information of the object in the next frame using a dynamic model representing the movement of the object, and compensates the predicted 3D information according to the 3D information of the object measured in the next frame. To perform the step of tracking the object,
And determining the height of the object from the boundary between the object and the road by using the information on the object identified from the front image by the 3D information measuring unit. delete The method of claim 5,
Measuring the three-dimensional information of the object,
An object tracking method for measuring the 3D information including the 3D position of the object and the size of the object by using depth information of a boundary to which the road profile is mapped. The method of claim 5,
The step of tracking the object,
Predicting 3D information of the object in the next frame by applying the predetermined velocity model to the measured 3D information of the object;
Checking a degree of association between the predicted 3D information and a plurality of 3D information measured in the next frame, and allocating the measured 3D information with the highest association degree to the object; And
Compensating the predicted 3D information according to the 3D information allocated to the object; Object tracking method comprising a.

Images