KR101964100B1 - Object detection apparatus based on neural network learning and method of the same - Google Patents

Abstract

Provided are an apparatus for detecting an object based on neural network learning and a method thereof which can effectively learn learning data for deep learning. To this end, the apparatus for detecting an object based on neural network learning comprises: a plurality of lidar units wherein a part of the lidar units collects first plane data by scanning a front object in a first direction and another part of the lidar units collects second plane data by scanning the front object in a second direction; and a neural network learning unit for performing neural network learning for the front object based on the first plane data and the second plane data.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an object detection apparatus and method based on neural network learning,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an object detection technique, and more particularly, to a neural network learning based object detection apparatus and method capable of effectively acquiring learning data for deep learning.

Generally, in order to detect the shape and position of an object distributed in space, a two-dimensional image of the object is measured through a camera, or information about the object is generated by measuring the position or movement speed of the object through Lidar . The generated information is applied to the learning model and can be used in the object detection process based on the information.

Conventional learning modeling technology extracts feature points from two-dimensional images from a camera and uses them for learning based on changes in brightness or crossing points of outlines, or uses three-dimensional information And can be used for learning. This conventional technique has a disadvantage in that data processing speed for learning is slow due to bringing up unnecessary data in the process of performing learning, and as a result, learning efficiency may be deteriorated.

Korean Patent No. 10-1737803 (Jul. 15, 2015) relates to an object information collecting apparatus, which is installed at a side of a crosswalk, and has a Lidar method for a plurality of pre- An object detection unit that detects a predetermined object based on the scan information and extracts feature information of the detected object, and an object detection unit that detects feature information of the object based on the feature information and pre- A fixture that is repeatedly detected at a predetermined position, and an object classifier that classifies at least one of a person and a vehicle having a predetermined size range, wherein the object classifier classifies the fixture information including the position or type of the fixture based on the feature information, And, when the fixture is detected, marking and providing the fixture.

Korean Patent Laid-Open Publication No. 10-2015-0045735 (Apr. 29, 2015) discloses a multi-wavelength image sensor device. The sensor device includes a transmitter for outputting a multi-wavelength optical pulse signal, a transmitter for outputting the multi- A transmission / reception optical section for converting the received optical signal into a signal and outputting it as a space, a transmission / reception optical section for transmitting a reception optical signal on which the transmission optical signal is reflected by an object in space, And a processor for calculating chromatic coordinate information of the received optical signal depending on the intensity of the reflected signal of each wavelength.

Korean Patent No. 10-1737803 (Jul. 2015) Korean Patent Laid-Open Publication No. 10-2015-0045735 (Apr. 29, 2015)

An embodiment of the present invention is to provide an apparatus and method for detecting an object based on a neural network that can effectively learn learning data for deep learning.

An object of the present invention is to provide an apparatus and method for detecting an object based on a neural network learning which has a cost reduction effect by acquiring a three-dimensional point cloud using a two-dimensional ray sensor.

An embodiment of the present invention is an apparatus and method for detecting an object based on a neural network that can acquire various data from a three dimensional Lidar sensor by removing unnecessary data by allowing various arrangements of a two dimensional Lidar sensor according to a mounting angle. .

Among the embodiments, the neural network learning-based object detecting apparatus performs scanning in a first direction with respect to a front object to collect first plane data, and another part performs scanning in a second direction to obtain second plane data And a neural network learning unit for performing neural network learning on the forward object based on the first and second plane data.

The plurality of ladders are mounted on the vehicle and the part performs scanning in a direction perpendicular to the traveling direction of the vehicle and the other part performs scanning in a direction parallel to the traveling direction of the vehicle .

The plurality of ladder units may comprise a plurality of the portions and the other portions may be configured in a single number.

And each of the plurality of ladder units may be 2D calibrated in advance to share the same 3D coordinate system.

The neural network learning unit may form the 3D point cloud by arranging the first and second plane data in the 3D coordinate system before learning the neural network.

The neural network learning unit may decompose the 3D point cloud through the xy plane, the yz plane, and the xz plane to generate a plurality of 2D projection data.

The neural network learning unit may perform neural network learning on the forward object based on each of the plurality of 2D projection data.

The neural network learning unit may perform DNN (Deep Neural Network) learning on the forward object.

The neural network learning-based object detecting apparatus may further include an object classifying unit that classifies an unknown object, which is acquired through a specific classifier and composed of a set of 3D points, as one of the forward objects learned as a neural network.

Among the embodiments, the object detection method based on the neural network learning is performed by the object detection apparatus based on the neural network learning. The neural network learning based object detection method includes the steps of (a) collecting first plane data by performing scanning in a first direction with respect to a front object, and scanning another object in a second direction to collect second plane data And (b) performing neural network learning on the forward object based on the first and second plane data.

Among the embodiments, a computer-executable recording medium records the method of claim 10.

The disclosed technique may have the following effects. It is to be understood, however, that the scope of the disclosed technology is not to be construed as limited thereby, as it is not meant to imply that a particular embodiment should include all of the following effects or only the following effects.

The neural network learning based object detecting apparatus and method according to an embodiment of the present invention can effectively learn learning data for deep learning.

The apparatus and method for detecting an object based on neural network learning according to an embodiment of the present invention have a cost reduction effect by acquiring a three-dimensional point cloud using a two-dimensional ray sensor.

The apparatus and method for detecting objects based on neural network learning according to an embodiment of the present invention can arrange various sensors in a variety of ways depending on the mounting angle, It is possible.

1 is a view for explaining an object detection apparatus based on a neural network learning according to an embodiment of the present invention.
FIG. 2 is a view showing an embodiment of a process of forming a 3D point cloud by the neural network learning unit shown in FIG. 1;
FIG. 3 is a diagram illustrating an exemplary process of the neural network learning unit of FIG. 1 performing neural network learning by analyzing a 3D point cloud.
FIG. 4 is a view showing an embodiment of a process of object classification by the object classification part in FIG. 1.
FIG. 5 is a flowchart illustrating a process of the neural network learning based object detection apparatus of FIG. 1 performing neural network learning.

The description of the present invention is merely an example for structural or functional explanation, and the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, the embodiments are to be construed as being variously embodied and having various forms, so that the scope of the present invention should be understood to include equivalents capable of realizing technical ideas. Also, the purpose or effect of the present invention should not be construed as limiting the scope of the present invention, since it does not mean that a specific embodiment should include all or only such effect.

Meanwhile, the meaning of the terms described in the present application should be understood as follows.

The terms "first "," second ", and the like are intended to distinguish one element from another, and the scope of the right should not be limited by these terms. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

It is to be understood that when an element is referred to as being "connected" to another element, it may be directly connected to the other element, but there may be other elements in between. On the other hand, when an element is referred to as being "directly connected" to another element, it should be understood that there are no other elements in between. On the other hand, other expressions that describe the relationship between components, such as "between" and "between" or "neighboring to" and "directly adjacent to" should be interpreted as well.

It is to be understood that the singular " include " or "have" are to be construed as including the stated feature, number, step, operation, It is to be understood that the combination is intended to specify that it does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

In each step, the identification code (e.g., a, b, c, etc.) is used for convenience of explanation, the identification code does not describe the order of each step, Unless otherwise stated, it may occur differently from the stated order. That is, each step may occur in the same order as described, may be performed substantially concurrently, or may be performed in reverse order.

The present invention can be embodied as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all kinds of recording devices for storing data that can be read by a computer system . Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like. In addition, the computer-readable recording medium may be distributed over network-connected computer systems so that computer readable codes can be stored and executed in a distributed manner.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. Commonly used predefined terms should be interpreted to be consistent with the meanings in the context of the related art and can not be interpreted as having ideal or overly formal meaning unless explicitly defined in the present application.

1 is a view for explaining an object detection apparatus based on a neural network learning according to an embodiment of the present invention.

1, the neural network learning based object detecting apparatus 100 may include a plurality of ladder units 110, a neural network learning unit 120, a control unit 130, and an object classifying unit 140, These can be interconnected.

The plurality of ladhardes 110 performs scanning in a first direction with respect to the front object 10 to collect first plane data and the other part performs scanning in a second direction to collect second plane data do. More specifically, each of the plurality of ladder units 110 is scanned in a first direction, which is one of x-axis, y-axis, and z-axis directions, with respect to the horizontal plane on which the corresponding line is arranged, And the other part or the other of the plurality of ladder sections 110 are arranged in a second direction that is the other one of the x-axis, y-axis, and z-axis directions To collect second plane data about the forward object 10. Here, the front object 10 may correspond to or include at least one object (e.g., a pedestrian, a pole, or other vehicle) in front of each of the plurality of ladder sections 110.

In one embodiment, each of the plurality of ladder sections 110 may be implemented as a two dimensional (LIDAR) laser radar that measures the distance between objects using a laser, and may be, for example, a 2D two-dimensional space) to generate two-dimensional data about the distance of each of at least one object in front of the vehicle.

The plurality of LDs 110 may include at least one first LD 112 performing the scanning in the first direction and at least one second LD 114 performing the scanning in the second direction. And in one embodiment, may further comprise scanning to perform scanning in a direction different from the first and second directions. The first and second ladders 112 and 114 may be disposed at different positions to collect plane data scanned in different directions with respect to the forward object 10.

In one embodiment, the plurality of columnar units 110 are mounted on the vehicle 20, some of which perform scanning in a direction perpendicular to the direction of travel of the vehicle, while others are in a direction parallel to the direction of travel of the vehicle Scanning can be performed. For example, the first line 112 may be disposed on the upper portion of the enclosure of the vehicle 20 in the front direction so as to perform scanning in a vertical direction perpendicular to the traveling direction of the vehicle 20, (112) may be disposed on the front surface of the enclosure of the vehicle (20) toward the front direction to perform scanning in a lateral direction horizontal to the traveling direction of the vehicle (20).

In the above embodiment, the plurality of ladder units 110 may constitute a plurality of the parts and constitute the other part in singular form. For example, the plurality of ladhardes 110 are coupled with the roof panel on top of the vehicle 20 and arranged side by side within a certain distance (e.g., 10 seconds) in equilibrium with the traveling direction of the vehicle 20 (112) coupled to the center of the front panel of the vehicle (20) and one second L (114) coupled to the center of the front panel of the vehicle (20).

Each of the plurality of ladder units 110 may be configured to be pre-calibrated 2D to share the same three-dimensional space coordinate system. More specifically, each of the plurality of ladder sections 110 may be calibrated in advance to a specific distance value to generate a combined 3D coordinate system based on the 2D plane data measured by the corresponding Later. In one embodiment, the values for calibration include a center point at which 2D plane data of the plurality of column units 110 can be combined, a respective location, a distance between the center point and the corresponding position, and a distance difference And can be set by a designer or a user.

The neural network learning unit 120 performs neural network learning on the forward object 10 based on the first and second plane data. In one embodiment, the neural network learning unit 120 may obtain the integrated 3D coordinate information based on the first and second plane data received from the plurality of column units 110, And neural network learning to extract feature data for creating or updating an object classification algorithm based on a neural network can be performed.

The neural network learning unit 120 may form the 3D point cloud by arranging the first and second plane data in the 3D coordinate system before learning the neural network. This will be described with reference to FIG. FIG. 2 is a view showing an embodiment of a process of forming a 3D point cloud by the neural network learning unit shown in FIG. 1;

2, the neural network learning unit 120 may integrate first and second plane data measured by a plurality of ladder units 110 into a 3D coordinate system according to a mounting point (refer to FIG. 2 a), and a 3D point cloud can be created based on the integrated 3D coordinate system (see FIG. 2 (b)). In one embodiment, the neural network learning unit 120 uses the first and second planar data, in which specific coordinate values are calibrated with respect to at least some of the xy, yz, and xz planes based on the mounting point in advance to share the same 3D coordinate system Can be integrated to form an integrated 3D coordinate system. In one embodiment, the neural network learning unit 120 matches the corresponding points between the first and second plane data to the integrated 3D coordinate system based on the pre-stored point cloud algorithm, as shown in FIG. 2 (b) To create a dense 3D point cloud.

The neural network learning unit 120 may perform neural network learning on the forward object 10 based on the generated 3D point cloud. This will be described with reference to FIG. FIG. 3 is a diagram illustrating an exemplary process of the neural network learning unit of FIG. 1 performing neural network learning by analyzing a 3D point cloud.

In FIG. 3A, the neural network learning unit 120 may decompose the 3D point cloud through the xy plane, the yz plane, and the xz plane to generate a plurality of 2D projection data. More specifically, the neural network learning unit 120 can project the generated 3D point cloud onto the two-dimensional coordinate system to generate corresponding 2D points of the xy plane data, the yz plane data, and the xz plane data, respectively, have.

3B, the neural network learning unit 120 can perform neural network learning on the forward object 10 based on each of the plurality of 2D projection data. More specifically, the neural network learning unit 120 may analyze each of the generated plurality of 2D projection data to extract the characteristic data from the corresponding 2D projection data, and based on the extracted characteristic data, Neural network learning algorithm based on neural network can be created or updated. In one embodiment, the neural network learning unit 120 may extract feature data including information about the shape of the corresponding plane based on the coordinate values of each of the plurality of 2D projection data.

3C shows an embodiment of a structural diagram of a neural network learning algorithm based on a neural network that is generated or updated by the neural network learning unit 120. [ The neural network learning algorithm shown in FIG. 3C may be composed of an input layer, at least one hidden layer, and an output layer, and in one embodiment, the hidden layer may be composed of one NN Neural Network based structure or DNN (Deep Neural Network) based structure composed of two or more hidden layers.

In FIG. 3D, the neural network learning unit 120 may perform neural network learning by applying feature data extracted from each of the plurality of 2D projection data to a pre-stored BackPropagation Algorithm. Here, the back propagation algorithm is a learning algorithm related to the neural network learning algorithm based on the neural network shown in FIG. 3B or 3C. The back propagation algorithm includes input data x for learning, output data y according to the input data, z. < / RTI >

More specifically, the neural network learning unit 120 can perform learning by calculating a weight through an error between a true value and an estimated value based on a back propagation algorithm. When input data x for learning is input, The output data y can be output by repeating the operations of multiplying and adding the weight of the neural network in the hidden layer, and the error (? = Zy) between the output data y and the desired output z is calculated, The weights can be sequentially updated in the contrary to the processing direction of the neural network. The neural network learning unit 120 updates the neural network weight of the neural network learning algorithm based on the neural network by reflecting characteristic data extracted from the plurality of 2D projection data as input to the back propagation algorithm, Can perform neural network learning.

In one embodiment, the neural network learning unit 120 can adjust the learning degree of the back propagation algorithm based on the learning rate (?), And determine the object detection performance of the neural network learning algorithm that is updated according to the adjusted learning degree Differences can occur. In one embodiment, the learning rate may be set and adjusted by the designer or user.

In one embodiment, the neural network learning unit 120 may determine the relevance between the target object and the dynamic object, and adjust the learning rate? Of the corresponding back propagation algorithm downward. Here, a dynamic object corresponds to an object to which a position of an object, including a vehicle, a pedestrian, and a bicycle occupant, can be mainly moved. For example, the neural network learning unit 120 may determine whether the forward object 10 corresponds to a dynamic object based on the input data x and the desired output z, and if so, The learning rate? Can be updated by reflecting the adjustment ratio r to the learning rate? ₀ . Here, the initially set learning rate eta ₀ and the adjustment ratio r can be set by the designer or the user, and the adjustment ratio r can be set to a value of 0.5 or more and less than 1.0.

[Equation 1]

η = r * η ₀

In one embodiment, the neural network learning unit 120 calculates the 2D projection data on the specific plane based on the traveling direction of the vehicle 20 in the process of performing the neural network learning based on the plurality of 2D projection data, . For example, the neural network learning unit 120 may calculate a reflection ratio of a neural network weight through learning in a learning process of 2D projection data of a specific plane, which is perpendicular to the traveling direction of the vehicle, among a plurality of 2D projection data, Can be set higher.

In one embodiment, the neural network learning unit 120 may perform Deep Neural Network (DNN) learning on the forward object 10. As described above, the neural network learning unit 120 can perform DNN learning through a neural network learning algorithm based on a DNN structure based on a neural network including two or more hidden layers, and a back propagation algorithm therefor.

The object classifying unit 140 may classify an unknown object, which is acquired through a specific line and is composed of a set of 3D points, as one of the forward objects learned as a neural network. In one embodiment, the object classifier 140 may generate a merged set of 3D points through a convolution of a plurality of 2D projection data when the neural network learning is performed, Point set can be constructed, and the set of 3D points can be applied to the neural network learning algorithm based on the neural network network updated through learning or analyzed through the probability calculation to finally perform object classification. For example, the object classification unit 140 determines whether the analysis result matches one of a plurality of object types stored through the learning of the neural network, and classifies the unknown object in the corresponding 3D point set as a specific object have.

In one embodiment, the object classifier 140 can classify objects based on a neural network learning algorithm based on a DNN structure learned through DNN learning, for example, as shown in FIG. 4 , The object type of an unknown object obtained through at least a part of the plurality of ladder units 110 with respect to the front object 10 and composed of a set of 3D points is selected as one of the object type candidates including the vehicle, the pedestrian, and the bicycle occupant You can decide.

In one embodiment, the neural network learning unit 120 may classify an unknown object into one of a dynamic object and a static object in the object classification process. Here, the dynamic object corresponds to an object in which the position of the object including the vehicle, the pedestrian, and the bicycle occupant can be mainly moved, and the static object corresponds to the position of the object such as a road or a graphical feature (for example, This corresponds mainly to objects that are not moved.

In one embodiment, the neural network learning unit 120 can easily detect an object that is difficult to detect through mathematical modeling using the neural network learning algorithm based on the neural network network learned through the neural network learning.

The control unit 130 can control the overall operation of the neural network learning based object detection apparatus 100 and can control the data flow between the plurality of ladder units 110, the neural network learning unit 120, and the object classifying unit 140 Can be controlled. The control unit 130 may be electrically connected to the plurality of ladder units 110, the neural network learning unit 120, and the object classifying unit 140 or may be connected through a network.

In one embodiment, the neural network learning unit 120, the control unit 130, and the object classifying unit 140 are integrally implemented in a computing device having a processor, a memory, a user input / output unit, and a network input / output unit, The control unit 130 may be implemented as a central processing unit (CPU) for controlling the control unit 130. The control unit 130 may be electrically or electronically connected to a plurality of ladder units 110 mounted at specific positions of the vehicle 20, And can be connected through a communication network. For example, the neural network learning-based object detection apparatus 100 can be implemented through a Linux PC (Personal Computer) installed in the vehicle 20, and can be implemented through a ROS (Robot Operation System) based platform installed in a Linux PC The neural network learning can be performed by receiving the measured plane data from the plurality of ladder units 110.

In another embodiment, the plurality of ladder units 110, the neural network learning unit 120, the controller 130, and the object classifying unit 140 may communicate with each other through CAN (Controller Area Network) communication.

FIG. 5 is a flowchart illustrating a process of the neural network learning based object detection apparatus of FIG. 1 performing neural network learning.

In FIG. 5, a plurality of rowizers 110 performs scanning in a first direction with respect to the front object 10 to collect first plane data, and the other part performs scanning in a second direction, The plane data is collected (step S510). The neural network learning unit 120 performs neural network learning on the forward object 10 based on the first and second plane data (step S520).

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as set forth in the following claims And changes may be made without departing from the spirit and scope of the invention.

100: object detection device based on neural network learning
10: front object 20: vehicle
110: plural ladder parts 120: neural network learning part
130: control unit 140: object classification unit

Claims (11)

A plurality of ladders for performing a scanning in a first direction with respect to a forward object to collect first plane data and a second to scan in a second direction to collect second plane data; And
And a neural network learning unit for performing neural network learning on the forward object based on the first and second plane data,
Wherein each of the plurality of ladder units is 2D which is previously calibrated to share the same 3D coordinate system.
2. The apparatus of claim 1, wherein the plurality of rows
Mounted on a vehicle
The part performs scanning in a direction perpendicular to the traveling direction of the vehicle,
And the other part performs scanning in a direction parallel to the traveling direction of the vehicle.
3. The apparatus of claim 2, wherein the plurality of rows
Wherein a plurality of said parts are constituted, and said other part is constituted by a singular number.
delete 2. The apparatus of claim 1, wherein the neural network learning unit
Wherein the 3D point cloud is formed by arranging the first and second plane data in a 3D coordinate system before learning the neural network.
6. The apparatus of claim 5, wherein the neural network learning unit
Wherein the 3D point cloud is decomposed through the xy plane, the yz plane, and the xz plane to generate a plurality of 2D projection data.
7. The apparatus of claim 6, wherein the neural network learning unit
And performs neural network learning on the forward object based on each of the plurality of 2D projection data.
8. The apparatus of claim 7, wherein the neural network learning unit
And performing DNN (Deep Neural Network) learning on the forward object.
The method according to claim 1,
Further comprising an object classifier for classifying an unknown object, which is acquired through a specific classifier and composed of a set of 3D points, as one of the forward objects learned through the neural network.
A method of detecting an object based on a neural network learning performed by an object detecting apparatus based on a neural network learning,
(a) a plurality of ladders, each consisting of 2D linearly calibrated to share the same 3D coordinate system, partly scanning in the first direction to collect the first plane data and Collecting second plane data by performing scanning in a second direction; And
(b) the neural network learning unit performs neural network learning on the forward object based on the first and second plane data.
A recording medium which can be executed by a computer recording the method of claim 10.

Images