KR102485099B1 - Method for data purification using meta data, and computer program recorded on record-medium for executing method therefor - Google Patents

Abstract

The present invention proposes a method for data purification using metadata, which is able to collect data for machine learning of an artificial intelligence (AI) and purify unnecessary data among the collected data. The method can comprise: a step in which a learning data collection device collects two-dimensional images for machine learning of AI; a step in which the learning data collection device extracts information on the collected two-dimensional images; and a step in which the learning data collection device determines the frames per second of the collected two-dimensional images based on the information on the collected two-dimensional images. As such, according to the present invention, unnecessary data can be purified among collected data and data with a low efficiency for learning can be reduced by determining the frames per second of collected two-dimensional images based on the information on the collected two-dimensional images.

Description

Method for data purification using meta data, and computer program recorded on record-medium for executing method therefor}

The present invention relates to the collection of data for artificial intelligence (AI) machine learning. More specifically, data purification method using metadata capable of collecting data for machine learning of artificial intelligence (AI) and purifying unnecessary data among the collected data, and a computer program recorded on a recording medium to execute the same will be.

Artificial intelligence (AI) refers to a technology that artificially implements some or all of human learning abilities, reasoning abilities, and perception abilities using computer programs. In relation to artificial intelligence (AI), machine learning refers to learning to optimize parameters with given data using a model composed of multiple parameters. Such machine learning is classified into supervised learning, unsupervised learning, and reinforcement learning according to the form of learning data.

In general, designing data for artificial intelligence (AI) machine learning proceeds in the steps of data structure design, data collection, data refinement, data processing, data expansion, and data verification.

To describe each step in more detail, data structure design is performed through ontology definition, classification system definition, and the like. Data collection is performed by collecting data through direct filming, web crawling, or associations/professional organizations. Data purification is performed by removing redundant data from collected data and de-identifying personal information. Data processing is performed by performing annotation and inputting metadata. Data extension is performed by performing ontology mapping and supplementing or extending the ontology as needed. In addition, data verification is performed by verifying validity according to the set target quality using various verification tools.

On the other hand, autonomous driving of a vehicle refers to a system that can judge and drive a vehicle by itself. Such autonomous driving may be classified into gradual stages from non-automation to complete automation according to the degree of involvement of the system in driving and the degree of control of the vehicle by the driver. In general, the level of autonomous driving is divided into six levels classified by the Society of Automotive Engineers (SAE) International. According to the six levels classified by the International Society of Automotive Engineers (SAE), level 0 is no automation, level 1 is driver assistance, level 2 is partial automation, and level 3 The stage is conditional automation, level 4 is high automation, and level 5 is full automation.

Autonomous driving of vehicles is performed through mechanisms of perception, localization, path planning, and control. Currently, several companies are developing to implement recognition and path planning among autonomous driving mechanisms using artificial intelligence (AI). In addition, the data used for machine learning of artificial intelligence (AI) that can be used for autonomous driving consists of a large number ranging from a few thousand to several million.

Data used for machine learning of artificial intelligence (AI) that can be used for autonomous driving of these vehicles is collected by various types of sensors installed in the vehicle. For example, the data used for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle is lidar, camera, radar, and ultrasonic sensor fixed to the vehicle. ) and GPS (Global Positioning System), etc., may be acquired, photographed, or sensed data, but are not limited thereto.

In particular, the 2D images acquired through the camera for artificial intelligence learning that can be used for autonomous driving of vehicles are composed of binary data, and the large capacity is very large, so there is a problem in transmitting the collected data. there was. In addition, since the 2D images obtained through the camera are obtained by real-time shooting, there is a problem in that they contain a large number of continuous 2D images without change, that is, data with low artificial intelligence learning efficiency.

Korean Patent Laid-open Publication No. 10-2020-0042629, ‘Method and apparatus for generating touch-based annotations and images in mobile devices for artificial intelligence learning’, (published on April 24, 2020)

One object of the present invention is to provide a data purification method using metadata that can collect data for machine learning of artificial intelligence (AI) and effectively refine unnecessary data among the collected data.

Another object of the present invention is to provide a computer program recorded on a recording medium for executing a data purification method using meta data capable of collecting machine learning data and effectively purifying unnecessary data among the collected data.

In order to achieve the technical problem as described above, the present invention comprises the steps of the learning data collection device, collecting 2D images for machine learning of artificial intelligence (Artificial Intelligence, AI); extracting, by the learning data collection device, information about the collected 2D images; and determining, by the learning data collection device, the number of frames per second of the collected 2D images based on the information on the collected 2D images.

Specifically, in the step of extracting information on the 2D images, a similarity between successive 2D images among the collected 2D images may be calculated, and the number of frames per second of the 2D images may be determined based on the similarity. .

In the step of extracting information on the 2D image, a similarity may be calculated by generating a red, green, blue (RGB) histogram for pixels in the continuous 2D image and comparing the generated RGB histograms.

The step of extracting information on the 2D images may include extracting an edge of each of the consecutive 2D images and calculating a similarity between the consecutive 2D images based on an edge change amount between the consecutive 2D images. there is.

In the step of extracting the information on the 2D image, a similarity may be calculated by extracting contours of objects included in the continuous 2D images and comparing moments of the extracted contours.

The step of extracting information on the 2D image calculates the degree of influence on machine learning through meta data collected together with the collected 2D image, and calculates the number of frames per second of the 2D images based on the degree of influence. can be determined.

In the step of extracting information about the 2D image, machine learning is performed through metadata including at least one of speed information, weather information, sensor operation information, and GPS coordinate information at the time when the learning information collection device collects the corresponding 2D image. The degree of influence on can be calculated.

In the step of extracting the information on the 2D image, when the moving speed of the learning data collection device is higher than a preset threshold value, a first directional degree may be calculated.

In the step of extracting the information on the 2D image, when the moving speed of the learning data collection device is lower than a preset threshold, a second influence lower than the first influence may be calculated.

In order to achieve the technical problem as described above, the present invention proposes a computer program recorded on a recording medium to execute a data purification method capable of collecting machine learning data and purifying unnecessary data among the collected data. The computer program may include a memory; transceiver; and a processor configured to process instructions resident in the memory, wherein the processor collects 2D images for machine learning of artificial intelligence (AI). step; extracting, by the processor, information about the collected 2D images; and determining, by the processor, the number of frames per second of the collected 2D images based on the information on the collected 2D images. It may be a computer program recorded on a recording medium. .

According to embodiments of the present invention, by determining the number of frames per second of the collected 2D images based on information on the collected 2D images, unnecessary data among the collected data is refined to increase learning efficiency can do.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a block diagram of an artificial intelligence learning system according to an embodiment of the present invention.
2 is a logical configuration diagram of a learning data collection device according to an embodiment of the present invention.
3 is a hardware configuration diagram of a learning data collection device according to an embodiment of the present invention.
4 is a logical configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.
5 is a hardware configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.
6 is a flowchart illustrating a guide providing method according to an embodiment of the present invention.
7 is a flowchart illustrating a data purification method according to an embodiment of the present invention.
8 is a flowchart illustrating a de-identification processing method according to an embodiment of the present invention.
9 and 10 are exemplary diagrams for explaining a guide providing method according to an embodiment of the present invention.
11 is an exemplary diagram for explaining a guide providing method according to another embodiment of the present invention.
12 is an exemplary diagram for explaining a data purification method according to an embodiment of the present invention.
13 to 16 are exemplary diagrams for explaining a de-identification processing method according to an embodiment of the present invention.
17 is a flowchart illustrating a data purification method according to an embodiment of the present invention.
18 is an exemplary diagram for explaining a process of refining data according to an embodiment of the present invention.

It should be noted that the technical terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. In addition, technical terms used in this specification should be interpreted in terms commonly understood by those of ordinary skill in the art to which the present invention belongs, unless specifically defined otherwise in this specification, and are overly inclusive. It should not be interpreted in a positive sense or in an excessively reduced sense. In addition, when the technical terms used in this specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be replaced with technical terms that those skilled in the art can correctly understand. In addition, general terms used in the present invention should be interpreted as defined in advance or according to context, and should not be interpreted in an excessively reduced sense.

Also, singular expressions used in this specification include plural expressions unless the context clearly indicates otherwise. In this application, terms such as "consisting of" or "having" should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or steps are included. It should be construed that it may not be, or may further include additional components or steps.

Also, terms including ordinal numbers such as first and second used in this specification may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. On the other hand, when a component is referred to as “directly connected” or “directly connected” to another component, it should be understood that no other component exists in the middle.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be construed as extending to all changes, equivalents or substitutes other than the accompanying drawings.

As described above, workers performing annotation work have to continuously perform repetitive work on similar images in the curation process of inputting meta data, which reduces work concentration. There was a problem. In addition, when there are many data of relatively low importance in the data used for machine learning of artificial intelligence (AI), there is a problem that the learning efficiency of machine learning is reduced. In addition, the de-identification processing method can prevent leakage of personal information, but there is a problem that the learning efficiency of machine learning is reduced because the object becomes unclear or is replaced with another image.

In order to overcome these limitations, the present invention provides guide information along with 2D images that can be helpful in the annotation of data for machine learning of artificial intelligence (AI), and collects 2D images in advance. We would like to suggest means of refining according to importance and de-identifying personal information in machine learning data.

1 is a block diagram of an artificial intelligence learning system according to an embodiment of the present invention.

As shown in FIG. 1, the artificial intelligence learning system according to an embodiment of the present invention includes a plurality of learning data collection devices (100a, 100b, ..., 100n; 100), a learning data generating device 200, a plurality of It may be configured to include annotation devices (300a, 300b, ..., 300n; 300) and an artificial intelligence learning device (400).

Since the components of the artificial intelligence learning system according to an embodiment are merely functionally distinct elements, two or more components are integrated and implemented in an actual physical environment, or one component is implemented in an actual physical environment. may be implemented separately from each other.

Describing each component, the learning data collection device 100 includes a lidar installed in a vehicle, a camera ( A device that collects data in real time from one or more of a camera, radar, ultrasonic sensor, rain sensor, position measurement sensor, and speed sensor.

Characteristically, the learning data collection apparatus 100 according to various embodiments of the present invention extracts information about the collected 2D images, and based on the information about the collected 2D images, frames per second of the collected 2D images The number of frames per second may be determined, and 2D images corresponding to the determined number of frames per second may be transmitted to the training data generating device 200 .

That is, the learning data collection apparatus 100 may determine the number of frames per second of 2D images to be transmitted to the learning data generation apparatus 200 based on the similarity between successive 2D images among the collected 2D images.

In addition, the learning data collection device 100 calculates the degree of influence on machine learning through metadata including at least one of speed information, weather information, sensor operation information, and GPS coordinate information at the time of collecting the 2D image, and calculates The number of frames per second of 2D images to be transmitted to the training data generating device 200 may be determined based on the determined influence.

Types of sensors that are controlled by the learning data collection device 100 and that are installed in the vehicle to obtain, photograph, or detect machine learning data include lidar, camera, radar, and ultrasonic sensors. One or more of an ultrasonic sensor, a rain sensor, a position measurement sensor, and a speed detection sensor may be included, but is not limited thereto. In addition, sensors that are controlled by the learning data collection device 100 and are installed in a vehicle to obtain, photograph, or detect machine learning data are not limited to being provided one by one for each type, and are provided in plural even if they are the same type of sensor. It can be.

With the following configuration, the learning data generating device 200 is a device that can be used to design and generate data for machine learning of artificial intelligence (AI) that can be used for autonomous driving of vehicles.

As such, the learning data generating device 200 is basically a device that is distinguished from the learning data verifying device 400, but in an actual physical environment, the learning data generating device 200 and the learning data verifying device 400 are integrated into one device. It may be integrated and implemented.

Characteristically, the learning data generation apparatus 200 according to embodiments of the present invention analyzes 2D images collected in advance for machine learning of artificial intelligence (AI) to extract singular points, The extracted singularity may be processed into guide information for the annotation device 300 to perform an annotation task, and may be provided together with 2D images.

In addition, the learning data generating device 200 according to embodiments of the present invention receives photographing information related to the 2D image together with the 2D image collected for machine learning of artificial intelligence (AI) from the learning data collection device, and captures the photographing information Singular points may be extracted by analyzing , and the extracted singular points may be processed into guide information for the annotation device 300 to perform annotation work, and may be provided to the annotation device 300 together with 2D images.

In addition, the learning data generation apparatus 200 according to embodiments of the present invention analyzes 2D images collected in advance for machine learning of artificial intelligence (AI) to evaluate the importance, and the 2D images collected according to the evaluated importance At least one of the 2D images may be refined.

In addition, the learning data generation apparatus 200 according to embodiments of the present invention identifies objects included in 2D images collected in advance for machine learning of artificial intelligence (AI), and identifies objects in correspondence to the types of the identified objects. Some of the identified objects may be de-identified.

The learning data generation device 200 having such characteristics transmits and receives data to and from the learning data collection device 100, the annotation device 300, and the artificial intelligence learning device 400, and performs calculations based on the transmitted and received data. Any device that can do this is acceptable. For example, the learning data generating device 200 may be any one of a fixed computing device such as a desktop, workstation, or server, but is not limited thereto.

With the following configuration, the annotation device 300 is a local computing device that can be used to perform annotation work on 2D images or 3D point cloud data distributed by the learning data generating device 200 . All or part of the annotation device 300 may be a device for performing annotation work by an annotation worker through a clouding service.

Specifically, the annotation device 300 may output one 2D image or 3D point cloud data to be annotated to a display from among 2D images or 3D point cloud data received from the learning data generating device 200 .

The annotation device 300 may select a tool according to a signal input from a user through an input/output device. Here, the tool is a tool for setting a bounding box specifying one or more objects included in 2D image or 3D point cloud data.

The annotation device 300 may receive coordinates according to the selected tool through an input/output device. In addition, the annotation device 300 may specify an object included in the 2D image or 3D point cloud data by setting a bounding box based on the input coordinates. Here, the bounding box is an area for specifying an object to be learned by artificial intelligence (AI) among objects included in the image. Such a bounding box may have a rectangle or cube shape, but is not limited thereto.

For example, the annotation device 300 receives two coordinates through an input/output device, and bounds the input two coordinates based on a rectangle having the coordinates of the upper left vertex and the coordinates of the lower right vertex in the 2D image. By setting a box, an object included in a 2D image can be specified. In this case, the two coordinates may be set by the user inputting one type of input signal twice (eg, mouse click) or by the user inputting two types of input signal once (eg, mouse drag). It may, but is not limited thereto.

The annotation device 300 may generate 2D image or 3D point cloud data to be annotated, or metadata for a set object according to a signal input from a user through an input/output device. Here, the metadata is 3D point cloud data or 2D image, and data for describing an object specified from the 3D point cloud data or 2D image. Such metadata includes the category of the object specified from 3D point cloud data or 2D image, the rate at which the object is cut by the angle of view, the rate at which the object is obscured by other objects or objects, the tracking ID of the object, the time the image was taken, the image It may include, but is not limited to, the weather conditions of the day on which the photo was taken, file size, image size, copyright holder, resolution, bit value, aperture transmission, exposure time, ISO sensitivity, focal length, aperture value, angle of view, white Balance, RGB depth, class name, tag, shooting location, road type, road surface information, or traffic jam information may be further included.

The annotation device 300 may generate an annotation work result based on an object set from 2D image or 3D point cloud data and generated metadata. In this case, the annotation work result may have a JSON (Java Script Object Notation) file format, but is not limited thereto. The annotation device 300 may transmit the generated annotation work result to the learning data generating device 200 . Also, the annotation device 300 may transmit 2D image or 3D point cloud data in which objects are set to the learning data generating device 200 for verification, in addition to the generated annotation work result.

Characteristically, the annotation device 300 according to an embodiment of the present invention, according to a signal input from a user through an input/output device, 2D image or 3D point cloud data to be annotated, or metadata for a set object ( metadata), guide information provided from the learning data generating device 200 may be output together.

Here, the guide information may be a section corresponding to a point in time when a photographing environment changes among 2D images, and may be displayed together with the 2D images so that a worker performing an annotation work can identify them.

The annotation device 300 according to an embodiment of the present invention does not transmit annotation work results and 2D image or 3D point cloud data in which objects are set to the learning data generating device 200, and input/output constituting the annotation device 300 Device control data may be transmitted to the learning data generating device 200 .

Here, the control data of the input/output device is data in which one or more signals input by the user to control the input/output device are time-sequentially stored in the process of the annotation device 300 performing an annotation operation on 2D image or 3D point cloud data. It can be. Here, the user may be referred to as a worker, performer, labeler, or data labeler, but is not limited thereto.

For example, when the annotation device 300 is driven by an operating system according to an event-driven architecture, one or more signals included in the control data of the input/output device may be used by the annotation device 200 It may be an event message generated by the operating system in response to the control of the input/output device. Also, the annotation device 300 may generate control data of the input/output device by duplicating a system queue in which event messages generated by the operating system are stored in a first-in-first-out structure.

As a more specific example, when the operating system of the annotation device 300 corresponds to Windows, the control data of the input/output device includes WM_LBUTONDOWN generated in response to a mouse left button click, WM_KEYDOWN generated in response to a keyboard input, and the like Event messages may be included.

The annotation device 300 having the above characteristics may be any device capable of transmitting/receiving data to/from the learning data generating device 200 and performing calculations based on the transmitted/received data. For example, the annotation device 300 may be a stationary computing device such as a desktop, workstation, or server, or a smart phone, laptop, tablet, phablet, or portable multimedia player. (Portable Multimedia Player, PMP), personal digital assistants (PDAs), or e-book readers (E-book reader).

With the following configuration, the artificial intelligence learning device 400 is a device that can be used for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

Specifically, the artificial intelligence learning device 400 may transmit requirements for achieving the purpose of artificial intelligence (AI) that can be used for autonomous driving of a vehicle to the learning data generating device 200 . The artificial intelligence learning device 400 may receive artificial intelligence (AI) learning data from the learning data generating device 200 . In addition, the artificial intelligence learning apparatus 400 may perform machine learning on artificial intelligence (AI) that can be used for autonomous driving of a vehicle using the received artificial intelligence (AI) learning data.

As such, the artificial intelligence learning device 400 may be any device capable of transmitting and receiving data to and from the learning data generating device 200 and performing calculations using the transmitted and received data. For example, the artificial intelligence learning device 400 may be any one of a fixed computing device such as a desktop, workstation, or server, but is not limited thereto.

As described above, the learning data collection device 100, the learning data generation device 200, a plurality of annotation devices 300, and the artificial intelligence learning device 400 are connected directly to each other through a secure line, common Data may be transmitted and received using a network in which one or more of a wired communication network or a mobile communication network is combined.

For example, public wired communication networks may include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). It may be, but is not limited thereto. In addition, in the mobile communication network, Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile telecommunication may be included, but is not limited thereto.

2 is a logical configuration diagram of a learning data collection device according to an embodiment of the present invention.

As shown in FIG. 2, the learning data collection device 100 according to an embodiment of the present invention includes a communication unit 105, an input/output unit 110, a similarity calculation unit 115, an influence calculation unit 120, It may include a frame determining unit 125, a data providing unit 130, and a storage unit 135.

Since the components of the learning data collection device 100 are only functionally distinct elements, two or more components are integrated and implemented in an actual physical environment, or one component is mutually exclusive in an actual physical environment. It could be implemented separately.

Describing each component, the communication unit 105 may transmit/receive data between multiple sensors installed in the vehicle and the learning data generating device 200 .

Specifically, the communication unit 105 detects from a lidar, a camera, a radar, an ultrasonic sensor, a rain sensor, a position measurement sensor, and a speed detection sensor installed in the vehicle. Data, 3D point cloud data, 2D image, distance information, weather information, location information, and speed information can be received.

In addition, the communication unit 105 may transmit sensing data, 3D point cloud data, 2D image, distance information, weather information, location information, and speed information to the learning data generating device 200 under the control of the data providing unit 130. .

Here, the communication unit 105 may transmit the 2D images to the training data generating device 200 according to the number of frames per second of the 2D images determined by the frame determining unit 125 .

With the following configuration, the input/output unit 110 may receive a signal from a user through a user interface (UI) or output an operation result to the outside.

Specifically, the input/output unit 110 determines the basic size of a buffer for storing sensing data, 3D point cloud data, 2D image, distance information, weather information, location information, and speed information, and the priority of data to be preferentially stored in the buffer. The ranking can be input from the user.

In addition, the input/output unit 110 may receive a threshold range, which is a size range for determining the number of frames per second of 2D images to be transmitted to the training data generating apparatus 200 among 2D images, from the user. That is, the input/output unit 110 determines the number of frames per second of 2D images to be transmitted to the learning data generating device 200 that matches the degree of similarity or influence calculated by the similarity calculator 115 or the influence calculator 120 from the user. can be input.

With the following configuration, the similarity calculation unit 115 may calculate a similarity between successive 2D images among collected 2D images. In particular, the similarity calculating unit 115 may generate a red, green, blue (RGB) histogram for pixels in consecutive 2D images, and compare the generated RGB histograms to calculate a similarity. Here, the RGB histogram is a graph representing the brightness distribution of each primary color (RGB) in an image. For example, in an RGB histogram, the horizontal axis indicates the brightness level of a color, and the vertical axis indicates the number of pixels allocated to the brightness level of a color. The more pixels there are, the brighter and darker the color can be expressed. As such, the similarity calculation unit 115 may calculate the similarity by comparing color saturation and gradation state, white balance tendency, etc. of consecutive 2D images through the RGB histogram.

Also, the similarity calculating unit 115 may extract an edge of each of the consecutive 2D images and calculate a similarity between the consecutive 2D images based on an edge change amount between the consecutive 2D images. Here, the similarity calculating unit 115 may extract an edge of the identified object area or an edge of an object included in the entire 2D image. At this time, the similarity calculation unit 115 may calculate the similarity by comparing moments of the extracted edges.

The influence calculation unit 120 may calculate the influence on machine learning through meta data collected together with the collected 2D images.

Specifically, the influence calculation unit 120 calculates the influence on machine learning through metadata including at least one of speed information, weather information, sensor operation information, and GPS coordinate information at the time of collecting the 2D image. can

That is, the influence calculation unit 120 determines the first influence when the moving speed of the vehicle collecting learning data is higher than a preset threshold, and the moving speed of the learning data collection device is lower than the preset threshold. In this case, a second influence degree lower than the first influence degree may be determined.

For example, when the speed of the vehicle for which learning data is collected through speed information is high, the influence calculation unit 120 calculates the influence low because there is a high possibility that the amount of change between consecutive 2D images is small, and the moving speed When is slow, the variation between 2D images is likely to be large, so the degree of influence can be calculated high.

The frame determining unit 125 determines the number of frames per second of 2D images to be transmitted to the learning data generating device 200 based on the similarity or influence calculated by the similarity calculating unit 115 and the influence calculating unit 120 described above. (frame per second) can be determined.

Here, the frame determination unit 125 may determine the number of frames per second for each section of the 2D images. That is, the frame determiner 125 may group 2D images according to similarity between consecutive 2D images, and apply a preset number of frames per second to the similarity of each group.

In addition, the frame determiner 125 may group 2D images according to the degree of influence between consecutive 2D images, and apply a preset number of frames per second to the degree of influence of each group.

Also, the frame determining unit 125 may determine the number of frames per second for all 2D images. That is, among the 2D images collected in the training data generation device 200, an average similarity value of consecutive 2D images may be calculated, and a preset number of frames per second matching the calculated average similarity value may be applied.

In addition, the frame determining unit 125 calculates an average influence value of consecutive 2D images among the 2D images collected in the learning data generating device 200, and calculates a preset number of frames per second matching the calculated average influence value. can be applied.

The data providing unit 130 may provide the 2D images corresponding to the number of frames per second determined by the frame determining unit 125 to the training data generating device 200 through the communication unit 105 . That is, the data providing unit 130 may apply the number of frames per second determined by the frame determining unit 125 to the collected 2D images and transmit the applied 2D images to the learning data generating device 200 .

The storage unit 135 may store data necessary for the operation of the learning data collection device 100 .

In detail, the storage unit 135 may include a buffer for storing sensing data, 3D point cloud data, 2D image, distance information, weather information, location information, and speed information. Also, the storage unit 135 may include a database for storing rules and basic data for calculating similarity or influence and determining the number of frames per second.

Hereinafter, hardware for implementing the above-described logical components of the learning data collection device 100 will be described in more detail.

3 is a hardware configuration diagram of a learning data collection device according to an embodiment of the present invention.

As shown in FIG. 3, the learning data collection device 100 includes a processor 150, a memory 155, a transceiver 160, an input/output device 165, and a data bus. (Bus, 170) and storage (Storage, 175) can be configured.

The processor 150 may implement the operation and function of the learning data collection device 100 based on instructions according to the software 180a in which the method according to the embodiments of the present invention is resident in the memory 155. . Software 180a in which a method according to embodiments of the present invention is implemented may be loaded in the memory 155 .

The transceiver 160 includes a lidar, a camera, a radar, an ultrasonic sensor, a rain sensor, a position measurement sensor, a speed detection sensor, and a learning data generating device 200 and can send and receive data. The input/output device 165 may receive data necessary for the operation of the learning data collection device 100 and output sensing data, 3D point cloud data, 2D image, distance information, weather information, location information, and speed information. The data bus 170 is connected to the processor 150, the memory 155, the transceiver 160, the input/output device 165, and the storage 175, and is a movement path for transferring data between each component. role can be fulfilled.

The storage 175 stores an application programming interface (API), a library file, a resource file, etc. necessary for the execution of the software 180a in which the method according to the embodiments of the present invention is implemented. can be saved The storage 175 may store the software 180b and the database 185 in which a method according to embodiments of the present invention is implemented.

The database 185 may store rules and basic data for calculating the degree of similarity or influence and determining the number of frames per second of 2D images according to the degree of similarity or influence.

According to one embodiment of the present invention, the software 180a, 180b for implementing a method of controlling sensors resident in the memory 155 or stored in the storage 175 is provided by the processor 150 using artificial intelligence (AI). The step of collecting 2D images for machine learning of ), the step of extracting information on the collected 2D images, and the number of frames per second (frame per second) of the 2D images collected based on the information on the collected 2D images. second) may be a computer program recorded on a recording medium to execute the determining step.

More specifically, the processor 150 may include an Application-Specific Integrated Circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 155 may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. The transceiver 160 may include a baseband circuit for processing wired/wireless signals. The input/output device 165 includes an input device such as a keyboard, a mouse, and/or a joystick, and a Liquid Crystal Display (LCD), an Organic LED (OLED), and/or a liquid crystal display (LCD). Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiments included in this specification are implemented as software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described functions. A module may reside in memory 155 and be executed by processor 150 . The memory 155 may be internal or external to the processor 150 and may be connected to the processor 150 by various well-known means.

Each component shown in FIG. 3 may be implemented by various means, eg, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one embodiment of the present invention includes one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), FPGAs ( Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored on a recording medium readable through various computer means. can be recorded. Here, the recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in computer software. For example, recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs (Compact Disk Read Only Memory) and DVDs (Digital Video Disks), floptical It includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, such as a floptical disk, and ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. These hardware devices may be configured to operate as one or more pieces of software to perform the operations of the present invention, and vice versa.

Hereinafter, the configuration of the learning data generating device 200 as described above will be described in more detail.

4 is a logical configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.

As shown in FIG. 4, the learning data generating device 200 includes a communication unit 205, an input/output unit 210, a singularity extraction unit 215, a guide information processing unit 220, an importance evaluation unit 225, an image It may include a refining unit 230, an object identification unit 235, a de-identification processing unit 240, and a storage unit 245.

Since the components of the learning data generation device 200 are merely functionally distinct elements, two or more components are integrated and implemented in an actual physical environment, or one component is mutually exclusive in an actual physical environment. It could be implemented separately.

Describing each component, the communication unit 205 may transmit/receive data with one or more of the learning data collection device 100, the annotation device 300, and the artificial intelligence learning device 400.

Specifically, the communication unit 205 may receive 2D images and 3D point cloud data from the learning data collection device 100 . Here, the 2D images may be images captured through a camera fixed to a vehicle in order to machine learn artificial intelligence (AI) that can be used for autonomous driving of a vehicle. In addition, the 3D point cloud data may be point cloud data obtained through a LIDAR fixed to a vehicle in order to machine learn artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

The communication unit 205 may distribute and transmit a plurality of 2D images or 3D point cloud data, which are objects of annotation work, to the plurality of annotation devices 300 . The communication unit 205 may receive annotation work results from each of the plurality of annotation devices 300 . Also, the communication unit 205 may receive control data of the input/output device from each of the plurality of annotation devices 300 . Here, the control data of the input/output device is the annotation device 300 receiving one or more of 2D images or 3D point cloud data. In the process of performing an annotation operation on the received 2D image or 3D point cloud data, the user It may be data that stores one or more signals inputted in order to control the input/output device constituting 300 in a time-sequential manner.

And, the communication unit 205 may transmit artificial intelligence (AI) learning data to the artificial intelligence learning device 300 .

With the following configuration, the input/output unit 210 may receive a signal from a user through a user interface (UI) or output an operation result to the outside.

Specifically, the input/output unit 210 may receive a control signal for designing a data structure for artificial intelligence (AI) learning from a user. The input/output unit 210 may receive an input of an allocation amount for distributing annotation work to a plurality of annotation devices 300 from a user.

With the following configuration, the singularity extractor 215 analyzes continuous 2D images among the collected 2D images and extracts a point in time when the photographing environment changes as a singularity. Specifically, the singularity extractor 215 may evaluate the similarity between consecutive 2D images among the collected 2D images, and determine a continuous 2D image having a similarity higher than a preset threshold as a point in time when the shooting environment changes. there is. At this time, the singularity extractor 215 may generate a red, green, blue (RGB) histogram for pixels in the continuous 2D image, and compare the generated RGB histograms to calculate a degree of similarity. Here, the RGB histogram is a graph representing the brightness distribution of each primary color (RGB) in an image. For example, in an RGB histogram, the horizontal axis indicates the brightness level of a color, and the vertical axis indicates the number of pixels allocated to the brightness level of a color. The more pixels there are, the brighter and darker the color can be expressed. As such, the singularity extractor 215 may calculate a similarity by comparing color saturation and gradation state, white balance tendency, and the like of consecutive 2D images through the RGB histogram.

In addition, the singularity extractor 215 may divide each of the collected 2D images into a plurality of regions, analyze each of the divided regions to identify environmental variables, and extract singularities based on changes in the environmental variables. In this case, the singularity extractor 215 may calculate environmental variables based on brightness or red, green, blue (RGB) values of the divided areas. For example, the singularity extractor 215 divides each of the collected 2D images into two regions in the vertical direction, and if the variation of the environmental variable in the upper region of the consecutive 2D images is higher than a preset threshold value, weather is detected. It is recognized as a point of time of change, and when the amount of change in the lower area environmental variable is higher than a preset threshold, it may be recognized as a point of time of change of the road.

In addition, the singularity extractor 215 may analyze the photographing information and extract a point in time when the photographing environment changes as a singularity. Here, the photographing information may be sensing values of sensors installed in the learning data collection device 100 . Specifically, the singularity extractor 215 determines whether the lighting device installed in the learning data collection device 100 is turned on or off through the illuminance sensor installed in the learning data collection device 100, and whether the lighting device is turned on or off. Through this, it is possible to determine the time when the 2D image was taken or the time of changing the location. For example, the singularity extractor 215 may recognize a time when a lighting device is turned on or off as a time when day and night change or as a time when entering a tunnel. In addition, the singularity extractor 215 may determine a time point at which weather conditions of the 2D image change through a rain sensor installed in the learning data collection device 100 . In addition, the singularity extractor 215 may determine a time point at which weather conditions of the 2D image change through a focus change amount of a camera installed in the learning data collection device 100 .

With the following configuration, the guide information processing unit 220 designates a section corresponding to a point in time when the shooting environment changes among the collected 2D images, and processes the designated section into guide information to form the annotation device 300 together with the 2D images can be provided to In addition, the guide information processing unit 220 designates a section in which the environment variable of each of the divided regions of the collected 2D images is higher than a preset threshold value, and processes the designated section as guide information to create an annotation device with the 2D images ( 300) can be provided. In addition, the guide information processing unit 220 may designate a section in which the shooting environment is changed, process the designated section into guide information, and provide the 2D images to the annotation device 300 . For example, when the guide information processing unit 220 outputs one 2D image or 3D point cloud data to be annotated on a display, the 2D image corresponding to the section where the viewpoint or shooting environment changes is identified as a 2D image whose viewpoint changes. It is possible to output possible indications together.

With the following configuration, the importance evaluation unit 225 extracts an object included in each of the collected 2D images, calculates a similarity between the extracted object and an object corresponding to a pre-set type require, and importance can be assessed. That is, the importance of each 2D image in machine learning may be determined according to whether or not it actually includes an object related to machine learning. Accordingly, the importance evaluation unit 225 may evaluate the importance by extracting objects included in the collected 2D images and comparing the extracted objects with objects actually required for machine learning.

In addition, the importance evaluator 225 may evaluate the importance based on each 2D image collected together with the 2D images and environmental factors at the time of photographing. That is, the importance evaluator 225 may evaluate a 2D image that is different from weather information, which is a target of machine learning, among 2D images as an image of low importance to be refined. In addition, the importance evaluator 225 may evaluate a 2D image that is different from a shooting point, which is a target of machine learning, among 2D images as an image of low importance to be refined. In addition, the importance evaluator 225 may evaluate a 2D image having GPS coordinates located within a preset threshold distance from Global Positioning System (GPS) coordinates, which are subject to machine learning, as images having low importance to be refined. In addition, the importance evaluation unit 225 extracts objects included in each of the collected 2D images, and converts 2D images in which the number of extracted objects is lower than a preset threshold number into images of low importance to be refined. can be evaluated In addition, the importance evaluator 225 selects a 2D image containing less than a preset required number of objects corresponding to a preset type of 2D images among 2D images of low importance to be refined. images can be evaluated.

The image refining unit 230 may refine at least one 2D image among the collected 2D images according to the level of importance evaluated by the importance evaluation unit 225 . Specifically, the image refiner 230 may refine 2D images in which the similarity between objects included in the collected 2D images and objects corresponding to the set request type is lower than a preset threshold. In addition, the image refiner 230 may refine 2D images whose importance is lower than a pre-set threshold based on environmental factors at the time the 2D image was captured, that is, weather information, shooting time, GPS coordinates, and the number of objects. .

With the following configuration, the object identification unit 235 may identify an object from sensing data collected by the learning data collection device 100, 3D point cloud data, 2D image, and distance information.

Basically, the object identification unit 235 according to an embodiment of the present invention sets an object region in which an object is predicted to exist in a 2D image based on the 3D coordinates of points included in 3D point cloud data. can In this case, the object region may be a two-dimensional region composed of vertices and edges connecting the vertices to each other.

The number of vertices constituting the object region may be configured in advance by the learning data generating apparatus 200 in correspondence with computing power.

However, the object identification unit 235 may adjust the number of vertices constituting the object area when the importance of the 2D image captured by the camera changes.

For example, the object identification unit 235 may increase the number of vertices constituting the object area in proportion to the size of a period during which a camera installed in a vehicle captures a 2D image.

As another example, the object identification unit 235 may increase the number of vertices constituting the object area in proportion to the moving speed of the vehicle in which the camera is installed.

Meanwhile, in order to identify an object in the 2D image from the 3D point cloud data, the object identification unit 235 may identify points forming clusters within a preset threshold range among points included in the 3D point cloud data.

The object identification unit 235 may identify the object type based on the X-axis width, Y-axis height, and Z-axis depth of the identified cluster. More specifically, the object identification unit 235 may determine the width on the X-axis and the height on the Y-axis of the identified cluster based on a rate relation of width, height, and depth prepared in advance for each type of object in the database. And the type of object corresponding to the depth on the Z axis can be identified.

The object identification unit 235 may three-dimensionally rotate a 3D model previously provided in the database according to the type of identified object according to the direction of an optical axis of a camera that has taken a corresponding 2D image. Also, the object identification unit 235 may identify a 2D shape of the 3D rotated 3D model viewed from the optical axis direction of the camera as an object area.

Also, the object identification unit 235 may set an object region in which an object is predicted to exist in the 2D image by configuring vertices and trunk lines by reflecting the 2D shape identified as the object region on the 2D image.

On the other hand, the object identification unit 235 extracts an edge for the identified object area, and determines whether there is an object detection rule corresponding to the type of object and the edge pattern extracted from the database. Based on this, the identified object area can be verified. In this case, the object detection rules are distributed by the training data generating device 200 and are rules enumerating edge patterns classified by object type so as to verify objects identified in the 2D image.

In addition, when no object is identified in the 2D image, the object identification unit 235 may remove 3D point cloud data acquired at the same time as when the 2D image at which no object is identified is captured from the storage unit 245. .

When the object identification unit 235 sets the object region using the 3D model, the de-identification processing unit 240 may perform de-identification processing on a region corresponding to the de-identification processing region previously assigned to the 3D model. .

Specifically, the de-identification processing unit 240 performs de-identification processing by blurring a part of the identified object, extracts a landmark from the identified object, and blurs the extracted landmark. Ring processing can be performed. For example, when the identified object is a person, the de-identification processing unit 240 extracts eyes, nose, and mouth corresponding to the person's landmarks, and selectively blurs only the extracted eyes, nose, and mouth, thereby processing the entire face. Compared to the blurring process, learning efficiency can be increased.

In addition, the de-identification processing unit 240 performs de-identification processing by blurring a part of the identified object, extracts an edge of the identified object, and performs blurring processing based on the extracted edge. can be done On the other hand, if the entire identified object is blurred, the edge of the object becomes unclear, which affects annotation work. Accordingly, the de-identification processing unit 240 may extract an edge of the object and perform a blurring process so as to be spaced apart from the extracted edge by the number of preset threshold pixels. For example, when the identified object is a person, the de-identification processing unit 240 extracts an edge corresponding to the shape of a person's face and blurs the edge so that it is separated from the edge by the number of preset threshold pixels, De-identification processing can be performed on a line that does not interfere with the shape of a person.

In addition, the de-identification processing unit 240 may de-identify a part of the identified object, extract an edge of the identified object, and change the pattern of the extracted edge. Specifically, the de-identification processing unit 240 may change the pattern of the extracted edge by considering at least one of a body shape and a face shape of a region to be machine learning for the extracted edge. For example, when the target region of machine learning is Korea, the de-identification processing unit 240 may change and apply the extracted edge pattern of the identified object to the average body shape and average face shape of Korea.

In addition, the de-identification processing unit 240 deep-fakes a part of the identified object to perform de-identification processing, extracts a landmark from the identified object, and marks the extracted landmark. can be replaced with another image appropriate for that landmark type. Specifically, the de-identification processing unit 240 may replace the extracted landmark with a landmark image having a similarity higher than a pre-set threshold among pre-stored landmark images according to the landmark type. For example, when the identified object is a face, the de-identification processing unit 240 extracts eyes, nose, and mouth, and compares the extracted eyes, nose, and mouth with pre-stored eye, nose, and mouth images that have a high degree of similarity. Eye, nose, and mouth images can be replaced with extracted eye, nose, and mouth images.

In addition, the de-identification processing unit 240 deep-fakes a part of the identified object to perform the first de-identification process, and then blurs a part of the first de-identification processed object. processing to perform secondary de-identification processing.

Hereinafter, hardware for implementing the above-described logical components of the learning data generating device 200 will be described in more detail.

5 is a logical configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.

As shown in FIG. 5, the learning data generating device 200 includes a processor 250, a memory 255, a transceiver 260, an input/output device 265, and a data bus. (Bus, 270) and storage (Storage, 275) can be configured.

The processor 250 may implement operations and functions of the learning data generating device 200 based on instructions according to the software 280a in which the method according to the embodiments of the present invention is resident in the memory 255. . Software 280a in which a method according to embodiments of the present invention is implemented may be loaded in the memory 255 . The transceiver 260 may transmit and receive data to and from the learning data collection device 100 , the annotation device 300 , and the artificial intelligence learning device 400 . The input/output device 265 may receive data necessary for the operation of the learning data generating device 200 and output collected and preprocessed 2D images, 3D point cloud data, and annotation work results. The data bus 270 is connected to the processor 250, the memory 255, the transceiver 260, the input/output device 265, and the storage 275, and is a movement path for transferring data between each component. role can be fulfilled.

The storage 275 stores an application programming interface (API), a library file, a resource file, etc. necessary for the execution of the software 280a in which the method according to the embodiments of the present invention is implemented. can be saved The storage 275 may store software 280b in which a method according to embodiments of the present invention is implemented. Also, the storage 275 may store information necessary for performing a method according to embodiments of the present invention.

According to one embodiment of the present invention, the software 280a, 280b for implementing the method of providing a guide, resident in the memory 255 or stored in the storage 275, the processor 250 artificial intelligence (Artificial Intelligence, AI) ) extracting singular points by analyzing 2D images collected in advance for machine learning, and processing the extracted singular points into guide information for the annotation device to perform annotation work, and the 2D It may be a computer program recorded on a recording medium to execute steps provided with images.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the guide providing method, which is resident in the memory 255 or stored in the storage 275, the processor 250 uses artificial intelligence from the learning data collection device. Receiving photographing information related to the collected 2D images together with collected 2D images for machine learning of (Artificial Intelligence, AI), extracting singular points by analyzing the photographing information, and extracting the extracted singularities It may be a computer program recorded on a recording medium in order to execute a step in which a singular point is processed by an annotation device into guide information for performing annotation work and provided together with the 2D images.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the data cleaning method, resident in the memory 255 or stored in the storage 275, the processor 250 Artificial Intelligence (AI) To execute the step of analyzing 2D images collected in advance for machine learning of ) to evaluate the importance and the step of refining at least one 2D image among the collected 2D images according to the evaluated importance It may be a computer program recorded on a recording medium.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the non-identification processing method, which is resident in the memory 255 or stored in the storage 275, is the processor 250 artificial intelligence (Artificial Intelligence). , AI) identifying an object included in a pre-collected 2D image for machine learning and de-identifying some of the identified object in correspondence to the type of the identified object It may be a computer program recorded on a recording medium to execute the processing steps.

More specifically, the processor 250 may include an Application-Specific Integrated Circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 255 may include read-only memory (ROM), random access memory (RAM), flash memory, a memory card, a storage medium, and/or other storage devices. The transceiver 260 may include a baseband circuit for processing wired/wireless signals. The input/output device 265 includes an input device such as a keyboard, a mouse, and/or a joystick, and a Liquid Crystal Display (LCD), an Organic LED (OLED), and/or a liquid crystal display (LCD). Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiments included in this specification are implemented as software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described functions. A module may reside in memory 255 and be executed by processor 250 . The memory 255 may be internal or external to the processor 250 and may be connected to the processor 250 by various well-known means.

Each component shown in FIG. 5 may be implemented by various means, eg, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one embodiment of the present invention includes one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), FPGAs ( Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored on a recording medium readable through various computer means. can be recorded. Here, the recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in computer software. For example, recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs (Compact Disk Read Only Memory) and DVDs (Digital Video Disks), floptical It includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, such as a floptical disk, and ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. These hardware devices may be configured to operate as one or more pieces of software to perform the operations of the present invention, and vice versa.

6 is a flowchart illustrating a guide providing method according to an embodiment of the present invention.

Referring to FIG. 6 , the learning data generation apparatus 200 according to an embodiment of the present invention analyzes continuous 2D images among collected 2D images and extracts a point in time when the shooting environment changes as a singular point ( S110). Specifically, the learning data generation device 200 evaluates the similarity between consecutive 2D images among the collected 2D images, and judges the consecutive 2D images having a similarity higher than a preset threshold as the point in time when the shooting environment changes. can In addition, the learning data generating apparatus 200 may divide each of the collected 2D images into a plurality of regions, analyze each of the divided regions to identify environmental variables, and extract a singular point based on the amount of change in the environmental variables. In addition, the learning data generating device 200 may analyze the photographing information and extract a point in time when the photographing environment changes as a singular point. Here, the photographing information may be sensing values of sensors installed in the learning data collection device 100 . Specifically, the learning data generating device 200 determines whether the lighting device installed in the learning data collecting device 100 is turned on or off through an illuminance sensor installed in the learning data collecting device 100, and turns on or off the lighting device. It is possible to determine the time of capturing the 2D image or the time of changing the location through presence or absence.

Next, the learning data generating device 200 designates a section corresponding to a point in time when the shooting environment changes among the collected 2D images, processes the designated section into guide information, and sends the selected section to the annotation device 300 along with the 2D images. It can be provided (S120). In addition, the learning data generating apparatus 200 may designate a section in which an environmental variable of each divided region of the collected 2D images is higher than a preset threshold value, and process the designated section into guide information. In addition, the learning data generating device 200 may designate a section in which the photographing environment changes and process the designated section into guide information.

Next, the learning data generating device 200 may provide the guide information processed in step S120 to the annotation device 300 together with the 2D image (S130).

7 is a flowchart illustrating a data purification method according to an embodiment of the present invention.

Referring to FIG. 7 , the learning data generating apparatus 200 extracts objects included in each of the collected 2D images, and determines the similarity between the extracted objects and objects corresponding to a pre-set type require. It is possible to evaluate the importance by calculating (S210). In addition, the learning data generation apparatus 200 may evaluate the importance based on each 2D image collected together with the 2D images and environmental factors at the time when they were captured. In addition, the learning data generation apparatus 200 may evaluate a 2D image different from a shooting point, which is an object of machine learning, among 2D images as an image of low importance to be refined. In addition, the learning data generating apparatus 200 may evaluate a 2D image having GPS coordinates located within a predetermined threshold distance from Global Positioning System (GPS) coordinates, which are subject to machine learning, as images of low importance to be refined. . In addition, the learning data generation apparatus 200 extracts objects included in each of the collected 2D images, and converts 2D images in which the number of extracted objects is lower than a preset threshold number into images of low importance to be refined. can be evaluated as In addition, the learning data generating apparatus 200 determines that the 2D image containing objects corresponding to a pre-set type require less than a pre-set required number among 2D images has a level of importance to be refined. It can be evaluated as a low image.

Next, the learning data generation apparatus 200 may refine at least one 2D image among the collected 2D images according to the importance evaluated in step S210 (S220). In detail, the learning data generating device 200 may refine 2D images in which the similarity between objects included in the collected 2D images and objects corresponding to the set request type is lower than a preset threshold. In addition, the learning data generating device 200 can refine 2D images whose importance is lower than a preset threshold based on environmental factors at the time the 2D image was captured, that is, weather information, shooting time, GPS coordinates, and the number of objects. there is.

Next, the training data generation device 200 may provide the 2D image refined in step S220 to the annotation device 300 (S230).

8 is a flowchart illustrating a de-identification processing method according to an embodiment of the present invention.

Referring to FIG. 8 , the learning data generating device 200 may identify an object from sensing data collected by the learning data collecting device 100, 3D point cloud data, 2D image, and distance information (S310). The learning data generation apparatus 200 may identify the type of object based on the width on the X axis, the height on the Y axis, and the depth on the Z axis of the identified cluster. In more detail, the learning data generating apparatus 200 uses the width on the X axis and the width on the Y axis of the identified cluster on the basis of a rate relation of the width, height, and depth previously provided for each type of object in the database. It is possible to identify the type of object corresponding to the height and the depth on the Z axis.

Next, when the learning data generating device 200 sets the object region using the 3D model in step S310, it may perform de-identification processing on a region corresponding to the de-identification processing region previously assigned to the 3D model. . Specifically, the learning data generating device 200 performs de-identification processing by blurring a part of the identified object, extracts a landmark from the identified object, and assigns the extracted landmark to the extracted landmark. Blurring processing can be performed. The learning data generation device 200 performs de-identification processing by blurring a part of the identified object, extracts an edge of the identified object, and performs blurring processing based on the extracted edge. can be done In addition, the learning data generation device 200 may de-identify a part of the identified object, extract an edge of the identified object, and change the pattern of the extracted edge. In detail, the learning data generating apparatus 200 may change the pattern of the extracted edge by considering at least one of a body shape and a face shape of an area to be machine learning for the extracted edge. In addition, the learning data generating device 200 performs de-identification processing by deep-fake processing a part of the identified object, extracts a landmark from the identified object, and marks the extracted landmark can be replaced with another image appropriate for that landmark type. Specifically, the learning data generating device 200 may replace the extracted landmark with a landmark image having a similarity higher than a preset threshold value among previously stored landmark images according to the corresponding landmark type.

Next, the training data generation device 200 may provide the 2D image de-identified in step S320 to the annotation device 300 (S330).

9 and 10 are exemplary diagrams for explaining a guide providing method according to an embodiment of the present invention.

9 and 10, the learning data generating device 200 according to an embodiment of the present invention analyzes collected 2D images to extract singular points, and the annotation device 300 annotates the extracted singular points. It is possible to improve the convenience and concentration of workers who perform annotation work by processing it into guide information and providing it together with 2D images.

For example, as shown in FIG. 9 , the learning data generation apparatus 200 may divide each of the consecutive 2D images (a) and (b) into two regions in the vertical direction. That is, the learning data generating device 200 may divide the upper regions A1 and B1 into regions capable of recognizing weather change timings and the lower regions A2 and B2 into regions capable of recognizing road change timings. .

After that, as shown in (a) of FIG. 10 , in order to recognize the upper area, that is, the time of weather change, the learning data generating device 200 may generate RGB histograms of the upper area for two consecutive images and compare the RGB histograms. there is.

And, as shown in (b) of FIG. 10, the learning data generating device 200 recognizes the point T at which the change in the RGB histogram is higher than a preset threshold as the point at which the weather changes, and the point at which the weather changes is a singular point. In this way, the annotation device may process guide information for performing annotation work and provide the information to the annotation device 300 together with 2D images.

11 is an exemplary diagram for explaining a guide providing method according to another embodiment of the present invention.

Referring to FIG. 11 , the learning data generation device 200 according to an embodiment of the present invention analyzes the captured information received together with the collected 2D images to extract singular points, and the extracted singular points are annotated by the device 300. By processing guide information for performing annotation work and providing it together with 2D images, it is possible to improve the convenience and concentration of workers performing annotation work.

Specifically, the learning data generating device 200 determines whether the lighting device installed in the learning data collecting device 100 is turned on or off through an illuminance sensor installed in the learning data collecting device 100, and turns on or off the lighting device. It is possible to determine the point of time when the 2D image was captured through the presence or absence.

For example, as shown in (a) of FIG. 11, when the time point at which the learning data collection device 100 is photographing is low, the lighting device is turned off, and the learning data collection device as shown in (b) When the time point 100 is photographing is night, it can be confirmed that the lighting device is turned on.

In this way, the learning data generating device 200 can check the shooting time of the corresponding 2D image through whether or not the lighting device installed in the learning data collecting device 100 that collects the learning data is turned on or off, and the shooting time is set as a singularity. An annotation device may process guide information for performing an annotation task and provide the information to the annotation device 300 together with 2D images.

12 is an exemplary diagram for explaining a data purification method according to an embodiment of the present invention.

Referring to FIG. 12 , the learning data generation apparatus 200 according to an embodiment of the present invention analyzes collected 2D images to evaluate importance, and at least one 2D image among the collected 2D images according to the evaluated importance. By refining , the learning efficiency of machine learning can be improved by refining 2D images of relatively low importance.

For example, as shown in (a), when the object to be machine learning is a car (object1), the learning data generating apparatus 200 extracts an object included in each of the collected 2D images, The importance can be evaluated by calculating the similarity between the extracted object and an object corresponding to a previously set request type, that is, a car.

At this time, as shown in (b), when the extracted object is a person, the learning data generating device 200 refines the corresponding 2D image because the similarity of the object corresponding to the set type is lower than the preset threshold. can do.

13 to 16 are exemplary diagrams for explaining a de-identification processing method according to an embodiment of the present invention.

13 to 16, the learning data generating apparatus 200 according to an embodiment of the present invention identifies an object included in a collected 2D image, and a part of the identified object corresponds to the type of the identified object. can be de-identified. That is, it is possible to improve learning efficiency of machine learning while preventing leakage of personal information by selectively de-identifying only a part of the identified object without de-identifying the entire identified object.

For example, as shown in FIG. 13, when a face is recognized as an object in a 2D image, if the recognized object is blurred, the face shape itself becomes unclear, so it is used as training data. When used, a problem in which learning efficiency is reduced may occur.

Therefore, as shown in FIG. 14, the learning data generation apparatus 200 performs de-identification processing by blurring a part of the identified object, extracts the edge of the identified object, and extracts Blurring processing may be performed based on the blurred edges. That is, the learning data generating apparatus 200 may increase learning efficiency by preventing the edge of a recognized object from being unclear by performing a blurring process to be spaced apart from the extracted edge by a preset threshold number of pixels.

Also, as shown in FIGS. 15 and 16 , the learning data generating device 200 may extract a landmark from the identified object. Further, the learning data generation apparatus 200 may increase learning efficiency by preventing an edge of a recognized object from being unclear by performing a blurring process on the extracted landmark.

17 is a flowchart illustrating a data purification method according to an embodiment of the present invention.

Referring to FIG. 17 , the learning data collection apparatus 100 according to an embodiment of the present invention may collect 2D images for machine learning of artificial intelligence (AI) (S100). In addition, the learning data collection device 100 may collect sensing data, 3D point cloud data, distance information, weather information, location information, and speed information.

Next, the learning data collection device 100 may extract information about the collected 2D images (S200). That is, the learning data collection apparatus 100 may calculate a similarity between successive 2D images among collected 2D images. In particular, the similarity calculation unit 115 may generate a red, green, blue (RGB) histogram for pixels in consecutive 2D images, and compare the generated RGB histograms to calculate a similarity. In addition, the learning data collection apparatus 100 may extract an edge of each of the consecutive 2D images and calculate a similarity between the consecutive 2D images based on the edge change amount between the consecutive 2D images. Here, the similarity calculating unit 115 may extract an edge of the identified object area or an edge of an object included in the entire 2D image. At this time, the similarity calculation unit 115 may calculate the similarity by comparing moments of the extracted edges. In addition, the learning data collection device 100 may calculate an influence on machine learning through meta data collected together with the collected 2D images. Specifically, the learning data collection device 100 calculates the influence on machine learning through metadata including at least one of speed information, weather information, sensor operation information, and GPS coordinate information at the time of collecting the 2D image. can

Next, the learning data collection apparatus 100 may determine the number of frames per second of the 2D images based on the calculated similarity or influence. Here, the learning data collection apparatus 100 may determine the number of frames per second for each section of the 2D images. That is, the frame determiner 125 may group 2D images according to similarity between consecutive 2D images and apply a preset number of frames per second to the group. Also, the learning data collection apparatus 100 may determine the number of frames per second for all 2D images. That is, among the 2D images collected in the training data generating device 200, an average similarity value of consecutive 2D images may be calculated, and a preset number of frames per second matching the similarity average value may be applied.

Next, the learning data collection device 100 may provide 2D images corresponding to the determined number of frames per second to the learning data generation device 200 . That is, the learning data collection apparatus 100 may apply the determined number of frames per second among the collected 2D images and transmit the applied 2D images to the learning data generation apparatus 200 .

18 is an exemplary diagram for explaining a process of refining data according to an embodiment of the present invention.

Referring to FIG. 18, the learning data collection apparatus 100 according to an embodiment of the present invention determines the number of frames per second of collected 2D images based on information on collected 2D images, Among the collected data, unnecessary data can be refined to increase learning efficiency.

More specifically, the learning data collection apparatus 100 calculates the similarity between two consecutive 2D images among the collected 2D images, or performs machine learning through meta data collected together with the collected 2D images. The number of frames per second of 2D images may be determined based on the degree of influence by calculating the degree of influence.

For example, as shown in (a) of FIG. 18 , the learning data collection apparatus 100 may calculate a similarity between two consecutive 2D images among collected 2D images. At this time, the similarity may be calculated by generating a red, green, blue (RGB) histogram for pixels in the continuous 2D image, and comparing the generated RGB histograms.

And, as shown in (b), the learning data collection apparatus 100 may control the number of frames per second of 2D images to be provided to the learning data generation apparatus 200 according to the similarity between two consecutive 2D images.

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention belongs that other modified examples based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein. It is self-evident to those skilled in the art. In addition, although specific terms have been used in the present specification and drawings, they are only used in a general sense to easily explain the technical content of the present invention and help understanding of the present invention, but are not intended to limit the scope of the present invention. Accordingly, the foregoing detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present invention should be selected by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Learning data collection unit: 100 Learning data generation unit: 200
Annotation Unit: 300 Artificial Intelligence Learning Unit: 400
Communication unit: 105, 205 Input/output unit: 110, 210
Similarity calculator: 115 Impact calculator: 120
Frame determination unit: 125 Data provision unit: 130
Singularity extraction unit: 215 Guide information provision unit: 220
Importance evaluation unit: 225 Image refinement unit: 230
Object identification unit: 235 De-identification processing unit: 240
Reservoir: 135, 245

Claims (10)

Collecting, by a learning data collection device, 2D images for machine learning of artificial intelligence (AI);
extracting, by the learning data collection device, information about the collected 2D images; and
Determining, by the learning data collection device, the number of frames per second of the collected 2D images based on the information on the collected 2D images;
The step of extracting information about the 2D image is
Characterized in that the number of frames per second of the 2D images is determined based on the degree of influence on machine learning by calculating the degree of influence on machine learning through meta data collected together with the collected 2D image,
The metadata is
Characterized in that the data for describing a specified object from the collected 2D image and 3D point cloud data obtained through lidar,
The metadata is
Category of the object, percentage of the object being clipped by the angle of view, percentage of the object being occluded by other objects or objects, tracking ID of the object, time the image was taken, weather conditions on the day the image was taken, file size, image size, copyright holder , resolution, bit value, aperture transmittance, exposure time, ISO sensitivity, focal length, aperture value, angle of view, white balance, RGB depth, class name, tag, shooting location, road type, road surface information and traffic jam information. Characterized in that it includes at least one,
In the step of extracting information about the 2D image,
Characterized in that the learning data collection device calculates the degree of influence on machine learning through metadata including at least one of speed information, weather information, sensor operation information, and GPS coordinate information at the time of collecting the corresponding 2D image, ,
In the step of extracting information about the 2D image,
It is characterized in that each of the collected 2D images is divided into a plurality of regions, each of the divided regions is analyzed to identify environmental variables, and a singular point is extracted based on the amount of change in the environmental variables,
In the step of extracting information about the 2D image,
Characterized in that the environmental variable is calculated based on the brightness of the divided area or RGB (Red, Green, Blue) values,
In the step of extracting information about the 2D image,
Each of the collected 2D images is divided into two regions in the vertical direction, and when the amount of change in environmental variables in the upper region of the consecutive 2D images is higher than a preset threshold, it is recognized as a time point when the weather changes, and the environment in the lower region Characterized in that, when the change amount of the variable is higher than a preset threshold value, it is recognized as a point in time when the road changes, a data refinement method.
The method of claim 1, wherein the step of extracting information about the 2D image
When the moving speed of the learning data collection device is higher than a preset threshold value, calculating a first influence degree, data purification method.
The method of claim 2, wherein the step of extracting information about the 2D image
When the moving speed of the learning data collection device is lower than a preset threshold value, calculating a second influence lower than the first influence, data purification method.
4. The method of claim 3, wherein determining the number of frames per second comprises:
2D images are grouped according to the degree of influence between the consecutive 2D images, and a predetermined number of frames per second is applied according to the degree of influence of each group.
4. The method of claim 3, wherein determining the number of frames per second comprises:
Characterized in that, an average value of influence between consecutive 2D images is calculated, and a preset number of frames per second matching the calculated average value of influence is applied.
A computer program recorded on a recording medium,
memory;
transceiver; and
In combination with a computing device configured to include a processor for processing instructions resident in the memory,
Collecting, by the processor, 2D images for machine learning of artificial intelligence (AI);
extracting, by the processor, information about the collected 2D images; and
By the processor, determining the number of frames per second of the collected 2D images based on the information on the collected 2D images,
The step of extracting information about the 2D image is
Characterized in that the number of frames per second of the 2D images is determined based on the degree of influence on machine learning by calculating the degree of influence on machine learning through meta data collected together with the collected 2D image,
The metadata is
Characterized in that it is data for describing a specified object from the collected 2D image and 3D point cloud data obtained through lidar,
The metadata is
Category of the object, percentage of the object being clipped by the angle of view, percentage of the object being occluded by other objects or objects, tracking ID of the object, time the image was taken, weather conditions on the day the image was taken, file size, image size, copyright holder , resolution, bit value, aperture transmission, exposure time, ISO sensitivity, focal length, aperture value, angle of view, white balance, RGB depth, class name, tag, shooting location, road type, road surface information and traffic jam information. Characterized in that it includes at least one,
In the step of extracting information about the 2D image,
Characterized in that the processor calculates the degree of influence on machine learning through metadata including at least one of speed information, weather information, sensor operation information, and GPS coordinate information at the time of collecting the corresponding 2D image,
In the step of extracting information about the 2D image,
It is characterized in that each of the collected 2D images is divided into a plurality of regions, each of the divided regions is analyzed to determine an environmental variable, and a singular point is extracted based on the amount of change in the environmental variable,
In the step of extracting information about the 2D image,
Characterized in that the environmental variable is calculated based on the brightness of the divided area or RGB (Red, Green, Blue) values,
In the step of extracting information about the 2D image,
Each of the collected 2D images is divided into two regions in the vertical direction, and when the amount of change in environmental variables in the upper region of the consecutive 2D images is higher than a preset threshold, it is recognized as a time point when the weather changes, and the environment in the lower region A computer program recorded on a computer-readable recording medium, characterized in that when the change amount of the variable is higher than a preset threshold value, it is recognized as a time point at which the road changes. delete delete delete delete

Images