CN113255879A

CN113255879A - Deep learning labeling method, system, computer equipment and storage medium

Info

Publication number: CN113255879A
Application number: CN202110288577.7A
Authority: CN
Inventors: 陈海波; 罗志鹏
Original assignee: Shenyan Technology Beijing Co ltd
Current assignee: Shenyan Technology Beijing Co ltd
Priority date: 2021-01-13
Filing date: 2021-03-18
Publication date: 2021-08-13
Anticipated expiration: 2041-03-18
Also published as: CN113255879B

Abstract

The invention discloses a deep learning labeling method, a deep learning labeling system, computer equipment and a storage medium. The method comprises the following steps: receiving an annotation task, wherein the annotation task comprises target data to be annotated and a category label of the target data; in the manual labeling mode, detecting that a preset intelligent labeling mode triggering condition is triggered, and converting the manual labeling mode into the intelligent labeling mode, wherein the manual labeling mode is used for labeling target data by acquiring a manual labeling operation instruction; and under an intelligent labeling mode, completing the labeling of the target data to be labeled by using a deep learning model. Through mutual cooperation among the task issuing unit, the labeling unit and the management unit, the whole labeling process is connected in series, the labeling process is simplified, different roles can work more efficiently in the classification and labeling of the target data, and the working efficiency is improved.

Description

Deep learning labeling method, system, computer equipment and storage medium

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a deep learning labeling method, system, computer device, and storage medium.

Background

With the continuous progress of the artificial intelligence technology, the application of the deep learning technology in various industries is more and more prominent. The supervised training of the deep learning model requires a large amount of training data, the quality of the data determines the upper limit of the model, the generation of the training data is inseparable from the data label, and the data label is used as an important ring in the machine learning engineering and is the basis for constructing the AI pyramid.

Currently, for annotation of data such as images, it is often the case that three parties are involved, including the algorithm engineer who issues the task, the annotator and the administrator. The existing working mode has the following problems: under the condition of participation of a plurality of people, the problems of mass deployment and maintenance are brought, and meanwhile, the problems of mass data distribution and distribution are involved, so that the method is complicated and is easy to introduce errors; user and authority management is an important requirement under participation of multiple persons, and is also a function that most marking tools are lacked; at present, a large amount of marking work needs to be finished manually, the working efficiency is low, and the labor cost is high.

Disclosure of Invention

In order to solve at least one of the above problems, a first aspect of the present invention provides a deep learning labeling method, including:

receiving an annotation task, wherein the annotation task comprises target data to be annotated and a category label of the target data;

in the manual labeling mode, detecting that a preset intelligent labeling mode triggering condition is triggered, and converting the manual labeling mode into the intelligent labeling mode, wherein the manual labeling mode is used for labeling target data by acquiring a manual labeling operation instruction;

and under an intelligent labeling mode, completing the labeling of the target data to be labeled by using a deep learning model.

In a specific embodiment, in the manual tagging mode, the method further includes:

retrieving whether a deep learning model containing the category label exists in a plurality of preset deep learning models;

if the deep learning model containing the class label is retrieved,

presenting the category label recommended by the deep learning model containing the category label in response to an operation instruction of manually marking target data to be marked in a marking interface;

and responding to a manual labeling instruction of the recommended category label, and acquiring labeled target data.

In a specific embodiment, the presenting the category label recommended by the deep learning model including the category label in response to the manual operation instruction on the target data to be labeled in the labeling interface includes:

presenting the category label with high k before the confidence degree predicted by the deep learning model containing the category label in response to an operation instruction of manually marking target data in a marking interface,

where k is a natural number.

if the deep learning model including the class label is not retrieved,

responding to an operation instruction of manually marking target data in a marking interface, and presenting all category labels in the marking task;

and responding to a manual marking instruction based on all the category labels, and acquiring marked target data.

In a specific embodiment, in the intelligent labeling mode, the completing labeling of the target data to be labeled by using a deep learning model includes:

and if the deep learning model containing the class label exists, completing the labeling of the target data to be labeled by using the deep learning model containing the class label.

In an embodiment, if the deep learning model including the category label is retrieved, the labeling of the target data to be labeled using the deep learning model including the category label includes:

predicting part of the target data to be labeled by using the deep learning model containing the class label;

reading the marked target data with the m-th maximum confidence level in the marked target data after the deep learning model containing the category label predicts as at least one part of a training set, and carrying out iterative training on the deep learning model containing the category label until a preset iteration ending condition is met to obtain a trained deep learning model;

completing the labeling of the target data to be labeled by using the trained deep learning model,

wherein m is a natural number.

and if the deep learning model containing the category label does not exist, completing the labeling of the target data to be labeled by using a deep learning model selected from the preset multiple deep learning models based on the labeling task.

In a specific embodiment, the labeling task is a target detection labeling task or a target classification labeling task.

In a specific embodiment, the triggering condition is that the manual labeling result reaches a predetermined number or the manual labeling result reaches a predetermined number and the authority of the intelligent mode is preset to be on.

In a specific embodiment, the method further comprises:

responding to a labeling task issuing instruction of a task issuer, and receiving the labeling task;

responding to an allocation operation instruction of an administrator, and allocating the issued annotation task to an account of an annotator;

and responding to a labeling operation instruction of a labeling operator, and labeling the target data.

The second aspect of the present invention provides a deep learning labeling system, including:

the task issuing unit is used for receiving an annotation task, wherein the annotation task comprises target data to be annotated and a category label of the target data;

the detection unit is used for detecting that a preset intelligent marking mode trigger condition is triggered in the manual marking mode and converting the manual marking mode into the intelligent marking mode, wherein the manual marking mode is used for marking target data by acquiring a manual marking operation instruction;

and the intelligent labeling unit is used for completing the labeling of the target data to be labeled by using a deep learning model in an intelligent labeling mode.

In a specific embodiment, the system further comprises: a retrieval unit and a manual labeling unit, wherein

The retrieval unit is used for retrieving whether a deep learning model containing the category label exists in a plurality of preset deep learning models or not;

if the retrieval unit retrieves that the deep learning model containing the category label exists, the artificial labeling unit is configured to:

In a specific embodiment, the manual labeling unit is further configured to:

where k is a natural number.

The retrieval unit is used for retrieving whether a deep learning model containing the category label exists in a plurality of preset deep learning models;

if the retrieval unit does not retrieve that the deep learning model containing the category label exists, the manual retrieval unit is configured to:

responding to an operation instruction of a target to be labeled in a labeling interface manually, and presenting all category labels in the labeling task;

In a specific embodiment, the system further comprises:

the intelligent labeling unit further comprises a model selection unit,

if the deep learning model containing the category label exists in the retrieval unit, the model selection unit finishes the labeling of the target data to be labeled by using the deep learning model containing the category label.

In a specific embodiment, the model selecting unit further includes:

the initial prediction unit is used for predicting part of target data in the target data to be labeled by using the deep learning model containing the class label;

the model iterative training unit is used for reading the labeled target data with the height m before the confidence level in the labeled target data after the deep learning model containing the class label predicts as at least one part of a training set, and performing iterative training on the deep learning model containing the class label until a preset iteration ending condition is met to obtain a trained deep learning model;

the intelligent labeling unit also comprises an intelligent labeling execution unit which is used for finishing the labeling of the target data to be labeled by using the trained deep learning model,

wherein m is a natural number.

In a specific embodiment, the system further comprises:

the intelligent labeling unit further comprises a model selection unit,

if the deep learning model containing the category label is not searched by the searching unit, the model selecting unit finishes the labeling of the target data to be labeled by using the deep learning model selected from the preset multiple deep learning models based on the labeling task.

In a specific embodiment, the system further comprises a manual marking unit and a management unit, wherein

The task issuing unit responds to an annotation task issuing instruction of a task issuer and receives the annotation task;

the management unit responds to an allocation operation instruction of an administrator and allocates the issued annotation task to an account of an annotator;

and the manual labeling unit responds to a labeling operation instruction of a labeling operator to label the target data.

A third aspect of the invention provides a computing device comprising a processor and a memory storing a program which, when executed, performs the method of the first aspect of the invention.

A fourth aspect of the present invention provides a computer readable storage medium storing a program which, when executed, performs the method of the first aspect of the present invention.

The invention has the following beneficial effects:

the embodiment of the invention provides a deep learning labeling method, a deep learning labeling system, computer equipment and a computer readable storage medium. The method can set a manual marking mode and an intelligent marking mode, is suitable for different occasions, and realizes full automation of the manual marking mode and the intelligent marking mode, so that the manual marking mode and the intelligent marking mode are seamlessly switched. Through the system, all the functional units are mutually cooperated, the whole labeling process is connected in series, the labeling process is simplified, different roles can more efficiently cooperate in the classification labeling of the target data, and the working efficiency is improved. The integrated process ensures that each role is concentrated on own responsibility, reduces the error rate when each step is connected and saves more time.

Drawings

The following describes embodiments of the present invention in further detail with reference to the accompanying drawings;

fig. 1 shows a scene schematic diagram for implementing a deep learning labeling method according to an embodiment of the invention.

Fig. 2 shows a flowchart of a deep learning labeling method according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of a task management interface corresponding to an image classification task type according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of a data management interface corresponding to an image classification task type according to an embodiment of the present invention.

FIG. 5 is a schematic diagram illustrating a task management interface corresponding to an image detection task type according to an embodiment of the present invention.

FIG. 6 illustrates a My task interface diagram corresponding to an image classification task type according to an embodiment of the invention.

FIG. 7 is a schematic diagram illustrating an annotation interface corresponding to an image classification task type according to an embodiment of the present invention.

FIG. 8 is a flow diagram illustrating a manual annotation mode according to an embodiment of the invention.

FIG. 9 is a schematic diagram illustrating an annotation interface corresponding to an image classification task type according to another embodiment of the invention.

FIG. 10 is a schematic diagram illustrating an annotation interface corresponding to an image detection task type according to an embodiment of the present invention.

FIG. 11 illustrates a flow diagram of an intelligent annotation schema in accordance with an embodiment of the present invention.

FIG. 12 shows a model iterative training flow diagram in accordance with an embodiment of the present invention.

FIG. 13 shows a system diagram according to an embodiment of the invention.

FIG. 14 is a diagram illustrating a task publishing unit composition according to an embodiment of the invention.

FIG. 15 is a schematic diagram of a composition of a manual labeling unit according to an embodiment of the present invention.

FIG. 16 is a schematic diagram of a work statistics interface of a manual labeling unit according to an embodiment of the present invention.

Fig. 17 shows a schematic diagram of a management unit composition according to an embodiment of the present invention.

Fig. 18 shows an overview interface diagram of a management unit according to an embodiment of the invention.

FIG. 19 shows a task management interface diagram of a management unit, according to an embodiment of the invention.

Fig. 20 shows a schematic view of a user management interface of a management unit according to an embodiment of the invention.

FIG. 21 shows a schematic diagram of a computer device implementing the method of the present invention.

Detailed Description

In order to more clearly illustrate the invention, the invention is further described below with reference to preferred embodiments and the accompanying drawings. Similar parts in the figures are denoted by the same reference numerals. It is to be understood by persons skilled in the art that the following detailed description is illustrative and not restrictive, and is not to be taken as limiting the scope of the invention.

First embodiment

A first embodiment of the present invention provides a deep learning annotation method, which can be implemented in the server 10 shown in fig. 1. Task issuers such as algorithm engineers, administrators and annotators log in their accounts via the

respective terminals

12, 14, 16.

The terminal can be hardware or software. When the terminal is hardware, it may be various electronic devices having a display screen and supporting image recognition, including but not limited to a smart phone, a tablet computer, a laptop portable computer, a desktop computer, and the like. The terminal can be provided with a client, which can be a browser client, an instant messaging client and the like, and the type of the client is not limited in the application. When the terminal device is software, the terminal device can be installed in the electronic devices listed above. It may be implemented as multiple pieces of software or software modules, or as a single piece of software or software module. And is not particularly limited herein.

The network serves as a medium for providing communication links between terminals and servers. The network may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few. The application is not limited thereto. A user may use the terminal to interact with a server over a network to receive or send messages, etc.

The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like. The server and the terminal may be directly or indirectly connected through a network, and the present application is not limited thereto.

It should be understood that the number of terminals, networks, and servers in fig. 1 are merely illustrative. There may be any suitable number of terminals, networks, and servers, as desired for an implementation.

As shown in fig. 2, the deep learning labeling method of the embodiment includes:

s100, receiving an annotation task, wherein the annotation task comprises target data to be annotated and a category label of the target data.

In one specific example, the method of the first embodiment of the present invention is described in conjunction with the system of the second embodiment of the present invention. And (3) the task publisher opens a tee to log in the system through the own account number, and a visual human-computer interaction interface shown in figure 3 is presented. And the tee issues a new labeling task by clicking a 'issue task' button.

In one example, Zhang three released a task regarding image classification of fish on 9, 25/2020.

When a task is released, the task needs to be explained, for example: the task type is as follows: classifying the images; customizing the category labels or selecting preset category labels (if any) from the system includes: finless eel, carp, crucian carp, tenuis, sturgeon, shark, whale and goldfish.

Then, the tee presents an interactive interface shown in fig. 4 by clicking a 'data management' menu, and uploads a corresponding data set to be labeled, namely a picture data set, on the interactive interface.

Thus, Zhang three completes a task release.

The method and the system of the invention are not limited to image classification labeling, and can also perform class labeling on targets such as voice, text, video and the like, for example, emotion labeling on voice, and the class labels are positive, negative and neutral, for example, in this case, the task type is voice classification.

Further, the method and system of the present invention are not limited to classification labeling tasks, but are also applicable to target detection tasks, such as image detection tasks. For example, zhang san uploads a picture containing a vehicle through a "data management" menu, and gives 4 tag categories, which are bicycle, motorcycle, car, and truck, respectively.

After the tasks are published, the system records the published tasks and notifies the administrator.

In one example, zhang san may select "task belongs" as itself and the task type is "image classification" or "image recognition" through a "task management" menu, so that all details about the image classification task that has been issued by itself may be viewed as shown in fig. 3, or all details about the image detection task that has been issued by itself may be viewed as shown in fig. 5.

In some examples, others may be selected in "task belonged" to view the task situation posted by others. This, of course, requires administrator authorization.

The task publisher can select other annotation tasks for viewing in the "task type," such as speech classification, text classification, and so on.

As shown in fig. 3, there are 3 image classification tasks currently released by three.

The state of the dog picture classification task is that the dog picture classification task is completed, the number of the total marked pictures is 200, and the total number of the dogs is 12.

The status of the fish picture classification task is marked, 20% of the fish picture classification task is completed, the current task takes 26 days, 23 hours, 15 minutes and 57 seconds, and the predicted completion time is 2021 years, 3 months and 9 days.

The current state of the bird picture classification task is that the bird picture classification task is returned by an administrator, and the marking is unqualified.

As shown in fig. 5, there are 3 image detection tasks currently issued by three.

The detection task 1 is completed, the number of the total marked pictures is 200, and the number of the labels is 12.

The status of test task 2 is marked and has been completed by 20%, and it takes 26 days, 23 hours, 15 minutes and 57 seconds currently, and the expected completion time is 2021 year, 3 month and 9 days.

The current status of the detection task 3 is returned by the administrator, which indicates that the label is not qualified.

And S200, under the manual marking mode, marking the target data by acquiring a manual marking operation instruction.

In a specific embodiment, the annotator lee logs in the system through the own account.

Labeling tasks as Classification labeling

When the annotator lee has logged into the system, the "my tasks" menu is displayed, as shown in FIG. 6.

Li four clicks on the My tasks menu on the main interface as shown in FIG. 6, the annotation task and task status can be seen.

For example, for a task named dog photo category, the task status is shown as completed; for the task named fish picture classification, the data volume is 2000, the types of the custom tags given by the task publisher are 8, and the custom tags are finless eel, common carp, crucian carp, tengyu, sturgeon, shark, whale and goldfish respectively.

The annotator clicks the "begin annotation" button and the page jumps to the annotation interface, which as shown in fig. 7, includes a data presentation area 2056 and a label area 2057. Wherein the data display area presents the pictures to be marked.

In one embodiment, as shown in fig. 8, S200 further includes:

s2001, searches whether or not a deep learning model including a category label included in the issued task exists among the plurality of deep learning models set in advance.

In one particular example, the system includes various target classification deep learning models, such as a CNN model suitable for image annotation, an LSTM model or BERT model suitable for text annotation, and so forth.

And S2003, if so, presenting the category label recommended by the deep learning model containing the category label in response to an operation instruction of manually marking the target data to be marked in the marking interface.

For example, when a deep learning model trained on the 8 fish classifications in the above example is retrieved, when lie four hovers a mouse over the picture as shown in fig. 7, the label area presents the category labels recommended by the model in the deep learning model trained on the 8 fish classifications.

Preferably, the top k labels with the highest confidence level, e.g., the top 3 labels, are recommended for selection by the annotator. As shown in fig. 7, the 3 tags with the highest confidence are goldfish: 0.9852, respectively; carp: 00103; shark: 0.0045.

and S2005, responding to a manual labeling instruction of the recommended category label, and acquiring labeled target data.

In a specific example, the Liqu selects a "goldfish" and clicks a "submit data" button to complete the labeling of the picture, so that the system obtains the standard result. And then, marking the next picture on the plum four until marking is finished, clicking a 'marking end' button, and uploading the data to the system.

In another embodiment, as also shown in fig. 8, step S200 further comprises:

and S2004, if the deep learning model containing the category labels is not retrieved, responding to an operation instruction of manually marking target data in a marking interface, and presenting all category labels in the marking task.

For example, when a deep learning model trained for classifying 8 fish in the above example is not retrieved, when the annotator operates on the data to be annotated in the data presentation area (for example, hovering the data over a picture as shown in fig. 9 with a mouse), all the category labels (here, 8 types) included in the annotation task appear in the label area.

And S2006, responding to a manual labeling instruction based on all the category labels, and acquiring labeled target data.

In one example, as shown in fig. 9, lei quan selects "bungius" and clicks the submit data button, completing the annotation of this picture. And then, marking the next picture on the plum four until marking is finished, clicking a 'marking end' button, and uploading the data to the system.

In the actual labeling work, the types of the labels included in the task are often not only 8, but may be dozens, so that there is a case that the label bar as shown in fig. 8 cannot be displayed on one screen and needs to be paged, which inevitably greatly reduces the work efficiency of manual labeling. In the scheme of intelligent recommendation of the existing model, the annotator can select the most appropriate label at the first time, and the working efficiency is greatly improved.

Labeling task as target detection label

In a specific example, in a scenario where the annotation task is image detection annotation, after the annotator li he logs in the system, a menu of "my task" is displayed, as shown in fig. 5.

For example, for a task named detection category 1, the task status is shown as completed; for the task named detection classification 2, the data volume is 2000 pieces, and the types of the custom tags given by the task publisher are 4, namely, bicycle, motorcycle, car and truck.

The annotator clicks the "begin annotation" button and the page jumps to the annotation interface, which, as shown in fig. 10, includes a picture presentation area 2058 and a label area 2059.

Similar to the classification label, in one embodiment, S200 further includes:

and S2010, searching whether a deep learning model containing the category label exists in a plurality of preset deep learning models.

In one particular example, the system includes various target detection deep learning models, such as Fast-R-CNN, YoloV4, and so forth.

S2013, if the deep learning model containing the category label is found, responding to an operation instruction of manually marking target data to be marked in a marking interface, and presenting the category label recommended by the deep learning model containing the category label.

In a specific example, in a case that a trained model for the task category is retrieved, when the annotator operates on the picture to be detected in the picture display area (for example, hovering the image over the image shown in fig. 10 with a mouse), a category label recommended by the model is presented in the label area.

Preferably, the k labels with the highest confidence are recommended for the annotator to select.

S2015, responding to a manual labeling instruction for the recommended category label, and acquiring labeled target data.

Specifically, the manual labeling result is obtained in response to manual selection of the recommended category label and setting of the bounding box of the target.

In a specific example, the label recommended at the head is "bicycle", the annotator selects the "bicycle" label and selects the "rectangle" function key from the toolbar, and clicks the "submit data" button after the selection of the driving bounding box, so as to complete the annotation of the picture. And then marking the next picture until marking is finished, clicking an end marking button, and uploading the data to the system.

The annotator can select a "move" button in the toolbar to adjust the position of the stroked bounding box.

In another embodiment, step S200 further comprises:

s2014, if the deep learning model containing the category labels is not retrieved, responding to an operation instruction of manually marking target data in a marking interface, and presenting all the category labels in the marking task.

For example, when the target detection deep learning model trained on 4 classes in the above example is not retrieved, when the annotator operates on the picture to be detected in the picture display area (for example, hovering the image on the image as shown in fig. 10 with a mouse), all the class labels (here, four types of bicycles, motorcycles, cars and trucks) included in the task are presented in the label area for the annotator to select.

And S2016, responding to a manual labeling instruction based on all the category labels, and acquiring labeled target data.

Specifically, the manual labeling result is obtained in response to manual selection from all the category labels and setting of the bounding box of the target.

And after the annotator selects the 'bicycle' label and selects the 'rectangle' function key from the toolbar to mark the driving boundary box, clicking a data submitting button to finish the marking of the picture. And then marking the next picture until marking is finished, clicking an end marking button, and uploading the data to the system.

In the actual labeling work, the types of the labels included in the task are not only 4, but may be dozens, so that the situation that the labels cannot be displayed on one screen in the label bar and need to be searched by turning pages exists, and the working efficiency of manual labeling is inevitably greatly reduced. In the scheme of intelligent recommendation of the existing model, the annotator can select the most appropriate label at the first time, and the working efficiency is greatly improved.

S300, under the manual labeling mode, detecting that a preset intelligent labeling mode triggering condition is triggered, and converting the manual labeling mode into the intelligent labeling mode.

In one specific example, a system administrator may set a trigger condition, such as manually annotating results to a predetermined number, such as 500 sheets. For example, after detecting that the annotator manually marks the category of 500 pictures, triggering to enter an intelligent marking mode.

On the other hand, the annotator can also be given the right to use the intelligent annotation mode. In the case where the administrator gives the annotator the use of the intelligent annotation mode, the intelligent pre-annotated switch is open as shown in the lower left corner of fig. 7.

And S400, under an intelligent labeling mode, completing labeling of the target data to be labeled by using a deep learning model.

In a specific example, as shown in fig. 11, S400 further includes:

and searching whether a deep learning model containing the category label exists in a plurality of preset deep learning models.

Even if the deep learning model contains the category labels (e.g., 8 fish in the above embodiment), it often contains other category labels (e.g., the above labeled dog category), and predicting a particular task directly with such a model increases the probability of error, and therefore, it is preferable to train it as well.

To this end, in one specific example, the method further comprises the step of iteratively training the model, as shown in fig. 12.

S4001, predicting part of the target data to be labeled by using the deep learning model containing the class label.

Wherein the number of the parts can be customized, for example 200.

S4003, reading the labeled target data with the highest m before the confidence level in the labeled target data after the deep learning model containing the category label predicts as at least a part of training set, and performing iterative training on the deep learning model containing the category label until a preset iteration ending condition is met to obtain a trained deep learning model.

In one specific example, S4003 includes:

s40031: a certain number (e.g. 60% confidence top, for example 120) of pictures are selected as a training set.

S40033: the model is trained using the training set.

S40035: predicting the residual pictures to be labeled according to the model obtained by training in the step S4003, selecting a certain number of pictures and a previous training set, and training the model;

s40037: and repeating the iteration steps, and obtaining the final labeling result of all the pictures by using the model trained for the last time when the preset iteration ending condition is met.

S4005, completing the labeling of the target data to be labeled by using the trained deep learning model.

In another specific example, as also shown in fig. 11, S400 further includes:

and if the deep learning model containing the category label does not exist in the retrieval unit, completing the labeling of the target data to be labeled by using the deep learning model selected from the preset multiple deep learning models based on the labeling task.

In particular, this step may also comprise a training step.

And training a deep learning model selected from the preset multiple deep learning models by taking the artificially labeled target data as a training set.

Specifically, if the image is not retrieved, the image which is manually marked before the intelligent marking is triggered is used as a training set to train the image classification deep learning model selected by the system, such as a CNN network.

As for how the system selects a suitable model, the system can be configured by a preset configuration file, for example, if the task is image classification, the system is configured to use a CNN network; if the task is speech recognition, it is configured to use the LSTM network, and so on.

Similar to S4003-4005, the selected model is trained in an incremental training manner until a preset iteration end condition is satisfied.

And S4013, marking the target data to be marked by using the trained deep learning model.

In a specific example, the preset iteration end condition includes one of:

all data sets enter a training set;

the number of the training sets reaches the number of the pictures defined by the user;

the number of model iterations reaches the user-defined number;

the decision to end the model iteration is made manually.

According to the deep learning labeling method provided by the embodiment of the invention, the manual labeling mode and the intelligent labeling mode can be set, different occasions are applicable, the switching between the manual labeling mode and the intelligent labeling mode is completely automatic, the manual labeling mode and the intelligent labeling mode are seamlessly switched, the labeling process is simplified, different roles can more efficiently cooperate in target classification labeling, and the working efficiency is improved.

Second embodiment

A second embodiment of the present invention provides a deep learning annotation system, which is used for annotation of a deep learning model, and as shown in fig. 13, includes: a task issuing unit 110, a manual labeling unit 112, a detection unit 114 and an intelligent labeling unit 116.

Task issuing unit

The task issuing unit is used for receiving the labeling task, wherein the labeling task comprises a target to be labeled and a category label of the target.

In a specific example, as shown in fig. 14, the task issuing unit includes a task management unit 1100 and a data management unit 1105.

And (4) the task publisher opens a tee to log in the system through an own account number. The task management unit comprises a visual human-computer interaction interface shown in fig. 3, and a tee issues a new labeling task by clicking a 'issue task' button.

In one example, Zhang three released a task regarding image classification of fish on 9, 25/2020. When a task is released, the task needs to be explained, for example: the task type is as follows: classifying the images; custom category tags or pre-set category tags (if any) selected from the system include finless eel, carp, crucian carp, tengrub, sturgeon, shark, whale and goldfish.

The data management unit includes a "data management" menu. And the tee presents an interactive interface shown in fig. 4 by clicking a 'data management' menu, and uploads a corresponding data set to be marked, namely a picture data set, on the interactive interface.

Thus, Zhang three completes a task release.

In another example, the method and system of the present invention are not limited to classification tagging tasks, but are also applicable to target recognition tasks, such as image recognition tasks. For example, zhang san uploads a picture containing a vehicle through a "data management" menu, and gives 4 tag categories, which are bicycle, motorcycle, car, and truck, respectively.

After the tasks are published, the system records the published tasks and notifies the administrator to assign the tasks.

Zhang III can select the task as self and the task type as image classification or image recognition through a task management menu, so that all the details of the tasks about image classification or image recognition issued by the user can be viewed, as shown in FIGS. 3 and 5.

As shown in FIG. 4, the data management unit includes a data management interface, and the task publisher can view the data set information, including the name of the data set, the type of the label, the status of the label, the creation time, the latest status, and optional operations.

In addition, the task publisher can perform preview, download or delete operation on the data set through the data management unit.

Manual labeling unit

The manual labeling unit is used for acquiring a manual labeling result in a manual labeling mode.

In one particular example, as shown in FIG. 15, the manual annotation unit includes a task presentation unit 1120 and a work statistics unit 1125.

Labeling tasks as Classification labeling

When the annotator li four logs in the system, the task display unit presents a menu of 'my task', as shown in fig. 6.

In one embodiment, the system further includes a retrieving unit configured to retrieve whether a deep learning model including the category label exists in a plurality of preset deep learning models.

and presenting the category label recommended by the deep learning model containing the category label in response to an operation instruction of manually marking target data to be marked in a marking interface.

For example, when a deep learning model trained on the 8 fish classifications in the above example is retrieved, when lie four hovers a mouse over the picture as shown in fig. 7, the label area presents the category labels recommended by the model in the deep learning model trained on the 8 fish classifications. Preferably, the top k labels with the highest confidence are recommended for selection by the annotator, for example, 3 labels. As shown in fig. 8, the 3 tags with the highest confidence are goldfish: 0.9852, respectively; carp: 00103; shark: 0.0045.

In another embodiment, if the retrieval unit does not retrieve that the deep learning model containing the category label exists, the manual retrieval unit is configured to:

and responding to an operation instruction of manually marking target data to be marked in the marking interface, and presenting all category labels in the marking task.

Labeling task as target detection label

In a specific example, in a scenario where the annotation task is image detection annotation, after the annotator li he logs in the system, the manual annotation unit includes a "my task" menu, as shown in fig. 5.

In one embodiment, the system further comprises:

and the retrieval unit is used for retrieving whether the deep learning model containing the category label exists in a plurality of preset deep learning models.

In a specific example, the label recommended at the head is "bicycle", and after the annotator selects the "bicycle" label and selects the "rectangle" function key from the toolbar to stroke the driving bounding box, the annotating of the picture is completed by clicking the "submit data" button. And then marking the next picture until marking is finished, clicking an end marking button, and uploading the data to the system.

In another embodiment, if the retrieving unit does not retrieve that the deep learning model containing the category label exists, the artificial labeling unit is configured to:

In the actual labeling work, the types of the labels included in the task are not only 4, but may be dozens, so that there are cases that one screen of the label bar cannot be displayed and needs to be paged for searching as shown in fig. 10, which inevitably greatly reduces the working efficiency of manual labeling. In the scheme of intelligent recommendation of the existing model, the annotator can select the most appropriate label at the first time, and the working efficiency is greatly improved.

In addition, the toolbar may include a "recall" function key for dismissing the most recently scribed bounding box. Clicking the function key of 'display label' can switch between hiding/displaying the boundary box on the picture. Selecting the "move" function key can adjust the position of the stroked bounding box. And selecting a full screen function key to display the pictures loaded in the display area in a full screen mode.

In a preferred example, the bounding boxes corresponding to different categories are represented by different colors, which is beneficial for a annotator to distinguish during annotation, and improves the work efficiency.

In one embodiment, the job statistics unit is configured to present the job statistics in response to a review operation by a annotator, as shown in FIG. 16.

Detection unit

The detection unit is used for detecting that a preset intelligent labeling mode trigger condition is triggered, and converting the manual labeling mode into an intelligent labeling mode.

In one specific example, a system administrator may set a trigger condition, such as manually annotating results to a predetermined number, such as 500 sheets. For example, when the detection unit detects that the annotator manually marks the category of 500 pictures, the intelligent annotation mode is triggered to enter.

Intelligent labeling unit

In a specific example, the intelligent annotation unit further comprises a model selection module, for example implemented as a configuration file as described hereinbefore. If the deep learning model containing the class label exists in the retrieval unit, the model selection module selects the deep learning model containing the class label to finish the labeling of the target to be labeled.

To this end, in a specific example, the model selecting unit further includes: :

and the initial prediction unit is used for predicting part of the target data to be labeled by using the deep learning model containing the class label.

Wherein the number of the parts can be customized, for example 200.

And the model iterative training unit is used for reading the labeled target data with the height m before the confidence level in the labeled target data after the deep learning model containing the class label predicts as at least one part of a training set, and performing iterative training on the deep learning model containing the class label until a preset iteration ending condition is met to obtain a trained deep learning model.

In one specific example, the model iteration training unit is configured to perform the following steps:

S40033: the model is trained using the training set.

The intelligent labeling unit also comprises an intelligent labeling execution unit which is used for finishing the labeling of the target data to be labeled by using the trained deep learning model.

In another specific example, if the deep learning model including the category label does not exist in the retrieval unit, the model selection unit completes labeling of the target data to be labeled by using the deep learning model selected from the preset multiple deep learning models based on the labeling task.

In particular, this step may also comprise a training step.

The model selection module may be implemented by a pre-configured configuration file. Pairing the tasks and the models through the configuration files, wherein for example, if the tasks are image classification, the tasks are configured to use a CNN network; if the task is speech recognition, it is configured to use the LSTM network, and so on.

And the intelligent marking execution unit finishes marking the target to be marked by using the trained deep learning model.

In a specific example, the preset iteration end condition includes one of:

all data sets enter a training set;

the number of model iterations reaches the user-defined number;

the decision to end the model iteration is made manually.

In a specific example, the management unit is used for an administrator to set a trigger condition of the intelligent annotation mode in the management unit.

In addition, as shown in fig. 17, the management unit further includes an overview unit 1180, a task management unit 1183, a user management unit 1186, and a setting unit 1189.

When the administrator logs in the system through the account of the administrator, the administrator enters the management unit 118.

As shown in fig. 18, the overview unit includes an overview interface for displaying the annotation status and the audit status of the data set in response to the viewing operation of the administrator. Selecting the task type as target classification/detection, and selecting the time range to check the task completion condition; manual annotation statistics and intelligent annotation statistics (not shown) can also be viewed separately.

The task management unit includes a task management menu, and jumps to a task management interface in response to a click operation by the administrator, as shown in fig. 19.

And in the task management interface, in response to the viewing operation of the administrator, the completion condition of each task is displayed, and the details of the tasks can be viewed. For the issued labeling task, the administrator can perform operations such as pause and end, and can also preview and audit the labeled data.

The user management unit includes a user management menu, and jumps to a user management interface in response to a click operation by the administrator, as shown in fig. 20.

In the user management interface, in response to the operation of the administrator, the roles of the users may be assigned, for example, to designate the users as algorithm engineers or annotators, and new users may be added or old users may be deleted.

The user management interface also has the functions of checking different roles and performing operations such as addition, deletion or modification on departments.

The setting unit comprises a setting interface, and in the setting interface, in response to the operation of an administrator, the setting unit can set the functions and product fields of the system and modify personal information at the same time, which is not described herein again.

The system of the implementation also comprises a database used for storing the data of the task issuing unit, the labeling unit and the management unit.

According to the system for integrating multiple functions, which is provided by the second embodiment of the invention, an administrator, a task publisher and a annotator share the same system to publish, label and manage tasks, and the labeling unit, the task publishing unit and the management unit are cooperated with each other to connect the whole labeling process in series, so that the labeling process is simplified, the error rate of the connection of all the steps is reduced, the time is saved, the target data detection and labeling can be performed by different roles more efficiently and cooperatively, and the working efficiency is improved greatly.

In addition, the invention can set both the manual marking mode and the intelligent marking mode, is suitable for different occasions, and the switching of the manual marking mode and the intelligent marking mode is completely automatic, so that the manual marking mode and the intelligent marking mode are seamlessly switched.

Third embodiment

As shown in fig. 21, a computer device adapted to be used to implement the method provided by the above-described embodiments includes a central processing module (CPU) that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) or a program loaded from a storage section into a Random Access Memory (RAM). In the RAM, various programs and data necessary for the operation of the computer system are also stored. The CPU, ROM, and RAM are connected thereto via a bus. An input/output (I/O) interface is also connected to the bus.

An input section including a keyboard, a mouse, and the like; an output section including a speaker and the like such as a Liquid Crystal Display (LCD); a storage section including a hard disk and the like; and a communication section including a network interface card such as a LAN card, a modem, or the like. The communication section performs communication processing via a network such as the internet. The drive is also connected to the I/O interface as needed. A removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive as necessary, so that a computer program read out therefrom is mounted into the storage section as necessary.

In particular, the processes described in the above flowcharts may be implemented as computer software programs according to the present embodiment. For example, the present embodiments include a computer program product comprising a computer program tangibly embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section, and/or installed from a removable medium.

The flowchart and schematic diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to the present embodiments. In this regard, each block in the flowchart or schematic diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the schematic and/or flowchart illustration, and combinations of blocks in the schematic and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The systems and units described in the present embodiment may be implemented by software or hardware. The described units may also be located in the processor.

Fourth embodiment

The embodiment also provides a nonvolatile computer storage medium, which may be the nonvolatile computer storage medium included in the above-mentioned apparatus in the above-mentioned embodiment, or may be a nonvolatile computer storage medium that exists separately and is not assembled in the terminal. The non-volatile computer storage medium stores one or more programs that, when executed by an apparatus, cause the apparatus to implement the method of the first embodiment.

It is to be noted that, in the description of the present invention, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion.

It should be understood that the above-mentioned embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention, and it will be obvious to those skilled in the art that other variations and modifications can be made on the basis of the above description, and all embodiments cannot be exhaustive, and all obvious variations and modifications belonging to the technical scheme of the present invention are within the protection scope of the present invention.

Claims

1. A deep learning labeling method is characterized by comprising the following steps:

2. The method of claim 1, wherein in manual annotation mode, the method further comprises:

if the deep learning model containing the class label is retrieved,

3. The method according to claim 2, wherein the presenting of the category label recommended by the deep learning model containing the category label in response to a manual operation instruction on target data to be labeled in a labeling interface comprises:

where k is a natural number.

4. The method of claim 1, wherein in manual annotation mode, the method further comprises:

if the deep learning model including the class label is not retrieved,

5. The method according to claim 1, wherein in the intelligent labeling mode, the labeling of the target data to be labeled is completed by using a deep learning model, and the method comprises the following steps:

6. The method of claim 5, wherein the using the deep learning model containing the class label to complete the labeling of the target data to be labeled if the deep learning model containing the class label is retrieved, comprises:

wherein m is a natural number.

7. The method according to claim 1, wherein in the intelligent labeling mode, the labeling of the target data to be labeled is completed by using a deep learning model, and the method comprises the following steps:

8. The method according to any one of claims 1 to 7,

the labeling task is a target detection labeling task or a target classification labeling task.

9. The method according to claim 1, wherein the trigger condition is that the manual labeling result reaches a predetermined number or the manual labeling result reaches a predetermined number and the authority of the intelligent mode is preset to be on.

10. The method of claim 1, further comprising:

11. A deep learning annotation system, comprising:

12. The system of claim 11, further comprising: a retrieval unit and a manual labeling unit, wherein

13. The system of claim 12, wherein the manual tagging unit is further configured to:

where k is a natural number.

14. The system of claim 11, further comprising: a retrieval unit and a manual labeling unit, wherein

15. The system of claim 11, further comprising:

the intelligent labeling unit further comprises a model selection unit,

16. The system of claim 15,

the model selection unit further includes:

wherein m is a natural number.

17. The system of claim 11, further comprising:

the intelligent labeling unit further comprises a model selection unit,

18. The system according to any one of claims 11-17,

19. The system of claim 11,

the triggering condition is that the manual labeling result reaches a preset number or the manual labeling result reaches the preset number and the authority of the intelligent mode is preset to be started.

20. The system according to claim 11, further comprising a manual labeling unit and a management unit, wherein

21. A computing device comprising a processor and a memory storing a program, wherein the program when executed implements the method of any of claims 1-10.

22. A computer-readable storage medium storing a program, characterized in that the program, when executed, implements the method of any one of claims 1-10.