KR102682639B1 - Server, method and computer program for automatically labelling image using feedback of post including image - Google Patents

Abstract

A server that automatically labels images using feedback from posts containing images receives feedback information related to the posts from a post register that registers posts containing images and text from a first user terminal, and a plurality of second user terminals. a feedback information receiving unit that analyzes the text to derive a first key keyword from the text, a text analysis unit that analyzes the feedback information to derive a second key keyword, and a labeling target included in the image by analyzing the image. Based on a match between an analysis unit including an image analysis unit that identifies and derives labeling-related keywords related to the identified labeling target, and at least one of the first core keyword, the second core keyword, and the labeling-related keyword and a labeling performing unit that performs labeling on the labeling object.

Description

Server, method, and computer program for automatically labeling images using feedback from posts containing images

The present invention relates to a server, method, and computer program for automatically labeling images using feedback from posts containing images.

Recently, research on methods for identifying objects using artificial intelligence technology has been actively conducted. Deep learning technology, which is mainly used in artificial intelligence technology, is characterized by high object discrimination and analysis capabilities.

This deep learning technology requires a task called labeling to proceed with learning, and in relation to deep learning learning technology through labeling, Korean Patent Publication No. 2020-0084411, a prior art, discloses a deep learning learning method and server.

However, in the past, a large amount of learning data was required for labeling, and a lot of people, money, and time were required to perform labeling. In addition, as labeling is performed manually, it incurs a lot of work costs, and if labeling is performed incorrectly due to operator error, deep learning learning results are poor.

Recently, a method has been used in which learning data is automatically labeled using automatic labeling technology, and then the developer checks the labeling and modifies the labeling. However, even if automatic labeling technology is used, the learning data is affected by other environmental variables (e.g., time of day/night, location, weather, indoor/outdoor, etc.), so the automatic labeling technology does not operate properly. It has the disadvantage of requiring manual re-labeling.

An object is to provide a server, a method, and a computer program for registering posts including images and text from a first user terminal and receiving feedback information related to the posts from a plurality of second user terminals.

Analyze text to derive first core keywords from the text, analyze feedback information to derive second core keywords, analyze images to identify labeling objects included in the image, and labeling-related keywords related to the identified labeling objects. The goal is to provide a server, method, and computer program that derives .

The present invention seeks to provide a server, method, and computer program that performs labeling of a labeling object based on a match between at least one of a first core keyword, a second core keyword, and a labeling-related keyword.

However, the technical challenges that this embodiment aims to achieve are not limited to the technical challenges described above, and other technical challenges may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention includes a post register that receives posts containing images and text from a first user terminal, and feedback information related to the posts from a plurality of second user terminals. a feedback information receiving unit that receives a text, a text analysis unit that analyzes the text to derive a first key keyword from the text, and analyzes the feedback information to derive a second key keyword; and an image analysis unit that analyzes the image to identify a labeling target included in the image and derives labeling-related keywords related to the identified labeling target, the first key keyword, the second key keyword, and An automatic image labeling server may be provided, including a labeling performing unit that performs labeling on the labeling object based on whether or not there is a match between at least one of the labeling-related keywords.

Another embodiment of the present invention includes the steps of registering a post including an image and text from a first user terminal, receiving feedback information related to the post from a plurality of second user terminals, analyzing the text and deriving a first key keyword from, analyzing the feedback information to derive a second key keyword, analyzing the image to identify a labeling object included in the image, and labeling related to the identified labeling object. Providing an automatic image labeling method comprising deriving a keyword and performing labeling on the labeling object based on a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword. can do.

In another embodiment of the present invention, when executed by a computing device, the computer program registers a post containing an image and text from a first user terminal and receives feedback information related to the post from a plurality of second user terminals. and analyze the text to derive a first key keyword from the text, analyze the feedback information to derive a second key keyword, analyze the image to identify a labeling target included in the image, and identify the labeling object included in the image. A sequence of instructions for deriving labeling-related keywords related to the labeling object and performing labeling on the labeling object based on a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword. A computer program stored on a medium containing can be provided.

The above-described means for solving the problem are merely illustrative and should not be construed as intended to limit the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description of the invention.

According to one of the above-described means for solving the problem of the present invention, a post containing an image and text is registered from a first user terminal, and feedback information related to the post is received from a plurality of second user terminals, thereby analyzing the text. Derive the first core keyword from the text, derive the second core keyword by analyzing the feedback information, analyze the image to identify the labeling object included in the image, and derive labeling-related keywords related to the labeling object, respectively. A server, a method, and a computer program that perform labeling on a labeling object based on a match between a 1 core keyword, a second core keyword, and a labeling-related keyword can be provided.

Conventional image identification technology using deep learning technology can only derive object names and cannot identify detailed information such as the breed of an animal or the model name of a car or machine, but the present invention A server, method, and computer program can be provided that allow detailed information, such as a model name, to be labeled in an area to be labeled on an image.

Conventionally, a large amount of learning data was required to identify a labeling target included in an image using deep learning technology, but the present invention uses labeling-related keywords derived from the image of the post, the first core keyword derived from the text of the post, and A server, method, and computer program that enable labeling to be performed using collective intelligence in various communities, SNS (Social Network Service), etc. based on second key keywords derived from comments on feedback information can be provided.

A server, method, and computer program can be provided that can determine whether the post is an authenticated post by determining the authenticity of the image included in the post.

1 is a configuration diagram of an automatic image labeling system according to an embodiment of the present invention.
Figure 2 is a configuration diagram of an automatic image labeling server according to an embodiment of the present invention.
3A and 3B are exemplary diagrams for explaining a process of analyzing text and images included in a post according to an embodiment of the present invention.
Figure 4 is an exemplary diagram illustrating a process of performing additional labeling on a labeling target when the sentence type of a post corresponds to a descriptive sentence according to an embodiment of the present invention.
Figure 5 is an exemplary diagram for explaining a process of labeling a labeling target included in a post when the sentence type of the post corresponds to a descriptive sentence according to an embodiment of the present invention.
FIGS. 6A and 6B are exemplary diagrams for explaining a process of labeling a labeling target included in a posting when the sentence type of the posting corresponds to a question according to an embodiment of the present invention.
FIGS. 7A and 7B are exemplary diagrams for explaining a process of not performing labeling on a labeling target included in a posting when the sentence type of the posting corresponds to a question according to an embodiment of the present invention.
Figure 8 is a flowchart of a method of automatically labeling images using feedback from posts containing images in an image automatic labeling server according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts unrelated to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, this means that it does not exclude other components, but may further include other components, unless specifically stated to the contrary, and one or more other features. It should be understood that it does not exclude in advance the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

In this specification, 'part' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized using one piece of hardware.

In this specification, some of the operations or functions described as being performed by a terminal or device may instead be performed on a server connected to the terminal or device. Likewise, some of the operations or functions described as being performed by the server may also be performed on a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings.

1 is a configuration diagram of an automatic image labeling system according to an embodiment of the present invention. Referring to FIG. 1 , the image automatic labeling system 1 may include an image automatic labeling server 100, a first user terminal 110, and a plurality of second user terminals 120. The automatic image labeling server 100, the first user terminal 110, and the plurality of second user terminals 120 illustrate components that can be controlled by the automatic image labeling system 1.

Each component of the automatic image labeling system 1 of FIG. 1 is generally connected through a network. For example, as shown in FIG. 1, the automatic image labeling server 100 may be connected to the first user terminal 110 or a plurality of second user terminals 120 simultaneously or at time intervals.

Network refers to a connection structure that allows information exchange between nodes such as terminals and servers, including Local Area Network (LAN), Wide Area Network (WAN), and World Wide Area Network (WWW). Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television communication networks, etc. Examples of wireless data communication networks include 3G, 4G, 5G, WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, ultrasonic communication, Visible Light Communication (VLC), and LiFi. ), etc. are included, but are not limited to this.

The automatic image labeling server 100 may learn a labeling object identification model that identifies the labeling object using training data including already collected images and label information of the labeling object.

The automatic image labeling server 100 may register a post including an image and text from the first user terminal 110 and receive feedback information related to the post from a plurality of second user terminals 120 . Here, the feedback information may include comments, views, preferences, reports, etc. for the post.

The automatic image labeling server 100 may input each text and feedback information into an emotion analysis model and classify each text and feedback information as positive, negative, or neutral.

The automatic image labeling server 100 may analyze the text, classify the sentence type of the post as either a descriptive sentence or a question sentence, and derive the first core keyword from the text.

The image automatic labeling server 100 may analyze the feedback information and derive the second key keyword. For example, the image automatic labeling server 100 may determine the second core keyword based on the positive rate, negative rate, and neutral rate of the feedback information.

The automatic image labeling server 100 may derive labeling-related keywords related to the identified labeling target and identify the labeling target from the image of the post using a labeling target identification model.

The automatic image labeling server 100 may perform labeling of the labeling target based on whether or not there is a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword. At this time, the automatic image labeling server 100 may perform labeling of the labeling target based on the sentence type of the classified post.

For example, if the sentence type of the post corresponds to a descriptive sentence and the labeling-related keywords, first core keyword, and second core keyword all match, the image automatic labeling server 100 places the post in the labeling target area corresponding to the image. Additional labeling of the labeling target may be performed based on at least one of the first core keyword and the second core keyword.

For another example, if the sentence type of the post corresponds to a descriptive sentence and the labeling-related keyword matches the first core keyword and does not match the second core keyword, the image automatic labeling server 100 uses the second core keyword based on the second core keyword. Labeling can be performed on the labeling target.

For another example, if the sentence type of the post corresponds to a question sentence and the positive rate for feedback information is more than a threshold, the image automatic labeling server 100 labels the labeling target area corresponding to the image based on the first core keyword. This allows labeling of the labeling object to be performed.

The automatic image labeling server 100 may determine that a post is an authenticated post for a labeling target based on whether or not there is a match between at least one of the related keywords. For example, the image automatic labeling server 100 calculates a first score based on the degree of relationship between the labeling-related keyword and the first core keyword, and calculates the second score based on the degree of relationship between the labeling-related keyword and the second core keyword. After calculating the third score based on the ratio to the number of views and preferences, calculating the authentication score for the post based on the first to third scores, and if the authentication score for the post is above the threshold, A post can be determined as a certified post for the labeling target.

Through this process, in the past it was possible to derive only the object name by identifying the labeling target included in the image of the post, but in the present invention, it is possible to derive only the object name through comments included in posts and feedback information uploaded to SNS, a specific web or app bulletin board. It is possible to add labeling or modify labeling for labeling objects included in the image. In other words, if the labeling target is an animal, it can provide the advantage of being able to label detailed information related to the labeling target, such as the breed of the animal and the model name if the labeling target is a car/electronic device. Additionally, the present invention can perform labeling using collective intelligence without requiring a large amount of learning data by performing additional labeling or labeling correction using comments on posts posted on SNS, a specific web, or app.

The first user terminal 110 may register a post containing text and images with the automatic image labeling server 100 through SNS (Social Network Service), a specific web, or an app. For example, when the user of the first user terminal 110 captures aquatic life through fishing activities, the first user terminal 110 automatically labels posts containing images of aquatic life and content related to fishing activities. It can be registered as server 100. For another example, when the user of the first user terminal 110 purchases a computer, the first user terminal 110 posts a post containing an image of the purchased computer and content related to the purchase to the image automatic labeling server 100. ) can be registered.

When the first user terminal 110 determines that the post is a certified post for the labeling target through the image automatic labeling server 100, the first user terminal 110 receives a notification message notifying that the post has been determined to be a certified post from the image automatic labeling server 100. You can.

The plurality of second user terminals 120 may receive comments, reports, preferences, etc. related to posts registered by the first user terminal 110 through SNS, a specific web, or an app.

The plurality of second user terminals 120 may transmit feedback information including comments, views, reports, preferences, etc. related to the post registered by the first user terminal 110 to the automatic image labeling server 100.

Figure 2 is a configuration diagram of an automatic image labeling server according to an embodiment of the present invention. Referring to FIG. 2, the image automatic labeling server 100 includes a learning unit 210, a post registration unit 220, a feedback information reception unit 230, an analysis unit 240, a labeling performance unit 250, and an authentication post determination unit. It may include (260).

The learning unit 210 may learn a labeling object identification model that identifies the labeling object using training data including already collected images and label information of the labeling object. Here, the labeling target identification model is Convolution Neural Network (CNN), Single Shot MultiBox Detector (SSD), Youn Only Look Once (YOLO), retinanet, K-nearest neighbor, K- It can be built based on K-means/K-modes, Gaussian Mixture Model, SVM (Support Vector Machien), BoW (Bag of Words), Viola-Jones, etc. , but is not limited to this.

The labeling target identification model uses Principal Component Analysis, Singular Value Decomposition algorithm, Latent Dirichlet Allocation algorithm, and Classification algorithm to further increase the identification accuracy of the labeling target. You can.

The post registering unit 220 may register a post including an image and text from the first user terminal 110. For example, the post registering unit 220 may receive a post including an image (or video) and text from the first user terminal 110 through a specific web or app. Here, a post may be composed of content including a title composed of text, text, and an image (or video).

The feedback information receiver 230 may receive feedback information related to the posting from the plurality of second user terminals 120. Here, the feedback information may include comments, views, preferences, reports, etc. for the post.

The analysis unit 240 may analyze the text, image, and feedback information included in the post, and derive keywords from each of the text, image, and feedback information.

This analysis unit 240 may include a text analysis unit 241 and an image analysis unit 242.

The text analysis unit 241 may input each of the text and feedback information into an emotion analysis model and classify each of the text and feedback information as positive, negative, or neutral. Here, the emotion analysis model may be, for example, a model created based on BERT or RNN (Recurrent Neural Network) that is trained to classify training data consisting of a plurality of texts into positive, negative, and neutral.

For example, the text analysis unit 241 may input the title and content of the post into an emotion analysis model to classify the post as positive, negative, or neutral. For another example, the text analysis unit 241 may input at least one comment included in the feedback information into an emotion analysis model and classify each of the at least one comment as one of positive, negative, and neutral.

The text analysis unit 241 may analyze the text and classify the sentence type of the post as either a descriptive sentence or a question sentence.

The text analysis unit 241 may analyze the text and derive the first key keyword from the text. For example, if the main content of the post includes the sentence “I caught a parrot sea bream at Sihwa Seawall,” the text analysis unit 241 uses a Named Entity Recognition algorithm to search for “fish” and “Sihwa Seawall.” At least one entity name such as ' can be derived. Here, entity name recognition may mean recognizing what type of word the word is by analyzing the word that means something. Thereafter, the text analysis unit 241 may derive a first key keyword, such as 'doldom', from among the at least one derived entity name.

The text analysis unit 241 may analyze the feedback information and derive the second key keyword. For example, when a comment in the feedback information includes the sentence “The parrotfish is really big,” the text analysis unit 241 derives at least one entity name such as ‘fish’ through an entity name recognition algorithm, and then derives Among at least one entity name, a second core keyword such as 'doldom' can be derived. At this time, if at least one comment included in the feedback information consists of a buzzword, compressed word, onomatopoeia, etc., the text analysis unit 241 may convert the comment into a similar keyword and then derive a second core keyword. .

This text analysis unit 241 can derive the first core keyword and the second core keyword using TextRank, WordRank, etc. Additionally, the text analysis unit 241 may derive first and second core keywords, respectively, using topic modeling. Here, topic modeling may refer to a text mining technique for extracting key keywords from text included in a post or comments included in feedback information.

The text analysis unit 241 may determine the second key keyword based on the positive rate, negative rate, and neutral rate of the feedback information. For example, the text analysis unit 241 determines the second core keyword when the positive rate or neutral rate is greater than or equal to the threshold based on the positive rate, negative rate, and neutral rate for the feedback information. In contrast, the negative rate is If it is above the threshold, the second core keyword may not be derived.

The image analysis unit 242 may analyze the image to identify a labeling target included in the image. For example, the image analysis unit 242 may identify a labeling target from an image of a post using a labeling target identification model.

The image analysis unit 242 may derive labeling-related keywords related to the identified labeling target. For example, the image analysis unit 242 may derive, for example, dog, cat, bicycle, laptop, etc., as labeling-related keywords in relation to the labeling target included in the image.

Hereinafter, the process of analyzing the text of the post to classify the sentence type and analyzing the image of the post to derive labeling-related keywords will be described through FIGS. 3A and 3B.

3A and 3B are exemplary diagrams for explaining a process of analyzing text and images included in a post according to an embodiment of the present invention.

Referring to FIG. 3A, the text analysis unit 241 may analyze the text corresponding to the title and body content of the post and classify the sentence type of the post as either a comment or a question. At this time, the text analysis unit 241 uses an intent classification algorithm to classify the intent based on the purpose or confirmation of the sentences included in the text, and then classifies the sentence type of the post as either a comment sentence or a question sentence. Can be classified. Here, the intention classification algorithm may be to classify, for example, into classes such as sales, purchase, information sharing, and advertising.

For example, if the text included in the post consists of the sentence “I caught a Dageumbari in Jeju Island,” the text analysis unit 241 will determine ‘Jeju Island-Purpose/In-Action and Status/Dageumbari-Purpose/Caught-Intention. After analyzing sentences such as '->Confirmation/.-Period (statement, request for command)', the post can be classified as a descriptive sentence (300).

For another example, if the text included in the post consists of the sentence 'What kind of fish is it?', the text analysis unit 241 may ask 'What-intention->Question/Fish-Purpose/Is-Intention->Question/? After analyzing the sentence as '-question mark->question', the post can be classified as a question (310).

For another example, if the text included in the post consists of the sentence 'Please give me one ice cream cake,' the text analysis unit 241 will analyze 'Ice cream-purpose condition/cake-purpose/one-quantity/please-intention- After analyzing sentences such as '>purchase/.-period (description, command, solicitation)', the intention of the post can be classified as purchase (320).

For another example, if the text included in the post consists of the sentence ‘What is the breed of the second dog?’, the text analysis unit 241 may use the sentence ‘Second-order/puppy-purpose, condition/breed-purpose/ After analyzing the sentence as ‘what happens-intention->question/?-question mark->question’, the post can be classified as a question (330).

Referring to FIG. 3B, the image analysis unit 242 may analyze the image to identify a labeling target included in the image and derive labeling-related keywords related to the identified labeling target. For example, the image analysis unit 242 inputs the image included in the post into the labeling target identification model through the object classification algorithm of (a) to classify the labeling target included in the image as a person 340, a sheep, etc. (341) and wolf (342), and after determining the image of the post through the Genetic Object Detection process in (b), a bounding box is displayed on the labeling object included in the image. And, after performing the semantic segmentation process of (c), the remaining background except the selected labeling target can be cropped through the object instance segmentation process of (d).

Through this process, the image analysis unit 242 can derive labeling-related keywords related to the labeling target, such as 'person', 'sheep', and 'wolf'.

Returning to FIG. 2 , the labeling unit 250 may perform labeling of the labeling target based on whether or not there is a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword. At this time, the labeling performing unit 250 may perform labeling of the labeling target based on the sentence type of the classified post. The process of labeling the labeling object will be described in detail with reference to FIGS. 4 to 7B.

Figure 4 is an exemplary diagram illustrating a process of performing additional labeling on a labeling target when the sentence type of a post corresponds to a descriptive sentence according to an embodiment of the present invention.

Referring to FIG. 4, if the sentence type of the post corresponds to a descriptive sentence and the labeling-related keywords, first core keyword, and second core keyword all match, the labeling performing unit 250 applies a label to the labeling target area corresponding to the image. Additional labeling of the labeling target may be performed based on at least one of the first core keyword and the second core keyword.

For example, assume that a post 400 including text 401 including a title and body content and an image 402 is registered from the first user terminal 110.

The text analysis unit 241 may analyze the text 401 of the post 400 and classify the sentence type of the post 400 as a 'commentary sentence'. For example, the text analysis unit 241 may analyze the text 401 of the post 400 and classify the sentence type as a 'commentary sentence' based on the period included in the sentence of the post 400.

The text analysis unit 241 inputs the text 401 of the post 400 into the sentiment analysis model to classify the text as 'positive', and selects the first key keyword from the text 401 as 'sports fold, cat' ), can be derived as ‘love’.

The image analysis unit 242 inputs the image 402 of the post 400 into the labeling object identification model to identify the labeling object included in the image 402, and sets the labeling-related keyword related to the identified labeling object to 'cat'. It can be derived as: At this time, the image analysis unit 242 may adjust the weight for identifying the labeling target based on whether metadata is included in the image 402.

At this time, in the post 400, the related keywords between the labeling-related keyword derived from the image 402 and the first core keyword derived from the text 401 may be 'Scottish fold' and 'cat'.

The text analysis unit 241 inputs a plurality of comments 411 included in the feedback information 410 into an emotion analysis model, classifies each of the plurality of comments 411 into positive, negative, and neutral, and then classifies the feedback information 410 into positive, negative, and neutral. ), the second key keyword can be derived by analyzing the comments (411) included in the comment (411). For example, the text analysis unit 241 is 'positive-Scottish fold' for user 1's comment, 'positive-doll' for user 2's comment, 'positive-X' for user 3's comment, and user 4 Multiple comments (411) were made, such as 'Positive - Niyang, Scottish Fold' to the comment of ', 'Positive - After classifying comments as positive, negative, and neutral, comment-related keywords can be derived from each comment. At this time, the text analysis unit 241 may convert 'niyang' in user 4's comment to 'cat' and 'Scottish fold' to 'Scottish fold' through similar keyword conversion.

Afterwards, the text analysis unit 241 determines that the positive and neutral ratios among the total '6' comments are '6', so the second key keywords are 'Scottish fold, doll, charisma, niyang, Scottish fold, cartilage, It can be derived as ‘arthritis’.

The labeling execution unit 250 determines that the sentence type of the post corresponds to a descriptive sentence, the labeling-related keyword is 'cat', the first core keyword is 'cat', 'Scottish fold', and 'love', and the second core keyword is 'cat'. If all keywords match 'cat' based on the labeling-related keywords derived from the image as 'Scottish Fold, cat', 'cat' is excluded from automatic labeling, and then the labeling target area corresponding to the image (labeling target on the image) is selected. Additional labeling may be performed on the labeling target, for example, 'Scottish fold', based on at least one of the first core keyword and the second core keyword in the designated area.

In the past, when trying to identify a labeling target by analyzing an image, in most cases only the object was judged because the training data set was not sufficient. For example, when labeling objects such as 'cat', 'fish', and 'car' are identified through images, it has the disadvantage that it is difficult to identify the breed and model name related to the labeling object.

However, the present invention uses a first key keyword derived from text related to the title and body content included in the post and a second key keyword derived from at least one comment included in the feedback information, for example, included in the post. Additional labeling can be performed on the labeling target, such as the breed of 'cat' being 'Scottish Fold'.

Figure 5 is an exemplary diagram for explaining a process of labeling a labeling target included in a post when the sentence type of the post corresponds to a descriptive sentence according to an embodiment of the present invention.

Referring to FIG. 5, if the sentence type of the post corresponds to a descriptive sentence and the labeling-related keyword matches the first core keyword and does not match the second core keyword, the labeling performing unit 250 performs the labeling function based on the second core keyword. Labeling can be performed on the labeling target.

For example, assume that a post 500 including text 501 including a title and body content and an image 502 is registered from the first user terminal 110.

The text analysis unit 241 may analyze the text 501 of the post 500 and classify the sentence type of the post as a 'commentary sentence'. For example, the text analysis unit 241 may analyze the text 501 of the post 500 and classify the sentence type as a 'commentary sentence' based on the period included in the sentence of the post 500.

The text analysis unit 241 inputs the text 501 of the post 500 into the emotion analysis model to classify the text as 'neutral', and selects the first core keyword from the text 501 as 'Seohae, Dageumbari, Jeju Island, Seohae'. , Muan, Haeseon-myeon, boat fishing’ can be derived.

The image analysis unit 242 inputs the image 502 of the post 500 into the labeling object identification model to identify the labeling object included in the image 502, and inputs the labeling-related keyword related to the identified labeling object into 'fish'. It can be derived as:

At this time, in the post 500, the matching keyword between the labeling-related keyword derived from the image 502 and the first core keyword derived from the text 501 may be 'dageumbari'.

The text analysis unit 241 inputs a plurality of comments 511 included in the feedback information 510 into an emotion analysis model, classifies each of the plurality of comments 511 into positive, negative, and neutral, and then classifies the feedback information 510 into positive, negative, and neutral. ), the second key keyword can be derived by analyzing the comments (511) included in the comment (511). For example, the text analysis unit 241 responds to User 1's comment with 'Negative - Dageumbari', to User 2's comment with 'Negative - Redbari, raw fish, 20,000 won', and to User 3's comment with 'Positive - Classification of multiple comments (511) into positive, negative, and neutral, such as 'Perciformes, Pteridae, Redbari', 'Positive - Redbari' for user 4's comment, 'Neutral - Jabari, Dageumbari' for user 5's comment, etc. After that, the second core keyword can be derived. At this time, the text analysis unit 241 determines that since User 1's comment is negative, the negative keyword 'Dageumbari' is excluded from the keyword, and although User 2's comment is negative, it is related to the fish identified from the image. Since it contains the word 'redbari', it is derived as a related keyword for the comment. Since the comments of user 3 and user 4 are positive, 'redbari' is derived as a related keyword, and user 5's comment is neutral. Therefore, ‘Jabari, Dageumbari’ can be derived as comment-related keywords.

Afterwards, the text analysis unit 241 can derive the second key keyword as 'red bari' because the positive and neutral ratios among the total '5' comments are '3'.

The labeling executing unit 250 determines that the sentence type of the post corresponds to a descriptive sentence, the labeling-related keyword is 'fish', the first core keyword is 'dageumbari', and the second core keyword is 'redbari', thus, If the sentence type of the post corresponds to a descriptive sentence and the labeling-related keyword matches the first core keyword and does not match the second core keyword, the labeling performing unit 250 performs labeling based on the second core keyword, 'redbari'. Labeling of objects can be performed.

In other words, if the keywords between the text related to the title and body content included in the post and the comments included in the feedback information are different from each other, labeling is performed based on the second core keyword, so that the labeling target is determined by the user. In case of a misjudgment, the labeling of the labeling target can be corrected using collective intelligence through comments.

FIGS. 6A and 6B are exemplary diagrams for explaining a process of labeling a labeling target included in a posting when the sentence type of the posting corresponds to a question according to an embodiment of the present invention.

Referring to FIG. 6A, assume that a post 600 including text 601 including a title and body content and an image 602 is registered from the first user terminal 110.

The text analysis unit 241 may analyze the text 601 of the post 600 and classify the sentence type of the post 600 as a 'question'. For example, the text analysis unit 241 may analyze the title of the post 600 and classify the sentence type as 'question' based on the question mark included in 'What is this bicycle model?'

The text analysis unit 241 may input the text 601 of the post 600 into an emotion analysis model to classify the text as 'neutral' and derive the first key keyword as 'bicycle' from the text 601. .

Here, briefly referring to FIG. 6B, the image analysis unit 242 inputs the image 602 of the post 600 into the labeling object identification model and labels the labeling object included in the image 602 as 'bicycle 620'. ', 'person 1' (621, person1), 'person 2' (622, person2), and the labeling-related keyword related to the identified labeling target can be derived as 'bicycle'.

At this time, in the post 600, the related keyword between the labeling-related keyword derived from the image 602 and the first core keyword derived from the text 601 may be 'bicycle'.

Returning to FIG. 6A, the text analysis unit 241 inputs a plurality of comments 611 included in the feedback information 610 into an emotion analysis model and classifies each of the plurality of comments 611 into positive, negative, and neutral. Afterwards, the second key keyword can be derived by analyzing the comment 611 included in the feedback information 610. For example, the text analysis unit 241 may use 'Neutral-Scott Scale Red' for user 1's comment, 'Positive-mountain bike/Scott scale/highest peak' for user 2's comment, and ' Comment-related keywords such as 'neutral-red scale' and 'positive-scale, scott, hardtail, mountain bike, cost-effectiveness' can be derived from user 4's comment.

At this time, the text analysis unit 241 derives the 'Scott scale' associated with the bicycle identified from the image because User 1's comment corresponds to neutral, and User 2's comment corresponds to positive, and the bicycle identified from the image is derived. Derive 'Mountain bike, Scott scale' related to bicycles, and since user 3's comment corresponds to neutral, 'red scale' related to bicycle identified from the image is derived, and user 4's comment corresponds to positive. Therefore, 'scale/scott/mountain bike/scott scale' related to the bicycle identified from the image can be derived.

Afterwards, the text analysis unit 241 can derive the second key keyword as 'Scott scale' because the positive and neutral ratios among the total '4' comments correspond to '4'.

The labeling performing unit 250 operates when the sentence type of the post corresponds to a question, the labeling-related keyword is 'bicycle', the first core keyword is 'bicycle', and the second core keyword is derived as 'Scott scale'. , Based on the second key keyword, 'Scott scale', labeling can be performed on the labeling target area (bicycle area on the image) corresponding to the image with 'Scott scale'. At this time, the bicycle area on the image is labeled with 'bicycle/Scott scale', so it can be used as learning data later.

On the other hand, if the sentence type of the post corresponds to a question sentence and the positive rate for the feedback information is more than the threshold, the labeling performing unit 250 applies the labeling target to the labeling target area corresponding to the image based on the first core keyword. Labeling can be performed.

For example, let's assume that the title of the post 600 includes "What model is this bike?" and the body content includes the question "Is this a Scott scale?" At this time, the text analysis unit 241 can derive the first key keywords as 'bicycle' and 'Scott scale' from the text of the post 600, and the labeling-related keyword as 'bicycle' from the image 602. there is. Afterwards, when the text analysis unit 241 has positive comments such as “yes,” “that’s right,” or “it looks like that” in the comments 611 of the feedback information 610 above the threshold, the labeling unit 250 selects the image. Labeling of the labeling target may be performed based on the first core keyword, 'Scott scale', in the labeling target area corresponding to .

FIGS. 7A and 7B are exemplary diagrams for explaining a process of not performing labeling on a labeling target included in a posting when the sentence type of the posting corresponds to a question according to an embodiment of the present invention.

Referring to FIG. 7A, assume that a post 700 including text 701 including a title and body content and an image 702 is registered from the first user terminal 110.

The text analysis unit 241 analyzes the text of the post 700 and, since the title of “Which office laptop do you like?” includes a question mark, the sentence type of the post 700 can be classified as a ‘questionnaire’.

The text analysis unit 241 inputs the text 701 of the post 700 into an emotion analysis model to classify the text as 'neutral' and selects the first core keyword from the text 701 as 'laptop/excel/powerpoint'. It can be derived as ‘/Netflix/YouTube’.

Here, briefly referring to FIG. 7B, the image analysis unit 242 inputs the image 702 of the post 700 into the labeling object identification model and selects the labeling object included in the image 702 as 'dog 1' (720, dog1), 'dog2'(721, dog2), and 'cat(722, cat), and labeling-related keywords related to the identified labeling target can be derived as 'dog' and 'cat'.

At this time, there is no matching keyword between the labeling-related keywords 'dog' and 'cat' derived from the image 702 of the post 700 and the first core keyword derived from the text 701 of the post 700. You can check it.

Returning to FIG. 7A, the text analysis unit 241 inputs a plurality of comments 711 included in the feedback information 710 into an emotion analysis model and classifies each of the plurality of comments 711 into positive, negative, and neutral. Afterwards, the second key keyword can be derived by analyzing the comment 711 included in the feedback information 710. For example, the text analysis unit 241 derives ‘Positive-Lenovo/IdeaPad/Ryzen’ for user 1’s comment, ‘positive-LG/gram’ for user 2’s comment, and user 3 'Neutral - You can.

At this time, the text analysis unit 241 determines that the comments of User 1, User 2, and User 4 are positive, and the comments of User 3 and User 5 are neutral, but the keywords extracted from each comment are keywords related to the image. Since does not exist, the second core keyword may not be derived.

The labeling performing unit 250 determines that the sentence type of the post corresponds to a question, the labeling-related keywords are 'dog' and 'cat', and the first key keyword is 'laptop/Excel/Powerpoint/Netflix/YouTube'. If the second core keyword does not exist, there is no relationship between the image 702 of the post 700 and the first core keyword and the second core keyword, so labeling may not be performed on the labeling target on the image.

Returning to FIG. 2 , the certified post determination unit 260 may determine that the post is a certified post for the labeling target based on whether or not there is a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword. Here, the process of determining a post as an authentication post for a labeling target will be briefly explained with reference to FIG. 3A.

The authentication post determination unit 260 calculates a first score based on the degree of relationship between the labeling-related keyword and the first core keyword, and calculates a second score based on the degree of relationship between the labeling-related keyword and the second core keyword, A third score can be calculated based on the ratio of the number of views and the number of preferences. Here, since the ratio of the number of views and preferences is 232/12012, it may correspond to 1% or more. If the ratio of the number of views and preferences is 1/12012, it may correspond to 0.01% or less. Additionally, the authentication post determination unit 260 may calculate the fourth score by further considering the number of reports compared to the number of views.

The authenticated post determination unit 260 calculates an authentication score for the post based on the first to third scores, and when the authentication score for the post is greater than or equal to a threshold, it may determine the post to be a certified post for the labeling target. Here, the authentication post determination unit 260 may calculate the authentication score for the post using, for example, Equation 1 below.

Referring to Equation 1, W may mean the post weight, x may mean the score for the post, y may mean the segmentation analysis score, and w may mean the detailed content weight.

For example, the weight for posts (W0=0.3), the weight for comments (W1=0.6), the weight for likes compared to views (W2=0.1), and the weight for reports compared to views (W3=0.2). Let's assume it's set up.

The authentication post determination unit 270 may calculate the first score based on the degree of relevance between the labeling-related keyword and the first core keyword. For example, the certified post judgment unit 270 calculates the score for the degree of relevance between the labeling-related keyword and the first core keyword as 0.8374, and the detailed weight (w0=0.7) for this and the score for the post classified as positive is calculated as 0.91, and considering the detailed weight (1-w0=0.3), the first score can be calculated as '0.85918' through (0.3*0.7*0.8374+0.3*0.3*0.91).

Additionally, the authentication post determination unit 270 may calculate the second score based on the degree of relevance between the labeling-related keyword and the second core keyword. For example, the authentication post judgment unit 270 calculates the score for the degree of relevance between the labeling-related keyword and the second core keyword as 0.97, based on the detailed weight (w1 = 0.6) and the positive rate of the feedback information. The positive score is calculated as 0.83, and considering the detailed weight (1-w1=0.4), the second score can be calculated as '0.914' through (0.6*0.6*0.97)+(0.6*0.4*0.83) .

Additionally, the authentication post determination unit 270 may calculate a third score based on the ratio of the number of views and the number of preferences. At this time, the detailed weight (w2) may be 0.1. For example, the authentication post determination unit 270 may derive the third score as '1' based on the ratio of the number of views and the number of preferences.

The authentication post determination unit 270 may further calculate a fourth score based on the ratio of the number of views and reports. At this time, the detailed weight (w3) may be '-0.2'. For example, the authentication post determination unit 270 may calculate the fourth score as '0' based on the ratio of the number of views and the number of reports.

Through this process, the authentication post judgment unit 270 sets the authentication score for the post to '0.906154' (0.85918*0.3+0.914*0.6+0.1*1+(-0.2)*0) based on the first to third scores. It can be derived as: At this time, the authentication post determination unit 270 may determine that the post is a certified post for the labeling target because the authentication score for the post is greater than or equal to the threshold of 0.7.

This automatic image labeling server 100 may be executed by a computer program stored in a medium containing a sequence of instructions for automatically labeling images using feedback from posts containing images. When executed by a computing device, the computer program registers a post containing an image and text from the first user terminal 110, receives feedback information related to the post from a plurality of second user terminals 120, and sends the text. Analysis to derive first key keywords from the text, analysis of feedback information to derive second key keywords, analysis of images to identify labeling objects included in the image, and derivation of labeling-related keywords related to the identified labeling objects. and may include a sequence of commands to perform labeling of the labeling target based on whether or not there is a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword.

Figure 8 is a flowchart of a method of automatically labeling images using feedback from posts containing images in an image automatic labeling server according to an embodiment of the present invention. Referring to FIG. 8, the method of automatically labeling images using feedback from posts containing images in the image automatic labeling server 100 includes steps processed in time series according to the embodiments shown in FIGS. 1 to 7B. Includes. Therefore, even if the content is omitted below, it also applies to the method of automatically labeling images using feedback from posts containing images performed by the automatic image labeling server 100 according to the embodiment shown in FIGS. 1 to 7B.

In step S810, the automatic image labeling server 100 may receive a post including an image and text from the first user terminal 110.

In step S820, the automatic image labeling server 100 may receive feedback information related to the posting from the plurality of second user terminals 120.

In step S830, the image automatic labeling server 100 may analyze the text and derive a first key keyword from the text.

In step S840, the image automatic labeling server 100 may analyze the feedback information and derive a second key keyword.

In step S850, the image automatic labeling server 100 may analyze the image to identify a labeling target included in the image and derive labeling-related keywords related to the identified labeling target.

In step S860, the image automatic labeling server 100 may perform labeling of the labeling target based on whether or not there is a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword.

In the above description, steps S810 to S860 may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present invention. Additionally, some steps may be omitted or the order between steps may be switched as needed.

The method of automatically labeling images using feedback from posts containing images performed in the image automatic labeling server described with reference to FIGS. 1 to 8 includes a computer program stored in a medium executed by a computer or instructions executable by a computer. It can also be implemented in the form of a recording medium containing. In addition, the method of automatically labeling images using feedback from posts containing images performed by the image automatic labeling server described with reference to FIGS. 1 to 8 can also be implemented in the form of a computer program stored in a medium executed by a computer. It can be.

Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include computer storage media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as single may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

100: Image automatic labeling server
110: Learning Department
120: Post register
130: Feedback information receiving unit
140: analysis department
141: Text analysis unit
142: Image analysis unit
150: Labeling execution unit
160: Verification post judgment unit

Claims (20)

In a server that automatically labels images using feedback from posts containing images,
A post register that receives posts including images and text from the first user terminal;
a feedback information receiving unit that receives feedback information related to the posting from a plurality of second user terminals;
a text analysis unit that analyzes the text to derive first key keywords from the text and analyzes the feedback information to derive second key keywords; and
An analysis unit including an image analysis unit that analyzes the image to identify a labeling target included in the image and derives labeling-related keywords related to the identified labeling target; and
A labeling performing unit that performs labeling on the labeling target based on a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword.
Including,
The text analysis unit analyzes the text and classifies the sentence type of the post as either a descriptive sentence or a question sentence,
The labeling performing unit performs labeling on the labeling target based on the sentence type of the classified post.
According to claim 1,
The text analysis unit inputs each of the text and the feedback information into an emotion analysis model to classify each of the text and the feedback information into one of positive, negative, and neutral.
According to claim 2,
The text analysis unit determines the second key keyword based on the positive rate, negative rate, and neutral rate for the feedback information.
According to claim 1,
The feedback information includes at least one of comments, views, preferences, and reports for the post.
According to claim 4,
A certified post determination unit that determines the post as a certified post for the labeling target based on a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword.
Further comprising: an image automatic labeling server.
In a server that automatically labels images using feedback from posts containing images,
A post register that receives posts including images and text from the first user terminal;
a feedback information receiving unit that receives feedback information related to the posting from a plurality of second user terminals;
a text analysis unit that analyzes the text to derive first key keywords from the text and analyzes the feedback information to derive second key keywords; and
An analysis unit including an image analysis unit that analyzes the image to identify a labeling target included in the image and derives labeling-related keywords related to the identified labeling target;
a labeling performing unit that performs labeling on the labeling target based on a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword; and
A certified post determination unit that determines the post as a certified post for the labeling target based on a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword.
Including,
The feedback information includes at least one of comments, views, preferences, and reports for the post,
The authentication post determination unit calculates a first score based on the degree of relationship between the labeling-related keyword and the first core keyword, and calculates a second score based on the degree of relationship between the labeling-related keyword and the second core keyword. , An image automatic labeling server that calculates a third score based on the ratio of the number of views and the number of preferences.
According to claim 6,
The authentication post determination unit calculates an authentication score for the post based on the first to third scores,
When the authentication score for the post is greater than or equal to a threshold, the automatic image labeling server determines the post as an authenticated post for the labeling target.
delete According to claim 1,
If the sentence type of the post corresponds to the descriptive sentence and the labeling-related keyword, the first core keyword, and the second core keyword all match, the labeling performing unit applies the first keyword to the labeling target area corresponding to the image. An automatic image labeling server that performs additional labeling on the labeling target based on at least one of a core keyword and the second core keyword.
According to claim 1,
If the sentence type of the post corresponds to the descriptive sentence and the labeling-related keyword matches the first core keyword and does not match the second core keyword, the labeling performing unit selects the labeling target based on the second core keyword. An image automatic labeling server that performs labeling.
According to clause 9,
If the sentence type of the post corresponds to the question and the positive rate for the feedback information is more than a threshold, the labeling performing unit assigns the labeling target to the labeling target area corresponding to the image based on the first key keyword. An image automatic labeling server that performs labeling.
According to claim 1,
It further includes a learning unit that learns a labeling object identification model to identify the labeling object using training data including already collected images and label information of the labeling object,
The image analysis unit identifies the labeling object from the image of the post using the labeling object identification model.
In a method of automatically labeling images using feedback from posts containing images in an image automatic labeling server,
Receiving a post including an image and text from a first user terminal;
Receiving feedback information related to the posting from a plurality of second user terminals;
Analyzing the text and deriving a first key keyword from the text;
Analyzing the feedback information to derive second key keywords;
Analyzing the image to identify a labeling target included in the image and deriving labeling-related keywords related to the identified labeling target; and
performing labeling on the labeling target based on whether or not there is a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword; and
Analyzing the text and classifying the sentence type of the post as either a descriptive sentence or a question sentence
Including,
The step of performing labeling on the labeling object is,
An image automatic labeling method comprising the step of performing labeling on the labeling target based further on the sentence type of the classified post.
According to claim 13,
The automatic image labeling method further comprising determining the post as an authenticated post for the labeling target based on a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword. .
In a method of automatically labeling images using feedback from posts containing images in an image automatic labeling server,
Receiving a post including an image and text from a first user terminal;
Receiving feedback information related to the posting from a plurality of second user terminals;
Analyzing the text and deriving a first key keyword from the text;
Analyzing the feedback information to derive second key keywords;
Analyzing the image to identify a labeling target included in the image and deriving labeling-related keywords related to the identified labeling target; and
Comprising the step of performing labeling on the labeling target based on whether or not there is a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword,
Further comprising determining the post as an authenticated post for the labeling target based on whether or not there is a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword;
The feedback information includes at least one of comments, views, preferences, and reports for the post,
The step of determining the authentication post is,
calculating a first score based on a degree of relevance between the labeling-related keywords and the first core keyword;
calculating a second score based on a degree of relevance between the labeling-related keyword and the second core keyword;
calculating a third score based on the ratio of the number of views and the number of preferences;
calculating an authentication score for the post based on the first to third scores; and
When the authentication score for the post is greater than or equal to a threshold, determining the post as an authenticated post for the labeling target.
delete According to claim 13,
The step of performing labeling on the labeling object is,
If the sentence type of the post corresponds to the descriptive sentence, and the labeling-related keyword, the first core keyword, and the second core keyword all match, the first core keyword and the second core keyword are placed in the labeling target area corresponding to the image. An image automatic labeling method further comprising performing additional labeling on the labeling object based on at least one of the second key keywords.
According to claim 17,
The step of performing labeling on the labeling object is,
If the sentence type of the post corresponds to the descriptive sentence and the labeling-related keyword matches the first core keyword and does not match the second core keyword, labeling the labeling target is performed based on the second core keyword. A method for automatically labeling images, further comprising the steps of:
According to claim 18,
The step of performing labeling on the labeling object is,
If the sentence type of the post corresponds to the question and the positive rate for the feedback information is more than a threshold, labeling the labeling target is performed based on the first key keyword in the labeling target area corresponding to the image. An image automatic labeling method further comprising the step of:
A computer program stored on a medium comprising a sequence of instructions for automatically labeling images using feedback from posts containing images, comprising:
When the computer program is executed by a computing device,
Register a post containing images and text from the first user terminal,
Receiving feedback information related to the posting from a plurality of second user terminals,
Analyzing the text to derive first key keywords from the text, and deriving second key keywords by analyzing the feedback information,
Analyzing the image to identify a labeling object included in the image, deriving labeling-related keywords related to the identified labeling object,
Perform labeling on the labeling target based on a match between at least one of the first core keyword, the second core keyword, and the labeling-related keyword,
Contains a sequence of commands to analyze the text and classify the sentence type of the post as either a descriptive sentence or a question sentence,
The command to perform labeling on the labeling object is
A computer program stored on a medium, comprising a sequence of instructions for performing labeling of the labeling object further based on the sentence type of the classified post.