KR20230083966A - Teeth condition analyzing method and system based on deep learning - Google Patents

Abstract

The present invention relates to a teeth condition analyzing method and system based on deep learning and can provide a tooth condition analyzing method based on deep learning, which comprises: a receiving step which receives a tooth image from a user terminal; a collecting step which collects a tooth image database from a dental clinic and web crawling; and an analyzing step which learns the collected tooth image database, and based on a learning result, analyzes a teeth condition of a user from the tooth image received in the receiving step. The analyzing step comprises: a decayed tooth analyzing step which uses an object detection method to identify a decayed tooth from the tooth image; and a plaque analyzing step which classifies the tooth image by pixel unit to identify a plaque. According to the present invention, a user can identify their decayed teeth and plaque and their locations to prevent oral cavity health from worsening.

Description

Deep learning based tooth condition analysis method and system {Teeth condition analyzing method and system based on deep learning}

The present invention relates to a deep learning-based dental condition analysis method and system, and more particularly, a deep learning-based dental condition analysis method that allows the user to determine the dental condition by determining the presence and location of tooth decay and calculus of a user. and system.

Oral diseases can have a high preventive effect through proper preventive treatment and steady management, but many patients with oral diseases are still unaware of oral prevention, such as not consulting a doctor or requesting treatment unless there is a problem with oral health. this is lacking.

In addition, in predicting the patient's current oral condition, accurate test and diagnosis results and objective numerical information are required, not the subjective opinions of dentists or dental hygienists, but there is no system that objectively calculates the oral health condition. .

In particular, tooth decay is one of the diseases that many modern people suffer from, and if not treated early, tooth extraction and nerve treatment are required. cause

It is important to identify and treat diseases such as tooth decay and tartar on a regular basis and at an early stage. Many people struggle with oral health management because it is difficult to determine whether they have cavities or tartar.

Therefore, the state of these oral diseases is diagnosed in advance, and based on this, various methods for improving oral health and preventing oral diseases have been studied and proposed.

However, conventional techniques have a problem in that it is difficult for the user to grasp the state of his/her teeth in real time.

Prior art includes Korean Patent Publication No. 10-2017-0050467 "User-customized oral care service providing method and oral care service providing server".

Therefore, in order to solve the above problems, the present invention aims to provide a deep learning-based tooth condition analysis method and system that determines the presence and location of the user's tooth decay and calculus and allows the user to grasp the tooth condition. .

In order to solve the above problems, a deep learning-based dental condition analysis method according to an embodiment of the present invention includes a receiving step of receiving a tooth image from a user terminal; A collection step of collecting a tooth image database from dentistry and web crawling; And an analysis step of learning the collected tooth image database and analyzing the user's tooth condition from the tooth image received in the receiving step based on the learning result, wherein the analysis step includes an object detection method from the tooth image. It may include a tooth decay analysis step of discriminating tooth decay by using the teeth and a calculus analysis step of discriminating calculus by classifying the tooth image in pixel units.

In addition, the analysis step is characterized by repeating the learning on the tooth image database when the learning result for accuracy is less than a preset value, and analyzing the user's tooth condition when it exceeds the preset value.

In addition, the cavity analysis step may include a cavity labeling step of securing a cavity data set by labeling the collected tooth image database; a caries determination pre-processing step of increasing the data amount of the caries data set and adjusting the resolution; A caries discrimination learning step of constructing a caries discrimination model by learning the preprocessed caries data set, and a caries discrimination step of discriminating caries from the tooth image through the caries discrimination model.

In addition, the dental caries labeling step is characterized in that the caries data set is secured in three classes by labeling the collected tooth image database with caries, gold, and amalgam, respectively.

In addition, the calculus analysis step may include a calculus labeling step of securing a calculus data set by storing an annotation indicating calculus from the collected tooth image database; Calculus determination pre-processing step of adjusting the resolution of the calculus data set marked with the annotation; It may include a calculus discrimination learning step of constructing a calculus discrimination model by learning the preprocessed calculus data set, and a calculus discrimination step of discriminating calculus from the tooth image through the calculus discrimination model.

In addition, the dental condition analysis system based on deep learning according to an embodiment of the present invention includes a user terminal receiving a tooth image from a user; It may include a server that receives a tooth image from the user terminal, analyzes the tooth state based on deep learning, generates tooth analysis information, and transmits the generated tooth analysis information to the user terminal.

In addition, the server, a receiving unit for receiving a tooth image; A collection unit that collects a tooth image database from dentistry and web crawling and an analysis unit that learns the collected tooth image database and analyzes a user's tooth condition from the tooth image received from the receiver based on a learning result, wherein the The analysis unit may include a tooth decay unit that determines tooth decay using an object detection method from the tooth image, and a calculus unit that classifies the tooth image in pixel units to determine calculus.

In addition, the tooth decay unit may include a tooth decay labeling unit for securing a tooth decay data set by labeling the collected tooth image database; a caries determination pre-processing unit increasing the data amount of the caries data set and adjusting the resolution; It may include a caries discrimination learning unit for constructing a caries discrimination model by learning the preprocessed caries data set, and a caries discrimination unit for discriminating caries from the tooth image through the caries discrimination model.

In addition, the calculus unit may include a calculus labeling unit for securing a calculus data set by storing annotations indicating calculus from the collected tooth image database; Calculus determination pre-processing unit for adjusting the resolution of the calculus data set marked with the annotation; It may include a calculus discrimination learning unit for learning the preprocessed calculus data set to build a calculus discrimination model, and a calculus discrimination unit for discriminating calculus from the tooth image through the calculus discrimination model.

According to the present invention, the deep learning-based dental condition analysis method and system provides tooth analysis information to the user, allowing the user to identify the presence and location of tooth decay and calculus, thereby preventing oral health deterioration in advance. there is

1 is a flowchart of a deep learning-based tooth condition analysis method according to an embodiment of the present invention.
Figure 2 is a flow chart of the caries analysis step of Figure 1;
Figure 3a is an exemplary view showing a tooth image, Figure 3b is an exemplary view showing the caries detected through the caries analysis step of FIG.
Figure 4 is a flow chart of the calculus analysis step of Figure 1;
5 is a configuration diagram of a tooth condition analysis system based on deep learning according to an embodiment of the present invention.

Hereinafter, the description of the present invention with reference to the drawings is not limited to specific embodiments, and various transformations may be applied and various embodiments may be applied. In addition, the content described below should be understood to include all conversions, equivalents, or substitutes included in the spirit and scope of the present invention.

In the following description, terms such as first and second are terms used to describe various components, and are not limited in meaning per se, and are used only for the purpose of distinguishing one component from another.

Like reference numbers used throughout this specification indicate like elements.

Singular expressions used in the present invention include plural expressions unless the context clearly dictates otherwise. In addition, terms such as "include", "include" or "have" described below are intended to designate that features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist. should be construed, and understood not to preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 5 attached.

1 is a flowchart of a tooth condition analysis method based on deep learning according to an embodiment of the present invention, FIG. 2 is a flowchart of a tooth decay analysis step in FIG. 1, FIG. 3A is an exemplary view showing a tooth image, and FIG. 3B is a diagram It is an exemplary view showing the tooth decay detected through the tooth decay analysis step of 2, and FIG. 4 is a flow chart of the calculus analysis step of FIG.

Referring to FIG. 1 , the deep learning-based tooth condition analysis method according to an embodiment of the present invention may include a receiving step (S10), a collecting step (S20), and an analysis step (S30).

Receiving step (S10) is a step of receiving a tooth image from the user terminal (10).

At this time, the user terminal 10 is a device capable of communication, input/output of information, and a camera, preferably in the form of a smartphone, but is not limited thereto and is capable of performing the functions described above, such as a desktop PC or a laptop computer. , a form of a tablet PC, etc. is also possible. Hereinafter, for convenience of description, it will be described assuming that the user terminal 10 is a smart phone.

In addition, the user terminal 10 may be pre-installed with an application for receiving a service according to the deep learning-based tooth condition analysis system of the present invention. In the above, the application refers to a general application based on Android and iOS created based on the deep learning-based dental condition analysis system of the present invention, and as a method of providing, the user terminal 10 is a deep learning-based dental condition analysis system. It can be downloaded by accessing the server 20 or downloaded and installed through an online application market (eg, Android Market, Apple Store, online market of telecommunication companies, etc.).

The tooth image may be obtained by photographing through the user terminal 10 as described above, or may be an image previously stored in the user terminal 10, but is not limited thereto.

Collecting step (S20) is a step of collecting a tooth image database from dentistry and web crawling.

In this case, the tooth image database may refer to tooth images excluding personal information.

The analysis step (S30) is a step of learning the collected tooth image database and analyzing the user's tooth condition from the tooth image received in step S10 based on the learning result.

At this time, in the analysis step (S30), if the learning result is less than or equal to a preset value, learning on the tooth image database is repeated, and if the value exceeds the preset value, the user's tooth condition may be analyzed.

More specifically, the analysis step repeats the preprocessing step and the learning step when the value is less than or equal to a predetermined value according to the learning result, and a detailed description of this will be given below.

As such, the present invention can increase the accuracy of the analysis step (S30) by repeating the process of collecting and learning the tooth image database when the tooth image database is not sufficiently secured.

In the analysis step (S30), tooth decay and tartar are analyzed and summed from the tooth image, respectively, and tooth analysis information may be generated. To this end, the analysis step (S30) may include a cavity analysis step (S31) and a calculus analysis step (S32).

First, referring to FIG. 2, the tooth decay analysis step (S31) is a step of determining tooth decay using an object detection method from a tooth image.

Here, the object detection method is a method of simultaneously grasping localization information and image classification information.

Step S31 according to the present invention uses a YOLO (You Only Look Once) model, one of the types of 1-Stage Detector that performs location discrimination and image classification at the same time, and YOLOv5, which is fast among them, is used.

YOLO is an object detection model that can extract bounding boxes and object classes that have location information through a single network operation.

In the present invention, YOLOv5 uses CSP-DarkNet as a backbone structure, which is a part that extracts feature maps, and extracts anchor box information from the head part of the extracted feature map and finally obtains a bounding box. At this time, the loss of YOLOv5 used BCE with logits loss, which is a combination of Binary Cross Entropy (BCE) loss and sigmoid, and Focal loss to solve the class imbalance problem during learning.

More specifically, step S31 may include a cavity labeling step (S310), a cavity determination preprocessing step (S311), a cavity determination learning step (S312), and a cavity determination step (S313).

The dental caries labeling step (S310) is a step of securing a dental caries data set by labeling the collected tooth image database.

More specifically, in step S310 , caries data sets of three classes may be secured by labeling the collected tooth image database as caries, gold, and amalgam, respectively.

The caries determination preprocessing step (S311) is a step of increasing the amount of caries data set and adjusting the resolution.

Step S311 is performed to overcome the disadvantage of a small amount of dental caries data set, and augmentation techniques such as Rotation, Scale, Flip, Translate, and Gamma can be used.

In addition, in step S311, the image resolution of the dental caries data set may be set to 500 to 700 ppi, and most preferably 640 ppi, but is not limited thereto.

At this time, if the image resolution of the dental caries data set is less than 500 ppi, there may be a problem with the accuracy of caries discrimination. It is not desirable to have

In addition, in step S311, the batch size may be set to 20 to 30, and most preferably 24, but is not limited thereto.

Here, the batch size means the number of data belonging to one small group when the entire training data set is divided into several small groups.

As such, in step S311, the best caries discrimination performance was confirmed by setting the batch size to the above numerical values.

The caries discrimination learning step (S312) is a step of building a caries discrimination model by learning the preprocessed caries data set.

More specifically, step S312 can be performed by setting the value of the SGD (Stochastoc gradient descent) optimizer and the learning rate to 0.01 to 0.05, and it is most preferable to set it to 0.01, but not limited thereto.

As such, in step S312, if the value is less than or equal to a predetermined value according to the learning result of the caries discrimination model built as described above, the preprocessing process of the caries data set may be performed again by returning to step S311. Thereafter, step S312 is performed again, and when the learning result exceeds a predetermined value, caries can be determined from the tooth image through the next step.

The caries determination step (S313) is a step of determining the presence and location of caries using the YOLOv5 model described above.

More specifically, in step S313, caries can be determined from the tooth image through a caries discrimination model, and the user can determine the presence and location of caries.

In this way, caries analysis step (S31) according to the present invention, as shown in Figure 3, can detect caries from the tooth image.

In this way, the caries discrimination model built through the above steps can measure the learning result for accuracy using Precision, mAP (mean Average Precision), and IoU (Intersection over Union), which are used as accuracy indicators in object detection. can

According to the above formula, the IoU value is TP (True Positive), FN (False Negative), FP (False Positive) and FP (False Positive) and TN (True Negative) value can be obtained.

Prediction Positive Prediction Negative Condition Positive True Positive False Positive Condition Negative False Positive True Positive

In addition, as shown in the above formula, Recall and Precision have an inversely proportional relationship. Average Precision (AP) can be obtained by dividing Recall into 11 values from 0 to 1 and averaging the Precison values. After obtaining AP values for all classes, the mAP value can be derived by averaging.

class Precision mAP@.5 cavity 0.9 81.6% gold 0.94 91.7% amalgam 0.85 89.6% Tooth decay + gold + amalgam 0.9 87.6%

Referring to Table 2, it was confirmed that the learning results of the dental caries discrimination model according to the present invention had overall high accuracy for all classes. In the case of caries, due to the existence of various shapes, it was confirmed that the value of mAP@.5 was slightly lower than that of other classes in which the color and shape were fixed to some extent.

Referring to Figure 4, the calculus analysis step (S32) is a step of determining the calculus by classifying the tooth image in pixel units.

More specifically, in step S32, calculus was separated from the tooth image database using the Instance Segmentation model among the Image Segmentation models, and calculus was discriminated by applying the 2-Stage Detector Mask R-CNN model.

Here, the Instance Segmentation model is a model that finds all objects in the image and classifies the instances in pixel units at the same time, and Mask R-CNN is an extended model from Faster R-CNN to find instances in pixel units.

At this time, Faster R-CNN is a model that solves the need to generate about 2,000 Region Proposals in the existing R-CNN with a neural network. Faster R-CNN is a model that puts the method of generating Region Proposal itself into a network structure inside CNN, and this network is called RPN (Region Proposal Network). Through RPN, a layer performing RoI pooling and a layer extracting a bounding box can share the same characteristics. In Faster R-CNN, only Classification and Bounding Box Regression are obtained, but in Mask R-CNN, a mask can be additionally obtained. This mask value could not be obtained using ROI pooling in which the original location information of the input image is distorted in Faster R-CNN, but in Mask R-CNN, the mask value can be obtained without loss of location information through ROI align. Specifically, Fast R-CNN was a model for object detection, so it was not important to include accurate location information in ROI Pooling. Therefore, to solve this problem, Mask R-CNN uses ROI Align instead of ROI Pooling.

Therefore, in the present invention, Mask R-CNN can extract the location information of calculus through RPN (Region Proposal Network), and detect calculus accurately through slow but accurate 2-Stage Detector, which is a method of obtaining classification and mask. There are advantages to being able to

Faster R-CNN of the calculus analysis step (S32) may include "branch to predict calculus", "branch to extract the bounding box of calculus" and "branch to extract the mask value of each calculus". "Branch extracting the mask value of each calculus" can be implemented through Fully Convolutional Network (FCN).

Here, FCN reduces the size of the input image by going through several convolution layers, extracts features in the image, and then goes through a Fully-Connected Layer to transform the image. can be classified.

Step S32 may include a calculus labeling step (S320), a tartar discrimination preprocessing step (S321), a tartar discrimination learning step (S322), and a tartar discrimination step (S323).

In the calculus labeling step (S320), a calculus data set may be secured by storing an annotation indicating calculus from the collected tooth image database.

The calculus discrimination preprocessing step (S321) is a step of adjusting and fixing the resolution of the calculus data set marked with annotations to increase the accuracy of learning.

At this time, step S321 may be performed in the same manner as step S311 of cavity analysis step (S31), but is not limited thereto.

The calculus discrimination learning step (S322) is a step of building a calculus discrimination model by learning the preprocessed calculus data set.

More specifically, in step S322, the Mask R-CNN model may be learned based on a loss function including a classification loss function, a bounding box loss function, and a mask loss function. At this time, the loss function can take a method of calculating the average after obtaining the classification loss function, the bounding box loss function, and the mask loss function.

In addition, in step S322, a calculus discrimination model can be built using ResNet-101 with the Backbone structure of the Mask R-CNN model described above.

Finally, the calculus discrimination step (S323) is a step of discriminating tartar from the tooth image through the calculus discrimination model.

More specifically, in step S323, calculus is determined from the tooth image through a calculus discrimination model, and the user can determine the presence and location of calculus.

Thus, the calculus analysis step (S32) constructed as above can derive the learning result for accuracy in the same way as the cavity analysis step (S31), and a detailed description thereof will be omitted.

5 is a configuration diagram of a tooth condition analysis system based on deep learning according to an embodiment of the present invention.

Referring to FIG. 5 , a deep learning-based dental condition analysis system according to an embodiment of the present invention may include a user terminal 10 and a server 20 .

The user terminal 10 may receive a tooth image from the user. Here, since the user terminal 10 and the tooth image are the same as those described in the deep learning-based tooth condition analysis method, descriptions thereof will be omitted.

The server 20 may receive a tooth image from the user terminal, analyze the tooth state based on deep learning, generate tooth analysis information, and transmit the generated tooth analysis information to the user terminal 10 . To this end, the server 20 may include a receiving unit 21, a collecting unit 22, and an analyzing unit 23.

The receiving unit 21 may receive a tooth image from the user terminal 10 .

The collecting unit 22 may collect tooth image databases from dentistry and web crawling. In this case, the tooth image database may refer to a tooth image excluding personal information.

The analysis unit 23 learns the collected tooth image database, analyzes the user's tooth condition from the tooth image received by the receiver 21 based on the learning result, generates tooth analysis information, and transmits it to the user terminal 10. can

More specifically, the analysis unit 23 may include a cavity and calculus.

The caries unit determines caries by using an object detection method from a tooth image, and may include a caries labeling unit, a caries determination pre-processing unit, a caries determination learning unit, and a caries determination unit.

The calculus unit classifies the tooth image in pixel units to determine calculus, and may include a calculus labeling unit, a calculus determination pre-processing unit, a calculus determination learning unit, and a calculus determination unit.

A detailed description of the cavity and calculus will be omitted since it has been described in the tooth analysis method.

As such, the deep learning-based dental condition analysis method and system according to an embodiment of the present invention is an advantage that anyone can easily, quickly and accurately manage oral health through a deep learning model using YOLOv5 and Mask R-CNN that discriminate between tooth decay and calculus. there is

The embodiments of the present invention described above can be implemented by various substitutions, modifications, and changes without departing from the technical spirit of the present invention, and such implementations can be made by experts in the technical field to which the present invention belongs from the description of the above-described embodiments. If so, it is easy to implement.

10: user terminal
20: server
21: receiver
22: collection unit
23: analysis unit

Claims (9)

A receiving step of receiving a tooth image from a user terminal;
A collection step of collecting a tooth image database from dentistry and web crawling;
An analysis step of learning the collected tooth image database and analyzing the user's tooth condition from the tooth image received in the receiving step based on the learning result,
The analysis step is
A tooth decay analysis step of determining tooth decay using an object detection method from the tooth image, and
Deep learning-based dental condition analysis method comprising a calculus analysis step of classifying the tooth image in pixel units to determine calculus.
According to claim 1,
The analysis step is
Deep learning-based tooth condition analysis method, characterized by repeating learning on the tooth image database when the learning result for accuracy is less than a preset value, and analyzing the user's tooth condition when it exceeds the preset value.
According to claim 1,
The cavity analysis step,
a cavity labeling step of securing a cavity data set by labeling the collected tooth image database;
a caries determination pre-processing step of increasing the data amount of the caries data set and adjusting the resolution;
A caries discrimination learning step of building a caries discrimination model by learning the preprocessed caries data set, and
A tooth condition analysis method based on deep learning comprising a tooth decay determination step of determining tooth decay from the tooth image through the tooth decay determination model.
According to claim 3,
The cavity labeling step,
Deep learning-based tooth condition analysis method, characterized in that for securing the cavity data set in three classes by labeling the collected tooth image database with caries, gold and amalgam, respectively.
According to claim 1,
The calculus analysis step,
A calculus labeling step of securing a calculus data set by storing an annotation indicating calculus from the collected tooth image database;
Calculus determination pre-processing step of adjusting the resolution of the calculus data set marked with the annotation;
A calculus discrimination learning step of building a calculus discrimination model by learning the preprocessed calculus data set, and
Deep learning-based dental condition analysis method comprising a calculus discrimination step of discriminating calculus from the tooth image through the calculus discrimination model.
A user terminal receiving a tooth image from a user;
A server for receiving a tooth image from the user terminal, analyzing a tooth state based on deep learning, generating tooth analysis information, and transmitting the generated tooth analysis information to the user terminal;
The server,
Receiving unit for receiving a tooth image;
A collection unit for collecting tooth image databases from dentistry and web crawling; and
A deep learning-based dental condition analysis system comprising an analysis unit that learns the collected tooth image database and analyzes the user's tooth condition from the tooth image received from the receiver based on the learning result.
According to claim 6,
The analysis unit,
A tooth decay unit for determining tooth decay using an object detection method from the tooth image, and
Deep learning-based dental condition analysis system comprising a calculus unit for classifying the tooth image in pixel units to determine calculus.
According to claim 7,
The tooth decay department,
a caries labeling unit to secure a caries data set by labeling the collected tooth image database;
a caries determination pre-processing unit increasing the data amount of the caries data set and adjusting the resolution;
A caries discrimination learning unit that learns the preprocessed caries data set to build a caries discrimination model; and
A dental condition analysis system based on deep learning including a caries discriminating unit for discriminating caries from the tooth image through the caries discriminating model.
According to claim 7,
The dental calculus,
A calculus labeling unit that secures a calculus data set by storing annotations indicating calculus from the collected tooth image database;
Calculus determination pre-processing unit for adjusting the resolution of the calculus data set marked with the annotation;
A calculus discrimination learning unit that learns the preprocessed calculus data set to build a calculus discrimination model, and
Deep learning-based dental condition analysis system including a calculus discriminating unit for discriminating calculus from the tooth image through the calculus discriminating model.

Images