CN117115352A - Method and device for generating orthodontic effect preview image - Google Patents
Method and device for generating orthodontic effect preview image Download PDFInfo
- Publication number
- CN117115352A CN117115352A CN202311106119.2A CN202311106119A CN117115352A CN 117115352 A CN117115352 A CN 117115352A CN 202311106119 A CN202311106119 A CN 202311106119A CN 117115352 A CN117115352 A CN 117115352A
- Authority
- CN
- China
- Prior art keywords
- tooth
- model
- dimensional
- photo
- semantic segmentation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 230000000694 effects Effects 0.000 title claims abstract description 36
- 230000011218 segmentation Effects 0.000 claims abstract description 90
- 210000000214 mouth Anatomy 0.000 claims abstract description 19
- 238000012545 processing Methods 0.000 claims abstract description 5
- 210000004513 dentition Anatomy 0.000 claims description 45
- 230000036346 tooth eruption Effects 0.000 claims description 45
- 230000008569 process Effects 0.000 claims description 22
- 230000006870 function Effects 0.000 claims description 18
- 230000005012 migration Effects 0.000 claims description 18
- 238000013508 migration Methods 0.000 claims description 18
- 239000013598 vector Substances 0.000 claims description 17
- 230000004927 fusion Effects 0.000 claims description 11
- 230000005484 gravity Effects 0.000 claims description 10
- 230000036544 posture Effects 0.000 claims description 8
- 238000005457 optimization Methods 0.000 claims description 5
- 238000011176 pooling Methods 0.000 claims description 4
- 238000004088 simulation Methods 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 238000005429 filling process Methods 0.000 claims 1
- 210000003464 cuspid Anatomy 0.000 description 3
- 210000002455 dental arch Anatomy 0.000 description 3
- 210000004195 gingiva Anatomy 0.000 description 3
- 230000000295 complement effect Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 238000009499 grossing Methods 0.000 description 2
- 210000004283 incisor Anatomy 0.000 description 2
- 238000012805 post-processing Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 241000287196 Asthenes Species 0.000 description 1
- 206010061274 Malocclusion Diseases 0.000 description 1
- 235000004240 Triticum spelta Nutrition 0.000 description 1
- 230000001154 acute effect Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000004763 bicuspid Anatomy 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000003628 erosive effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 210000001847 jaw Anatomy 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 210000004357 third molar Anatomy 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30036—Dental; Teeth
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Geometry (AREA)
- Multimedia (AREA)
- Computer Graphics (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Dental Tools And Instruments Or Auxiliary Dental Instruments (AREA)
Abstract
The invention relates to a method and a device for generating an orthodontic effect preview image, wherein the method comprises the following steps: step S1: establishing a three-dimensional tooth template; step S2: constructing a semantic segmentation model and an intra-oral region segmentation model; step S3: sequentially processing the original front photo of the user by using the semantic segmentation model and the intra-oral area segmentation model to obtain a tooth profile with tooth position information and an intra-oral area mask; step S4: iteratively optimizing camera parameters and dental template parameters to reconstruct a user-personalized three-dimensional dental model; step S5: performing tooth arrangement and filling, and obtaining a tooth semantic segmentation map; step S6: and inputting the tooth semantic segmentation map, the mask of the oral cavity inner area and the part of the original front photo of the user corresponding to the mask of the oral cavity inner area into the trained front photo orthodontic effect preview to obtain a front smile map or flaring photo after tooth orthodontic. Compared with the prior art, the method has the advantages that the orthodontic image can be generated only by using the front photo.
Description
Technical Field
The invention relates to the field of image processing, in particular to a method and a device for generating an orthodontic effect preview image.
Background
Orthodontic is a process of straightening teeth, improving tooth appearance and relieving malocclusions, aims to align teeth and restore normal occlusion, and is mainly applicable to the situation that teeth are uneven but jawbones are normal. In the orthodontic treatment process, smile photos and intraoral flaring photos with different angles are taken, so that doctors and patients can be helped to know the treatment process, and the change and effect before and after treatment are compared. By viewing the front smile photograph, a doctor can better understand the patient's dental problems and facial features and formulate a more effective treatment plan. The front smile photo can also clearly display the tooth and the facial outline of the patient, can evaluate the treatment effect more accurately, and is convenient for doctors to adjust the treatment plan in time. In addition, before orthodontic treatment, the patient may feel spelt for the unsightly teeth, and by taking a front smile photograph, the patient can see changes and progress of his own treatment before and after, enhancing self-confidence and improving quality of life. However, frontal smile photographs are often taken by doctors at regular intervals and recorded, i.e. the patient who is not undergoing orthodontic and just beginning orthodontic treatment cannot give clear knowledge of the intended orthodontic treatment effect. Therefore, through deep learning and other technologies, according to front smile photos or intraoral flaring front images input by a user and priori knowledge of orthodontic treatment process, the oral cavity pictures and three-dimensional tooth models with orderly arranged teeth after orthodontic treatment are predicted, and the method has guiding significance for treatment of orthodontic patients and decision making of potential orthodontic patients.
Disclosure of Invention
The invention aims to provide a method and a device for generating an orthodontic effect preview image.
The aim of the invention can be achieved by the following technical scheme:
a method for generating an orthodontic effect preview image, comprising:
step S1: establishing a three-dimensional tooth template based on a digital three-dimensional tooth model obtained by intraoral scanning;
step S2: constructing a semantic segmentation model for performing tooth semantic segmentation on a mouth region of a front smile photo or a front intraoral flare photo of the single Zhang Yuanshi, and constructing an intraoral region segmentation model for segmenting an intraoral region and extracting a two-dimensional tooth contour with tooth position information;
step S3: acquiring an original front photo of a user, and sequentially processing the original front photo of the user by using a semantic segmentation model and an intra-oral area segmentation model to obtain a tooth profile with tooth position information and an intra-oral area mask, wherein the front photo is smiling photo or flaring photo;
step S4: iteratively optimizing a camera parameter and a tooth template parameter to optimize a contour matching loss function, deforming the three-dimensional tooth template to enable the tooth contour of the corresponding tooth position obtained by projection to fit the tooth contour with tooth position information so as to reconstruct a user personalized three-dimensional tooth model;
step S5: performing tooth arrangement and filling on the personalized three-dimensional tooth model of the user, and projecting the personalized three-dimensional tooth model after tooth arrangement under the same camera parameters to obtain a tooth semantic segmentation map;
step S6: and inputting the tooth semantic segmentation map, the mask of the oral cavity inner area and the part of the original front photo of the user corresponding to the mask of the oral cavity inner area into a trained front photo orthodontic effect preview model to obtain a front smile map or a flaring photo after tooth orthodontic.
The step S1 includes:
step S11: acquiring a plurality of groups of digital three-dimensional tooth models, and performing segmentation, numbering and filling operations on each digital three-dimensional tooth model to obtain a three-dimensional tooth model with tooth position information;
step S12: aligning the upper dentition and the lower dentition of each three-dimensional tooth model according to the corresponding dentition according to the gravity center of each tooth, and calculating the average gravity center of the tooth corresponding to each dentition;
step S13: performing similar transformation registration on different teeth belonging to the same tooth position by using a point cloud registration algorithm of consistent point drift, and recording the tooth pose and corresponding point relation;
step S14: and establishing a statistical shape model of each tooth position single tooth according to the corresponding point relation, recording the average shape and the average pose, and establishing a three-dimensional tooth template by using the average shapes and the average poses of the teeth with different tooth positions.
The process of dividing is to divide the triangular patch grids of the gingiva and different teeth, the process of numbering is to label the triangular patch grids of the different teeth obtained by dividing according to the tooth numbering specification, and the process of complementing is to complement the crown area of the triangular patch grids of the teeth to be watertight triangular patch grids.
The semantic segmentation model comprises an image encoder, a tooth semantic segmentation decoder, a tooth binary contour segmentation decoder and a region-contour fusion module,
the output end of the image encoder is respectively connected with the input ends of the tooth semantic segmentation decoder and the tooth binary contour segmentation decoder, the output ends of the tooth semantic segmentation decoder and the tooth binary contour segmentation decoder are stacked and then connected with the input end of the region-contour fusion module, and the output of the region-contour fusion module is a tooth semantic segmentation graph.
The image encoder, the tooth semantic segmentation decoder and the tooth binary contour segmentation decoding are all standard U-Net3+ models.
The step S4 specifically includes:
step S41: initializing camera parameters by using experience values based on the positions of the original front photos, initializing a tooth three-dimensional template by using the average shapes and the average postures of the teeth with different tooth positions, and initializing relative position parameters of upper and lower dentitions;
step S42: projecting vertexes in a three-dimensional tooth template of the current tooth template parameters based on the current camera parameters and a standard small-hole camera model, and extracting visible edge contour points of each tooth;
step S43: performing corresponding point relation matching on the visible edge contour points obtained by projection and the tooth contour points with tooth number information;
step S44: optimizing an objective function obtained by weighting the matching loss of the corresponding points and the negative log likelihood loss of the probability distribution of the tooth shape and the pose in the tooth parameter model, and updating the camera parameters, the tooth template parameters and the relative position parameters of the upper dentition and the lower dentition;
step S45: iterating the steps S42 to S44 for a plurality of times until the objective function in the step S44 converges, wherein the value of the objective function tends to be stable, and the optimized camera parameters, the optimized tooth template parameters and the relative positions of the upper dentition and the lower dentition are obtained;
step S46: and (3) carrying out surface reconstruction on the three-dimensional tooth template after the optimization deformation by using a poisson surface reconstruction algorithm to obtain a three-dimensional tooth model expressed by a triangular patch grid.
The tooth arrangement process in the step S5 specifically includes: the position and posture of the different teeth at each stage are simulated according to the current tooth state and the expected treatment period.
The frontal photo orthodontic effect preview model generates an countermeasure network for an image style migration class based on a StyleGAN2 image style migration decoder.
In step S6, the tooth semantic segmentation map and the intra-oral area mask map are input to an image structure encoder formed by sequentially connecting 6 convolution layers and pooling layers in series after being stacked through channels, a part of the original front photo corresponding to the intra-oral area mask is input to an image style encoder formed by connecting 6 groups of different scale residual modules in series, the final output size of the image structure encoder is the same as that of the image style encoder, the output of the image structure encoder and the final output size of the image style encoder are input to a StyleGAN2 image style migration decoder through channel stacking, style vectors of an image style migration class generation countermeasure network are generated through residual connection and different convolution networks by different scale side path outputs of the image style encoder, the output of the image style migration decoder is the intra-oral area in the front photo after simulation of the orthodontic treatment, and the output of the image style migration decoder is the intra-oral area in the simulated orthodontic.
An orthodontic effect preview image generating device comprises a memory, a processor and a program stored in the memory, wherein the processor realizes the method when executing the program.
Compared with the prior art, the invention has the following beneficial effects: the method can realize the prediction of the tooth arrangement in the oral cavity and the generation of the front smile photo or the front intraoral flaring photo after the orthodontic treatment of the user on the premise of not requiring a digital three-dimensional tooth model of the user before the orthodontic treatment stage starts, can help the user and doctor participating in the orthodontic treatment to predict the orthodontic effect in advance and help other users to make decisions about whether to participate in the orthodontic treatment.
Drawings
FIG. 1 is a flow chart of a method according to an embodiment of the invention.
Fig. 2 is a schematic structural diagram of a semantic segmentation model.
Fig. 3 is a schematic diagram of a front view orthodontic effect preview model.
Detailed Description
The invention will now be described in detail with reference to the drawings and specific examples. The present embodiment is implemented on the premise of the technical scheme of the present invention, and a detailed implementation manner and a specific operation process are given, but the protection scope of the present invention is not limited to the following examples.
A method for generating an orthodontic effect preview image, as shown in fig. 1, includes:
step S1: based on the digitized three-dimensional tooth model obtained by intraoral scanning, a three-dimensional tooth template is established, and the main contents of the process are as follows: according to a plurality of groups of existing oral cavity scanning three-dimensional tooth models, carrying out statistical modeling on tooth shapes and tooth pose distribution in dentition, constructing tooth statistical shape models of different tooth positions to describe tooth morphology changes, constructing parameterized three-dimensional tooth templates, and describing tooth size and pose distribution in dentition.
Specifically, step S1 includes:
step S11: obtaining a plurality of groups of digital three-dimensional tooth models, performing segmentation, numbering and filling operations on each digital three-dimensional tooth model to obtain a three-dimensional tooth model with tooth position information, specifically, collecting a batch of at least 50 digital three-dimensional tooth models, wherein each three-dimensional model consists of triangular patch grids representing teeth and gingiva, and performing segmentation, numbering and filling operations on each three-dimensional model to obtain the three-dimensional tooth model with tooth position information.
Specifically, 130 digitized three-dimensional models are collected in the embodiment, the digitized three-dimensional tooth models can be obtained by an oral scanner, the segmentation operation refers to segmentation of triangular patch grids belonging to gums and different teeth, the numbering operation refers to marking of the triangular patch grids of the different teeth obtained by segmentation according to the tooth numbering specification, the completion refers to completion of crown areas of the triangular patch grids of the teeth to enable the crown areas to be watertight triangular patch grids, and the segmentation, numbering and completion operations can be manually completed by manual operation patch grid editing software.
Step S12: aligning the upper dentition and the lower dentition of each three-dimensional tooth model according to the corresponding dentition according to the gravity center of each tooth, and calculating the average gravity center of the tooth corresponding to each dentition;
specifically, in this embodiment, the three-dimensional model of 130 upper dentition teeth obtained in the previous step is calculated according to the number, the center of gravity of the mesh vertex of the triangular face piece of the tooth corresponding to each tooth position is calculated, the third molar is not considered, the obtained center of gravity is averaged, and the same operation is adopted for the obtained three-dimensional model of the lower dentition teeth.
Step S13: performing similar transformation registration on different teeth belonging to the same tooth position by using a point cloud registration algorithm of consistent point drift, recording the tooth pose and corresponding point relation, and performing probability modeling on the tooth size and the pose by using multi-element normal distribution;
specifically, the sizes of all teeth of the upper dentition and the lower dentition are subjected to probability modeling by using a multi-element normal distribution, the pose of each dentition tooth comprises a three-degree-of-freedom position vector and a three-degree-of-freedom rotation vector by using the multi-element normal distribution, probability modeling is performed, the mean value and the variance of the probability distribution are described by using a sample mean value and a sample variance, and the established mean value of the multi-element normal distribution is standardized so that the size mean value of each dentition tooth is 1, the position vector mean value is [0, 0], and the rotation vector mean value is [0, 0].
Step S14: and establishing a statistical shape model of each tooth position single tooth according to the corresponding point relation, recording the average shape and the average pose, and establishing a three-dimensional tooth template by using the average shapes and the average poses of the teeth with different tooth positions.
Specifically, the standard three-dimensional tooth template comprises an average three-dimensional tooth model of an upper dentition and a lower dentition, the template has deformability, deformation parameters comprise the relative pose of the upper dentition and the lower dentition, the size, the pose and the deformation parameters of each tooth position tooth, the number of points of the tooth model of each tooth position in the template is 1500, and the first 10 shape components of the statistical shape model of the teeth of different tooth positions are reserved.
The dividing process is specifically to divide the triangular patch grids of the gingiva and different teeth, the numbering process is specifically to label the triangular patch grids of the different teeth obtained by dividing according to the tooth numbering specification, and the complementing process is specifically to complement the crown area of the triangular patch grids of the teeth to enable the triangular patch grids to be watertight.
Step S2: constructing a semantic segmentation model for carrying out tooth semantic segmentation on a mouth region of a single original front smile photo or a front intraoral flaring photo based on a convolutional neural network, and constructing an intraoral region segmentation model for segmenting an intraoral region and extracting a two-dimensional tooth contour with tooth position information;
the semantic segmentation model comprises an image encoder, a tooth semantic segmentation decoder, a tooth binary contour segmentation decoder and a region-contour fusion module,
the output end of the image encoder is respectively connected with the input ends of the tooth semantic segmentation decoder and the tooth binary contour segmentation decoder, the output ends of the tooth semantic segmentation decoder and the tooth binary contour segmentation decoder are stacked and then are connected with the input end of the region-contour fusion module, and the output of the region-contour fusion module is a tooth semantic segmentation graph.
The image encoder, the tooth semantic segmentation decoder and the tooth binary contour segmentation decoding are all standard U-Net3+ models.
The construction process of the semantic segmentation model and the intra-oral area segmentation model is specifically as follows:
step S21: the model with the best effect is stored on a test set by using a standard U-Net network as an intra-oral area segmentation model, training the model, and designing a front photo tooth semantic segmentation model based on a U-Net3+ network structure in deep learning, a multi-scale cavity space convolution pooling pyramid module and a multi-task learning double-branch structure.
Specifically, as shown in fig. 2, the network structure of the front photo tooth semantic segmentation model is that the output of an image encoder of standard U-net3+ is connected in parallel with a tooth semantic segmentation decoder and a tooth binary contour segmentation decoder in series, and then a region-contour fusion module based on multi-scale cavity space convolution is connected in series, wherein the decoder structure is that of the decoder of standard U-net3+, the output of the decoder is that of the decoder of standard U-net3+ and the output of the decoder is that of the region-contour fusion module after being stacked, and the output of the module is a tooth semantic segmentation graph. The size of the input picture of the front photo tooth semantic segmentation model is 256,256,3, the size of the output tooth semantic segmentation graph is 256,256,33, and the tooth semantic segmentation graph adopts single-heat coding, and comprises 33 categories including 1 background category and 32 tooth categories.
Step S22: the output of the tooth semantic segmentation model is adjusted by using a post-processing algorithm, the post-processing algorithm comprises dividing the connected areas of the tooth semantic segmentation graph according to the binary tooth profile subjected to expansion operation, unifying the tooth numbers in the same connected area, extracting the largest connected area of the tooth numbers of different teeth, sorting the tooth numbers of the connected areas according to the sequence from the middle to the two sides of the graph, modifying the tooth numbers of the redundant connected areas, and smoothing the result by using morphological algorithms such as erosion, expansion and the like.
Step S23: judging the occlusion characteristics of teeth of a patient based on the relative area relation of the upper dentition tooth areas and the lower dentition tooth areas, if the segmentation area of the upper dentition tooth areas is larger than that of the lower dentition, preferentially extracting the visible contours of the teeth of the upper dentition, otherwise, preferentially extracting the tooth contours of the lower dentition, and classifying the extracted tooth contours according to the tooth numbers according to the sequence from the middle to the left and right sides when extracting the contours.
Step S3: acquiring an original front photo of a user, and sequentially processing the original front photo of the user by using a semantic segmentation model and an intra-oral area segmentation model to obtain a tooth profile with tooth position information and an intra-oral area mask, wherein the front photo is smiling photo or flaring photo;
step S4: iteratively optimizing a camera parameter and a tooth template parameter to optimize a contour matching loss function, fitting a tooth contour of a corresponding tooth position obtained by deforming a three-dimensional tooth template to project the three-dimensional tooth template to a tooth contour with tooth position information so as to reconstruct a user-personalized three-dimensional tooth model, wherein the method specifically comprises the following steps of:
step S41: the camera parameters are initialized by using empirical values based on the positions of the original front photos, the tooth three-dimensional templates are initialized by using the average shapes and the average postures of the teeth with different tooth positions, the relative position parameters of the upper dentition and the lower dentition are initialized, so that the three-dimensional model presents a standard normal occlusion relationship, specifically, the initial relative rotation vector of the relative positions of the upper dentition and the lower dentition is [0, 0], the relative position relationship is that the upper dentition is positioned at a position 7mm above the lower dentition, the position is 2mm in front, and the left-right direction offset is 0.
Step S42: and projecting the vertexes in the three-dimensional tooth template of the current tooth template parameters and extracting visible edge contour points of each tooth based on the current camera parameters and the standard small-hole camera model, wherein the camera parameters comprise a global camera pose, a camera focal length and a camera main point, and judging the front-back position relation of the projection points of each tooth according to the distance between each tooth and a camera plane in the three-dimensional tooth model so as to extract the visible contour points, wherein the camera model does not consider the influence of lens distortion.
Step S43: the corresponding point relation matching is carried out on the visible edge contour point obtained by projection and the tooth contour point with tooth number information, specifically, the tooth contour of the tooth position tau extracted from the photo in the step 2-3) is expressed as a set { c } of points on a contour line in a pixel coordinate system i τ -representing the visible contour of the corresponding tooth position τ projected in step 3-2) asThen for the extracted toothSome contour point c of bit τ i τ The corresponding point on the contour line obtained by projection is +.>Can be calculated from the following formula;
wherein: sigma is a super parameter, which is obtained by multiple experimental adjustment, the specific value is 0.3, n i τ To extract the normal vector at the contour point i of the tooth position tau,the normal vector at the corresponding point j on the contour line obtained by projection;
step S44: optimizing objective functions obtained by weighting corresponding point matching loss and negative log likelihood loss of probability distribution of tooth shape and pose in tooth parameter model, updating camera parameters, tooth template parameters and relative position parameters of upper and lower dentitions, in particular, calculating loss function to be optimizedIt may be expressed as that,
wherein lambda is n And lambda (lambda) p Is a weight constant, L p For projection error, describe the relative position deviation of the corresponding point, L n Describing the relative deviation of the profile normal line of the corresponding point as the normal error, L prior Is a penalty term based on the pose and shape probability distribution of teeth in the tooth parameterized model, and in particular,
wherein N is the total number of tooth contour points obtained by segmentation in the photo, T is the number of tooth categories obtained by segmentation in the photo, s is the size vector of all teeth projected in the photo, and p τ Is the pose vector of the tau th tooth, h τ Is the shape characteristic vector of the tau-th tooth, D size ，D pose And D shape The Mahalanobis distance in the probability distribution of the size, pose and shape features of the projected teeth for each photo, respectively, are used for describing the deviation degree of the current parameter estimation and the mean value of the probability distribution, and lambda is used in the three-dimensional reconstruction process n =50, and λ p ＝25。
Step S45: iterating the steps S42 to S44 for a plurality of times until the objective function in the step S44 converges, wherein the value of the objective function tends to be stable, obtaining optimized camera parameters, tooth template parameters and the relative positions of upper and lower dentitions, firstly keeping the tooth template parameters fixed, iterating the relative positions of the camera parameters and the upper and lower dentitions for 10 rounds, then simultaneously optimizing all the parameters for 20 rounds, pushing the gradient of the determined optimization function through display, and adopting a sequence least square method as an optimization algorithm;
step S46: and carrying out surface reconstruction on the three-dimensional tooth template after optimization deformation by using a poisson surface reconstruction algorithm to obtain a three-dimensional tooth model represented by a triangular patch grid, wherein the normal vector of a point cloud required by poisson surface reconstruction is determined by the normal line of a plane formed by 30 points in the neighborhood of the point, and the specific direction is defined as the direction with an acute angle with the vector of the center of gravity of the tooth point cloud pointing to the point cloud.
Step S5: performing tooth arrangement and filling on the personalized three-dimensional tooth model of the user, and projecting the personalized three-dimensional tooth model after tooth arrangement under the same camera parameters to obtain a tooth semantic segmentation graph, wherein the tooth arrangement process specifically comprises the following steps: the position and posture of the different teeth at each stage are simulated according to the current tooth state and the expected treatment period.
The process uses an automatic tooth arranging algorithm to arrange the reconstructed three-dimensional teeth, simulates the orthodontic treatment process of the teeth to obtain a three-dimensional teeth model after simulated tooth arranging, and particularly, the automatic tooth arranging algorithm is derived from tooth arranging software used in the orthodontic treatment process, and simulates the positions and postures of different teeth at each stage according to the current tooth state and the expected treatment period.
Specifically, the midpoints of incisors and side incisors in upper and lower dentitions, the cuspids of the cuspids, the buccal cuspids of the premolars and the proximal and distal buccal cusps of the molars in the three-dimensional tooth model are extracted as characteristic points, dental arches of the upper and lower dentitions are fitted respectively according to the characteristic points using symmetrical Beta functions, the characteristic points of each tooth of the reconstructed three-dimensional tooth model are translated to corresponding positions of the dental arches, the collision relationship is considered, the tooth arrangement is made as compact as possible, the rotation vector of each tooth is adjusted to an average posture, the size and shape vector of each tooth are kept unchanged, and the plane formed by the centers of gravity of each tooth is made parallel to the plane of the dental jaw, wherein the expression of the symmetrical Beta functions is as follows,
wherein x is the distance from the feature point to the center line of the dentition, t is the corresponding coordinate of the dental arch curve, D and W are parameters to be fitted, the extracted feature point coordinate is substituted into the above formula, and the least square fitting is used to obtain D and W.
Step S6: and inputting the tooth semantic segmentation map, the mask of the oral cavity inner area and the part of the original front photo of the user corresponding to the mask of the oral cavity inner area into a trained front photo orthodontic effect preview model to obtain a front smile map or a flaring photo after tooth orthodontic treatment, thereby realizing the function of orthodontic effect preview.
The frontal photo orthodontic effect preview model generates an countermeasure network for the image style migration class based on the StyleGAN2 image style migration decoder, namely: the StyleGAN2 image style migration class based on the residual connection and encoder-decoder structure generates an countermeasure network, the structure of which is shown in fig. 3, and trains, and stores the best-performing model on the test set, which is input as the tooth semantic segmentation map, the intra-oral area mask, and the intra-oral area of the front smile photograph or the intra-oral flare photograph.
Specifically, the specific structure and flow of the orthodontic effect preview model of the front photo are that after being stacked through channels, a tooth semantic segmentation graph and an intraoral region mask are input into an image structure encoder formed by sequentially connecting 6 convolution layers and pooling layers in series, the intraoral region of the front smile photo or the intraoral flaring photo is input into an image style encoder formed by connecting 6 groups of different scale residual modules in series, the final output size of the image structure encoder is the same as that of the image style encoder, the output of the image structure encoder and the final output size of the image style encoder are input into a StyleGAN2 image style migration decoder through channel stacking, style vectors of the image style migration decoder are generated through residual connection and different convolution networks through different scale side-channel outputs of the image style encoder, and are injected into the image style migration decoder through a neural network weight modulation and demodulation algorithm, and the output of the image migration decoder is the intraoral region in the front photo after orthodontic simulation. The model inputs are the dimensions of the tooth semantic segmentation map (256,256,33), the intraoral area mask (256,256,1), and the intraoral area of the frontal smile photograph or intraoral flare photograph (256,256,3), respectively.
The generating the preview image route based on the front photo orthodontic effect preview model specifically comprises:
step S61: constructing a front photo orthodontic effect preview model;
step S62: projecting the three-dimensional tooth model simulated and aligned by the automatic tooth arrangement algorithm according to the obtained camera parameters, coding the projection graph according to the tooth position information, and obtaining an orthodontic tooth semantic segmentation graph, wherein the mutual shielding condition is considered according to the relative position relation between each tooth and the projection plane.
Step S63: inputting a mouth region of the front smile photo or a front intraoral flaring photo into an intraoral region segmentation model, obtaining a binary intraoral region mask map, and extracting a corresponding intraoral region color picture according to the mask by using Boolean operation.
Step S63: and inputting the obtained orthodontic tooth semantic segmentation graph, the mask of the oral cavity inner region and the corresponding oral cavity inner region in the corresponding orthodontic front photo into an orthodontic effect preview model, and outputting the oral cavity inner region in the orthodontic simulated front photo.
Step S64: step S64 is performed using a bicubic interpolation algorithm: and the obtained oral cavity inner area in the simulated orthodontic front photo is up-sampled, the resolution of the original photo is improved, the original photo is covered at a corresponding position in the original front photo, the boundary at the picture splicing position is subjected to smoothing treatment by using Gaussian filtering, the treated picture is used as an orthodontic effect preview picture of the front photo, and the preview picture describes the tooth arrangement condition in the front smile photo of the user after the simulated orthodontic treatment.
The above functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Claims (10)
1. A method for generating an orthodontic effect preview image, comprising:
step S1: establishing a three-dimensional tooth template based on a digital three-dimensional tooth model obtained by intraoral scanning;
step S2: constructing a semantic segmentation model for performing tooth semantic segmentation on a mouth region of a front smile photo or a front intraoral flare photo of the single Zhang Yuanshi, and constructing an intraoral region segmentation model for segmenting an intraoral region and extracting a two-dimensional tooth contour with tooth position information;
step S3: acquiring an original front photo of a user, and sequentially processing the original front photo of the user by using a semantic segmentation model and an intra-oral area segmentation model to obtain a tooth profile with tooth position information and an intra-oral area mask, wherein the front photo is smiling photo or flaring photo;
step S4: iteratively optimizing a camera parameter and a tooth template parameter to optimize a contour matching loss function, deforming the three-dimensional tooth template to enable the tooth contour of the corresponding tooth position obtained by projection to fit the tooth contour with tooth position information so as to reconstruct a user personalized three-dimensional tooth model;
step S5: performing tooth arrangement and filling on the personalized three-dimensional tooth model of the user, and projecting the personalized three-dimensional tooth model after tooth arrangement under the same camera parameters to obtain a tooth semantic segmentation map;
step S6: and inputting the tooth semantic segmentation map, the mask of the oral cavity inner area and the part of the original front photo of the user corresponding to the mask of the oral cavity inner area into a trained front photo orthodontic effect preview model to obtain a front smile map or a flaring photo after tooth orthodontic.
2. The method for generating an orthodontic effect preview image according to claim 1, wherein said step S1 comprises:
step S11: acquiring a plurality of groups of digital three-dimensional tooth models, and performing segmentation, numbering and filling operations on each digital three-dimensional tooth model to obtain a three-dimensional tooth model with tooth position information;
step S12: aligning the upper dentition and the lower dentition of each three-dimensional tooth model according to the corresponding dentition according to the gravity center of each tooth, and calculating the average gravity center of the tooth corresponding to each dentition;
step S13: performing similar transformation registration on different teeth belonging to the same tooth position by using a point cloud registration algorithm of consistent point drift, and recording the tooth pose and corresponding point relation;
step S14: and establishing a statistical shape model of each tooth position single tooth according to the corresponding point relation, recording the average shape and the average pose, and establishing a three-dimensional tooth template by using the average shapes and the average poses of the teeth with different tooth positions.
3. The method for generating an orthodontic effect preview image according to claim 2, wherein the dividing process is specifically dividing triangular patch grids of gums and different teeth, the numbering process is specifically marking the triangular patch grids of the different teeth obtained by dividing according to a tooth numbering specification, and the filling process is specifically filling crown areas of the triangular patch grids of the teeth to be watertight triangular patch grids.
4. The method for generating an orthodontic effect preview image according to claim 1, wherein said semantic segmentation model comprises an image encoder, a tooth semantic segmentation decoder, a tooth binary contour segmentation decoder, a region-contour fusion module,
the output end of the image encoder is respectively connected with the input ends of the tooth semantic segmentation decoder and the tooth binary contour segmentation decoder, the output ends of the tooth semantic segmentation decoder and the tooth binary contour segmentation decoder are stacked and then connected with the input end of the region-contour fusion module, and the output of the region-contour fusion module is a tooth semantic segmentation graph.
5. The method for generating an orthodontic effect preview image according to claim 4, wherein said image encoder, said tooth semantic segmentation decoder and said tooth binary contour segmentation decoder are all standard U-net3+ models.
6. The method for generating an orthodontic effect preview image according to claim 2, wherein the step S4 specifically includes:
step S41: initializing camera parameters by using experience values based on the positions of the original front photos, initializing a tooth three-dimensional template by using the average shapes and the average postures of the teeth with different tooth positions, and initializing relative position parameters of upper and lower dentitions;
step S42: projecting vertexes in a three-dimensional tooth template of the current tooth template parameters based on the current camera parameters and a standard small-hole camera model, and extracting visible edge contour points of each tooth;
step S43: performing corresponding point relation matching on the visible edge contour points obtained by projection and the tooth contour points with tooth number information;
step S44: optimizing an objective function obtained by weighting the matching loss of the corresponding points and the negative log likelihood loss of the probability distribution of the tooth shape and the pose in the tooth parameter model, and updating the camera parameters, the tooth template parameters and the relative position parameters of the upper dentition and the lower dentition;
step S45: iterating the steps S42 to S44 for a plurality of times until the objective function in the step S44 converges, wherein the value of the objective function tends to be stable, and the optimized camera parameters, the optimized tooth template parameters and the relative positions of the upper dentition and the lower dentition are obtained;
step S46: and (3) carrying out surface reconstruction on the three-dimensional tooth template after the optimization deformation by using a poisson surface reconstruction algorithm to obtain a three-dimensional tooth model expressed by a triangular patch grid.
7. The method for generating an orthodontic effect preview image according to claim 1, wherein the tooth arranging process in step S5 specifically includes: the position and posture of the different teeth at each stage are simulated according to the current tooth state and the expected treatment period.
8. The method for generating an orthodontic effect preview image according to claim 1, wherein the front photo orthodontic effect preview model generates an countermeasure network for an image style migration class based on a StyleGAN2 image style migration decoder.
9. The method according to claim 8, wherein in the step S6, the tooth semantic segmentation map and the intra-oral region mask map are input to an image structure encoder composed of 6 convolution layers and pooling layers sequentially connected in series after being stacked through channels, the portion of the original front photo corresponding to the intra-oral region mask is input to an image style encoder composed of 6 sets of different scale residual modules connected in series, the final output size of the image structure encoder is the same as that of the image style encoder, the outputs of the image structure encoder and the final output size of the image style encoder are input to a StyleGAN2 image style migration decoder through channel stacking, style vectors of an image style migration class generation countermeasure network are generated by different scale side path outputs of the image style encoder through residual connection and different convolution networks, and the output of the image style migration decoder is the intra-oral region in the front photo after simulation of the orthodontic.
10. An orthodontic effect preview image generating device comprising a memory, a processor, and a program stored in the memory, wherein the processor implements the method of any one of claims 1-9 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311106119.2A CN117115352A (en) | 2023-08-30 | 2023-08-30 | Method and device for generating orthodontic effect preview image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311106119.2A CN117115352A (en) | 2023-08-30 | 2023-08-30 | Method and device for generating orthodontic effect preview image |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117115352A true CN117115352A (en) | 2023-11-24 |
Family
ID=88810725
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311106119.2A Pending CN117115352A (en) | 2023-08-30 | 2023-08-30 | Method and device for generating orthodontic effect preview image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117115352A (en) |
-
2023
- 2023-08-30 CN CN202311106119.2A patent/CN117115352A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111784754B (en) | Tooth orthodontic method, device, equipment and storage medium based on computer vision | |
| US11759291B2 (en) | Tooth segmentation based on anatomical edge information | |
| US11694418B2 (en) | Systems and methods for constructing a three-dimensional model from two-dimensional images | |
| US11735306B2 (en) | Method, system and computer readable storage media for creating three-dimensional dental restorations from two dimensional sketches | |
| US20220028162A1 (en) | Systems and methods for constructing a dental arch image using a machine learning model | |
| US11850113B2 (en) | Systems and methods for constructing a three-dimensional model from two-dimensional images | |
| CN113728363B (en) | Method for generating dental model based on objective function | |
| US20230206451A1 (en) | Method for automatic segmentation of a dental arch | |
| WO2023185405A1 (en) | Design method for 3d printed denture framework, and apparatus and storable medium | |
| CN114586069A (en) | Method for generating dental images | |
| CN115457198A (en) | Tooth model generation method and device, electronic equipment and storage medium | |
| TW202409874A (en) | Dental restoration automation | |
| WO2022125433A1 (en) | Systems and methods for constructing a three-dimensional model from two-dimensional images | |
| WO2024114343A1 (en) | Gum line extraction method, dental instrument manufacturing method and electronic device | |
| US20240144480A1 (en) | Dental treatment video | |
| US20240065807A1 (en) | 3d facial reconstruction and visualization in dental treatment planning | |
| CN117115352A (en) | Method and device for generating orthodontic effect preview image | |
| WO2024119138A2 (en) | Augmented video generation with dental modifications | |
| EP4307229A1 (en) | Method and system for tooth pose estimation | |
| Mostafa et al. | Statistical morphable model for human teeth restoration | |
| CN118212376A (en) | Tooth three-dimensional model reconstruction method based on tooth parameterized model and tooth photo | |
| US12144701B2 (en) | Tooth segmentation based on anatomical edge information | |
| CN117137660A (en) | Method for determining occlusion relation between upper dentition and lower dentition of digitized intraoral scanning model | |
| KR102610716B1 (en) | Automated method for generating prosthesis from three dimensional scan data and computer readable medium having program for performing the method | |
| WO2024097673A1 (en) | Dental treatment video |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |