CN104850633B - A kind of three-dimensional model searching system and method based on the segmentation of cartographical sketching component - Google Patents

Abstract

The embodiment of the present invention discloses a three-dimensional model retrieval system and method based on hand-drawn sketch part segmentation, wherein the system includes: a preprocessing module, which is used to denoise the hand-drawn query sketch to obtain a grayscale image, and Binary processing, boundary expansion processing, and image hole filling processing are performed on the grayscale image to obtain the processed image; the component marking module is used to sample the processed image at equal intervals, and add component labels to the sampling points; sampling points The feature extraction module is used to extract various feature vectors of sampling points; the component segmentation module is used to perform segmentation model training according to various feature vectors of sampling points after adding component labels; the similarity calculation and total score sorting module is used for Calculate the partial similarity of parts, sort according to the total score, and return the sorting results to the client. The implementation of the embodiments of the present invention can make the retrieval of 3D models based on hand-drawn sketches more accurate and effective.

Description

A 3D model retrieval system and method based on hand-drawn sketch parts segmentation

technical field

The invention relates to the technical field of computer image processing, in particular to a three-dimensional model retrieval system and method based on hand-drawn sketch component segmentation.

Background technique

In recent years, with the rapid development of computer-aided design, computer-aided manufacturing, virtual reality, 3D animation and 3D games, the number of 3D models on the Internet has increased dramatically. However, the 3D model is different from traditional multimedia information such as pictures, audio or video, and it contains many detailed information that is difficult to express in words.

However, current 3D model retrieval methods are still unsatisfactory in application. On the one hand, when users need some kind of 3D model resources, they often do not have ready-made model files at hand; on the other hand, with the rapid popularization of touch screens and electronic pens, users can easily outline the model by hand sketching . A hand-drawn sketch of a 3D model that can be thought of as the outline of a projected view from a certain perspective. Freehand sketches can be simple outlines or include detailed information on inner outlines. Due to the different artistic foundations of users of hand-drawn sketches and different input devices, the degree of detail of the drawn models is naturally different. However, the hand-drawn sketches of 3D models usually contain overlapping, separated or unclosed component outlines. Existing related research is usually based on manual segmentation or labeling of hand-drawn sketches, although these manually specified information is helpful for computers to analyze hand-drawn sketches. Sketches are analyzed, but it usually requires users to follow certain rules and constraints when drawing sketches, which limits the freedom of users' drawing to some extent, or puts forward requirements on the user's drawing foundation.

According to the classification of retrieval methods, the current retrieval of 3D models is mainly divided into two categories, namely the traditional text-based retrieval (Text-based Retrieval) method and the content-based retrieval (Content-based Retrieval) method.

(1) Text-based 3D model retrieval method

The text-based 3D model retrieval method is based on keywords and is currently the most common retrieval method. This requires artificially adding keywords to describe the 3D model in the database, such as SketchUp's 3D model library (3D Warehouse), TurboSquid's professional model library, and China National Taiwan University's 3D protein retrieval system (3DProtein Retrieval System), etc. Now, some large-scale commercial model retrieval platforms can be found on the Internet, most of which are keyword-based 3D model retrieval methods.

(2) Content-based 3D model retrieval method

Content-based retrieval is a research hotspot in 3D model retrieval. As shown in Figure 1, the basic framework of content-based 3D model retrieval is divided into an offline part and an online part. For the offline part, each 3D model needs to be labeled with a shape descriptor. In order to improve retrieval efficiency, an index is usually established for each model feature descriptor in the database. For the online part, there are two main ways to input the query expression: one is to provide a 3D model similar to the target model; the other is to draw a sketch of the target model. After calculating the feature descriptors, compare the similarity between the descriptors retrieved by the user and the model feature descriptors in the database, and then return the results in descending order of similarity and present them visually to the user.

Shortcomings existing in the prior art:

(1) Text-based 3D model retrieval method

The traditional method based on text keywords cannot be well applied in the scene of 3D model retrieval. There are three main reasons for this: First, 3D models have complex topological structures and shape features, and there are many kinds of them. Detailed information is difficult to express clearly with a few keywords. Second, the tagging process of adding text keywords to 3D models needs to be done manually, and the number of 3D models on the Internet is increasing rapidly, and the manual adding method is cumbersome and the workload is also heavy. Third, because different people have different understandings of various 3D models, the keywords they think of to describe them are also quite different, which may easily lead to inconsistencies between the search keywords and the labels of the target model, and the way of manually adding keyword labels is affected It is limited to the types of label languages, and it is not convenient for international promotion. It is for these reasons that if only simple keywords are used for retrieval, the success rate will be very low, and the desired results will not be obtained in many cases. For example, if a user wants to search for a car with a specific shape and pattern, it is difficult to search for accurate and satisfactory results only by keywords.

(2) Content-based 3D model retrieval method

For model retrieval based on 3D model instances, the disadvantage is that it is usually difficult for the user to find a very suitable model instance as input when the user initiates the retrieval, because if the user has a very suitable target model at hand, then there is no need for retrieval up.

For 3D model retrieval based on hand-drawn sketches, its disadvantage is that it usually does not consider the overall structure of the sketch, it is only considered based on the local area; another disadvantage is that it is sensitive to the style of the user's sketch, if the user is in the local outline If the line drawing style is quite different, the difference in the extracted results will be enlarged, which will inevitably affect the final retrieval results.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies of the prior art. The present invention provides a 3D model retrieval system and method based on hand-drawn sketch parts segmentation, which can make the 3D model retrieval based on hand-drawn sketches more accurate and effective.

In order to solve the above problems, the present invention proposes a three-dimensional model retrieval system based on hand-drawn sketch parts segmentation, the system includes:

A preprocessing module, configured to receive a hand-drawn query sketch, perform denoising processing on the hand-drawn query sketch to obtain a grayscale image, and perform binarization processing, boundary extension processing, and image hole filling processing on the grayscale image to obtain a processed after the image;

A component labeling module, configured to sample the processed images at equal intervals, obtain sampling points, and add component labels to the sampling points;

Sampling point feature extraction module, used to extract various feature vectors of the sampling point;

The component segmentation module is used to perform segmentation model training according to various feature vectors of the sampling points after adding component labels;

The similarity calculation and total score sorting module is used to extract local features of parts and calculate local similarity of parts based on the segmentation model, and perform view global feature extraction and view global similarity calculation on the processed image, according to the total score Sort and return the sorted results to the client.

Preferably, the preprocessing module includes:

A sketch denoising processing unit, configured to perform denoising processing on the hand-drawn query sketch to obtain a grayscale image;

a binarization processing unit, configured to perform binarization processing on the grayscale image;

The boundary extension processing unit is used to perform blank filling processing around the binarized image;

The image hole filling processing unit is configured to perform image hole filling processing on the image after blank filling processing.

Preferably, the component marking module includes:

A contour extraction unit, configured to extract contours from the processed image;

The sampling unit is used to sample the image at equal intervals after the contour line is extracted to obtain sampling points;

The component labeling unit is used for adding component labels to the sampling points.

Preferably, the sampling point feature extraction module includes:

A unitary feature extraction unit, which is used to extract a unitary feature from the sampling point after adding the component label;

The binary feature extraction unit is used to perform binary feature extraction on the sampling points after adding component labels.

Preferably, the component segmentation module includes:

The segmentation model training unit is used to perform segmentation model training according to various feature vectors of the sampling points after adding the part label;

The component segmentation unit is configured to perform component segmentation on the sampling points after adding component labels according to the segmentation model.

Correspondingly, the present invention also provides a 3D model retrieval method based on hand-drawn sketch component segmentation, the method comprising:

receiving a hand-drawn query sketch, performing denoising processing on the hand-drawn query sketch to obtain a grayscale image, and performing binarization processing, boundary extension processing, and image hole filling processing on the grayscale image to obtain a processed image;

Sampling the processed image at equal intervals to obtain sampling points, and adding component labels to the sampling points;

Extracting various feature vectors of the sampling points;

Carry out segmentation model training according to various feature vectors of sampling points after adding component labels;

Based on the segmentation model, extract local features of parts and calculate local similarity of parts, and perform global view feature extraction and global view similarity calculation on the processed images, sort according to the total score, and return the sorting results to the client.

Preferably, the step of performing denoising processing on the hand-drawn query sketch to obtain a grayscale image, and performing binarization processing, boundary extension processing, and image hole filling processing on the grayscale image to obtain a processed image, include:

Denoising the hand-drawn query sketch to obtain a grayscale image;

Perform binarization processing on the grayscale image;

Perform blank filling processing around the binarized image;

Carry out image hole filling processing on the image after blank filling processing.

Preferably, the step of sampling the processed images at equal intervals to obtain sampling points, and adding component labels to the sampling points includes:

Contour extraction is performed on the processed image;

Sampling at equal intervals on the image after extracting the contour line to obtain sampling points;

Add component labels to the sampling points.

Preferably, the step of extracting various feature vectors of the sampling points includes:

Perform unary feature extraction on the sampling points after adding component labels;

Binary feature extraction is performed on the sampling points after adding part labels.

Preferably, the step of performing segmentation model training according to various feature vectors of sampling points after adding part labels includes:

Carry out segmentation model training according to various feature vectors of sampling points after adding component labels;

Segment the sampling points after adding component labels according to the segmentation model.

In the embodiment of the present invention, the geometric information of the hand-drawn sketch components, the topology information between the components and the global information of the entire view are comprehensively used, and a dynamic weighting mechanism for three views is set up to amplify the impact of important perspectives on the total score , so that the retrieval of 3D models based on hand-drawn sketches is more accurate and effective; in addition, it can be used alone in the application scenarios of hand-drawn drawing parts segmentation such as sketch understanding and sketch classification.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is a schematic diagram of a basic framework of content-based 3D model retrieval in the prior art;

2 is a schematic diagram of the structural composition of a three-dimensional model retrieval system based on hand-drawn sketch parts segmentation according to an embodiment of the present invention;

3 is a schematic diagram of the internal processing process of the three-dimensional model retrieval system based on hand-drawn sketch parts segmentation according to an embodiment of the present invention;

Fig. 4 is a schematic diagram of the effect of adding labels in the embodiment of the present invention;

FIG. 5 is a schematic flowchart of a three-dimensional model retrieval method based on hand-drawn sketch component segmentation according to an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

Fig. 2 is a schematic diagram of the structural composition of a three-dimensional model retrieval system based on hand-drawn sketch parts segmentation according to an embodiment of the present invention. As shown in Fig. 2, the system includes:

The preprocessing module 1 is used to receive the hand-drawn query sketch, perform denoising processing on the hand-drawn query sketch to obtain a grayscale image, and perform binarization processing, boundary expansion processing, and image hole filling processing on the grayscale image to obtain a processed image ;

The component labeling module 2 is used to sample the images processed by the preprocessing module 1 at equal intervals, obtain sampling points, and add component labels to the sampling points;

The sampling point feature extraction module 3 is used to extract various feature vectors of the sampling points obtained by the part marking module 2;

Part segmentation module 4, for performing segmentation model training according to various feature vectors of sampling points after adding part labels;

Similarity Calculation and Total Score Sorting Module 5, which is used to extract local features of parts and calculate local similarity of parts based on the segmentation model, and perform view global feature extraction and view global similarity calculation on processed images, and sort according to the total score , and returns the sorted results to the client.

Fig. 3 is the internal processing process of the 3D model retrieval system based on hand-drawn sketch parts segmentation according to the embodiment of the present invention. The system according to the embodiment of the present invention will be described in detail below in conjunction with Fig. 2 and Fig. 3 .

The hand-drawn sketch entered by the user at the time of retrieval differs from the projected view of the 3D model. Since users draw sketches through mouse, touch screen or stylus, hand-drawn sketches are often not precise enough, and their contours often contain noise. Therefore, necessary preprocessing is required for user-drawn sketches before subsequent processing procedures can be performed.

Wherein, the preprocessing module 1 includes:

A sketch denoising processing unit, configured to denoise the hand-drawn query sketch to obtain a grayscale image;

A binarization processing unit, configured to binarize the grayscale image;

The boundary extension processing unit is used to perform blank filling processing around the binarized image;

The image hole filling processing unit is configured to perform image hole filling processing on the image after blank filling processing.

Sketch denoising processing: Since the end of the hand-drawn sketch is not closed, it will affect the subsequent feature extraction. In the embodiment of the present invention, preprocessing is performed to record the starting and ending coordinates of each stroke when the user draws. If the endpoint coordinates of a certain stroke are different from other If the Euclidean distance between the coordinates of the endpoints of the stroke is less than the threshold of 30 pixels, then a line is connected between these two points.

Bounding box processing: Due to the different sizes of the drawn parts of different sketches, in order to effectively and consistently perform subsequent feature extraction and machine learning, it is necessary to perform bounding box processing on the view to obtain the local area with sketch lines in the original image.

Maintain scaling processing: In order to satisfy image scale invariance and minimize the blank borders of all images as much as possible, scaling processing is required. Image scaling is performed by dividing 180 by the length of the longest side of the bounding box as the scale factor for scaling.

Binarization processing: After zooming, the hand-drawn sketch becomes a grayscale image, which needs to be binarized to facilitate global feature extraction in the model retrieval process.

Boundary extension processing: In order to center the sketch drawing area, boundary extension processing is required to fill in blanks around the image so that the image size is unified to 200*200 pixels.

Image hole filling processing: Since the contour line of the sketch includes the outer contour line and the inner contour line, for the model retrieval scene, the outer contour line is usually enough to distinguish different types of models, so in order to make the hand-drawn sketch and the projected view show similar details For comparison, it needs to be filled and preprocessed first.

Further, the component marking module 2 includes:

Contour line extracting unit, is used for carrying out contour line extraction to processed image;

The sampling unit is used to sample the image at equal intervals after the contour line is extracted to obtain sampling points;

The component labeling unit is used to add component labels to the sampling points.

(1) Classification

There are many types of models in the database, and the topological structure varies greatly, so it is necessary to classify the models before part marking. The criteria for class division lie in two aspects: on the one hand, as many models with similar topological structures as possible should be classified into the same class, and the training and learning of component segmentation should be performed together; on the other hand, all models in the database should be divided The number of major categories should be as few as possible, so as to reduce the total number of divisions of online hand-drawn sketches according to each major category.

(2) Based on equally spaced sampling point design and component labeling

Considering the subsequent feature extraction of each sampling point, the size of the sampling point must ensure the following three aspects: the sampling point should be dense enough to prevent different components from falling into the same sampling point, resulting in inaccurate component labels; the sampling point should not be too large Intensive, in order to avoid the high total operation cost of feature extraction, resulting in the inability to respond in real time during retrieval; the size of sampling points should take into account the size of sampling points during extraction of various unary features, so that the difference between the side lengths of sampling points for extracting different features There is an integer multiple relationship between them, so as to ensure that each sampling point can obtain a reasonable and effective combination of features.

Based on the above reasons, after a lot of experimental attempts, the method of equal interval sampling is finally adopted, and one sampling point is extracted for every 10 pixels along the contour line distance of each contour line. Add part labels to the sampling points, such as head, body, limbs, tail, etc., to segment the views of the large class model, and use the local feature vectors of each view sampling point in the current class for training.

(3) Automatic pre-segmentation based on discrete curve evolution model and skeleton information

In the research of part segmentation based on view, an intuitive idea is to analyze the geometric features of the view contour line to determine the dividing line between adjacent parts, and then perform part segmentation. In order to improve the efficiency of adding labels, in the embodiment of the present invention, the automatic segmentation of view components based on the discrete curve evolution model and the skeleton is used to automatically perform pre-segmentation of components.

The difference between the end point set of the simplified polygon and the end point set of the skeleton of the discrete curve evolution model, and all other tangent points of the inscribed circle with the central axis as the center of the inscribed circle with each element in the set as the tangent point, so that Constitute the pre-segmentation point set. Through experiments, it is found that this set is a set of potential segmentation points of most models, so this set is used as the basis for contour segmentation, and marks are added in segments in the order of contour sampling points, eliminating the need to circle polygons for each part The bounding box process can effectively improve the marking efficiency.

(4) Design of component marking widget

In order to add component labels to the sampling point data conveniently, an interactive way of adding and modifying labels is adopted. As shown in Figure 4, it is a schematic diagram of adding labels. The sampling point where the component is located can be easily marked by clicking the mouse to delineate the polygonal bounding box of the component.

In the output file, no category labels and model labels are added. The main reason is to continuously adjust the feature extraction method and parameters in the later stage. As long as the processing methods of the bounding boxes and sampling points remain unchanged, the component labels of the sampling points will not be affected. Adjust the effect of the feature extraction method. Therefore, reduce the coupling degree between the label and the feature extraction part as much as possible, separate the two parts, follow the unified file naming rules, and write them into two files in the same directory to facilitate post-processing. The labeling work is completely done in the interactive widget without interfering with the automatic feature extraction process.

Further, sampling point feature extraction module 3 includes:

A unitary feature extraction unit, which is used to extract a unitary feature from the sampling point after adding the component label;

The binary feature extraction unit is used to perform binary feature extraction on the sampling points after adding component labels.

(1) The unary feature of a single sampling point itself

Unary features are used to characterize the features inside each sampling point. The unary features used in the embodiments of the present invention are all based on sampling points, and various feature vectors of each sampling point are calculated. The details of this feature extraction process will be described separately below.

2D shape diameter feature: the calculation of each sampling point is based on the angle between it and the line between adjacent sampling points, calculate the tangent direction of the point, and then emit rays in the image mask along the direction perpendicular to the tangent , intersect at the edge point on the other side of the image, and calculate the length of the ray part inside the shape. Similarly, calculate the edge point where the ray drawn from the direction with the angle of 30° and 60° on both sides of the vertical line intersects the edge point on the other side of the image, also calculate the length value, and finally calculate the average value of these distance values as the The 2D shape diameter feature at the sampling point, with a total of 1 dimension.

The distance feature from the sampling point to the center point of the image: the Euclidean distance between the sampling point and the center point of the image is used as a part of the unary feature.

Average Euclidean distance feature: The average Euclidean distance metric based on sampling points is used to characterize how far each sampling point is from other sampling points. For example, in a view of an insect, the average Euclidean distance of the insect's legs is usually farther than other parts. The average Euclidean distance of each sampling point is obtained through the distance matrix of SC sampling points. If there are multiple sampling points in each sampling point, the average Euclidean distance of the sampling points is used. The Euclidean distance is obtained by calculating the average Euclidean distance from each sample point to other sample points. At the same time, the square of the mean value and the data of the 10th, 20th, 30th, and 90th percentiles are calculated, and then these 11 statistical measures are uniformly divided by the maximum value of the Euclidean distance of all sampling points in the current image to perform normalization. Unified to form an 11-dimensional vector.

Shape context histogram feature: The shape context algorithm takes sampling points at equal intervals on the edge line of the object, and calculates the Euclidean distance and angle of each sampling point relative to other sampling points.

Proportion feature of connected branches: Since some hand-drawn drawings and views are composed of multiple contour lines, each contour line segment usually represents a semantically independent component. In the embodiment of the present invention, a 1-dimensional vector is used to represent the proportion of the connected branch where the current sampling point is located in the entire image. First, record the number of sampling points of each contour line in the stage of extracting the contour line and down-sampling, and then determine the proportion of the sampling points of the contour line where the current sampling point is located in the total sampling points, so as to obtain a sampling point independent of the sampling step size The proportional feature of is used to represent the proportion of the contour line where the current sampling point is located in the total contour line. In the experiments on insects, it is found that the sampling points with the proportion characteristics of smaller connected branches are usually narrow and long parts such as legs and antennae around the object.

(2) Binary features between adjacent sampling points

Binary features measure the label consistency between each sampling point and adjacent sampling points. Therefore, before calculating the binary features, it is necessary to obtain the adjacent point information of each sampling point on the contour line. Since the sampling point sequence is ordered, the adjacency relationship can be determined by checking whether the Euclidean distance of adjacent points in the sampling point sequence is greater than the sampling step threshold. If it is greater than the threshold, it is not an adjacent point, otherwise it is an adjacent point. In addition, considering that some models have multiple contour line segments, it is also necessary to determine whether the start sampling point and the end sampling point of each contour line are adjacent. The binary features need to have sufficient discrimination, that is, the binary features of the sampling points at the component junction and the binary features of the sampling points at the non-component junction should be quite different. The absolute value of the difference between the diameters of the 2D shape and the absolute value of the difference between the tangent directions, as a binary feature. The value of each component of the binary eigenvector of the sampling point at the intersection of components is larger; the value of each component of the binary feature vector of the sampling point not near the intersection of components is smaller, so there is a significant degree of discrimination for the intersection of components.

Further, the component segmentation module 4 includes:

The segmentation model training unit is used to perform segmentation model training according to various feature vectors of the sampling points after adding the part label;

The component segmentation unit is configured to perform component segmentation on the sampling points after adding component labels according to the segmentation model.

In a specific implementation, after adding component labels, a conditional random field (CRF) model is used to train the segmentation model.

(1) Conditional random field model

The target energy function of the CRF model is composed of unary and binary terms. The unary item measures the unary feature of the sampling point and the consistency of its label; the binary item measures the label compatibility between the sampling point and its neighbors by the binary feature.

Here, the component segmentation and labeling methods based on the CRF model are used to train and learn the component segmentation of the internal model views of each category. Calculating the best labels for all sampling points requires minimizing the objective function, as shown in formula (1):

The objective function consists of two parts, where E ₁ is the unary energy item, and E ₂ is the binary energy item.

Unary energy term E ₁ : used to evaluate a classifier. The classifier takes the feature vector x of the sampling point as input, and outputs the probability distribution P(c|x,θ ₁ ) of the label of the sampling point under the characteristic condition. Machine learning with JointBoost classifier. The calculation method of the unary energy item is shown in formula (2):

E ₁ (c; x,θ ₁ )=-logP(c|x,θ ₁ ) (2)

In formula (2), x is a unary feature, and the unary energy item of each label c is equal to the negative logarithm of the probability distribution of the label of the sampling point under the condition of the feature vector.

Binary energy item E ₂ : used to characterize the probability that each sampling point is labeled as a different label from adjacent sampling points on the contour line, and its definition is shown in formula (3):

E ₂ (c,c'; y,θ ₂ )=W(c,c')·[-κlogP(c≠c'|y,ξ)+μ] (3)

In formula (3), y is a binary feature. P(c≠c'|y,ξ) characterizes the different likelihood of labels, which is a function of geometric features. The label penalty matrix W(c,c') represents the degree of compatibility between labels c and c'. It is a symmetric matrix, each element in the matrix is initialized to 9999, and the penalty value between each pair of labels is obtained through an iterative learning process.

(2) Iterative optimization learning method of conditional random field model parameters

After extracting the unary and binary features of the sampling points, the parameters of the CRF model are learned by iterative optimization, and the set of marked training sampling points is randomly divided into 5 equal parts, of which 4 parts are used as training sets and 1 part is used as the training set. as a validation set. First, the JointBoost classifier is trained with the sample set, and some parameters of the unary and binary items are learned. Then, the segmentation results are iteratively optimized with the validation set to learn the remaining parameters in the binary term of the CRF model.

The parts divided by the three projection views of each model are trained separately for the classifier, and the three views respectively calculate the number of votes falling in the corresponding category during retrieval. In this way, the accuracy of training is higher than that of putting all the features of the view components together, and it reduces the calculation amount of the online part, thereby improving the retrieval efficiency.

Parts segmentation can be used to obtain the part label sequence of sampling points. The sequence records the part label for each sampling point and its coordinate position in the image.

Since the sampling point marking results are arranged in an orderly manner along the contour line, the embodiment of the present invention designs a part map generation algorithm based on this feature, reads each sampling point mark and its image coordinate point in sequence, and records the upper part of each part label. A sampling point coordinate position, based on which the Euclidean distance between adjacent sampling points with the same label is calculated. If the distance is less than the contour line equidistant sampling step, a line segment is connected between the two points. If a new component label that has not generated a component diagram is read in, the component diagram matrix is created accordingly. While reading the sampling point marks, the number of sampling points of each component is also recorded. This process is carried out iteratively until the sequence of sampling point markers is read. Finally, the bounding box and extended boundary are calculated for the image of each part, so that the part is centered, thereby generating a part map with a size of 100*100 pixels. The weight of each component is the proportion of the sampling points of the component to the total sampling points.

The Zernike moment feature descriptor used constitutes the global feature of the view. The Zernike moment satisfies the invariance of rotation and has a good degree of discrimination for different shape contours. The Zernike moment has two parameters, namely n and m, n represents the order of the Zernike moment, and m represents the repetition number of the Zernike moment. The combination of each group of n and m can obtain a complex Zernike moment value, and the magnitude of the Zernike moment is used as the component of the global eigenvector. The more components selected for this kind of feature descriptor, the more accurate the result, but the more components, the more time-consuming calculation will increase. Considering both retrieval accuracy and calculation time consumption, a 10-dimensional Zernike moment is selected, and its n and m value combinations are shown in Table 1.

Table 1 The combination of n and m values of Zernike moment global features

n value 3 5 7 9 11 4 6 8 10 12 m value 3 3 3 3 3 4 4 4 4 4

The image is divided into 32 parts horizontally and vertically, and the blocks are moved in steps of 200/32 pixels. In the process of moving the block, if there is no contour line in the block, the statistical data of the block will be discarded directly, so as to avoid the situation that a large number of feature components are 0 due to the local blank of the image.

In the offline part, the K-means clustering algorithm is used to cluster the Gabor features to form 128 clusters, and the centers of each cluster are saved to form a dictionary. Finally, count the number of Gabor eigenvectors closest to the center of each cluster.

The direct adjacency of part labels provides the topology between parts, and the topological features are robust to part twisting. For example, the geometry of the legs of a running horse is quite different from that of a standing horse, but the legs are contiguous to the body parts regardless of whether they are running or not. Based on this feature, a graph model of the topological structure between components is established, and components are used as nodes of the graph, and adjacent components are connected by edges, which is realized by an adjacency list. If the part in the hand-drawn sketch is different from the adjacent part of the corresponding part in the projected view, the difference in the topological level of the two graphs is penalized, that is, the similarity score is reduced.

Considering the underlying geometric information and topological structure information between components, the weight value of the component is determined according to the proportion of the sampling points of the component to the total sampling points of all components and the proportion of the component in all model libraries. Equation (4) shows the calculation formula of component weight:

Formula (5) shows the calculation formula of the similarity score of the view angle between the sketch and the model:

The component topology of the sketch may not match the topology of the projected view, and a reasonable penalty is used here to account for the topology difference. Comparing the corresponding components in the topology adjacency list, if there is an inconsistency in the topology, the similarity score is punished according to the weight of the component in the projection view.

The amount of information provided by the projection views of different viewing angles of the 3D model is different. For example, it is almost impossible to obtain component information from the top view of a person, and it is impossible to see that it belongs to the model projection of a person, while the front view of a person clearly shows semantic components such as head, body, and limbs. Only the number of parts divided by the view objectively reflects the amount of information that the projected view can provide. Based on this feature, as shown in formula (6), calculate the proportion of the number of parts segmented in the front view, side view, and top view to the total number of parts segmented in the three views, and use them as the front view weight w _i,front and the side view weight w _i,side and top view weight w _i,top , using the formula (7) to show the calculation formula of the overall similarity score of each model in the current category:

Finally, the top 200 models are returned to the user according to the ascending order of the total score, and the corresponding thumbnails are displayed in the browser by paging, which completes a retrieval process.

Correspondingly, the embodiment of the present invention also provides a 3D model retrieval method based on hand-drawn sketch component segmentation, as shown in FIG. 5 , the method includes:

S501. Receive a hand-drawn query sketch, perform denoising processing on the hand-drawn query sketch to obtain a grayscale image, and perform binarization processing, boundary extension processing, and image hole filling processing on the grayscale image to obtain a processed image;

S502. Sampling the processed images at equal intervals to obtain sampling points, and adding component labels to the sampling points;

S503, extracting various feature vectors of sampling points;

S504, perform segmentation model training according to various feature vectors of the sampling points after adding component labels;

S505. Extract local features of components and calculate local similarity of components based on the segmentation model, and perform global view feature extraction and global view similarity calculation on the processed images, sort according to the total score, and return the sorting results to the client.

Among them, S501 further includes:

Denoise the hand-drawn query sketch to obtain a grayscale image;

Binarize the grayscale image;

Perform blank filling processing around the binarized image;

Carry out image hole filling processing on the image after blank filling processing.

S502 further includes:

Extract the outline of the processed image;

Sampling at equal intervals on the image after extracting the contour line to obtain sampling points;

Add part labels to sampling points.

S503 further includes:

Perform unary feature extraction on the sampling points after adding component labels;

Binary feature extraction is performed on the sampling points after adding part labels.

S504 further includes:

Carry out segmentation model training according to various feature vectors of sampling points after adding component labels;

Segment the sampling points after adding component labels according to the segmentation model.

Specifically, for the implementation process of the method embodiment of the present invention, reference may be made to the relevant description of the working principle of the system-related functional modules, which will not be repeated here.

In the embodiment of the present invention, the geometric information of the hand-drawn sketch components, the topology information between the components and the global information of the entire view are comprehensively used, and a dynamic weighting mechanism for three views is set up to amplify the impact of important perspectives on the total score , so that the retrieval of 3D models based on hand-drawn sketches is more accurate and effective; in addition, it can be used alone in the application scenarios of hand-drawn drawing parts segmentation such as sketch understanding and sketch classification.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium can include: Read Only Memory (ROM, Read Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

In addition, the 3D model retrieval system and method based on hand-drawn sketch parts segmentation provided by the embodiments of the present invention have been introduced in detail above. In this paper, specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments It is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and scope of application. In summary, The contents of this description should not be construed as limiting the present invention.

Claims (6)

1. A three-dimensional model retrieval system based on hand-drawn sketch parts segmentation, comprising a preprocessing module, a part labeling module, a sampling point feature extraction module, a parts segmentation module, and a similarity calculation and total score sorting module, characterized in that, The preprocessing module is configured to receive a hand-drawn query sketch, perform denoising processing on the hand-drawn query sketch to obtain a grayscale image, and perform binarization processing, boundary expansion processing, and image hole filling processing on the grayscale image, Obtain the processed image; the preprocessing module includes: A sketch denoising processing unit, configured to perform denoising processing on the hand-drawn query sketch to obtain a grayscale image; and a binarization processing unit, configured to perform binarization processing on the grayscale image; and A boundary extension processing unit, configured to perform blank filling processing around the binarized image; and An image hole filling processing unit, configured to perform image hole filling processing on the image after blank filling processing; The component labeling module is used to sample the processed image at equal intervals to obtain sampling points, and add component labels to the sampling points; The sampling point feature extraction module is used to extract various feature vectors of the sampling point; The part segmentation module includes: A segmentation model training unit, for performing segmentation model training according to various feature vectors of sampling points after adding part labels; and The component segmentation unit is used to segment the sampling points after adding component labels according to the segmentation model; The similarity calculation and total score sorting module is used to extract local features of parts and calculate local similarity of parts based on the segmentation model, and perform view global feature extraction and view global similarity calculation on the processed image, according to the total Scores are sorted and the sorted results are returned to the client. 2. The 3D model retrieval system based on hand-drawn sketch parts segmentation according to claim 1, wherein the part marking module includes: A contour extraction unit, configured to extract contours from the processed image; The sampling unit is used to sample the image at equal intervals after the contour line is extracted to obtain sampling points; The component labeling unit is used for adding component labels to the sampling points. 3. The 3D model retrieval system based on hand-drawn sketch parts segmentation according to claim 1, characterized in that the sampling point feature extraction module includes: A unitary feature extraction unit, which is used to extract a unitary feature from the sampling point after adding the component label; The binary feature extraction unit is used to perform binary feature extraction on the sampling points after adding component labels. 4. A three-dimensional model retrieval method based on hand-drawn sketch parts segmentation, is characterized in that, comprises the steps: receiving a hand-drawn query sketch, performing denoising processing on the hand-drawn query sketch to obtain a grayscale image, and performing binarization processing, boundary extension processing, and image hole filling processing on the grayscale image to obtain a processed image; Sampling the processed image at equal intervals to obtain sampling points, and adding component labels to the sampling points; Perform unary feature extraction on the sampling points after adding component labels; Perform binary feature extraction on the sampling points after adding component labels; Carry out segmentation model training according to various feature vectors of sampling points after adding component labels; Segment the sampling points after adding the component labels according to the segmentation model; Based on the segmentation model, extract local features of parts and calculate local similarity of parts, and perform global view feature extraction and global view similarity calculation on the processed images, sort according to the total score, and return the sorting results to the client. 5. The 3D model retrieval method based on hand-drawn sketch parts segmentation as claimed in claim 4, wherein said hand-drawn query sketch is subjected to denoising processing to obtain a grayscale image, and said grayscale image is subjected to binary Value processing, boundary extension processing, image hole filling processing, and steps to obtain a processed image include: Denoising the hand-drawn query sketch to obtain a grayscale image; Perform binarization processing on the grayscale image; Perform blank filling processing around the binarized image; Carry out image hole filling processing on the image after blank filling processing. 6. The 3D model retrieval method based on hand-drawn sketch parts segmentation as claimed in claim 4, wherein said processed images are sampled at equal intervals to obtain sampling points, and parts are added to said sampling points Labeling steps, including: Contour extraction is performed on the processed image; Sampling at equal intervals on the image after extracting the contour line to obtain sampling points; Add component labels to the sampling points.