CN113888751A - Method and device for identifying key points of joints and computer equipment - Google Patents
Method and device for identifying key points of joints and computer equipment Download PDFInfo
- Publication number
- CN113888751A CN113888751A CN202111214042.1A CN202111214042A CN113888751A CN 113888751 A CN113888751 A CN 113888751A CN 202111214042 A CN202111214042 A CN 202111214042A CN 113888751 A CN113888751 A CN 113888751A
- Authority
- CN
- China
- Prior art keywords
- image data
- joint
- convolution
- feature
- characteristic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000003062 neural network model Methods 0.000 claims abstract description 52
- 230000004913 activation Effects 0.000 claims abstract description 34
- 238000000605 extraction Methods 0.000 claims abstract description 28
- 238000003709 image segmentation Methods 0.000 claims abstract description 15
- 238000005070 sampling Methods 0.000 claims description 40
- 230000006870 function Effects 0.000 claims description 36
- 238000011176 pooling Methods 0.000 claims description 30
- 230000011218 segmentation Effects 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 10
- 238000012216 screening Methods 0.000 claims description 9
- 210000001503 joint Anatomy 0.000 description 15
- 238000010586 diagram Methods 0.000 description 10
- 210000000588 acetabulum Anatomy 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 5
- 238000002591 computed tomography Methods 0.000 description 3
- 238000011882 arthroplasty Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 210000001624 hip Anatomy 0.000 description 2
- 210000004394 hip joint Anatomy 0.000 description 2
- 210000003141 lower extremity Anatomy 0.000 description 2
- 238000005498 polishing Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 210000002414 leg Anatomy 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 210000004197 pelvis Anatomy 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10081—Computed x-ray tomography [CT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30008—Bone
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Molecular Biology (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Computational Biology (AREA)
- Medical Informatics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method, a device and computer equipment for identifying joint key points, wherein the method comprises the following steps: performing image feature extraction on the image data of the plurality of joints through a first neural network model to obtain first feature image data corresponding to the image data of each joint; selecting a plurality of pieces of target characteristic image data which accord with the characteristics of the joint image data from the first characteristic image data according to the output result of the preset activation function; performing image segmentation on each target characteristic image data through a second neural network model to obtain second characteristic image data, wherein the second characteristic image data comprises a plurality of sub-regions with different image characteristics; carrying out image stacking processing according to the plurality of sub-regions to obtain a three-dimensional joint model; and determining joint key points according to the three-dimensional joint model. And establishing a model to calculate the positions of key points of the joints by the aid of a neural network model. The joint key points are accurately and quickly positioned, errors are reduced, labor consumption is reduced, and efficiency is greatly improved.
Description
Technical Field
The invention relates to the technical field of machine learning, in particular to a method and a device for identifying joint key points and computer equipment.
Background
In the medical field, whether the location of key points can be accurately determined is an important factor in the success of some medical procedures. For example, in total hip arthroplasty, the location of the central point of the acetabulum determines the placement of the prosthesis and the degree of polishing of the acetabulum.
At present, in the prior art, only a doctor can select key points manually, but due to the complex joint structure, the selected key points have the problems of large error, large manual time consumption and low efficiency.
Disclosure of Invention
In view of the defects of the prior art, the application provides a method, a device and computer equipment for identifying key points of joints, so that accurate identification and positioning of the key points of the joints are realized.
In a first aspect, the present application provides a method for identifying joint key points, the method comprising:
performing image feature extraction on the image data of the plurality of joints through a first neural network model to obtain first feature image data corresponding to the image data of each joint;
selecting a plurality of pieces of target characteristic image data which accord with the characteristics of the joint image data from the first characteristic image data according to the output result of a preset activation function;
performing image segmentation on each target characteristic image data through a second neural network model to obtain second characteristic image data, wherein the second characteristic image data comprises a plurality of sub-regions with different image characteristics;
carrying out image stacking processing according to the plurality of sub-regions to obtain a three-dimensional joint model;
and determining joint key points according to the three-dimensional joint model.
Optionally, the image segmentation is performed on each target feature image data through a second neural network model to obtain second feature image data, including:
performing image feature extraction on each target feature image data through the second neural network model to obtain third feature image data;
performing up-sampling on the third characteristic image data to obtain up-sampled characteristic image data;
splicing the third characteristic image data with the up-sampling characteristic image data to obtain characteristic spliced image data;
adjusting the size of the feature stitching image data to be the same as that of the joint image data to obtain fourth feature image data;
and screening the plurality of fourth characteristic image data through a first activation function to obtain the second characteristic image data.
Optionally, the first neural network model includes: the system comprises a convolution module, a pooling module and a full-connection module;
the image feature extraction of the multiple joint image data through the first neural network model to obtain first feature image data corresponding to each joint image data includes:
performing convolution processing on the image data of each joint through a convolution core of the convolution module to obtain convolution image data;
performing maximal pooling on the convolution image data through the pooling module to obtain pooled image data;
and integrating the pooled image data through the full-connection module to obtain the first characteristic image data.
Optionally, the convolution module includes: convolutional layer, BN layer and Mish activation function; the obtaining of the convolution image data by performing convolution processing on the image data of each joint through the convolution kernel of the convolution module includes:
performing image feature extraction on the image data of each joint through the convolution layer to obtain the image data of the joint after convolution;
normalizing the joint image data after convolution through the BN layer to obtain normalized image data;
and carrying out nonlinear processing on the normalized image data through the Mish activation function to obtain the convolution image data.
Optionally, the performing image stacking processing according to the plurality of sub-regions to obtain a three-dimensional joint model includes:
randomly sampling the pixel points in the sub-regions to obtain a plurality of sampling points;
and obtaining the three-dimensional joint model under a world coordinate system according to the plurality of sampling points.
Optionally, the three-dimensional joint model is a hemispherical model, and the joint key point is a joint center coordinate;
determining joint key points according to the three-dimensional joint model comprises the following steps:
obtaining a set of standard spherical formulas corresponding to the three-dimensional joint model under the world coordinate system;
acquiring coordinates of a plurality of points on the three-dimensional joint model;
determining the joint center coordinates from the set of standard sphere equations and the coordinates of the plurality of points.
In a second aspect, the present application provides a joint keypoint identification device, comprising:
the first feature extraction module is used for extracting image features of the image data of the multiple joints through the first neural network model to obtain first feature image data corresponding to the image data of each joint;
the selection module is used for selecting target characteristic image data which accord with the characteristics of the joint image data from the first characteristic image data according to the output result of a preset activation function;
the segmentation module is used for carrying out image segmentation on each target characteristic image data through a second neural network model to obtain second characteristic image data, and the second characteristic image data comprises a plurality of sub-regions with different image characteristics;
the modeling module is used for carrying out image stacking processing according to the plurality of sub-regions with different image characteristics to obtain a three-dimensional joint model;
and the key point determining module is used for determining joint key points according to the three-dimensional joint model.
Optionally, the segmentation module includes:
the second feature extraction submodule is used for carrying out image feature extraction on each target feature image data through the second neural network model to obtain third feature image data;
the sampling submodule is used for performing up-sampling on the third characteristic image data to obtain up-sampling characteristic image data;
the splicing submodule is used for splicing the third characteristic image data and the up-sampling characteristic image data to obtain characteristic spliced image data;
the adjusting submodule is used for adjusting the size of the feature splicing image data to be the same as that of the joint image data to obtain fourth feature image data;
and the screening submodule is used for screening the fourth feature image data through a first activation function to obtain the second feature image data.
In a third aspect, the present application provides a computer device comprising a processor and a memory, said memory storing a computer program, said computer program when executed by said processor performing the method of identifying a joint keypoint of any of the above.
In a fourth aspect, the present application provides a computer readable storage medium storing a computer program which, when run on a processor, performs the method of identifying joint keypoints as described in any of the above.
According to the method, the device and the computer equipment for identifying the joint key points, image features of a plurality of pieces of joint image data are extracted through a first neural network model, and first feature image data corresponding to each piece of joint image data are obtained; selecting a plurality of pieces of target characteristic image data which accord with the characteristics of the joint image data from the first characteristic image data according to the output result of a preset activation function; performing image segmentation on each target characteristic image data through a second neural network model to obtain second characteristic image data, wherein the second characteristic image data comprises a plurality of sub-regions with different image characteristics; carrying out image stacking processing according to the plurality of sub-regions to obtain a three-dimensional joint model; and determining joint key points according to the three-dimensional joint model. Therefore, the three-dimensional joint model is established by the aid of the neural network model, the positions of the key points of the joints are calculated according to the three-dimensional joint model, the key points of the joints can be accurately and quickly positioned, errors are reduced, labor time is reduced, and efficiency is greatly improved.
Drawings
In order to more clearly illustrate the technical solution of the present invention, the drawings required to be used in the embodiments will be briefly described below, and it should be understood that the following drawings only illustrate some embodiments of the present invention, and therefore should not be considered as limiting the scope of the present invention. Like components are numbered similarly in the various figures.
FIG. 1 is a flow chart diagram illustrating a method for identifying key points of a joint;
FIG. 2 is a schematic diagram of a first neural network model structure of a joint key point identification method;
FIG. 3 is a structural diagram of a second neural network model of a joint key point identification method;
fig. 4 is a block diagram of a joint key point recognition apparatus.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments.
The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.
Hereinafter, the terms "including", "having", and their derivatives, which may be used in various embodiments of the present invention, are only intended to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing, and should not be construed as first excluding the existence of, or adding to, one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing.
Furthermore, the terms "first," "second," "third," and the like are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of the present invention belong. The terms (such as those defined in commonly used dictionaries) should be interpreted as having a meaning that is consistent with their contextual meaning in the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein in various embodiments of the present invention.
In the prior art, whether some joint key points can be accurately determined is an unimportant influence factor of success of some medical operations. For example, in total hip arthroplasty, the location of the central point of the acetabulum determines the placement of the prosthesis and the degree of polishing of the acetabulum. Therefore, the accurate positioning of the acetabulum center point can judge the length difference of the lower limbs before operation, and avoids various discomfort conditions of the patient after operation, such as the problems that the length difference of the two lower limbs is too large, the two legs are not as long, the patient cannot walk or limp after the operation. However, at present, through digital reconstruction of the radiographic image, doctors can only calculate points by manual pointing, but because of the complexity of the acetabulum fossa and the position of the center of the acetabulum is at a position which is high relative to the pelvis, the manual pointing by the doctors easily causes large errors and is difficult to meet the actual requirement.
Example 1
The embodiment of the application provides a method for identifying key points of joints, which realizes accurate identification of the key points of the joints. As shown in fig. 1, the method includes:
s101, image feature extraction is carried out on a plurality of joint image data through a first neural network model, and first feature image data corresponding to each joint image data are obtained;
specifically, the plurality of joint images are CT original images of a plurality of joints previously labeled by a doctor. Taking the hip joint as an example, an axial CT (Computed Tomography)) original image of the hip joint is used. The first neural network model is a variation of the basic structure of the vgg16 network model. Fig. 1 is a schematic diagram of a first neural network model, which includes a process of five-segment convolution, wherein each segment convolution process has 2 or 3 convolution modules, and a maximum pooling module is connected to the end of each segment convolution. For example, the first convolution: conv3-64 and conv3-64, representing a convolution kernel of 3 × 3, performed two convolution operations with channel number 64, followed by a "maxporoling" maximum pooling operation. And starting full connection operation after five-section convolution processes, wherein the full connection stage comprises three full connection layers including FC-4096, FC-4096 and FC-2, wherein FC-4096 represents that 4096 nodes of the full connection layers are connected. Each node of the full link layer is connected with all nodes of the previous layer, all outputs and inputs of the full link layer are connected, and parameters of the full link layer are the most. Finally, using a Softmax activation function to Output, and using Softmax processing to obtain the probability of classification Output (Output). And extracting the characteristic data of each joint image through a first neural network to obtain first characteristic image data.
S102, selecting a plurality of pieces of target characteristic image data which accord with the characteristics of the joint image data from the first characteristic image data according to the output result of a preset activation function;
specifically, the classification task is realized through the first neural network model, a floating point number is output through an activation function at the end of the first neural network model, the floating point number is a decimal number, the numeric value range of the decimal number is between 0 and 1, and a threshold value is preset for judgment. If the threshold is set to 0.5 by default, the output value of the image a from the first neural network model is 0.7, if 0.7 is greater than 0.5, the image a is considered to be closer to 1, the category of the image a is marked as 1, which indicates that the image a meets the standard, and the image a is considered to belong to the image data meeting the characteristics of the joint image data. And if the value of the image B is less than or equal to 0.5, the image B is considered to be image data which does not accord with the characteristics of the joint image data. The activation function may be selected as follows in this embodiment:
formula (1):
the x is a feature value input by the joint image data.
Through the steps, the image data which accords with the characteristics of the joint image data are searched, the range to be processed in the segmentation process of the subsequent step S103 is narrowed, irrelevant parts are eliminated, the efficiency is improved, and unnecessary processing processes are reduced.
S103, carrying out image segmentation on each target characteristic image data through a second neural network model to obtain second characteristic image data, wherein the second characteristic image data comprises a plurality of sub-regions with different image characteristics;
specifically, in addition to S101 and S103, a plurality of pieces of target feature image data that match the features of the joint image data are subjected to segmentation processing, and 2D segmentation is performed by the second neural network model. As shown in fig. 3, the structural schematic diagram of the second neural network model adopts the deformation of the U-Net network model, the second neural network model includes two parts of image feature extraction and upsampling, the feature extraction part performs convolution first and then pooling, and one scale is obtained after pooling every time; and every time sampling is carried out, merging with the same scale of the number of channels corresponding to the feature extraction part, and outputting after splicing and merging to obtain second feature image data. The image segmentation is that the second characteristic image data obtained after the image segmentation is segmented into a plurality of different sub-regions, each small region has different image characteristics and can be segmented according to the pixel level, and the pixel level of the same sub-region is the same.
The segmentation requirements of this step may not be very strict, false negative results may occur, but false positive results may not occur. A false negative result means that this sample is a positive sample but the predicted result is a negative sample. A false positive result refers to a sample being a negative sample but the predicted result being a positive sample. The positive sample refers to a sample belonging to an image data category conforming to the characteristics of the joint image data, and the negative sample refers to a sample not belonging to an image data category conforming to the characteristics of the joint image data.
S104, carrying out image stacking processing according to the plurality of sub-regions to obtain a three-dimensional joint model;
specifically, a plurality of sub-regions obtained by segmentation are stacked to form a three-dimensional joint model in a world coordinate system, and the world coordinate system is a three-dimensional coordinate system. So that the true positive coordinates in the three-dimensional joint model can be acquired in the subsequent step S105. The true positive coordinate refers to a sample belonging to an image data class conforming to the characteristics of the joint image data, and the prediction is also a sample of an image data class conforming to the characteristics of the joint image data.
S105, determining joint key points according to the three-dimensional joint model.
After a three-dimensional joint model in a world coordinate system is obtained, a plurality of coordinate points exist on the three-dimensional key model, and joint key points are calculated in the world coordinate system through a traditional algorithm according to the coordinate points. The joint key points can be efficiently and accurately determined, and the key points can be accurately identified.
In a specific embodiment, the image segmentation of each target feature image data through the second neural network model to obtain second feature image data includes:
performing image feature extraction on each target feature image data through the second neural network model to obtain third feature image data;
performing up-sampling on the third characteristic image data to obtain up-sampled characteristic image data;
splicing the third characteristic image data with the up-sampling characteristic image data to obtain characteristic spliced image data;
adjusting the size of the feature stitching image data to be the same as that of the joint image data to obtain fourth feature image data;
and screening the plurality of fourth characteristic image data through a first activation function to obtain the second characteristic image data.
Specifically, as shown in fig. 3, after the target feature image data is input, the processing procedure of the second neural network model is as follows: firstly, extracting the features of the image through the left half part, and performing convolution of a first section: conv3-64 and conv3-64, representing that a convolution kernel of 3 × 3 is subjected to convolution operation with 64 channels twice, and then maximum pooling is carried out; and a second convolution: conv3-128 and conv3-128, representing that a convolution kernel of 3 × 3 performs two convolution operations with the channel number of 128, and then performs one maximum pooling; and (3) convolution of a third section: conv3-256 and conv3-256, representing that a convolution kernel of 3 × 3 performs convolution operation with 256 channels twice, and then performs maximum pooling once; and a fourth convolution: conv3-512 and conv3-512, representing that a convolution kernel of 3 × 3 is subjected to convolution operation with 512 channels twice, and then to maximum pooling once; and a fifth convolution stage: conv3-1024 and conv3-1024, representing that the convolution kernel of 3 × 3 performs two times of convolution operation with 1024 channels. Wherein the maximum pooling is a maximum pooling with a step size of 2, and is subjected to a total of 4 times of maximum pooling, and the number of channels is doubled after each pooling until the number of channels finally becomes 1024. And the convolved feature maps will be saved before each pooling.
Then, upsampling is performed, bilinear interpolation is used in the upsampling (Up-conv), convolution operation is also performed before each upsampling, and the feature map saved before pooling is spliced (Concat) with the upsampled feature map with the same length and width, as shown in fig. 3, for example: and performing classified output after the size of the original image is up-sampled by adding the front and back features and fusing, and performing activation output on a final result by using a sigmoid activation function.
In addition, the number of times of network training is reduced, the result of false positive is greatly reduced, the result of true positive is also reduced, the final result is not influenced, and only more than 100 true positive coordinate points are required to be ensured to appear in the world coordinate system, so that the method is used as a basis for accurately establishing the three-dimensional joint model subsequently.
In a specific embodiment, the first neural network model comprises: the system comprises a convolution module, a pooling module and a full-connection module;
the image feature extraction of the multiple joint image data through the first neural network model to obtain first feature image data corresponding to each joint image data includes:
performing convolution processing on the image data of each joint through a convolution core of the convolution module to obtain convolution image data;
performing maximal pooling on the convolution image data through the pooling module to obtain pooled image data;
and integrating the pooled image data through the full-connection module to obtain the first characteristic image data.
Specifically, as shown in fig. 2, the first convolution: conv3-64 and conv3-64, representing that a convolution kernel of 3 × 3 is subjected to convolution operation with 64 channels twice, and then maximum pooling is carried out; and a second convolution: conv3-128 and conv3-128, representing that a convolution kernel of 3 × 3 performs two convolution operations with the channel number of 128, and then performs one maximum pooling; and (3) convolution of a third section: conv3-256, conv3-256 and conv1-256, representing that a convolution kernel of 3 × 3 performs convolution operation of 256 channels twice, and a convolution kernel of 1 × 1 performs convolution operation of 256 channels once, and then performs maximum pooling once; and a fourth convolution: conv3-512, conv3-512 and conv1-512, representing that a convolution kernel of 3 × 3 performs two convolution operations with the channel number of 512, and a convolution kernel of 1 × 1 performs one convolution operation with the channel number of 512 and then performs one maximum pooling; and a fifth convolution stage: conv3-512 and conv3-512, representing that a convolution kernel of 3 × 3 performs a convolution operation of 512 channels twice, and a convolution kernel of 1 × 1 performs a convolution operation of 512 channels once.
Then, after the second section of convolution, the third section of convolution and the fourth section of convolution, performing hollow sampling (scaled), wherein the hollow sampling is adopted, and the specific process is as follows: taking an 8 × 8 picture as an example, 64 features in total, using a hole sampling mode to divide the picture into 4 × 4 feature maps in sequence, then extracting (0, 0) in each feature map to form a feature map, then extracting (1, 1) to form a feature map, and so on, and finally obtaining 4 × 4 feature maps, but the relative position relationship of each element is unchanged, reducing the feature maps under the condition of ensuring the original picture structure, then splicing the feature maps to form 4 × 4 × 4 feature maps, wherein the number of the features is also 64, but the size can be spliced with the subsequent smaller feature maps. Compared with the original neural network model, the cavity sampling method has the advantages that the receptive field is increased, strong representation information is obtained, and feature fusion is enhanced.
In a specific embodiment, the convolution module includes: convolutional layer, BN layer and Mish activation function; the obtaining of the convolution image data by performing convolution processing on the image data of each joint through the convolution kernel of the convolution module includes:
performing image feature extraction on the image data of each joint through the convolution layer to obtain the image data of the joint after convolution;
normalizing the joint image data after convolution through the BN layer to obtain normalized image data;
and carrying out nonlinear processing on the normalized image data through the Mish activation function to obtain the convolution image data.
Specifically, the structure of one convolution module is convolution layer + BN layer + activation function, where the BN layer is used to normalize the data distribution and has a slight over-fitting prevention effect, i.e. a regularization effect, and the setting of the BN layer can be selected according to actual requirements.
The Mish activation function is smoother compared with the relu activation function, the U-net network model uses the relu activation function, the property of the relu activation function is relu (x) max (0, x), and a phenomenon that all neurons become 0 in a certain layer is possibly caused to be called dead relu.
Formula (2): mix ═ x tanh (ln (1+ e)x))
X is a characteristic value of the input joint image data, and e represents a natural constant e.
In a specific embodiment, the performing image stacking processing according to the plurality of sub-regions to obtain a three-dimensional joint model includes:
randomly sampling the pixel points in the sub-regions to obtain a plurality of sampling points;
and obtaining the three-dimensional joint model under a world coordinate system according to the plurality of sampling points.
Specifically, the random sampling is performed 10 times of decimation, which is completely random, and is cycled N times. Specifically, a Central Processing Unit (Central Processing Unit CPU) can be used for random sampling, the operation speed can reach ten-digit millisecond level, and the result error of the two methods is less than 0.001 mm when N > 12. Compared with other schemes, for example, all coordinate points are subjected to permutation and combination of extracting 10 points at a time, and all obtained results are averaged. The amount of data generated is huge, a Graphics Processing Unit (GPU) is necessary for calculation, and the operation speed is still slow. According to the embodiment, the sampling calculation precision is improved by adopting random sampling, the calculation amount is not too large, the consumed time is too long, and the calculation efficiency is improved.
In a specific embodiment, the three-dimensional joint model is a hemispherical surface model, and the joint key point is a joint center coordinate;
determining joint key points according to the three-dimensional joint model comprises the following steps:
obtaining a set of standard spherical formulas corresponding to the three-dimensional joint model under the world coordinate system;
acquiring coordinates of a plurality of points on the three-dimensional joint model;
determining the joint center coordinates from the set of standard sphere equations and the coordinates of the plurality of points.
Wherein, the standard spherical center equation is as follows:
formula (3): (x-a)2+(y-b)2+(z-c)2＝R2
Wherein, x, y and z refer to three coordinates of the selected point in the random sampling, a, b, c and R are unknowns, a, b and c represent world coordinates of a sphere center, R represents a semi-sphere radius, then 10 points are selected to be substituted to form a group of quadric quadratic equations consisting of 10 equations, the quadric quadratic equations are solved by a Runge Kutta method, and a Central Processing Unit (CPU) can be used for calculation. Thereby determining the central coordinates of the hemispherical surface, namely the coordinates of the key points of the joints.
In the method for identifying key points of joints provided by this embodiment, image features of a plurality of pieces of joint image data are extracted through a first neural network model, and first feature image data corresponding to each piece of joint image data is obtained; selecting a plurality of pieces of target characteristic image data which accord with the characteristics of the joint image data from the first characteristic image data according to the output result of a preset activation function; performing image segmentation on each target characteristic image data through a second neural network model to obtain second characteristic image data, wherein the second characteristic image data comprises a plurality of sub-regions with different image characteristics; carrying out image stacking processing according to the plurality of sub-regions to obtain a three-dimensional joint model; and determining joint key points according to the three-dimensional joint model. And establishing a model to calculate the positions of key points of the joints by the aid of a neural network model. The joint key points are accurately and quickly positioned, errors are reduced, labor consumption is reduced, and efficiency is greatly improved.
Example 2
An embodiment of the present application provides an apparatus for identifying a joint key point, as shown in fig. 4, the apparatus 400 for identifying a joint key point includes:
the first feature extraction module 401 is configured to perform image feature extraction on multiple pieces of joint image data through a first neural network model to obtain first feature image data corresponding to each piece of joint image data;
a selecting module 402, configured to select, from the first feature image data, target feature image data that meets a feature of the joint image data according to an output result of a preset activation function;
a segmentation module 403, configured to perform image segmentation on each target feature image data through a second neural network model to obtain second feature image data, where the second feature image data includes a plurality of sub-regions with different image features;
the modeling module 404 is configured to perform image stacking processing according to the plurality of sub-regions with different image characteristics to obtain a three-dimensional joint model;
and a key point determining module 405, configured to determine a joint key point according to the three-dimensional joint model.
Optionally, the segmentation module 403 includes:
the second feature extraction submodule is used for carrying out image feature extraction on each target feature image data through the second neural network model to obtain third feature image data;
the sampling submodule is used for performing up-sampling on the third characteristic image data to obtain up-sampling characteristic image data;
the splicing submodule is used for splicing the third characteristic image data and the up-sampling characteristic image data to obtain characteristic spliced image data;
the adjusting submodule is used for adjusting the size of the feature splicing image data to be the same as that of the joint image data to obtain fourth feature image data;
and the screening submodule is used for screening the fourth feature image data through a first activation function to obtain the second feature image data.
In this embodiment, the first neural network model includes: the system comprises a convolution module, a pooling module and a full-connection module; a first feature extraction module 401, configured to perform convolution processing on each joint image data through a convolution kernel of the convolution module to obtain convolution image data; performing maximal pooling on the convolution image data through the pooling module to obtain pooled image data; and integrating the pooled image data through the full-connection module to obtain the first characteristic image data.
In this embodiment, the convolution module includes: convolutional layer, BN layer and Mish activation function; the obtaining of the convolution image data by performing convolution processing on the image data of each joint through the convolution kernel of the convolution module includes: performing image feature extraction on the image data of each joint through the convolution layer to obtain the image data of the joint after convolution; normalizing the joint image data after convolution through the BN layer to obtain normalized image data; and carrying out nonlinear processing on the normalized image data through the Mish activation function to obtain the convolution image data.
The modeling module 404 in this embodiment is configured to perform random sampling on the pixel points in the multiple sub-regions to obtain multiple sampling points; and obtaining the three-dimensional joint model under a world coordinate system according to the plurality of sampling points.
A key point determining module 405 in this embodiment, configured to obtain a set of standard spherical formulas corresponding to the three-dimensional joint model in the world coordinate system; acquiring coordinates of a plurality of points on the three-dimensional joint model; determining the joint center coordinates from the set of standard sphere equations and the coordinates of the plurality of points.
In the device for identifying key points of joints provided by this embodiment, image features of a plurality of pieces of joint image data are extracted through a first neural network model, and first feature image data corresponding to each piece of joint image data is obtained; selecting a plurality of pieces of target characteristic image data which accord with the characteristics of the joint image data from the first characteristic image data according to the output result of a preset activation function; performing image segmentation on each target characteristic image data through a second neural network model to obtain second characteristic image data, wherein the second characteristic image data comprises a plurality of sub-regions with different image characteristics; carrying out image stacking processing according to the plurality of sub-regions to obtain a three-dimensional joint model; and determining joint key points according to the three-dimensional joint model. And establishing a model to calculate the positions of key points of the joints by the aid of a neural network model. The joint key points are accurately and quickly positioned, errors are reduced, labor consumption is reduced, and efficiency is greatly improved.
Example 3
The embodiment of the present application provides a computer device, which includes a processor and a memory, where the memory stores a computer program, and the computer program executes the method for identifying joint key points according to any one of the above embodiments 1 when the computer program runs on the processor.
For specific implementation steps, reference may be made to the description related to the joint key point identification method provided in embodiment 1, and details are not repeated here in order to avoid repetition.
Example 4
Embodiments of the present application provide a computer-readable storage medium storing a computer program which, when run on a processor, performs a method of identifying joint keypoints as described in any one of embodiments 1 above.
For specific implementation steps, reference may be made to the description related to the joint key point identification method provided in embodiment 1, and details are not repeated here in order to avoid repetition.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative and, for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, each functional module or unit in each embodiment of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention or a part of the technical solution that contributes to the prior art in essence can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention.
Claims (10)
1. A method for identifying key points of a joint, the method comprising:
performing image feature extraction on the image data of the plurality of joints through a first neural network model to obtain first feature image data corresponding to the image data of each joint;
selecting a plurality of pieces of target characteristic image data which accord with the characteristics of the joint image data from the first characteristic image data according to the output result of a preset activation function;
performing image segmentation on each target characteristic image data through a second neural network model to obtain second characteristic image data, wherein the second characteristic image data comprises a plurality of sub-regions with different image characteristics;
carrying out image stacking processing according to the plurality of sub-regions to obtain a three-dimensional joint model;
and determining joint key points according to the three-dimensional joint model.
2. The method of claim 1, wherein performing image segmentation on each target feature image data through a second neural network model to obtain second feature image data comprises:
performing image feature extraction on each target feature image data through the second neural network model to obtain third feature image data;
performing up-sampling on the third characteristic image data to obtain up-sampled characteristic image data;
splicing the third characteristic image data with the up-sampling characteristic image data to obtain characteristic spliced image data;
adjusting the size of the feature stitching image data to be the same as that of the joint image data to obtain fourth feature image data;
and screening the plurality of fourth characteristic image data through a first activation function to obtain the second characteristic image data.
3. The method of claim 1, wherein the first neural network model comprises: the system comprises a convolution module, a pooling module and a full-connection module;
the image feature extraction of the multiple joint image data through the first neural network model to obtain first feature image data corresponding to each joint image data includes:
performing convolution processing on the image data of each joint through a convolution core of the convolution module to obtain convolution image data;
performing maximal pooling on the convolution image data through the pooling module to obtain pooled image data;
and integrating the pooled image data through the full-connection module to obtain the first characteristic image data.
4. The method of claim 3, wherein the convolution module comprises: convolutional layer, BN layer and Mish activation function; the obtaining of the convolution image data by performing convolution processing on the image data of each joint through the convolution kernel of the convolution module includes:
performing image feature extraction on the image data of each joint through the convolution layer to obtain the image data of the joint after convolution;
normalizing the joint image data after convolution through the BN layer to obtain normalized image data;
and carrying out nonlinear processing on the normalized image data through the Mish activation function to obtain the convolution image data.
5. The method of claim 1, wherein the image stacking process from the plurality of sub-regions to obtain a three-dimensional joint model comprises:
randomly sampling the pixel points in the sub-regions to obtain a plurality of sampling points;
and obtaining the three-dimensional joint model under a world coordinate system according to the plurality of sampling points.
6. The method of claim 5, wherein the three-dimensional joint model is a hemispherical model, and the joint key points are joint center coordinates;
determining joint key points according to the three-dimensional joint model comprises the following steps:
obtaining a set of standard spherical formulas corresponding to the three-dimensional joint model under the world coordinate system;
acquiring coordinates of a plurality of points on the three-dimensional joint model;
determining the joint center coordinates from the set of standard sphere equations and the coordinates of the plurality of points.
7. An apparatus for identifying key points of a joint, the apparatus comprising:
the first feature extraction module is used for extracting image features of the image data of the multiple joints through the first neural network model to obtain first feature image data corresponding to the image data of each joint;
the selection module is used for selecting target characteristic image data which accord with the characteristics of the joint image data from the first characteristic image data according to the output result of a preset activation function;
the segmentation module is used for carrying out image segmentation on each target characteristic image data through a second neural network model to obtain second characteristic image data, and the second characteristic image data comprises a plurality of sub-regions with different image characteristics;
the modeling module is used for carrying out image stacking processing according to the plurality of sub-regions with different image characteristics to obtain a three-dimensional joint model;
and the key point determining module is used for determining joint key points according to the three-dimensional joint model.
8. The apparatus of claim 7, wherein the segmentation module comprises:
the second feature extraction submodule is used for carrying out image feature extraction on each target feature image data through the second neural network model to obtain third feature image data;
the sampling submodule is used for performing up-sampling on the third characteristic image data to obtain up-sampling characteristic image data;
the splicing submodule is used for splicing the third characteristic image data and the up-sampling characteristic image data to obtain characteristic spliced image data;
the adjusting submodule is used for adjusting the size of the feature splicing image data to be the same as that of the joint image data to obtain fourth feature image data;
and the screening submodule is used for screening the fourth feature image data through a first activation function to obtain the second feature image data.
9. A computer device comprising a processor and a memory, the memory storing a computer program which, when executed by the processor, performs the method of keypoint identification of a joint according to any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that it stores a computer program which, when run on a processor, performs the method of keypoint identification of a joint according to any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111214042.1A CN113888751A (en) | 2021-10-19 | 2021-10-19 | Method and device for identifying key points of joints and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111214042.1A CN113888751A (en) | 2021-10-19 | 2021-10-19 | Method and device for identifying key points of joints and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113888751A true CN113888751A (en) | 2022-01-04 |
Family
ID=79003383
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111214042.1A Pending CN113888751A (en) | 2021-10-19 | 2021-10-19 | Method and device for identifying key points of joints and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113888751A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024066852A1 (en) * | 2022-09-29 | 2024-04-04 | 中国电信股份有限公司 | Image downsampling method and device, and computer readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180061058A1 (en) * | 2016-08-26 | 2018-03-01 | Elekta, Inc. | Image segmentation using neural network method |
| CN110648337A (en) * | 2019-09-23 | 2020-01-03 | 武汉联影医疗科技有限公司 | Hip joint segmentation method, hip joint segmentation device, electronic apparatus, and storage medium |
| CN111223146A (en) * | 2020-02-13 | 2020-06-02 | 张逸凌 | Processing method and computing device for hip joint image |
| CN111898622A (en) * | 2019-05-05 | 2020-11-06 | 阿里巴巴集团控股有限公司 | Information processing method, information display method, model training method, information display system, model training system and equipment |
| CN113012155A (en) * | 2021-05-07 | 2021-06-22 | 刘慧烨 | Bone segmentation method in hip image, electronic device, and storage medium |
-
2021
- 2021-10-19 CN CN202111214042.1A patent/CN113888751A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180061058A1 (en) * | 2016-08-26 | 2018-03-01 | Elekta, Inc. | Image segmentation using neural network method |
| CN111898622A (en) * | 2019-05-05 | 2020-11-06 | 阿里巴巴集团控股有限公司 | Information processing method, information display method, model training method, information display system, model training system and equipment |
| CN110648337A (en) * | 2019-09-23 | 2020-01-03 | 武汉联影医疗科技有限公司 | Hip joint segmentation method, hip joint segmentation device, electronic apparatus, and storage medium |
| CN111223146A (en) * | 2020-02-13 | 2020-06-02 | 张逸凌 | Processing method and computing device for hip joint image |
| CN113012155A (en) * | 2021-05-07 | 2021-06-22 | 刘慧烨 | Bone segmentation method in hip image, electronic device, and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| SHUJUN LIANG 等: "Multi-View Spatial Aggregation Framework for Joint Localization and Segmentation of Organs at Risk in Head and Neck CT Images", 《IEEE TRANSACTIONS ON MEDICAL IMAGING》, vol. 39, no. 9, 24 February 2020 (2020-02-24), pages 2794 - 2805 * |
| 马安邦 等: "基于贝叶斯决策的髋关节自动分割方法", 《中国组织工程研究》, vol. 20, no. 39, 23 September 2016 (2016-09-23), pages 5873 - 5878 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024066852A1 (en) * | 2022-09-29 | 2024-04-04 | 中国电信股份有限公司 | Image downsampling method and device, and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110838124B (en) | Method, system, and medium for segmenting images of objects having sparse distribution | |
| US20210365717A1 (en) | Method and apparatus for segmenting a medical image, and storage medium | |
| CN113313234A (en) | Neural network system and method for image segmentation | |
| CN108986891A (en) | Medical imaging processing method and processing device, electronic equipment and storage medium | |
| CN113689402A (en) | Deep learning-based femoral medullary cavity form identification method, device and storage medium | |
| JP2023520846A (en) | Image processing method, image processing apparatus, computer program and computer equipment based on artificial intelligence | |
| CN113034495B (en) | Spine image segmentation method, medium and electronic device | |
| CN115393272B (en) | Knee joint patella replacement three-dimensional preoperative planning system and method based on deep learning | |
| CN113706463A (en) | Method, device and equipment for automatically detecting key points of joint image based on deep learning and storage medium | |
| CN113221869B (en) | Medical invoice structured information extraction method, device equipment and storage medium | |
| Shan et al. | SCA-Net: A spatial and channel attention network for medical image segmentation | |
| CN111667459A (en) | Medical sign detection method, system, terminal and storage medium based on 3D variable convolution and time sequence feature fusion | |
| CN115546270A (en) | Image registration method, model training method and equipment for multi-scale feature fusion | |
| CN113888751A (en) | Method and device for identifying key points of joints and computer equipment | |
| Zhang et al. | Consecutive context perceive generative adversarial networks for serial sections inpainting | |
| CN114627299A (en) | Method for detecting and segmenting camouflage target by simulating human visual system | |
| CN117710760B (en) | Method for detecting chest X-ray focus by using residual noted neural network | |
| CN112750110A (en) | Evaluation system for evaluating lung lesion based on neural network and related products | |
| CN112766332A (en) | Medical image detection model training method, medical image detection method and device | |
| CN113327221B (en) | Image synthesis method, device, electronic equipment and medium for fusing ROI (region of interest) | |
| CN113837192B (en) | Image segmentation method and device, and neural network training method and device | |
| CN110147715A (en) | A kind of retina OCT image Bruch film angle of release automatic testing method | |
| CN118541728A (en) | Image generation device, image generation method, image generation program, learning device, and learning data | |
| CN114881930A (en) | 3D target detection method, device, equipment and storage medium based on dimension reduction positioning | |
| CN112884706A (en) | Image evaluation system based on neural network model and related product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |