TWI850670B - System and method for cardiovascular risk prediction and computer readable medium thereof - Google Patents

Abstract

Provided are a system and a method for cardiovascular risk prediction, where artificial intelligence is utilized to perform segmentation on non-contrast or contrast medical images to identify precise regions of the heart, pericardium, and aorta of a subject, such that the adipose tissue volume and calcium score can be derived from the medical images to assist in cardiovascular risk prediction. Also provided is a computer readable medium for storing a computer executable code to implement the method.

Description

Systems and methods for cardiovascular risk prediction and computer-readable media thereof

The present invention relates to medical image analysis, and more particularly to a system and method for cardiovascular risk prediction and a computer-readable medium thereof.

In the field of medical image analysis, quantification of calcification in the heart/aorta and the adipose tissue surrounding or within the heart/aorta (e.g., epicardial adipose tissue, EAT) is an important predictor of future cardiovascular risk. In particular, quantification of heart/aorta calcification can reflect the amount of calcified plaques in the aorta and coronary arteries, and quantification of intrapericardial EAT may reflect the regional thickness of the fat surrounding the coronary arteries, which can lead to inflammation and/or coronary atherosclerosis.

Existing technologies for identifying the above variables, such as invasive diagnostic tests (e.g., cardiac catheterization), may help obtain accurate lesion identification, but the above technologies are associated with additional surgical risks and medical expenses. Therefore, non-invasive methods for diagnosing cardiovascular risk are highly valued in the market.

Based on the above reasons, there is an urgent need to use artificial intelligence to segment the heart, aorta and/or pericardium from non-contrast or contrast medical images, and derive EAT and calcification integral from them to predict cardiovascular risk.

In summary, the present disclosure provides a system for cardiovascular risk prediction, which includes: a segmentation module for segmenting a region from a medical image; and an extraction module for extracting analysis results from the region of the medical image.

The present disclosure also provides a cardiovascular risk prediction method, comprising: configuring a segmentation module to segment a region from a medical image; and configuring an extraction module to extract an analysis result from the region of the medical image.

In at least one embodiment of the present disclosure, the medical image is a non-developing CT scan image. In at least one embodiment of the present disclosure, the medical image is a developing medical image.

In at least one embodiment of the present disclosure, the segmentation module is executed by a machine learning model to segment regions from a medical image, and the machine learning model has a network architecture including an encoder part, a decoder part, an attention mechanism, and a variational self-encoder-decoder branch.

In at least one embodiment of the present disclosure, an attention mechanism is configured to highlight salient features through residual connections between an encoder portion and a decoder portion, and a variational self-encoder-decoder branch is configured to reconstruct a medical image based on features of endpoints of the encoder portion during training of a machine learning model.

In at least one embodiment of the present disclosure, a model training module is included for providing training for a machine learning model by the following steps: preprocessing training data to a predetermined consistency; enhancing the training data by performing random cropping, random spatial flipping and/or random intensity scaling or translation on the training data; training the machine learning model using the training data; and verifying the training results of the machine learning model using a loss function.

In at least one embodiment of the present disclosure, the training data is generated by manually and/or with the help of an auxiliary annotation model to label non-visual or visual medical images.

In at least one embodiment of the present disclosure, the analysis result includes the volume of fat tissue in the region, and the extraction module includes a fat extraction unit configured to quantify the volume of fat tissue in the pericardium in the region by the following steps: calculating the Hounsfield unit value of the pericardium based on the attenuation coefficient under computer tomography; defining the positive and negative standard deviation range of the Hounsfield unit value according to the noise tolerance limit; and determining the volume of the fat tissue in the pericardium according to the range. In some embodiments, the fat tissue can be epicardial fat tissue, but the present disclosure is not limited thereto.

In at least one embodiment of the present disclosure, the analysis result includes a calcification integral of the region, and wherein the extraction module includes a calcium extraction unit configured to quantify the calcification integral of the heart or aorta from the region by the following steps: identifying a calcium region from the region based on a cut-off point defined by a cardiovascular calcification integral (Agatston score); capturing a 3D image of the calcium region; analyzing the 3D image by a classifier to determine a classification of the calcium region; assigning a calcification integral to the calcium region; and generating a heat map to illustrate the calcium region and the calcification integral.

In at least one embodiment of the present disclosure, a preprocessing module is further included for preprocessing the medical image to a predetermined consistency by: resampling the 3D volume of the medical image to a spacing of 2×2×2 mm; normalizing the intensity of the 3D volume to a unit standard deviation with a mean of zero; and converting the 3D volume to a channel-first matrix form.

In at least one embodiment of the present disclosure, an output module is further included that is configured to present a cardiovascular risk prediction score based on the analysis results.

The present disclosure further provides a computer-readable medium that stores computer-executable code, and the computer-executable code is executed to implement the above method.

1: System

10: Image acquisition module

121: Starting point

122: Apex of the Heart

123: Pericardium

131: Edge of the aortic arch

132: The edge of the ascending aorta

133: The edge of the descending aorta

134: Ascending aorta

135: Aortic arch

136: First descending aorta

137: Second descending aorta

20: Preprocessing module

30: Split module

30A: Encoder part

30B: Decoder part

30C: Attention mechanism

30D: Variational autoencoder-decoder branch

31: Pericardium and heart

32: Ascending aorta

33: descending aorta

34: Pericardium

301: Residual volume block

302: Attention valve

40: Extraction module

41: Fat extraction unit

42: Calcium extraction unit

50: Output module

60: Model training module

501: Original image

502: Segmentation result

503:Results

504:Survival analysis

CL: Center Line

F1-F2: Base point

H1-H3: Hospital

S1-S5: Steps

S61-S65: Steps

S621-S624: Steps

x: Skipped features

x': previously skipped feature

p: Pay attention to the logic

p': previous attention logic

g: Valve control signal

FIG1 is a schematic diagram of a system for cardiovascular risk prediction according to the present disclosure.

FIG2 is a flow chart of the steps for cardiovascular risk prediction according to the present disclosure.

Figures 3A and 3B are schematic diagrams of images captured by scanning devices with different imaging capabilities according to the present disclosure.

Figures 4A and 4B are schematic diagrams of the network architecture of the machine learning model disclosed herein.

Figures 5A to 5E are schematic diagrams of regions segmented from medical images according to the present disclosure.

FIG6 is a schematic diagram of epicardial adipose tissue (EAT) extracted from medical images according to the present disclosure.

FIG. 7 is a schematic diagram of the calcification integral extracted from the medical image according to the present disclosure.

FIG8 is a schematic diagram of the network architecture of a classifier for classifying calcium regions according to the present disclosure.

FIG9 is a schematic diagram of a cardiovascular risk prediction report according to the present disclosure.

FIG10 is a flow chart of the steps for training a machine learning model according to the present disclosure.

Figure 11 is a schematic diagram of the concept of federated learning.

Figures 12A-1 to 12D-3 are schematic diagrams of the process of manually marking the training data of the heart.

Figures 13A to 13G-2 are schematic diagrams of the process of manually marking the training data of the aorta.

FIG. 14 is a flow chart of the steps for training a machine learning model according to the present disclosure.

In order to explain the present disclosure in detail, the following implementations are provided. After reading the disclosure of this specification, a person skilled in the art can easily understand the advantages and effects of the present disclosure, and can also implement or apply it in other different implementations. Therefore, the following implementations for executing the present disclosure can be modified and/or changed without violating the scope of different aspects and applications, and any element or method within the scope of the present disclosure disclosed herein can be combined with any other element or method disclosed in any implementation of the present disclosure.

The proportions, structures, dimensions and other features shown in the attached figures of this disclosure are only used to illustrate the implementation described herein so that those skilled in the art can read and understand this disclosure, and are not intended to limit the scope of this disclosure. Any changes, modifications or adjustments to the above features without affecting the design purpose and effect of this disclosure shall fall within the scope of the technical content of this disclosure.

As used herein, ordinal terms such as "first", "second", etc. are only used to facilitate description or distinction of limitations such as elements, components, structures, regions, parts, devices, systems, etc., and are not intended to limit the scope of the present disclosure, nor are they intended to limit the spatial sequence between these limitations. In addition, unless otherwise specified, singular terms such as "one", "the" and the like also belong to the plural form, and terms such as "or" and "and/or" can be used interchangeably.

As used herein, the terms "subject," "individual," and "patient" are interchangeable and refer to animals, such as mammals including the human species. Unless a gender is explicitly indicated, "subject" is intended to refer to both males and females.

The terms "include", "comprises", "has", "contains" or other similar terms described herein are not intended to exclude other elements. For example, when an object is described as "including" a limitation, unless otherwise stated, it may additionally include other elements, components, structures, regions, parts, devices, systems, steps or connections, etc., and other limitations should not be excluded.

As used herein, the phrase "at least one" referring to a list of one or more elements should be understood to mean at least one element selected from any one or more elements in the list of elements, but does not necessarily include at least one of each element listed in the list of elements, and does not exclude any combination of elements in the list of elements. This definition also allows that in addition to the elements identified in the list of elements to which the phrase "at least one" refers, elements may optionally be present, whether related or unrelated to those identified elements. Thus, as a non-limiting example, "at least one of A and B" (or equivalently, "at least one of A or B", or equivalently, "at least one of A and/or B") may refer, in one embodiment, to at least one optionally including more than one A and no B (and optionally including elements other than B); in another embodiment, to at least one optionally including more than one B and no A (and optionally including elements other than A); in yet another embodiment, to at least one optionally including more than one A and at least one optionally including more than one B (and optionally including other elements).

As used herein, "one or more" and "at least one" may have the same meaning and include one, two, three or more.

Referring to FIG. 1 , the present application discloses a system 1 for cardiovascular risk prediction. In at least one embodiment, the system 1 mainly includes an image acquisition module 10, a preprocessing module 20, a segmentation module 30, an extraction module 40, an output module 50, and a model training module 60. The arrows between the elements in the system 1 represent the operational relationship and information transmission direction between them, which can be implemented by any suitable wired or wireless method.

In some embodiments, the image acquisition module 10 can be coupled to or embedded in a scanning device to acquire medical images of an individual (e.g., a patient). In a preferred embodiment, the scanning device is a computer tomography (CT) scanner provided by Philips (Brilliance iCT, Brilliance CT), General Electric (Lightspeed VCT, Revolution CT), Siemens (Somation DefinitionAS), Canon (Aquilion PRIME), or the like, and the acquired medical images are non-developmental computer tomography images. In the embodiments described herein, the image acquisition module 10 can also be coupled or implanted in a medical image storage and transmission system (picture archiving and communication system, PACS) installed in any hospital, so that the medical images stored by the PACS can be connected by the image acquisition module 10 for subsequent processing in the system 1. In a further embodiment, the image acquisition module 10 can also manually receive individual medical images by providing an interactive interface for uploading/inputting medical images. However, the scanning device, medical image and image acquisition module 10 discussed above can also be implemented in other suitable forms, and therefore, it is not meant to limit the scope of the present disclosure. The detailed functions of the image acquisition module 10 will be further described later in the present disclosure.

In some embodiments, the preprocessing module 20 is configured to maintain the consistency of the medical image acquired by the image acquisition module 10 before performing the analysis process in the system 1. In the embodiments described herein, the tasks performed by the preprocessing module 20 may include but are not limited to resampling, normalization, and transformation of the medical image. The detailed functions of the preprocessing module 20 will be further described later in this disclosure.

In some embodiments, the segmentation module 30 is configured to segment the heart, pericardium and/or aorta regions from a given medical image (after preprocessing by the preprocessing module 20) for further analysis of cardiovascular risk prediction. In the embodiments described herein, the segmentation module 30 is configured to be implemented by a machine learning model that performs the segmentation task. The machine learning model can be developed based on algorithms such as decision trees, convolutional neural networks (CNNs), recursive neural networks (RNNs), or any combination thereof, but the present disclosure is not limited thereto. The detailed functions of the segmentation module 30 will be further described later in the present disclosure.

In some embodiments, the extraction module 40 is configured to extract analysis results from the segmented medical image. In the embodiments described herein, the extraction module 40 includes a fat extraction unit 41 configured to calculate the volume of fat tissue from the medical image, and a calcium extraction unit 42 configured to calculate the calcification integral from the medical image. It should be noted that the extraction module 40 may also include other units for extracting other information from the medical image to assist in cardiovascular risk prediction, but the present disclosure is not limited thereto. The detailed functions of the extraction module 40 will be further described later in the present disclosure.

In some embodiments, the output module 50 is configured to output analysis results about individual cardiovascular risk after analyzing the medical image. In the embodiments described herein, the analysis results can be implemented in the form of a report and indicate information such as segmented regions, fat tissue volume, calcification integral, etc. from the medical image. However, other forms for presenting the analysis results may also be used and should not limit the scope of the present disclosure. The detailed functions of the output module 50 will be further described later in the present disclosure.

In some embodiments, the model training module 60 is configured to provide training of the machine learning model before deployment to the segmentation module 30. In the embodiments described herein, the training of the machine learning model is performed based on joint learning and/or adaptive learning, so that the machine learning model can continuously improve its segmentation accuracy based on updated medical images and parameter settings collected from clinical practices of different institutions and/or scanning devices, even if the machine learning model has been deployed in actual use. The detailed functions of the model training module 60 will be further described later in this disclosure.

In some embodiments, the components of system 1 can be individually implemented as any suitable computer equipment, devices, programs, systems, etc., but the present disclosure is not limited thereto. In some embodiments, any two or more of the image acquisition module 10, the preprocessing module 20, the segmentation module 30, the extraction module 40, the output module 50, and the model training module 60 can be integrated rather than implemented as different units. In some embodiments, the components can also be implemented in a cloud computer environment. However, without departing from the operating concept of the present disclosure, the configuration of the components of system 1 can be implemented in any suitable form and should not limit the scope of the present disclosure.

Referring to FIG. 2 , a flow chart describing the steps of cardiovascular risk prediction using the elements of system 1 is disclosed, and FIGS. 3A-3B , 4A-4B , 5A-5E and 6-9 are also cited to illustrate the execution details of each step by reference. It should be understood that the steps shown in FIG. 2 are performed on the basis that the machine learning model of the segmentation module 30 is fully trained and ready for actual use. However, the training process of the machine learning model can also be performed in the steps of FIG. 2 under the concept of adaptive learning, and therefore will not interfere with the steps in FIG. 2 .

In step S1, the image acquisition module 10 obtains one or more medical images (for example, from a scanning device and/or PACS). In the embodiment described herein, the image acquisition module 10 is configured to receive medical images in accordance with the digital imaging communications in medicine (DICOM) standard, and the image acquisition module 10 may provide a graphical user interface (GUI) for users to manually or automatically upload and input medical images. As described above, the medical images are not limited to computer tomography (CT) images, but may also be magnetic resonance imaging (MRI) images, single photon emission computed tomography (SPECT) images, positron emission tomography (PET) images, etc., but the present disclosure is not limited thereto. In addition, due to the region of interest for analysis purposes, the obtained medical image should at least include the area around the individual's heart (e.g., chest CT image), but the content of the medical image may vary depending on the imaging capability (e.g., see Figures 3A and 3B, where the scanning device of Hospital A can only image the individual's heart, while the scanning device of Hospital B can image the individual's entire chest), but the present disclosure is not limited to this. In an alternative embodiment, the image acquisition module 10 can implement a filtering mechanism to ensure that the medical image is indeed associated with the region of interest in the individual before analysis.

In step S2, the preprocessing module 20 preprocesses one or more medical images to a predetermined consistency before analysis. In some embodiments described herein, the preprocessing of the medical images includes the following steps: resampling the three-dimensional (3D) volume of the medical image to a spacing of 2×2×2 mm; normalizing the intensity of the resampled 3D volume of the medical image to a unit standard deviation with a mean of zero (i.e., the mean is zero and the standard deviation is one); and converting the normalized 3D volume of the intermediate image to a channel-first matrix form. In at least one embodiment, the resampling of the 3D volume of the medical image can avoid insufficient storage space during subsequent processing by the segmentation module 30. In at least one embodiment, the normalized intensity of the resampled 3D volume of the medical image can make the medical images from different scanning devices consistent, so that the segmentation module 30 can obtain better segmentation results. In at least one embodiment, the normalized 3D volume of the converted medical image can help speed up the calculation speed in the subsequent processing of the segmentation module 30. After the medical image is pre-processed, it can be used as the input of the segmentation module 30 for further processing.

In step S3, one or more medical images are segmented by the machine learning model of the segmentation module 30 to identify the regions of the heart, pericardium, and aorta of the individual. In the embodiment described herein, the output from the machine learning model of the segmentation module 30 is a medical image of the individual with the regions of the heart, pericardium, ascending aorta, and descending aorta labeled, and they are processed separately to determine the fat tissue volume and calcification integral therefrom in subsequent processing.

Referring to FIG. 4A , the network architecture of the machine learning model used in this paper is disclosed, which includes an encoder part 30A, a decoder part 30B, an attention mechanism 30C, and a variational autoencoder (VAE) decoder branch 30D. From the network architecture shown, it can be seen that the machine learning model is based on a U-Net structure, in which each residual convolution block 301 is composed of 6 arithmetic operations stacked, namely group normalization (Norm as shown in FIG. 4A ), rectified linear unit (ReLU) (Act as shown in FIG. 4A ), convolution (Conv as shown in FIG. 4A ), group normalization, ReLU and convolution, using shortcuts (shortcut connecting) to connect and the initial 16 filters (filter). In at least one embodiment, the encoder portion 30A is configured to gradually reduce the image size of the input (medical image) (see the downward arrow of the encoder portion 30A) while increasing the feature size of the input during the feature extraction process of the input. In at least one embodiment, the decoder portion 30B is configured to gradually reduce the feature size while gradually increasing the image size of the feature from the end point of the encoder portion 30A (see the upward arrow of the decoder portion 30B) until an output having the same spatial size as the input is generated. In at least one embodiment, the attention mechanism 30C is configured to highlight the salient features transmitted through the residual connection between the encoder portion 30A and the decoder portion 30B (see the horizontal arrows around the attention mechanism 30C). In at least one implementation, the variational autoencoder (VAE) decoder branch 30D is configured to reconstruct the input (medical image) based on features from the endpoints of the encoder portion 30A, following the same architecture of the decoder portion 30B, which helps add additional guidance and regularization to the encoder portion 30A during training if the training data is limited.

4A and 4B , the attention mechanism 30C includes a plurality of attention gates 302, which correspond to the stages of the decoder part 30B and are stacked by upsampling, ReLU, convolution and sigmoid functions, wherein the previous attention logit p' carries the information learned from the previous layer, the gate control signal g carries the contextual information from the previous coarse scale, and the previously skipped features. feature)x' (i.e., the skipped feature x passes through the two residual convolution blocks 301 of the corresponding level of the encoder part 30A) and generates an attention logic feature p by each attention valve 302 before entering the next attention valve 302 and/or connecting to the decoder part 30B for output.

Referring to FIGS. 5A-5E, examples of segmentation results of medical images processed by the segmentation module 30 are shown, wherein FIGS. 5A, 5B and 5C show segmented regions of the pericardium and heart 31, the ascending aorta 32 and the descending aorta 33 respectively observed from the sagittal plane, the coronal plane and the horizontal plane of the individual; FIG. 5D shows a segmented region of the pericardium 34 presented in a 2D view observed from the horizontal plane of the individual; and FIG. 5E shows a segmented region of the pericardium and heart 31, the ascending aorta 32 and the descending aorta 33 presented in a 3D view of the chest cavity of the individual.

In step S4 of FIG. 2 , the extraction module 40 analyzes one or more medical images to extract the volume and calcification integral of fat tissue (e.g., EAT) based on the segmentation results. In the embodiment described herein, the extraction module 40 is configured to exclude unimportant parts (e.g., vertebrae and sternum) from the medical image in an orderly manner, and then quantify the volume and calcification integral of the fat tissue based on the Hounsfield Unit (HU) value of the segmented region presented in the medical image (e.g., via the fat extraction unit 41 and the calcium extraction unit 42).

In at least one embodiment, the fat extraction unit 41 is configured to quantify the EAT volume from the medical image by: calculating the HU value of the pericardium segmented by the segmentation module 30 based on the attenuation coefficient under CT (e.g., the attenuation coefficient of water, air and/or calcium); defining the positive and negative standard deviation range of the HU value based on the noise tolerance; and determining the EAT volume and position in the pericardium based on the range. An example of the EAT volume and position extracted by the fat extraction unit 41 is shown in FIG6 , where the position of the EAT is represented by a point on the pericardium, and the extracted EAT volume is 140 cm ³ . In some embodiments, different electron energies of X-rays during scanning may result in different measured attenuations of water, air and/or calcification. The algorithm disclosed herein can be calibrated based on the parameters of electron energy to obtain a more accurate calcification integral or classification thereof.

In at least one embodiment, the calcium extraction unit 42 is configured to quantify the calcification integral from the medical image by: based on the cardiovascular calcification integral of 130HU (Agatston score); capturing a plurality of 3D images of the calcium regions; analyzing the plurality of 3D images by a classifier (e.g., DenseNet) to classify the calcium regions, wherein the classification of the calcium regions may include but is not limited to those described in Table 1 below (wherein the columns represent categories, the left columns represent the main categories of the categories, and the right columns represent the subcategories of the categories); specifying a calcification integral for each calcium region; and generating a heat map (e.g., by a Gradient-weighted Class Activation Mapping technique) to display the calcium regions and their corresponding calcification integrals. An example of the calcium area extracted by the calcium extraction unit 42 and its corresponding calcification integral is shown in FIG7 , wherein the calcium areas on the heart, ascending aorta, and descending aorta are presented as colored areas, and the corresponding calcification integrals of the areas are 52, 450, and 1282, respectively.

Table 1. Calcium region categories identified by the classifier

Referring to FIG8 , the classifier used in the calcium extraction unit 42 is developed based on a network architecture, wherein the convolution blocks within the neural network of the classifier are designed to be interconnected (e.g., via a residual connection, skip connection) so that the classifier can learn the best transmission path to effectively determine the calcium area from the medical image. In the embodiment described herein, the neural network of the classifier is composed of a total of 121 layers, but the number of layers of the neural network of the classifier can be changed as needed, and the present disclosure is not limited thereto.

In step S5 of FIG2 , the output module 50 organizes the segmentation results from the segmentation module 30 and the adipose tissue volume and calcification integral extracted by the extraction module 40 to generate an analysis result. In the embodiment described herein, the output module 50 is further configured to calculate a cardiovascular risk prediction score (survival probability) based on the information analyzed in the previous processing steps (steps S1-S4) of the system 1. Referring to FIG. 9 , an example of an analysis result report generated by the output module 50 is disclosed, wherein an original image 501 of an individual's medical image, a segmentation result 502 (generated by the segmentation module 30 ) of a segmented region from the original image 501 , a result 503 of a quantitative value of the segmentation result 502 (calculated by the extraction module 40 ), and a survival analysis 504 of the individual's cardiovascular risk prediction score. It should be noted that the format of the analysis result is not limited to the report shown, but can be presented in any suitable physical or virtual form.

In the embodiments described herein, the formula for the cardiovascular risk prediction score is based on a study sample of patients who underwent chest computed tomography scans from the National Health Insurance database, through the following steps: collecting study samples from a total of 1,970 patients, wherein the study samples include patient imaging information, outpatient information, hospitalization information, medication information, insurance (death) records, and other data; linking the study samples together to conduct a generation of follow-up studies on the above patients, approximately 2,633.2 people per year; collecting (e.g., using imaging data The invention discloses a method for detecting calcification integral and fat tissue volume of the heart, ascending aorta and descending aorta in the imaging information of the research sample (previously through the machine learning model of the segmentation module 30, or the collected records that have been marked by the clinician on the imaging information from the National Health Insurance database); collecting basic demographic information (such as gender, age, etc.) and comorbidity information (about 57 comorbidities are defined) from the outpatient information, hospitalization information and medication information of the research sample; smoothing continuous variables (such as age, calcification integral, etc.) from the information collected via restricted cubic spline; and constructing a cardiovascular risk prediction score formula using Cox regression analysis based on the smoothed continuous variables. As shown in Figure 9, the cardiovascular risk prediction score is presented as the individual's probability of experiencing an event (e.g., rehospitalization or death) calculated as the number of years from the report creation date.

Referring to FIG. 10 , a flowchart of the steps for training the machine learning model of the segmentation module 30 executed by the model training module 60 is disclosed, wherein FIG. 11 , FIG. 12A-1 to FIG. 12D-3 , FIG. 13A to FIG. 13G-2 and FIG. 14 are cited to illustrate the execution details of each step. It should be understood that the training of the machine learning model can be operated independently of other components of the system 1 (i.e., steps S1 to S5 shown in FIG. 2 ), and therefore will not interfere with the operation of the system 1 during actual use.

In the embodiments described herein, the training of the machine learning model can be implemented in any suitable development platform, such as NVIDIA DGX, NVIDIA EGX, TensorFlow, Caffe, etc., and can utilize any suitable architecture, such as NVIDIA Clara imaging, Horovod, etc., but the present disclosure is not limited thereto. In some embodiments, each step described in FIG. 10 can be implemented by a separate or designated unit disposed in the model training module 60, but the present disclosure is not limited thereto. In other embodiments, the model training module 60 is also used to provide a GUI or other instruction mechanism to guide the user to complete the steps described in FIG. 10, but the present disclosure is not limited thereto.

In at least one implementation, the training of the machine learning model is performed based on federated learning. Generally speaking, it is difficult to collect a large amount of medical data to train a powerful neural network under the constraints of privacy protection, and it is usually not feasible to share data between multiple medical institutions (such as hospitals). Therefore, federated learning solves the above problems by distributing the training of the neural network among various medical institutions and only sharing the training weights among various medical institutions to complete the training of the machine learning model.

In the embodiment described herein, as shown in FIG. 11 , three hospitals (respectively indicated as H1, H2 and H3) participate in the development of the machine learning model of the segmentation module 30, whereby joint learning will establish a relationship between the server side (developer) and the client side (hospital) to train the machine learning model.

As shown in Figure 11, the server first distributes the initialized/pre-trained global weights to the client. Then, the client's hospital will train a local model based on the global weights using the patient data in the corresponding hospital H1/H2/H3. Next, the local weights derived from the client during local model training will be applied to differential privacy (e.g., selective parameter sharing, sparse vector technology, etc.) to prevent model conversion before returning to the server. Finally, the client transmits the local weights back to the server, and the server will aggregate the local weights based on their contributions (e.g., the contribution is determined by the amount of data provided by each hospital) as updated global weights for training the machine learning model. Once training is complete, the updated global weights can be sent back to the client for actual deployment of the machine learning model.

Based on the concept of joint learning explained in FIG11 , the steps of training the machine learning model of the segmentation module 30 will be described as follows.

In step S61 of FIG. 10 , the training data of the machine learning model uses an AI-Assisted Annotation (AIAA) tool (such as the NVIDIA Clara train SDK provided by NVIDIA Clara Imaging) or a labeling tool (such as 3D slicer (three-dimensional reconstruction software), Medical Imaging Interaction Toolkit (MITK), etc.). However, the tools used to label the training data are not limited to those described herein, and other suitable tools may be used as needed, and the present disclosure is not limited thereto.

In addition, in the case of joint learning, considering privacy protection, non-visual or visual medical images can be de-identified before selecting labels, thereby removing any identifiers (such as but not limited to name, address, date of birth, hospitalization date, discharge date, date of death, telephone number, fax number, email address, social security number, medical record number, health insurance card number, ID number, driver's license number, license plate number, medical material product number, personal URL address, personal IP address, biometric data, full face image or other) that may reveal the identity of the individual who contributed the medical image.

It should be noted that the training data selection rules of the machine learning model can have more or fewer rules, and the requirements described are only for example and should not limit the scope of this disclosure.

In the embodiments described herein, FIGS. 12A-1 to 12D-3 and FIGS. 13A to 13G-2 illustrate the process of marking training data, wherein FIGS. 12A-1 to 12D-3 illustrate the process of manually marking the pericardium from a medical image of a specific individual, and FIGS. 13A to 13G-2 illustrate the process of manually marking the aorta from a medical image of a specific individual.

In the case of manually marking the pericardium of a particular individual from medical images, the process begins by drawing the upper edge of the heart starting from the starting point 121 of the pulmonary trunk at the bottom of the right ventricle. Figures 12A-1 to 12A-3 show the location of the starting point 121 for drawing the upper edge of the pericardium as observed from the horizontal plane (Figure 12A-1), sagittal plane (Figure 12A-2), and coronal plane (Figure 12A-3) of the individual, respectively, where the area of the upper edge observed at the starting point 121 in the horizontal plane (Figure 12A-1) is then circled. At this stage, care should be taken not to include the esophagus in the area of the upper edge of the pericardium.

The process then continues with drawing the inferior margin of the pericardium ending at the apex 122 of the heart. Figures 12B-1 to 12B-3 show the location of the apex 122 used to draw the inferior margin of the heart as viewed from the horizontal plane (Figure 12B-1), sagittal plane (Figure 12B-2), and coronal plane (Figure 12B-3) of the individual, respectively, where the inferior margin area observed at the apex 122 in the horizontal plane (Figure 12B-1) is then circled. At this stage, care should be taken not to include the liver in the inferior margin area of the pericardium.

Next, the process is completed by drawing the heart border along the pericardium 123 of the heart (connecting the upper and lower edges). Figures 12C-1 to 12C-3 show the location of the pericardium 123 drawn from the horizontal plane (Figure 12C-1), sagittal plane (Figure 12C-2), and coronal plane (Figure 12C-3) of the target, respectively. At this stage, care should be taken not to include the sternum in the area of the heart border.

Figures 12D-1 to 12D-3 show examples of an individual's heart and pericardium after labeling, with the circled area of the heart displayed in the individual's horizontal plane (Figure 12D-1), sagittal plane (Figure 12D-2), and coronal plane (Figure 12D-3). From here, a smoothing function can be additionally applied to balance the roughness of the circled area before using it as training data.

In the case of manually marking the aorta of a particular individual from imaging, the process begins by determining the reference point F2 of the aorta where it intersects the capitulum brachial artery. Figure 13A shows the location of reference point F2 as viewed from the sagittal plane of the individual.

The process then continues by identifying another reference point F1 of the aorta that intersects the left subclavian artery. Figure 13B shows the location of reference point F1 as viewed from the sagittal plane of the subject.

Next, the process continues with drawing the edge 131 of the aortic arch. Figures 13C-1 to 13C-3 show the position of the edge 131 used to draw the aortic arch as viewed from the horizontal plane (Figure 13C-1), sagittal plane (Figure 13C-2), and coronal plane (Figure 13C-3) of the individual.

In addition, the process continues to draw the edge 132 of the ascending aorta. Figures 13D-1 to 13D-3 show the position of the edge 132 used to draw the ascending aorta observed from the horizontal plane (Figure 13D-1), sagittal plane (Figure 13D-2), and coronal plane (Figure 13D-3) of the individual.

The process then continues with marking the edge 133 of the descending aorta in a similar procedure as described in FIGS. 13C-1 to 13D-3, which are not described in detail herein. FIG. 13E is included to illustrate an example of the edge 133 of the descending aorta as viewed from the sagittal plane of the individual after marking, where the border of the descending aorta terminates (see the lower border of the descending aorta edge) above the common iliac artery.

Next, the process continues to extract the centerline CL from the determined aortic edge (via the extraction function of the 3D slicer) to ensure accurate marking of the aorta. FIG13F shows the extracted centerline CL and its relationship with the ascending aorta 134, the aortic arch 135, the first descending aorta 136, and the second descending aorta 137 in a 3D view. Specifically, the process continues to determine the cross-section of the aorta based on the above-mentioned reference points F1 and F2, wherein the aorta segment between the heart and the reference point F2 (the aorta root) is defined as the ascending aorta 134, the aorta segment between the reference points F1 and F2 is defined as the aorta arch 135, the aorta segment descending from the reference point F1 to the other end (far from the aorta root) of the aorta corresponding to the height of the reference point F2 (when the aorta is observed from the horizontal plane at this height of the reference point F2) is defined as the first descending aorta 136, and the remaining segment of the aorta is defined as the second descending aorta 137.

Figures 13G-1 and 13G-2 further show examples of the positions of the ascending aorta 134, the aortic arch 135, the first descending aorta 136, and the second descending aorta 137 observed from the horizontal plane (Figure 13G-1) and the sagittal plane (Figure 13G-2) of the individual.

After marking the individual aortas, a smoothing function can be applied to balance the roughness of the selected region before using it as training data.

In the embodiment described herein, for the purpose of collecting sufficient training data, a more detailed process of labeling the training data includes the following steps (see the relationship between steps S61 and S62 in FIG. 10 ): manual labeling (by 3D slicer) 10 medical image examples (see the process described in Figures 12A-1 to 12D-3 and Figures 13A to 13G-2); train the first version of the machine learning model based on the 10 medical image examples; use the first version of the machine learning model as an auxiliary annotation model of the AIAA tool to assist in manual annotation of more medical images; train the second version of the machine learning model based on more medical images; repeat the above annotation through manual annotation and the auxiliary annotation model to train the second version of the machine learning model until the training data is sufficient.

In step S62 of FIG10 , the machine learning model (second version machine learning model) is trained using training data. FIG14 shows a flow chart of the steps for training the machine learning model, which will be described below.

In step S621, the training data is preprocessed to a predetermined consistency by resampling, normalizing and transforming the training data. The preprocessing of the training data is similar to that described in the above-mentioned preprocessing module 20 and will not be described in detail here.

In step S622, the training data is further reinforced to prevent overfitting. In the embodiments described herein, the 3D volume of medical images preprocessed from the training data can first be randomly cropped to a random size (e.g., a maximum of 160×160×160 pixels and/or a minimum of 64×64×64 pixels). The randomly cropped 3D volume of medical images can then be padded with a fixed size (e.g., 160×160×160 pixels) to ensure smooth training of the machine learning model. Other augmentation techniques can also be used here, such as but not limited to: augmenting the 3D volume of medical images with random spatial flips during training to enhance the bias of the training data, scaling or moving the strength of the 3D volume of medical images without background to regularize the machine learning model. Therefore, even if the scanning equipment from different hospitals has different imaging capabilities, the machine learning model can still generalize to process various types of medical images as training data.

In step S623, the training data is sent to the machine learning model for training. It should be noted that the network architecture of the machine learning model during training is the same as that described in Figures 4A and 4B, and will not be repeated here.

In step S624, the training results of the machine learning model (e.g., the segmentation results output by the machine learning model during the training process) are output for subsequent processing and verification.

Continuing from step S63 of FIG. 10 , the machine learning model being trained is verified to quantify its segmentation performance (based on the training results of step S624 in FIG. 14 ). In the implementation described herein, a loss function is used to verify the segmentation performance of the machine learning model, which can be expressed as a dice loss designed according to the DICE similarity coefficient (DSC), which is defined as follows:

Dice Loss = 1- DSC

表示在訓練期間機器學習模型實際分割的預測區域。 Where y represents the target region to be segmented by the machine learning model (i.e., the heart/pericardium and aorta regions marked in the training data).

represents the predicted region of the actual segmentation by the machine learning model during training.

From the above, we can see that DSC aims to measure the similarity between the target region recognized by the machine learning model and the predicted region, so it is a quantifiable measure for evaluating the segmentation performance of the machine learning model.

Based on the loss function, the model training module 60 can retrain the machine learning model (see the relationship between steps S63 and S62 in FIG. 10 ) or prepare the machine learning model for actual use (i.e., perform step S64 in FIG. 10 ). In the implementation described herein, the machine learning model should at least achieve the DSC listed in Table 2 below before being put into actual use.

Table 2 Training results of machine learning model

The machine learning model with sufficient segmentation performance is output in step S64 and deployed (e.g., via the NVIDIA Clara deploy SDK provided by NVIDIA Clara Imaging) to the segmentation module 30 in step S65 at the end of training. If the machine learning model is first deployed to the segmentation module 30, the components of the system 1 can therefore be put into practical use to predict cardiovascular risk for any individual in real time. However, the model training module 60 can still optimize the performance of the machine learning model of the segmentation module 30 in real time based on updated training data and/or parameter settings during clinical practice.

In a further embodiment described herein, a computer-readable medium is further provided, which stores computer-executable code, and the computer-executable code is configured to implement steps S1 to S5 of FIG. 2 , steps S61 to S65 of FIG. 10 , and/or steps S621 to S624 of FIG. 14 as described above after execution.

In summary, the present disclosure utilizes artificial intelligence to segment medical images to identify the precise regions of the heart, aorta and/or pericardium of an individual, thereby deriving the fat tissue volume and calcification integral from contrast-free or contrast-enhanced medical images for cardiovascular risk prediction.

1: System

10: Image acquisition module

20: Preprocessing module

30: Split module

40: Extraction module

41: Fat extraction unit

42: Calcium extraction unit

50: Output module

60: Model training module

Claims (20)

A system for predicting cardiovascular risk, comprising: a segmentation module configured to segment a region from a medical image, wherein a machine learning model of the segmentation module identifies the region of the heart, pericardium, and aorta of an individual, so as to output a segmentation result of the medical image of the individual with the region of the heart, pericardium, ascending aorta, and descending aorta marked; an extraction module configured to analyze the region of the medical image, wherein the extraction module excludes non-fat tissue and calcified parts from the medical image based on the segmentation result, and then extracts the region of the medical image based on the segmentation results presented in the medical image. The method comprises: quantifying the fat tissue volume and the calcification integral of the region by using the Hounsfield unit value of the region to extract the fat tissue volume and the calcification integral; and an output module configured to organize the segmentation result from the segmentation module and the fat tissue volume and the calcification integral extracted by the extraction module to generate an analysis result, wherein the machine learning model has a network architecture including an encoder part, a decoder part and an attention mechanism, and the attention mechanism is configured to pick out significant features through residual connections between the encoder part and the decoder part. A system as described in claim 1, wherein the medical image is a non-developing or developing CT scan image. A system as described in claim 1, wherein the segmentation module is executed by the machine learning model to segment the region from the medical image, and wherein the network architecture of the machine learning model further includes a variational autoencoder-decoder branch. The system of claim 3, wherein the variational self-encoder decoder branch is configured to reconstruct the medical image based on features of the endpoints of the encoder portion during training of the machine learning model. The system as described in claim 3 further includes a model training module configured to provide the machine learning model training by the following steps: preprocessing the training data to a predetermined consistency; enhancing the training data by performing random cropping, random spatial flipping and/or random intensity scaling or translation on the training data; training the machine learning model using the training data; and verifying the training result of the machine learning model using a loss function. A system as claimed in claim 5, wherein the training data is generated by labeling the medical image manually and/or with the help of an auxiliary annotation model. A system as described in claim 1, wherein the analysis result includes the volume of the fat tissue in the region, and wherein the extraction module includes a fat extraction unit configured to quantify the volume of the fat tissue in the pericardium in the region by: calculating the Hounsfield unit value of the pericardium based on the attenuation coefficient under computer tomography; defining the positive and negative standard deviation range of the Hounsfield unit value based on the noise tolerance limit; and determining the volume of the fat tissue in the pericardium based on the range. The system of claim 1, wherein the analysis result includes the calcification integral of the region, and wherein the extraction module includes a calcium extraction unit configured to quantify the calcification integral of the heart or aorta from the region by: identifying a calcium region from the region based on a cut-off point defined by a cardiovascular calcification integral (Agatston score); capturing a 3D image of the calcium region; Analyzing the 3D image by a classifier to determine the classification of the calcium region; assigning the calcification integral of the calcium region; and generating a heat map to illustrate the calcium region and the calcification integral. The system as described in claim 1 further includes a pre-processing module configured to pre-process the medical image to a predetermined consistency by: resampling the 3D volume of the medical image to a spacing of 2×2×2 mm or any predetermined size; and normalizing the intensity of the 3D volume to a unit standard deviation with a mean of zero. A system as described in claim 1, wherein the output module presents a cardiovascular risk prediction score based on the analysis results. A cardiovascular risk prediction method is executed on a computer or a server. The cardiovascular risk prediction method includes the following steps: a segmentation module is used to segment a region from a medical image. The machine learning model of the segmentation module identifies the region of the heart, pericardium and aorta of the individual to output the segmentation result of the medical image of the individual with the region of the heart, pericardium, ascending aorta and descending aorta marked; an extraction module is used to analyze the region of the medical image. Based on the segmentation result, non-fat tissue and calcification are excluded from the medical image, and then the region of the medical image is extracted based on the segmentation result. quantifying the fat tissue volume and the calcification integral according to the Hounsfield unit value of the segmented region presented in the image to extract the fat tissue volume and the calcification integral; and arranging the segmentation result from the segmentation module and the fat tissue volume and the calcification integral extracted by the extraction module through an output module to generate an analysis result, wherein the machine learning model has a network architecture including an encoder part, a decoder part and an attention mechanism, and the attention mechanism is configured to pick out significant features through residual connections between the encoder part and the decoder part. The method as described in claim 11, wherein the medical image is a computer tomography image. The method as claimed in claim 11, wherein the segmentation module is executed by the machine learning model to segment the region from the medical image, and wherein the network architecture of the machine learning model further includes a variational autoencoder-decoder branch. The method of claim 13, wherein the variational self-encoder decoder branch is configured to reconstruct the medical image based on features of the endpoints of the encoder portion during training of the machine learning model. The method as described in claim 13 further includes a model training module configured to provide the machine learning model training by the following steps: preprocessing the training data to a predetermined consistency, wherein the training data is generated by manually and/or with the help of an auxiliary annotation model to label the medical image; enhancing the training data by performing random cropping, random spatial flipping and/or random intensity scaling or translation on the training data; training the machine learning model using the training data; and verifying the training result of the machine learning model using a loss function. The method as claimed in claim 11, wherein the analysis result includes the volume of the fat tissue in the region, and wherein the extraction module includes a fat extraction unit configured to quantify the volume of the fat tissue in the pericardium in the region by: calculating the Hounsfield unit value of the pericardium based on the attenuation coefficient under computer tomography; defining the positive and negative standard deviation range of the Hounsfield unit value based on the noise tolerance limit; and determining the volume of the fat tissue in the pericardium based on the range. The method of claim 11, wherein the analysis result includes the calcification integral of the region, and wherein the extraction module includes a calcium extraction unit configured to quantify the calcification integral of the heart or aorta from the region by: identifying a calcium region from the region based on a cut point defined by a cardiac vascular calcification integral; capturing a 3D image of the calcium region; analyzing the 3D image by a classifier to determine a classification of the calcium region; assigning a calcification integral to the calcium region; and generating a heat map to illustrate the calcium region and the calcification integral. The method as claimed in claim 11, further comprising a preprocessing module configured to preprocess the medical image to a predetermined consistency by: resampling the 3D volume of the medical image to a spacing of 2×2×2 mm or any predetermined size; and normalizing the intensity of the 3D volume to a unit standard deviation with a mean of zero. The method as described in claim 11, wherein the output module presents a cardiovascular risk prediction score based on the analysis results. A computer-readable medium storing a computer-executable code, and executing the computer-executable code to implement the method described in claim 11.

Images