KR20230133230A - Method, program and apparatus for contrasitive pre-training of neural network model based on electrocardiogram - Google Patents

Abstract

The present invention relates to a contrastive pre-learning method of a neural network model based on electrocardiogram, which is performed by a computing device including at least one processor. The contrastive pre-learning method comprises: a local information generation step of generating a local representation context using a self-supervised learning-based model when an unlabeled raw electrocardiogram signal is input; a global information generation step of generating a global representation context by applying time average pooling to the local representation context; and a learning step of pre-learning a neural network model through contrast learning based on learning data including the local and global representation contexts. Therefore, excellent performance in diagnostic classification and patient identification can be exhibited.

Description

Method, program and apparatus for contrastive pre-training of neural network model based on electrocardiogram {METHOD, PROGRAM AND APPARATUS FOR CONTRASITIVE PRE-TRAINING OF NEURAL NETWORK MODEL BASED ON ELECTROCARDIOGRAM}

The present disclosure relates to contrasting pre-training methods for electrocardiogram-based neural network models, specifically learning local and global features (representation) with large amounts of unlabeled electrocardiogram signals and then fine-tuning them with small amounts of labeled data. It's about how to do it.

An electrocardiogram (ECG) is a signal that measures electrical signals generated in the heart and checks for abnormalities in the conduction system from the heart to the electrodes to determine the presence or absence of disease.

The heartbeat, which is the cause of the electrocardiogram, is an impulse that originates from the sinus node located in the right atrium, first depolarizes the right and left atrium, and after a brief delay in the atrioventricular node, Activates the ventricles.

The right ventricle, which has the fastest septum and thin walls, activates before the left ventricle, which has thick walls. The depolarization wave transmitted to the Purkinje fibers spreads from the endocardium to the epicardium like a wavefront in the myocardium, causing ventricular contraction. Because electrical impulses are normally conducted through the heart, the heart contracts approximately 60 to 100 times per minute. Each contraction is represented by one heart beat.

Such an electrocardiogram can be detected through a bipolar lead, which records the potential difference between two parts, and a unipolar lead, which records the potential of the area where the electrode is attached. Methods for measuring an electrocardiogram include the bipolar lead. There is a standard limb lead, a unipolar limb lead, and a unipolar thoracic lead (precordial lead).

The electrical activity stage of the heart is largely divided into atrial depolarization, ventricular depolarization, and ventricular repolarization, and each of these stages is reflected in the form of several waves called P, Q, R, S, and T waves, as shown in Figure 1.

These waves must have a standard shape for the heart's electrical activity to be considered normal. In order to determine whether it is a standard form or not, it is necessary to check whether characteristics such as the time each wave is maintained, the interval between each wave, the amplitude of each wave, and kurtosis are within the normal range.

These electrocardiograms are measured with expensive measuring equipment and used as an auxiliary tool to measure the patient's health status. In general, electrocardiogram measuring equipment only displays measurement results and diagnosis is entirely up to the doctor.

Currently, research is continuing to quickly and accurately diagnose diseases using artificial intelligence based on electrocardiograms to reduce dependence on doctors. In addition, with the development of wearable and lifestyle electrocardiogram measurement devices, the possibility of diagnosing and monitoring not only heart disease but also various other diseases based on electrocardiogram is emerging.

Non-invasively assessing heart function by measuring electrocardiograms helps diagnose numerous heart diseases, including arrhythmia, myocardial infarction, and arterial disease. In all medical facilities, cardiac-related symptoms include electrocardiogram (ECG) measurements, which are stored in daily accumulation of ECG records.

Prior art deep learning-based ECG signal analysis methods have involved supervised learning of ECG data with labels that can only be annotated by clinical practitioners or expert cardiologists. Because of this, the number of samples that can be used for learning is limited, which makes it difficult to learn a model based on deep learning with a small amount of training data, and there is a problem that it cannot be learned on unlabeled data.

Republic of Korea Patent Publication No. 10-2021-0061769 (May 28, 2021)

The present disclosure was created in response to the above-mentioned background technology, and the contrasting dictionary learning method of the neural network model based on the electrocardiogram according to an embodiment of the present disclosure is a method of learning local characteristics and local characteristics using a large amount of unlabeled electrocardiogram signals through self-supervised learning. The goal is to perform fine-tuning with a small amount of labeled data after learning all global features.

However, the problems to be solved by this disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood based on the description below.

As a contrastive dictionary learning method of a neural network model based on an electrocardiogram, performed by a computing device including at least one processor, according to an embodiment of the present disclosure for realizing the above-described task, , When an unlabeled raw ECG signal is input, a local information generation step of generating a local representation context (Local Representation Context) using a model based on self-supervised learning; A global information generation step of generating a global characteristic context from the local characteristic context; and a learning step of pre-training a neural network model through contrast learning based on learning data including the local feature context and the global feature context.

Alternatively, it is intended to provide a method in which the local information generation step includes using a Wav2Vec model, and the global information generation step includes using a CMSC (Contrastive Multi-segment Coding) technique.

Alternatively, the local information generation step divides the raw ECG signal for a preset number of leads into 2×N (a natural number where N>0) sample data, and performs RLM (Random Lead Masking) on the divided sample data. It is intended to provide a method further including the step of generating a masked lead electrocardiogram signal by applying .

Alternatively, we would like to provide a method that further includes an evaluation step of fine tuning and evaluating the pre-trained neural network model using a labeled dataset for cardiac arrhythmia classification and patient identification.

Alternatively, the Wav2Vec model includes at least one convolutional neural network (CNN) and a transformer corresponding to the CNN, and the local characteristic context is generated as the raw ECG signal passes through the CNN and the transformer. The aim is to provide a method in which the global characteristic context is generated by applying time average pooling while the local characteristic context passes through the transformer.

Alternatively, the local information generation step may include inputting the raw ECG signal to generate a latent vector (Z) in a latent space using a CNN, and encoding the latent vector (Z) to create the local feature context (C). Including generating a , wherein the learning step is to quantize a potential vector (Z) at a preset time step in a pre-learning process to generate a quantized feature (Q), which is masked (m) before being provided to the transformer. We would like to provide a method that includes this.

Alternatively, the neural network model is characterized in that the cosine similarity between the local feature context and the quantized feature is learned to have a maximum value at each masked time step.

Alternatively, the global information generation step may include a duration Si and time segment characteristics of adjacent, non-overlapping time zones within the duration Si for the ith ECG signal in the ECG record of the first patient. defining a pair of positive numbers; and defining time segment characteristics of the ECG signal of the second patient, which are different from the time segment characteristics of the ECG signal of the first patient, as a negative pair.

Alternatively, the learning step may be characterized in that each patient is learned through self-supervised learning using the correlation between the electrocardiogram records of the first patient and the second patient in the prior learning process.

A computer program stored in a computer-readable storage medium according to another embodiment of the present disclosure, wherein the computer program, when executed on one or more processors, performs operations for contrastive pre-learning of a neural network model based on the electrocardiogram. The operations include: generating a local representation context using a model based on self-supervised learning when an unlabeled raw ECG signal is input; generating a global feature context by applying time average pooling to the local feature context; and pre-training a neural network model through contrast learning based on learning data including the local feature context and the global feature context.

A computing device for contrastive pre-learning of a neural network model based on an electrocardiogram according to another embodiment of the present disclosure, comprising: a processor including at least one core; and a memory containing program codes executable on the processor; It includes, wherein, according to execution of the program code, when an unlabeled raw ECG signal is input, the processor generates at least one local characteristic context (Local Representation Context) using a model based on self-supervised learning, Provides a device that generates a global feature context by applying time average pooling to the local feature context and pre-trains a neural network model through contrastive learning based on learning data including the local feature context and the global feature context. I want to do it.

The contrasting pre-learning method of the ECG-based neural network model according to an embodiment of the present disclosure uses a neural network model integrating the Wavc2Vec model, CMSC technique, and RLM technique to obtain both global and local characteristics for the unlabeled ECG signal. By pre-training and fine-tuning the pre-trained neural network model through downstream operations with a random read set, there is an effect of providing a neural network model that can show excellent performance in diagnostic classification and patient identification even with a small amount of data. .

1 is a diagram showing an electrocardiogram signal according to the present disclosure.
2 is a block diagram of a computing device according to an embodiment of the present disclosure.
3 is a flowchart showing a contrasting dictionary learning method for a neural network model based on an electrocardiogram according to an embodiment of the present disclosure.
Figure 4 is a diagram showing the structure of a neural network model according to an embodiment of the present disclosure.
Figure 5 is a flowchart explaining the process of performing the learning step according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art (hereinafter referred to as skilled in the art) can easily implement the present disclosure. The embodiments presented in this disclosure are provided to enable any person skilled in the art to use or practice the subject matter of this disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure can be implemented in various different forms and is not limited to the following embodiments.

The same or similar reference numerals refer to the same or similar elements throughout the specification of this disclosure. Additionally, in order to clearly describe the present disclosure, reference numerals in the drawings may be omitted for parts that are not related to the description of the present disclosure.

As used in this disclosure, the term “or” is intended to mean an inclusive “or” and not an exclusive “or.” That is, unless otherwise specified in the present disclosure or the meaning is not clear from the context, “x uses a or b” should be understood to mean one of natural implicit substitutions. For example, unless otherwise specified in the present disclosure or the meaning is not clear from the context, “x uses a or b” means that x uses a, x uses b, or x uses a and It can be interpreted as one of the cases where both b are used.

The term “and/or” as used in this disclosure should be understood to refer to and include all possible combinations of one or more of the listed related concepts.

The terms “comprise” and/or “comprising” as used in this disclosure should be understood to mean that certain features and/or elements are present. However, the terms "comprise" and/or "including" should be understood as not excluding the presence or addition of one or more other features, other components, and/or combinations thereof.

Unless otherwise specified in this disclosure or the context is clear to indicate a singular form, the singular should generally be construed to include “one or more.”

The term "th nth (n is a natural number)" used in the present disclosure can be understood as an expression used to distinguish the components of the present disclosure according to a predetermined standard such as a functional perspective, a structural perspective, or explanatory convenience. there is. For example, in the present disclosure, components performing different functional roles may be distinguished as first components or second components. However, components that are substantially the same within the technical spirit of the present disclosure but must be distinguished for convenience of explanation may also be distinguished as first components or second components.

The term “acquisition” used in this disclosure is understood to mean not only receiving data through a wired or wireless communication network with an external device or system, but also generating data in an on-device form. It can be.

Meanwhile, the term "module" or "unit" used in this disclosure refers to a computer-related entity, firmware, software or part thereof, hardware or part thereof. , can be understood as a term referring to an independent functional unit that processes computing resources, such as a combination of software and hardware. At this time, the “module” or “unit” may be a unit composed of a single element, or may be a unit expressed as a combination or set of multiple elements. For example, a "module" or "part" in the narrow sense is a hardware element or set of components of a computing device, an application program that performs a specific function of software, a process implemented through the execution of software, or a program. It can refer to a set of instructions for execution, etc. Additionally, as a broad concept, “module” or “unit” may refer to the computing device itself constituting the system, or an application running on the computing device. However, since the above-described concept is only an example, the concept of “module” or “unit” may be defined in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

As used in this disclosure, the term "model" refers to a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or a process to solve a specific problem. It can be understood as an abstract model of a process. For example, a neural network “model” may refer to an overall system implemented as a neural network that has problem-solving capabilities through learning. At this time, the neural network can have problem-solving capabilities by optimizing parameters connecting nodes or neurons through learning. A neural network “model” may include a single neural network or a neural network set in which multiple neural networks are combined.

“Data” used in this disclosure may include “image”, signals, etc. The term “image” used in this disclosure may refer to multidimensional data composed of discrete image elements. In other words, “image” can be understood as a term referring to a digital representation of an object that can be seen by the human eye. For example, “image” may refer to multidimensional data consisting of elements corresponding to pixels in a two-dimensional image. “Image” may refer to multidimensional data consisting of elements corresponding to voxels in a three-dimensional image.

The term “block” used in the present disclosure can be understood as a set of components divided based on various criteria such as type, function, etc. Accordingly, the configuration classified as one “block” can be changed in various ways depending on the standard. For example, a neural network “block” can be understood as a set of neural networks containing at least one neural network. At this time, it can be assumed that the neural networks included in the neural network “block” perform the same specific operation. The explanation of the foregoing terms is intended to aid understanding of the present disclosure. Therefore, if the above-mentioned terms are not explicitly described as limiting the content of the present disclosure, it should be noted that the content of the present disclosure is not used in the sense of limiting the technical idea.

Figure 2 is a block diagram of a computing device according to an embodiment of the present disclosure.

The computing device 100 according to an embodiment of the present disclosure may be a hardware device or part of a hardware device that performs comprehensive processing and calculation of data, or may be a software-based computing environment connected to a communication network. For example, the computing device 100 may be a server that performs intensive data processing functions and shares resources, or it may be a client that shares resources through interaction with the server. Additionally, the computing device 100 may be a cloud system in which a plurality of servers and clients interact to comprehensively process data. Since the above description is only an example related to the type of computing device 100, the type of computing device 100 may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

Referring to FIG. 2, the computing device 100 according to an embodiment of the present disclosure may include a processor 110, a memory 120, and a network unit 130. there is. However, since FIG. 2 is only an example, the computing device 100 may include other components for implementing a computing environment. Additionally, only some of the configurations disclosed above may be included in computing device 100.

The processor 110 according to an embodiment of the present disclosure may be understood as a structural unit including hardware and/or software for performing computing operations. For example, the processor 110 may read a computer program and perform data processing for machine learning. The processor 110 may process computational processes such as processing input data for machine learning, extracting features for machine learning, and calculating errors based on backpropagation. The processor 110 for performing such data processing includes a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), and a custom processing unit (TPU). It may include a semiconductor (ASICc: application specific integrated circuit), or a field programmable gate array (FPGA: field programmable gate array). Since the type of processor 110 described above is only an example, the type of processor 110 may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

The processor 110 may learn a neural network model that exhibits excellent performance in diagnosis identification and patient classification based on the ECG signal. For example, the processor 110 may train a neural network model to estimate diagnosis identification and patient classification based on biological data including information such as gender, age, weight, height, etc., along with electrocardiogram signals. The processor 110 may perform an operation representing at least one neural network block included in the neural network model during the learning process of the neural network model. This processor 110 performs contrastive dictionary learning that considers both local feature context and global feature context based on the ECG signal, thereby outperforming other ECG-related dictionary learning for two downstream tasks: cardiac arrhythmia classification and patient identification. It can provide excellent neural network models. The processor 110 uses the neural network model generated through the above-described learning process to estimate diagnostic identification and patient classification from ECG signals.

In addition to the examples described above, the types of medical data including electrocardiogram signals and the output of the neural network model may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

The memory 120 according to an embodiment of the present disclosure may be understood as a structural unit including hardware and/or software for storing and managing data processed in the computing device 100. That is, the memory 120 can store any type of data generated or determined by the processor 110 and any type of data received by the network unit 130. For example, the memory 120 may be a flash memory type, hard disk type, multimedia card micro type, card type memory, or random access memory (RAM). ), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (prom: programmable read-only memory), magnetic memory , may include at least one type of storage medium among a magnetic disk and an optical disk. Additionally, the memory 120 may include a database system that controls and manages data in a predetermined system. Since the type of memory 120 described above is only an example, the type of memory 120 may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

The memory 120 can structure, organize, and manage data necessary for the processor 110 to perform operations, combinations of data, and program codes executable on the processor 110. For example, the memory 120 may store an electrocardiogram signal received through the network unit 130, which will be described later. The memory 120 includes a program code that operates the neural network model to receive an electrocardiogram signal and perform learning, a program code that operates the neural network model to receive an electrocardiogram signal and perform inference according to the purpose of use of the computing device 100, and Processed data generated as the program code is executed can be saved.

The network unit 130 according to an embodiment of the present disclosure may be understood as a structural unit that transmits and receives data through any type of known wired or wireless communication system. For example, the network unit 130 may be connected to a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), or wireless (WIBRO). broadband internet, 5th generation mobile communication (5g), ultra wide-band wireless communication, zigbee, radio frequency (RF) communication, wireless LAN, wireless fidelity ), data transmission and reception can be performed using a wired or wireless communication system such as near field communication (NFC), or Bluetooth. Since the above-described communication systems are only examples, the wired and wireless communication systems for data transmission and reception of the network unit 130 may be applied in various ways other than the above-described examples.

The network unit 130 may receive data necessary for the processor 110 to perform calculations through wired or wireless communication with any system or client. Additionally, the network unit 130 may transmit data generated through the calculation of the processor 110 through wired or wireless communication with any system or any client. For example, the network unit 130 may receive medical data through communication with a database in a hospital environment, a cloud server that performs tasks such as standardizing medical data including electrocardiogram signals, or a computing device. The network unit 130 may transmit output data of the neural network model, intermediate data derived from the calculation process of the processor 110, processed data, etc. through communication with the above-described database, server, or computing device.

3 is a flowchart showing a contrasting dictionary learning method for a neural network model based on an electrocardiogram according to an embodiment of the present disclosure.

Referring to FIG. 3, as a contrastive pre-training method of a neural network model based on an electrocardiogram, which is performed by a computing device including at least one processor, a step (S100) of first acquiring a raw electrocardiogram signal may be performed. .

The ECG signal may be obtained directly as measured through an ECG measurement device, or may be obtained through network communication from the ECG measurement device. Specifically, electrocardiogram signals are acquired through multiple electrodes placed on specific surfaces of the body, and the heart condition is evaluated by measuring the voltage between the electrodes.

Typically, the standard electrocardiogram measurement method consists of 12 leads (channels), each of which measures a specific potential difference, but because patients need to have at least 10 electrodes placed on their skin in a medical facility, all 12 leads must be maintained. Because it is difficult to do, many medical practitioners and cardiologists actually use a reduced channel system.

Next, a step (S200) can be performed by inputting the sample data sampled for the raw ECG signal into a neural network model and performing contrastive dictionary learning considering both the global feature context and the local feature context through self-supervised learning. there is. This pre-trained neural network model can perform better in diagnostic classification and patient identification compared to other neural network models in downstream tasks.

In addition, the learning step (S200) inputs biological data including at least one of age, gender, weight, and height along with the electrocardiogram signal into a neural network model as variables affecting the characteristics of heart disease, and determines the relationship between the biological data and heart disease. A step of learning based on correlation may be further included.

Figure 4 is a diagram showing the structure of a neural network model according to an embodiment of the present disclosure.

As shown in Figure 4, the neural network model uses Wav2Vec 2.0. In general, the Wav2Vec model has a structure in which two convolution neural networks are stacked, and f: , h : Z ⇒ C can be called a context network.

In the neural network model, the raw ECG signal is divided into two segments using the CMSC (Contrastive Multi-Segment Coding) technique, and each segment sequentially passes through Wav2Vec's CNN and transformer. In other words, the neural network model receives one segment as input, outputs n local feature contexts (C), and creates a global feature context (G) using time average pooling of the n local feature contexts.

Specifically, the encoder network consists of several convolutional encoder blocks that encode the raw ECG signal. The context network consists of multiple transformer encoder blocks to derive local characteristics appropriate for the context from the CNN output. Additionally, the context network performs the task of collating the latent features derived from the convolutional encoder block with the local feature context by quantizing them. Here, quantization can be the process of mapping continuous-valued feature vectors to a finite set of discrete-valued vectors (i.e., codes) through a learnable codebook. The codebook contains vectors where latent features are replaced by the closest code in the codebook selected via Gumbel-softmax.

In particular, the encoder network is a convolutional encoder where, given an input X, a raw electrocardiogram signal (ECG), f: Create a local property context 'C'. At this time, the quantization module q : Z → Q quantizes the latent function. During pre-training, the latent function is quantized to q at a random time step, and as shown in the upper part of the CNN layer, m before Q is provided to the transformer. is masked.

Next, a neural network model is trained to maximize the cosine similarity between the local feature context and the corresponding quantized features at each masked time step.

Meanwhile, CMSC is a patient-specific learning method through effective self-supervised learning of unlabeled patients, where time segments as positive pairs, especially when the duration is the recording of the ith ECG signal Si seconds, pairs of time segments do not overlap each other. can be sampled in segments, such as Si/2 second time zones or Si/4 second time zones (see duration below the raw ECG signal in Figure 4). In the present disclosure, for simplicity, the duration Si = 10 is fixed for each ith ECG signal, so that adjacent segments can be used as positive pairs, and other time segments of other electrocardiogram (ECG) samples can be used as negative pairs. .

The contrastive dictionary learning method using the CMSC technique focuses on the global characteristics of ECG records through comparison of ECG records between different patients.

Under the assumption that ECG records reflect both temporal and spatial information, the neural network model must be able to utilize both temporal invariance and spatial invariance. Here, temporal invariance means that sudden changes to the ECG are unlikely to occur on the order of seconds, so adjacent segments of shorter duration continue to share the context, and temporal invariance means that even in different leads, the same heart Because it reflects functionality, it means sharing context.

Therefore, CMSC utilizes the temporal invariance of the ECG signal by defining the characteristics of adjacent but non-overlapping temporal segments as positive pairs. Given the characteristics of heart signals from completely different patients, negative pair characteristics that repel each other can be formed, and this framework allows learning characteristics that do not change with temporal changes in the ECG signal.

As such, the present disclosure provides a framework for implementing a neural network model that exhibits excellent performance in diagnostic classification and patient identification using a pre-trained neural network model, and this framework combines the local characteristics of each ECG signal and the quantized It can effectively capture local information that is important for both features, and global information that is important for global features for patient identification.

For example, the framework according to an embodiment of the present disclosure is an artificial intelligence framework implemented in Fairseq, and includes a feature extractor composed of four blocks, each block consisting of a convolutional layer, a layer normalization, and a GELU activation function. It can be configured. In addition, the convolution layer of each block has 256 channels with a stride of 2 and a kernel length of 2, and the transformer settings are set based on the BERTBASE model, so the number of transformer block layers is 12 and the dimension of the model is 768. , the number of self-attention heads can be set to 12, and the dimension of the feedforward network can be set to 3,072. When fine-tuning, an additional fully connected layer projects the feature vector into a number of classes for each task, and for identification tasks, a class can represent a unique patient in the training dataset.

Figure 5 is a flowchart explaining the process of performing the learning step according to an embodiment of the present invention.

Specifically, Figure 5 shows one embodiment of step S200 of Figure 3 in more detail. The learning step (S200) integrates the framework of Wav2Vec 2.0, CMSC technique, and RLM method to learn both local feature context and global feature context through contrast learning.

In the learning step, the 12-channel raw ECG signal is divided into 2 × N sample data (S210). Here, N represents the number of samples and is used as input to the local contrast task shown in Figure 4. Sample data may include a number of segments that can reflect the characteristics of the ECG signal, that is, various temporal and spatial characteristic values, such as QRS complex, P wave, T wave, and ST segment.

At this time, the 12-lead ECG signal is used as the standard for various ECG-related downstream tasks, but there is an increasing demand for neural network models that can provide robust performance for reduced lead ECG signals (using reduced channel systems). This is due to limited accessibility of standard 12-lead ECG recordings. Additionally, the increasing popularity of wearable electrocardiography devices (e.g., smart watches, etc.) that typically measure lead-reduced ECG signals is likely to result in even greater volumes of lead-reduced ECGs. Considering this, the obvious self-supervised learning method is to pre-train and then fine-tune individual neural network models for each lead set, but this has the problem that there are too many possible combinations with 12 leads.

Therefore, in the present disclosure, an ECG-specific enlargement method that individually and randomly masks each lead with a probability of p = 0.5, applies Random Lead Masking (RLM) to the segmented sample data to generate a masked lead ECG signal and transmits it to the CNN. Provided (S220). RLM can mimic a pre-training setup in which a neural network model for 12 leads uses various combinations of leads by stochastically masking random leads at every iteration. In other words, the neural network model to which RLM-applied sample data is input is exposed to various lead combinations in the pre-learning stage, thereby reducing the dependence on characteristics of all 12 leads of the ECG signal. Therefore, neural network models pre-trained using RLM can show robust performance when fine-tuned in downstream tasks with arbitrary lead sets.

The neural network model goes through a local minimization process using Wav2Vec 2.0, and defines the contrast loss between the local feature and its quantized feature (S230). At this time, the local feature context is created through CNN before the transformer of Wav2Vec 2.0.

Specifically, given the local feature vector c _t and the quantized latent feature q _t at masked time step t, the local contrast loss is defined as Equation 1 below.

In Equation 1, sim(a, b) is the cosine similarity between two vectors a and b, Q is the set of quantized candidate latent features consisting of quantized features of masked time steps different from q _t , and M is the masked is a set of time steps.

In order to learn the global feature context along with the local feature context, the neural network model applies time-averaged pooling to the local feature context (see what is shown at the top of the transformer block in FIG. 4) to generate the global feature context (S240). Next, the estimated loss compared to noise for each patient is used to learn the global characteristics of the ECG signal, and the global compared loss is defined as Equation 2 below.

In Equation 2, g _i = 1, t∈T, c _t represents the global characteristic of the ith sample, and P+ represents the index set of positive pairs in the batch. Finally, the overall objective function of the neural network model can be expressed as Equation 3 below.

The local contrast learning task focuses on the internal relationships within each ECG signal, while the global contrast learning task exploits the interrelationships between ECG records from different patients during learning. At this time, the local characteristics may correspond to some segments of the ECG, that is, parameters less than one bit, and the global characteristics may correspond to the entire segment of the ECG signal consisting of a plurality of bits. At this time, the segment may or may not correspond to one bit.

In this way, both the local feature context and the global feature context for the neural network model are learned through local contrast learning tasks and global contrast learning (S250), and this pre-trained neural network model is used for downstream tasks using the labeled dataset. The neural network model can be evaluated by fine-tuning (S260).

At this point, the downstream task is to evaluate the neural network model by fine-tuning the labeled data for cardiac arrhythmia classification and patient identification. During the fine-tuning process, time-averaged pooling is applied to the output of the transformer (i.e., contextualized local features (ct's)) to extract feature vectors for the entire ECG signal. We then add a randomly initialized fully connected layer after the temporary average pooling layer to perform downstream tasks.

As an example, there are five different lead combinations of leads available: 12-lead, 6-lead (I, II, III, aVF, aVL, aVR), 3-lead (I, II, V2), 2-lead (I, II) and 1 lead (I). Lead III can be obtained by the simple equation of leads I and II (Lead III = -Lead I + Lead II), and the 4-lead combination does not provide additional information compared to the 3-lead combination. Because many wearable ECG measurement devices, such as smart watches, only measure 1-lead ECG signals, it is desirable to add 1-lead (I) experiments.

Cardiac arrhythmia classification is a task to predict diagnosis. It can be classified into 26 SNOMED-CT multi-labels with ECG scores, and the CinC Score introduced in PhysioNet/Computing in Cardiology can be used for evaluation. By partially acknowledging misdiagnoses with symptoms similar to the true diagnosis (or diagnosis) and partially penalizing misdiagnoses with symptoms very different from the true diagnosis, this weighted score can be used to clinically treat all misdiagnoses as equal. It can reflect a reality that is not considered.

A neural network model that outputs the correct class for all samples receives a score of 1, and a neural network model that always outputs a normal class (i.e., no abnormalities) receives a score of 0, and the score range can be set from -1 to 1. .

Patient identification is a necessary task for neural network models for effective learning, allowing ECG signals to be represented so that the similarity between two different ECGs from the same patient is relatively high. Here, the fully connected layer is treated as a collection of weight vectors for each patient (class). That is, when performing a classification task using a conventional softmax classifier, the feature vector is learned to have high similarity (i.e., high dot product) with the corresponding weight vector and low similarity (i.e., low dot product) with other weight vectors. . However, because the softmax classifier optimizes internal similarity, it does not force high similarity between ECGs within patients and diversity between ECGs between patients. To overcome this problem, the Arc-Face loss formulated as Equation 4 below is used.

는 특징 벡터와 해당 가중치 벡터 사이의 각도이고, m은

에 대한 각도 마진 페널티이며, s는 softmax를 적용하기 전의 배율 인수이다. 추가 각도 마진 페널티로 코사인 유사성을 최적화함으로써 ArcFace 손실은 환자 간 ECG를 최소화하면서 환자 내 ECG의 유사성을 최대화할 수 있다.In equation 4,

is the angle between the feature vector and the corresponding weight vector, and m is

is the angle margin penalty for , and s is the scaling factor before applying softmax. By optimizing cosine similarity with an additional angular margin penalty, ArcFace loss can maximize the similarity of within-patient ECGs while minimizing the between-patient ECGs.

For evaluation purposes, the test set consists of two subsets, called gallery and probe set. Subsets consist of unique ECG samples but may share the same patient. For example, if the gallery set contains an ECG sample from Patient A, the probe set must also contain an ECG recording from the same Patient A. During testing, we discard the additional fully connected layers, take only the representation vectors of the ECG samples from each layer, and consider the subset.

We then calculate the cosine similarity of all possible pairs between the gallery and the probe set, and define the closest pair as having the same ID.

Hereinafter, a verification study of a neural network model according to an embodiment of the present disclosure will be described.

In the first experiment, we pre-trained the proposed neural network model and other baselines on a large-scale ECG dataset using all 12 leads, followed by 5 reduced lead combinations by padding unavailable leads with zeros for the 2 downstream tasks. Fine-tune a single 12-lead model.

In the second experiment, the proposed neural network model and other baselines for each lead combination are pretrained and fine-tuned to demonstrate the theoretical upper bound performance of the first experiment. This experiment requires a huge amount of training time and computational resources as the model is pre-trained separately for each read set.

In the third experiment, we can observe the results of using RLM with other base models by performing an ablation study on RLM. The improved performance using RLM demonstrates the effectiveness of RLM as an ECG-specific augmentation for self-supervised contrastive learning.

Since the main goal of 3KG is to convert ECG to VCG and generate positive pairs by applying stochastic 3D perturbations, we do not perform the second and third experiments with 3KG. If fewer ECG leads are used during conversion to VCG, 3D structures cannot be constructed and perturbations do not have contextual significance.

In a validation study of the neural network model, pre-training experiments are performed on six datasets: CPSC, CPSC-Extra, PTB-XL, Georgia, Ningbo, and Chapman from PhysioNet 2021 (Reyna et al., 2021).

Each sample has a sampling frequency of 500 Hz and a range of 5 to 144 seconds. For each ith data sample, we repeatedly segment Si = 10 second time segments. Additionally, to facilitate data processing for global contrast operations, each 10-second sample is split into two Si/2 = 5-second segments. This results in 189,051 samples of 12-lead ECG recordings, with a sample size of 2500 each. Utilizing the CPSC and Georgia datasets in fine-tuning for cardiac arrhythmia classification. This is because these two datasets are used for evaluation in PhysioNet/Computing. These sample data are divided into training, validation, and test sets.

Table 1 below shows the results of testing the performance when pre-training a single 12-lead model and fine-tuning 5 lead combinations. At this time, leads that cannot be used in fine tuning are filled with 0, and this is called a P-N-lead (Padded-N-lead). Here, W2V stands for Wav2Vec 2.0. CinC score for diagnostic classification (Dx.) and accuracy for patient identification (Id.) are measured. Means and 95% confidence intervals are shown across three seeds. For each lead combination for both tasks, the best performance is highlighted in bold font.

It generates 32640, 4079, and 4079 samples, respectively, at an 8:1:1 ratio. Validation and test sets are excluded from the pre-training dataset. In fine-tuning for the patient identification task, we use a pre-training dataset for training and validation. The training set and validation set consist of 147,444 and 17,670 samples, and the ratio of the pre-training dataset is 8:2. Since most PhysioNet 2021 datasets do not contain patient identity information, samples segmented from the same ECG are assumed to have the same identity. However, after fine-tuning, we test the model performance on PTB-XL with patient ID, which is explained below.

To properly test our model for the PTB-XL patient identification task, we utilize the PTB-XL dataset (Wagner et al., 2020), which is a subset of PhysioNet 2021, but with patient IDs for each ECG sample. Therefore, it is possible to identify ECG samples from different sessions, which is a more appropriate way to evaluate the model. Patients with at least two ECG sessions are selected and randomly split into 5-second intervals. Afterwards, for each unique patient, two ECG sessions are randomly selected and used as gallery and probe set, respectively. As a result, we retain 2127 unique patient pairs of 12-lead ECG records that were excluded from the pre-training dataset.

Table 2 shows the results of testing the model performance of various pre-training methods when pre-training and fine-tuning individual data. The CinC score for diagnostic classification (Dx.) and the accuracy for patient identification (Id.) are measured for each model for each lead combination. The best performance is highlighted in bold. Here, the 12 lead results in Table 1 are taken as reference.

For local contrast learning, we follow the hyperparameter settings of the original Wav2Vec 2.0, select each token as the start of the range to be masked with probability 0.065, and mask the subsequent 10 time steps. Additionally, the quantization module includes two groups of 320 codes. For ArcFace in identification, a scale factor s = 192 and a margin m = 1.0 are used.

Optimizing the neural network model using Adam (Kingma and Ba, 2014) with a learning rate of 5 Х 10-5 for pre-training and fine-tuning for the classification task and 3 Х 10-5 for fine-tuning for the discrimination task. do.

For pre-training, the batch size is 512 ECG samples of 10 seconds, split into 1024 ECG samples of 5 seconds across four RTX A6000 GPUs, giving a training time of 24 hours. For fine-tuning, the batch size is 128 ECG samples in 5 seconds on a single RTX 3090 GPU, with training times of 4 hours for cardiac arrhythmia classification and 18 hours for patient identification, respectively.

Table 1 shows experimental results when pretraining a single 12-lead model and fine-tuning various lead combinations. That is, a single model pre-trained with 12 leads is fine-tuned on reduced leads by filling in unusable leads with zeros.

As a result of the experiment, it can be seen that the neural network model, which is pre-trained by separating local feature context and global feature context using the Wav2Vec model, CMSC technique, and neural network model with RLM, shows superior performance compared to other models. For the classification task, the Wav2Vec model, CMSC technique, and neural network model with RLM showed an average improvement of 0.0298 and 0.1366 in CinC score, respectively, compared to the Wav2Vec model and the model with CMSC technique in all read combinations. Similarly, in the case of identification tasks, the accuracy of the Wav2Vec model, the CMSC technique, and the neural network model with RLM increases by an average of 6.4%p and 6.67%p, respectively, compared to the Wav2Vec model and the model with the CMSC technique in all lead combinations. .

Table 3 shows the results of testing the performance when applying random read masking to a model that integrates the Wav2Vec model and the CMSC technique, measuring CinC. Diagnostic classification score (Dx.) and patient identification accuracy (Id.). If performance is improved with RLM, it is indicated in bold.

As shown in Table 3, it can be seen that the performance of P-3-read is consistently higher than that of P-2-read or P-6-read in all methods. This is because the two limb leads are measured in the same frontal (coronal) plane and can therefore be used to calculate the other four leads. Therefore, 2-lead and 6-lead contain the same amount of ECG information. On the other hand, 3-lead contains the prespinal lead (V2) along with two extremity leads (I, II), so 3-lead has additional ECG information compared to 2-lead or 6-lead.

In addition, when comparing the results of the model that integrates the Wav2Vec model, CMSC technique, and RLM technique, and the model that integrates the Wav2Vec model and CMSC technique, it can be seen that there is a significant difference in all lead combinations. This shows that using RLM along with the Wav2Vec model and CMSC technique is superior in terms of performance when pre-learning the 12-lead ECG signal.

The fine-tuning results of the pre-trained models separately for each lead combination are shown in Table 2. For classification tasks, the Wav2Vec model shows the best scores for all lead combinations other than 12-lead. This classification task focuses on local feature context for accurate diagnosis. Therefore, the Wav2Vec model, which is a local contrast method, can demonstrate more powerful performance than a neural network model that integrates the global contrast method and the Wav2Vec model, which is a local contrast method, CMSC technique, and RLM technique.

For identification tasks, the model combining the Wav2Vec model and CMSC technique shows optimal performance for all lead combinations other than 12-lead. In this case, the global and local features learned for each read combination may outperform the local or global features.

In this experimental setup, the performance of the model combining the Wav2Vec model, CMSC technique, and RLM technique was consistently better than the model combining the Wav2Vec model and CMSC technique for all reduced lead combinations (6-lead, 3-lead, 2-lead). It can be seen that it is low. This can be assumed to be due to the use of fewer leads in pre-training with RLM, compromising the expressiveness of the model compared to using all 12 leads. In other words, by reducing the available leads, the possible lead combinations are also greatly reduced. The efficacy of RLM comes from the fact that the model can learn different combinations of leads in the pre-training phase, so reducing the available leads is not appropriate for a model that learns a robust representation for an arbitrary set of leads.

As can be seen in Table 3, the model combining the Wav2Vec model and the RLM technique shows improved performance for all read combinations in the classification task. However, when applying RLM, the performance of some read combinations in the identification task deteriorates. It can be assumed that the main reason for these results is that using RLM with the Wav2Vec model improves the ability of the Wac2Vec model to capture local feature context rather than global feature context. Therefore, the model that integrates the Wav2Vec model and the RLM technique shows superior performance in classification tasks that require local feature context compared to the Wav2Vec model, but performs poorly in identification tasks that require global feature context.

Table 4 shows the results of performance testing when applying various augmentations to a model that integrates the Wav2Vec model and the CMSC technique. Physio(4) represents power line noise, electromyography noise, baseline wander, and baseline shift. Physio(3) represents power line noise, electromyography noise, and baseline wander. CinC score for diagnostic classification (Dx.) and accuracy for patient identification (Id.) are measured.

On the other hand, the model incorporating the CMSC and RLM techniques shows significantly improved performance compared to the model to which only the CMSC technique is applied in all cases. We assume that the contrast task of the original CMSC technique, which maximizes the match between temporally adjacent ECG segments, is too simple for the model to learn valuable representations. RLM techniques help the model learn different combinations of leads. This means that the model can explore a large amount of augmented ECG samples for each patient during pre-training.

This allows the CMSC technique to learn useful characteristics of patients that perform significantly better for all lead combinations in the two downstream tasks.

As such, this disclosure provides a self-supervised learning method for global features and local features by integrating the Wav2Vec model, which is a contrasting dictionary learning method, and the CMSC technique. Additionally, in this disclosure, an augmentation technique that randomly masks each read during the learning phase so that a pre-trained neural network model can be successfully fine-tuned with random read combinations in cases where all 12 reads are not available through downstream tasks. RLM is being applied. As a result, it can be seen that the neural network model that integrates the Wav2Vec model, CMSC technique, and RLM technique provided in this disclosure shows excellent performance in terms of diagnosis classification and patient identification.

The various embodiments of the present disclosure described above may be combined with additional embodiments and may be changed within the scope understandable to those skilled in the art in light of the above detailed description. The embodiments of the present disclosure should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form. Accordingly, all changes or modified forms derived from the meaning and scope of the claims of the present disclosure and their equivalent concepts should be construed as being included in the scope of the present disclosure.

Claims (10)

1. A method for contrastive dictionary training of a neural network model based on an electrocardiogram, performed by a computing device comprising at least one processor, comprising:
When an unlabeled raw ECG signal is input, a local information generation step of generating a local representation context using a model based on self-supervised learning;
A global information generation step of generating a global characteristic context from the local characteristic context; and
A learning step of pre-training a neural network model through contrast learning based on learning data including the local feature context and the global feature context,
method.
According to paragraph 1,
The local information generation step includes using the Wav2Vec model,
The global information generation step includes using CMSC (Contrastive Multi-segment Coding) technique,
method.
According to paragraph 1,
The local information generation step is,
The raw ECG signal for a preset number of leads is divided into 2 further comprising steps,
method.
According to paragraph 1,
Further comprising an evaluation step of fine tuning the pre-trained neural network model using a labeled dataset for cardiac arrhythmia classification and patient identification,
method.
According to paragraph 1,
The local information generation step is,
Inputting the raw ECG signal to generate a latent vector (Z) in a latent space using a CNN, and encoding the latent vector (Z) to generate the local feature context (C),
The learning step is,
In the pre-learning process, the latent vector (Z) is quantized at a preset time step to generate a quantized feature (Q), which includes masking (m) before being provided to the transformer.
method.
According to clause 5,
The neural network model is,
wherein the cosine similarity between the local feature context and the quantized feature is learned to have a maximum value at each masked time step,
method.
According to paragraph 1,
The global information generation step is,
Defining a duration (Si) and time segment characteristics of adjacent but non-overlapping time zones within the duration (Si) for the ith ECG signal in the ECG record of the first patient as positive pairs; and
Comprising the step of defining time segment characteristics of the ECG signal of the second patient, which are different from the time segment characteristics of the ECG signal of the first patient, as a negative pair,
method.
In clause 7,
The learning step is,
In the pre-learning process, each patient is learned through self-supervised learning using the correlation between the ECG records of the first patient and the second patient,
method.
A computer program stored in a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs operations for contrastive dictionary learning of a neural network model based on an electrocardiogram,
The above operations are:
When an unlabeled raw ECG signal is input, an operation of generating a local representation context using a model based on self-supervised learning;
generating a global feature context by applying time average pooling to the local feature context; and
Comprising the operation of pre-training a neural network model through contrast learning based on learning data including the local feature context and the global feature context,
computer program.
A computing device for a contrastive dictionary learning method of a neural network model based on electrocardiogram, comprising:
A processor including at least one core; and
a memory containing program codes executable on the processor;
Including,
The processor, according to execution of the program code,
When an unlabeled raw ECG signal is input, at least one local representation context is created using a model based on self-supervised learning,
Create a global feature context by applying time average pooling to the local feature context,
Pre-training a neural network model through contrast learning based on learning data including the local feature context and the global feature context,
Device.

Images