1 Introduction

Electronic health records (EHRs) are becoming increasingly valuable in precision medicine as they provide diverse and comprehensive information on patients’ health conditions, which can facilitate clinical decision-making using decision support systems (DSSs). EHRs comprise a wide range of information from various sources, such as patient demographics, diagnoses, laboratory test results, prescribed medications, clinical notes, and medical images (Johnson et al., 2016). The explosive growth of EHRs has created an urgent need for innovative methods in DSSs that can efficiently utilize large amounts of high-dimensional EHR data to stratify patients. Patient stratification plays a crucial role in advancing precision medicine and personalized treatment. By stratifying patients into subgroups based on their health conditions, healthcare providers can better understand each individual’s specific medical needs, allowing for the development of customized healthcare interventions that minimize risk. Furthermore, patient stratification can help identify patients who are likely to benefit from particular treatments, enabling healthcare providers to make informed decisions and deliver personalized care that is optimized for each patient.

Patient stratification refers to the identification of subgroups of patients with similar health conditions. It is important to note that, for the sake of explainability, patient stratification also explains each subgroup using terms such as disease severity, phenotype, and diagnosis that are commonly shared among patients within the same group. This task is illustrated in Fig. 1, where the input EHR is represented as a patient-word matrix. In this matrix, the 0/1 value in the i-th row and j-th column indicates the absence/presence of the j-th word (e.g., clinical concept) in patient i. By applying computational methods, the input matrix is transformed into two matrices: the patient-subgroup matrix and the subgroup-word matrix. Each row in the patient-subgroup matrix indicates the probability of a patient being assigned to different subgroups, while each row in the subgroup-word matrix explains each subgroup by a distribution of words. Using these two matrices, we can detect the potential patients and words in each subgroup, respectively.

Fig. 1
figure 1

The illustration of patient stratification. The patient-word matrix is the representation of the raw input EHR. With the application of patient stratification models, the input data can be decomposed into two sub-matrices: the patient-subgroup matrix and the subgroup-word matrix (darker colours indicate larger values). (Color figure online)

Full size image

Conventional approaches for patient stratification often use topic modelling to extract explainable topics as patient subgroups. Latent Dirichlet Allocation (LDA) (Blei, 2012) is a widely used probabilistic modelling technique for discovering latent topics from EHR data. LDA assumes that each EHR is a mixture of some topics. Recent approaches to improving the performance of LDA have combined topic modelling with neural networks. For example, the neural topic model (NTM) (Zhao et al., 2021) uses neural variational inference to create parameterized topic distributions during training. One of the challenges of NTM is to obtain good topics when there are large narratives and big vocabulary sizes. The embedded topic model (ETM) (Dieng et al., 2020), a recent approach to NTM, overcomes this issue by integrating neural topical modelling with word embeddings (Mikolov et al., 2013). This generative model assumes that each document (e.g., EHR) is produced from a blend of topics, and each observed word (e.g., clinical concept) in the document is drawn from some topics. All topics and observed words are embedded with numeric vectors, and similarities between these vectors can indicate the similarities among words and topics.

In this paper, our focus is on developing a patient stratification model that leverages EHRs with multiple modalities. However, one challenge of applying the aforementioned methods to model multimodal EHRs is how to align and unify heterogeneous information from different modalities for learning consensus topics. Common multimodal data integration methods include early, hybrid, and late fusions, where the work in Xu et al. (2021) adopted the neural architecture search to find an optimal data fusion strategy. Moreover, there are increasing efforts being made nowadays to ensure different modalities are encoded into a unified semantic space. Among them, multi-modality contrastive learning (CL) (Grill et al., 2020; Oord et al., 2018; Caron et al., 2020; Li et al., 2021) has shown great potential. For example, Coca (Yu et al., 2022) utilized a dual encoder model like CLIP (Radford et al., 2021) to fuse image and text data, while FLAVA (Singh et al., 2022) aligned multimodal data at a fine-grain level via cross-attention. All these methods adopted the idea of CL by augmenting data with positive and negative sample pairs. However, most of them are not specifically designed to handle multimodal data from EHRs.

Here, we develop a Contrastive Multimodal learning model for EHR (ConMEHR) for patient stratification. Our study is centred on the two most frequently encountered types of modalities in EHRs: unstructured texts and structured medical terms. Unstructured texts refer to free-form narrative data, such as medical notes and reports, while structured medical terms are structured data elements containing standard medical terms like the International Classification of Diseases (ICD) codes (Organization et al., 1978) and medication names. For illustration purposes, we provide information from four representative modalities of these two types found in EHRs: Note, Disease, Symptom, and Medication. Note is an unstructured text modality, while Disease, Symptom, and Medication are the structured medical term modalities. ConMEHR addresses the challenge of unifying heterogeneous information from diverse modalities by utilizing modality-level CL. This approach enables the learning of a unified latent representation space for various modalities, which can then be integrated to obtain patient subgroups. As a topic modelling based approach, the performance of ConMEHR would also be influenced by the ability to learn diverse topics (Ding et al., 2015). Previous works examined the topic diversity by identifying unique words associated with each topic (Benson et al., 2014; Arora et al., 2013) while ignoring the semantic-level separation of topics. Hence, to address this issue, we consider imposing a topic-level CL module to diversify topics by separating their latent representations.

To demonstrate the applicability of ConMEHR, we use two EHR datasets: MIMIC-III, a publicly accessible EHR dataset in English, and a Chinese EHR dataset collected from Chinese medical clinics. To encode EHR characters in English and Chinese, we utilized BioBERT (Lee et al., 2020) and Chinese-BERT (Cui et al., 2021), respectively. We focus on four representative modalities of EHRs, namely, Note, Disease, Symptom, and Medication, to illustrate our approach. Considering the difficulty of modelling data with high complexity, we incorporate a regularization module as a training trick in our model. Our regularization module is based on the one proposed in RDrop (Wu et al., 2021). This module is trained by minimizing the bidirectional Kullback–Leibler (KL) divergence between the output distributions of two sub-models after dropout. It reduces the freedom of the model parameters and hence improves the model’s adaptability to high-complexity multiple modalities data. Our contributions in this paper can be summarized as follows:

  • We propose ConMEHR, a deep learning model for patient stratification that utilizes both unstructured and structured modalities of EHRs to identify patient subgroups.

  • To effectively integrate diverse information from multiple modalities, we propose using modality-level and topic-level contrastive learning mechanisms to obtain a unified latent representation space for the various modalities, while also diversifying patient subgroups.

  • We evaluate ConMEHR on two real-world EHR datasets, using both quantitative and qualitative methods. Case studies, such as patient subgroup interpretation and visualization, demonstrate the significant power of our model in DSS.

2 Related work

2.1 Patient stratification using EHRs

Patient stratification using EHRs is a crucial task in precision medicine that provides decision support. This task involves grouping patients into different categories and explaining each subgroup using shared terms among patients in the same group. This approach allows clinicians and researchers to gain insights into the underlying factors contributing to a specific patient’s condition and provide accurate prognoses. This is in contrast to classification models such as Bagging CART and XGBoost, which only focus on predicting patients to predefined classes. There are primarily two branches of methods used for patient stratification: tensor factorization and topic modelling.

Tensor factorization (Kim et al., 2017; Wang et al., 2015) is an efficient approach for patient stratification, where massive EHRs are converted into meaningful concepts. For example, PARAFAC2 (Harshman, 1972) automated the stratification for patients who need intensive medical care. Additionally, in Henderson et al. (2017), a diversified and sparse nonnegative tensor factorization method was developed to derive patient subgroups from EHRs. Another work in Ho et al. (2014) stratified patients by decomposing the tensor representation of EHR using the CP-APR algorithm (Chi & Kolda, 2012).

Topic models are commonly used to deal with unstructured texts and explain patient groups by learning topics so that patients with similar observations can be grouped together based on shared topics. LDA (Blei, 2012) is a classical topic modelling approach that uses bag-of-words (BoW) inputs to find patient subgroups from documents. There have been numerous efforts to enhance its performance. For example, methods introduced in (Shi et al., 2017; Zhao et al., 2017) modify the prior distributions over topics in LDA. In Xun et al. (2017), a topic model is incorporated with word embedding to first convert the discrete texts into continuous representations. The work in Bunk and Krestel (2018) involved replacing topic-drawn words with Gaussian-distributed embeddings in a random manner. The emergence of neural networks has led to a surge in the number of approaches that combine probabilistic topic models with deep neural networks (Srivastava & Sutton, 2017; Cong et al., 2017; Zhang et al., yyyy). The majority of these approaches employ variational auto-encoder and amortized inference to reduce input data dimensionality (Rezende et al., 2014; Dieng et al., 2020). Among them, ETM (Dieng et al., 2020) and its extensions (Zou et al., 2022; Wang et al., 2022) are neural topic models that use word embeddings and also learn topic embeddings. In our study, the technical framework is based on the ETM approach, which leverages the explanatory power of topic modelling to explain subgroups while also incorporating semantic features through the use of embedding representations.

2.2 Contrastive representation learning

CL is a type of self-supervised learning that has gained increasing attention due to its tremendous impact on representation learning (Mikolov et al., 2013; Chopra et al., 2005; Logeswaran & Lee, 2018; Weinberger & Saul, 2009; Chechik et al., 2010; Hoffer & Ailon, 2015; Oh Song et al., 2016; Mikolov et al., 2013; Henaff, 2020; Hjelm et al., 2018; Wu et al., 2018; He et al., 2020; Zhang et al., 2022). CL works based on the InfoMax principle (Linsker, 1988) by maximizing the mutual information between two augmented views of a sample (e.g., an image with different rotations and shifts) (Tian et al., 2020; Bachman et al., 2019). In CL, an anchor and a positive sample should be drawn closer together, while an anchor and a negative sample should be drawn farther apart (Chen et al., 2020; He et al., 2020; Pan et al., 2021; Gao et al., 2021).

One of the research areas of interest for CL is the design of positive pairs (Arora et al., 2019; Gao et al., 2021). For single-modality CL approaches, a typical practice is to optimize by an auxiliary set gained through data augmentation (Wu et al., 2018; Ho & Nvasconcelos, 2020; He et al., 2020; Chen et al., 2020). These methods disperse different instances apart while implicitly bringing similar instances together. In multi-modality CL approaches, for each modality of a certain sample, other modalities from the same sample will be regarded as positives, while any modalities from different samples would be negatives (Grill et al., 2020; Oord et al., 2018; Caron et al., 2020; Li et al., 2021; You et al., 2021; Zhang et al., 2022). These methods guarantee that the features from different modalities of the same sample map to proximate points in the latent representation space. Among them, MCSE (Zhang et al., 2022) is the state-of-the-art multimodal contrastive model used for learning sentence embeddings. It employs a multimodal contrastive objective that aligns data from different modalities in an embedding space. This generic multimodal objective can be integrated into various sentence embedding techniques, potentially improving their effectiveness.

3 Methodology

3.1 Patient stratification system

As shown in Fig. 2, the proposed patient stratification system consists of three key steps, which are data preprocessing, the ConMEHR model and applications in precision medicine.

Fig. 2
figure 2

The pipeline of our patient stratification system

Full size image

The data preprocessing is responsible for receiving heterogeneous data from EHR data warehouses and preparing it for subsequent analyses. In the patient stratification system presented in this paper, the module takes information from four representative modalities of EHRs: Note (N), Disease (D), Symptom (S), and Medication (M). However, it can be naturally extended in the future to handle other modalities. Among these modalities, Disease, Symptom, and Medication are structured medical term modalities that consist of structured data. In contrast, Note contains unstructured texts that are more likely to have noisy information resulting from synonyms, abbreviations, and informal descriptions. Therefore, a range of natural language processing (NLP) techniques, such as word segmentation, stop word removal, and abbreviation replacement methods will be adopted to preprocess the unstructured texts. Word segmentation involves breaking down lengthy texts, such as paragraphs and articles, into word units to facilitate easier analysis and processing. For English text, we usually split it according to space, while for Chinese text with different written rules, we use the integrated toolkit jiebaFootnote 1 to split sentences. Stop words, which are frequently used words such as ‘the’, ‘a’, ‘an’, or ‘in’ are ignored by search engines. Since we do not intend to utilize these words in our modelling and analysis processes, we can easily remove them by creating a list of potential stop words. We utilize the Natural Language Toolkit (NLTK)Footnote 2 in Python, which includes stopping word lists in 16 different languages. For abbreviation replacement, there are mainly two steps: abbreviation detection and replacement. The AbbreviationDetector component in Spacy,Footnote 3 a free open-source library for NLP in Python, can support abbreviation detection. We apply AbbreviationDetector to identify abbreviations and replace them with their full names.

The ConMEHR model is used to stratify patients based on preprocessed multimodal EHR data. This approach extends the idea of topic modelling, which assumes that patients belong to latent topics or subgroups, and the patient-to-topic distribution can determine the probabilities of assigning patients to different subgroups. To achieve this, we use the ETM model, which employs deep learning in representation learning to generate the representations of learned latent topics, ensuring explainable results. ETM is used as our backbone model and improved to integrate the heterogeneous information from multiple modalities of EHRs via CL. To encode each modality, neural networks first project raw data into a latent representation space. It is important to note that multiple modalities from the same EHR are potentially correlated and can provide complementary information to each other. Therefore, to ensure that their latent representations share some common characteristics, we propose modality-level CL. This technique is widely used to obtain robust representations in the manner of self-supervised learning, and we will explain how we adapt it to our problem in the following section. In addition to modality-level CL, we also design topic-level CL to diversify topics or subgroups and learn separable topic representations.

The results of the ConMEHR model are promising for various applications in precision medicine, including personalized treatment plans, targeted interventions to reduce individual risk, and efficient allocation of medical resources. In the following subsection, we will provide detailed information about the ConMEHR model.

3.2 Our ConMEHR model

As shown in Fig. 3, ConMEHR consists of two modelling processes: the inference process and the generative process. We will first describe the inference process to get the patient subgroup distributions from the multimodal EHR data via modality-level CL. Then, we introduce the generative process which adopts topic-level CL to get distinguished subgroup representations via BoW reconstruction. Finally, the optimization objective used in the model training will be defined. The main notations used in this section are summarized in Table 1.

Fig. 3
figure 3

The structure of the ConMEHR model. For the d-th EHR, the inference process encodes its four modalities to get the topic distribution \(\varvec{\theta }_{d}\). The four modalities are Note, Disease, Symptom, and Medication, labelled as N, D, S, and M, respectively. The text encoder in the generative process is BioBERT or Chinese-BERT which depends on whether the input data is Chinese or English text. The modality-level CL is used to align and unify those four modalities in the same semantic space. The dropout-based regularization module is used here to improve the model’s representation ability via dropout technology. The generative process uses the topic distribution \(\varvec{\theta }_d\) and embeddings of vocabularies to learn subgroup representations and reconstruct the BoW representation of inputs. The topic-level CL is used to get separable topic representations

Full size image
Table 1 Symbols and descriptions
Full size table
Fig. 4
figure 4

The illustration of modality-level CL. In the embedding space, for an anchor point \( \varvec{h}_i^t\), the modality-level CL aims to pull positive samples like \(\varvec{h}_i^{t'}\) and \(\varvec{h}_i^{t''}\) closer and push apart negative ones like \(\varvec{h}_j^{t}\), \(\varvec{h}_j^{t'}\) and \(\varvec{h}_j^{t''}\). Here, \(\varvec{h}_i^{t'}\) and \(\varvec{h}_i^{t''}\) are from the EHR i same as the anchor \(\varvec{h}_i^t\) but of different modalities, while \(\varvec{h}_j^{t}\), \(\varvec{h}_j^{t'}\) and \(\varvec{h}_j^{t''}\) are from a different EHR j

Full size image

3.2.1 Inference process for patient subgrouping through contrastive multimodal learning

For each EHR d, the inference process encodes its multimodal information to approximate the posterior distribution of the latent representation \(\varvec{x}_d\). The posterior distribution is denoted as \(q(\varvec{x}_d \vert \varvec{w}_d^*)\), where \(\varvec{w}^*_d = \{\varvec{w}_{d}^N,\varvec{w}_{d}^D,\varvec{w}_{d}^S,\varvec{w}_{d}^M\}\) contains the normalized BoW representations of four modalities. For each modality, a separated MultiLayer Perceptron (MLP) module is employed to map the normalized BoW inputs into the latent space. Let \(\varvec{h}^N_d\), \(\varvec{h}^D_d\), \(\varvec{h}^S_d\), and \(\varvec{h}^M_d\) denote the latent representations of four modalities respectively. One challenge of integrating these latent representations lies in aligning and unifying them in the same semantic space.

Inspired by the work in Li et al. (2021), we introduce the modality-level CL mechanism to facilitate multimodal data integration. CL is a self-supervised learning approach for learning robust representations by encouraging augmented samples of the same input to have relatively similar representations to other augmented samples. It is based on the concept of positive and negative samples in relation to an anchor point. This entails pulling the anchor and positive sample closer in the embedding space while simultaneously pushing the anchor away from multiple negative samples. The modality-level CL mechanism we proposed is to encourage representations from different modalities of the same sample to become more similar than those from different samples. To adopt the modality-level CL mechanism, we construct positive and negative pairs for the training. The positive pair (\(\varvec{h}_i^t\), \(\varvec{h}_i^{t'}\)) is constructed by selecting an anchor point \(\varvec{h}_i^t\) from one modality t and pairing it with \(\varvec{h}_i^{t'}\), which is derived from the same EHR i but belongs to a different modality. The negative pairs can be classified into two distinct types. The first type consists of (\(\varvec{h}_i^t\), \(\varvec{h}_j^t\)) pairs, where \(j\ne i\) represents a different EHR index within the current batch, but both vectors belong to the same modality t. The second type of negative pair comprises (\(\varvec{h}_i^t\), \(\varvec{h}_j^{t'}\)), where \(\varvec{h}_j^{t'}\) is distinct not only in the EHR index but also in the modality index from the anchor point \(\varvec{h}_i^t\). Specifically, for a batch of EHRs, the modality-level contrastive learning loss is as shown in Fig. 4 and defined as follows:

$$\begin{aligned} \begin{aligned} \mathcal {L}_{mcl} = -\sum _{t=1}^{4} \sum _{t' \geqslant t}^{4}\sum _{i=1}^{B}log\frac{exp(sim(\varvec{h}^{t}_{i},\varvec{h}^{t'}_{i})/\tau )}{{\textstyle \sum _{j=1}^{B}\mathbb {1}_{j\ne i}\cdot exp(sim(\varvec{h}^{t}_{i},\varvec{h}^{t'}_{j}))/\tau )}}, \end{aligned} \end{aligned}$$
(1)

where t and \(t'\) are the indices of modalities, B is the batch size, and \(\mathbb {1}_{j\ne i}\) is an indicator function which equals to 1 when \(j\ne i\) and equals to 0 when \(j = i\). \(\tau \) denotes the temperature parameter which is used to regulate the severity of penalties imposed on the hard negative samples. sim(.) measures the cosine similarity between two vectors (Chen et al., 2020). \(\varvec{h}^{t}_{i}\) is the latent representations of modality t from the i-th EHR. The numerator of Eq.(1) represents the similarity of two modalities of the same sample, while the denominator represents the similarity of two modalities from different samples in the batch.

With the constraints imposed by the modality-level CL, four latent representations are concatenated into one vector via:

$$\begin{aligned} \begin{aligned}&\varvec{h}^{mix}_d=\varvec{h}^N_d\oplus \varvec{h}^D_d\oplus \varvec{h}^S_d\oplus \varvec{h}^M_d, \end{aligned} \end{aligned}$$
(2)

where \(\oplus \) is the concatenation operator. \(\varvec{h}^{mix}_d\) can be regarded as an integrated representation of the multimodal EHR. Then, we approximate the posterior distribution of \(\varvec{x}_d\) using \(\varvec{h}^{mix}_d\). Following the framework of neural variational inference approaches (Miao et al., 2016; Kingma & Welling, 2013; Rezende et al., 2014), the posterior \(q(\varvec{x}_d \vert \varvec{w}^*_d)\) is assumed to be Gaussian and it’s mean vector \(\varvec{\mu }_d\) and standard deviation vector \(\varvec{\sigma }_d\) are approximated via two separated MLP modules respectively. A sample of \(\varvec{x}_d\) can be obtained from \(q(\varvec{x}_d \vert \varvec{w}^*_d)\) as follows:

$$\begin{aligned} \hat{\varvec{x}}_d = \varvec{\mu }_d + \epsilon \cdot \varvec{\sigma }^2_d \end{aligned}$$
(3)

where \(\epsilon \in {\mathcal {N}} (0,\varvec{I})\). With \(\hat{\varvec{x}}_d\), the per EHR topic distribution \(\varvec{\theta }_d\) can be generated as:

$$\begin{aligned} \varvec{\theta }_d = softmax(\hat{\varvec{x}}_d), \end{aligned}$$
(4)

where softmax(.) is the Softmax activation function (Nwankpa et al., 2021) to transform the Gaussian sample \(\hat{\varvec{x}}_d\) into the topic distribution \(\varvec{\theta }_d\).

3.2.2 Generative process for subgroup representation learning via BoW reconstruction

Similar to our backbone model ETM, the generative process in the ConMEHR model utilizes the embeddings of vocabularies and also learns the embedding matrix of topics. Each word is assigned with an L-dimensional embedding vector by pretrained language models (PLMs). PLMs are extensive neural networks that find application in a broad range of NLP tasks. These models operate using a pretrain-finetune approach, where they undergo pretraining on a vast corpus of text, following which they are refined for a downstream task. PLMs are considered to be effective language encoders as they offer fundamental language comprehension abilities that can be leveraged across various downstream applications. Elazar et al. (2021). Bidirectional Encoder Representations from Transformers (BERT) (Kenton & Toutanova, 2019) is one classical PLM that utilizes Transformer (Vaswani et al., 2017), a novel neural network architecture based on a self-attention mechanism, for language understanding. There are several different BERT models available. For example, Chinese-BERT (Cui et al., 2021) is a language model pretrained on a large Chinese text corpus for encoding Chinese texts. BioBERT (Lee et al., 2020) is a pretrained biomedical language model which adopts the architecture of BERT (Devlin et al., 2019) and is fine-tuned on PubMedFootnote 4 abstracts and PMC articles. SentenceBERT (Reimers & Gurevych, 2019) is a language model which has fine-tuned BERT (Devlin et al., 2019) for better measuring sentence similarities. In our paper, we focus on using Chinese-BERTFootnote 5 and BioBERTFootnote 6 to encode terms in the Chinese EHR dataset and the English EHR dataset respectively.

Suppose the latent representation of each topic is represented as an embedding vector \(\varvec{\alpha }_k \in \mathbb {R}^L\) for \(k \in \{1,..., K\}\), where K is the total number of topics. In our model, \(\varvec{\alpha }_k\) will be randomly initialized and updated via end-to-end training. To learn separable topic representations, we follow the work in Zhang et al. (2021) and introduce topic-level CL. In our topic-level CL module, for each topic k we regard its embedding vector \(\varvec{\alpha }_k\) as the anchor and the embeddings from other topics as negatives. Specifically, positive pairs are formed by using identical embedding vectors, where both vectors in the positive pair (\(\varvec{\alpha }_k\), \(\varvec{\alpha }_k\)) correspond to the same topic k. On the other hand, negative pairs are formed by using non-identical embedding vectors and consist of pairs (\(\varvec{\alpha }_k\), \(\varvec{\alpha }_{k'}\)) where k and \(k'\) represent different topics. The topic-level contrastive learning loss is defined as follows:

$$\begin{aligned} \begin{aligned} \mathcal {L}_{tcl} = -\sum _{k=1}^{K}log\frac{exp(sim(\varvec{\alpha }_{k},\varvec{\alpha }_{k})/\tau )}{{\textstyle \sum _{k'=1}^{K}\mathbb {1}_{k'\ne k}\cdot exp(sim(\varvec{\alpha }_{k},\varvec{\alpha }_{k'})/\tau )}}, \end{aligned} \end{aligned}$$
(5)

where the numerator of Eq. (5) is a constant value equal to \(exp(1/\tau )\), while the denominator represents the similarity of the samples in negative pairs. Therefore, Eq. (5) can be rewritten as \(\mathcal {L}_{tcl} = -(1/\tau )+2\sum _{k=1}^{K}log\sum _{k'> k}exp(sim(\varvec{\alpha }_{k},\varvec{\alpha }_{k'})/\tau )\), implying that the training goal is to distinguish representations from different topics.

In the generative process, the BoW representations of EHRs are reconstructed using all modalities’ word probability distributions for all topics obtained from the embeddings of vocabularies and topics. The detailed steps are described as follows. For each EHR d, using the topic distribution \(\varvec{\theta }_d\) from Eq. (4) in the inference process, the topic assignment of the n-th word in modality t is drawn from:

$$\begin{aligned} z_{d,n}\sim \ Cat(\varvec{\theta }_d), \end{aligned}$$
(6)

where Cat(.) refers to the categorical distribution and \(z_{d,n} \in \{1, 2,..., K \} \). The word probability distribution for the assigned topic \(z_{d,n}\) can be obtained from:

$$\begin{aligned} \varvec{\beta }_{z_{d,n}}^t = softmax(({\varvec{\rho }^t})^T \varvec{\alpha }_{z_{d,n}}) \end{aligned}$$
(7)

through calculating the inner product of the topic embedding vector and the embedding of each word in the vocabulary. The operator \((.)^T\) indicates the matrix transposition and softmax normalizes \( \vert V_t \vert \) real numbers resulting from the inner product into a probability distribution of \( \vert V_t \vert \) possible outcomes. With \(\varvec{\beta }_{z_{d,n}}^t\) and \(z_{d,n}\), the n-th word \(w_{d,n}^t \in \{1,..., \vert V_t \vert \}\) can be drawn from:

$$\begin{aligned} w_{d,n}^t\sim \ Cat(\varvec{\beta }_{z_{d,n}}^t). \end{aligned}$$
(8)

The log marginal likelihood of \(w_{d,n}^t\) is represented as:

$$\begin{aligned} \log p(w_{d,n}^t \vert \varvec{x}_d) = log\sum _{z_{d,n}}[p(w_{d,n}^t \vert \varvec{\beta }_{z_{d,n}}^t)p(z_{d,n} \vert \varvec{\theta }_{d})] = \log {\mathcal {B}^t} \varvec{\theta }_{d}, \end{aligned}$$
(9)

where \(\mathcal {B}^t=[ \varvec{\beta }_1^t,..., \varvec{\beta }_k^t,..., \varvec{\beta }_K^t]\) is the modality t’s word probability distributions for all topics (Miao et al., 2017).

3.3 Optimization objective

We adopt the framework of variational inference (Blei et al., 2017) so that the evidence lower bound (ELBO) (Yang, 2017) for minimization is defined as:

$$\begin{aligned} \begin{aligned}&\mathcal {L}_d = -\mathbb {E}_{q(\varvec{x}_d \vert \varvec{w}^*_d)} [\sum _{t=1}^{4}\sum _{n=1}^{N_{d}^t} \log p(w_{d,n}^t \vert \varvec{x}_d)]+D_{KL}[q(\varvec{x}_d \vert \varvec{w}^*_d) \vert \vert p(\varvec{x}_d)], \end{aligned} \end{aligned}$$
(10)

where \(N_{d}^t\) is the total number of words in modality t from the d-th EHR, and \(D_{KL}[.]\) measures the KL divergence (Joyce, 2011) between the prior \(p(\varvec{x}_d)\) and the variational approximation \(q(\varvec{x}_d \vert \varvec{w}^*_d)\). Here, the prior \(p(\varvec{x}_d)\) is assumed to follow the normal distribution N(0, I) (Miao et al., 2016).

It is worth noting that due to the high complexity of EHRs, it is not a trivial task to generalize our model to unseen EHR data. Therefore, a dropout-based regularization mechanism is introduced as a training trick to improve the representation ability. \(\varvec{h}^{mix}_d\) is fed into two parallel sub-modules of the same architecture (composed of MLP modules with dropout) but with distinct parameters. The outputs of these two sub-modules follow two distributions \(q(\varvec{x}_d^1 \vert \varvec{w}^*_d) = N(\varvec{\mu }^1_d,(\varvec{\sigma }^{1}_d)^2)\) and \(q(\varvec{x}_d^2 \vert \varvec{w}^*_d) = N(\varvec{\mu }^2_d,(\varvec{\sigma }^{2}_d)^2)\). The samples are generated as follows:

$$\begin{aligned} \hat{\varvec{x}}^1_d = \varvec{\mu }^1_d+\epsilon _1\cdot \varvec{\sigma }^1_d, \end{aligned}$$
(11)

and

$$\begin{aligned} \hat{\varvec{x}}^2_d = \varvec{\mu }^2_d+\epsilon _2\cdot \varvec{\sigma }^2_d, \end{aligned}$$
(12)

where \(\epsilon _1\) and \(\epsilon _2\) are small random noises following normal distribution \({\mathcal {N}} (0, I)\). Then the average of \(\hat{\varvec{x}}_d^1\) and \(\hat{\varvec{x}}_d^2\) gives \(\hat{\varvec{x}}_{d}\), that is:

$$\begin{aligned} \hat{\varvec{x}}_d = \frac{1}{2}(\hat{\varvec{x}}_d^1 + \hat{\varvec{x}}_d^2). \end{aligned}$$
(13)

Therefore, with the adoption of the dropout-based regularization mechanism, \(\hat{\varvec{x}}_d\) will be obtained from Eq. (13) instead of Eq. (3) The bidirectional KL divergence from the outputs of the dropout-based regularization module will also be included in our optimization objective as:

$$\begin{aligned} \begin{aligned}&\mathcal {L} = \mathcal {L}_d + \lambda _{kl}*\mathcal {L}_{KL} + \lambda _t*\mathcal {L}_{tcl} + \lambda _m*\mathcal {L}_{mcl}, \end{aligned} \end{aligned}$$
(14)

where

$$\begin{aligned} \begin{aligned}&\mathcal {L}_{KL} = \frac{1}{2} D_{KL}[q(\varvec{x}_d^1 \vert \varvec{w}^*_d) \vert \vert q(\varvec{x}_d^2 \vert \varvec{w}^*_d)] + \frac{1}{2}D_{KL}[q(\varvec{x}_d^2 \vert \varvec{w}^*_d) \vert \vert q(\varvec{x}_d^1 \vert \varvec{w}^*_d)]. \end{aligned} \end{aligned}$$
(15)

\(\lambda _{kl}\), \(\lambda _t\) and \(\lambda _m\) are the weights to balance various losses. We select their values from {0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000} using the grid search method. The whole training process of ConMEHR is shown in Algorithm 1.

Algorithm 1
figure a

The training process of ConMEHR

Full size image

4 Experiments

This section conducts extensive experiments to demonstrate the performance of our proposed model on patient stratification. In Sect. 4.1, we introduce two real-world datasets, comparative methods, and evaluation metrics. In Sect. 4.2, the optimal values of hyperparameters for each comparative method are determined. In Sect. 4.3, we show quantitative evaluation results to demonstrate the superiority of our ConMEHR model on both datasets. In Sect. 4.4, we further illustrate the impacts of our method in decision support through qualitative experiments. In Sect. 4.5, we conduct ablation studies to understand the effectiveness of core modules in our model.

4.1 Experience setup

4.1.1 Dataset

To demonstrate the applicability of our proposed model in real-world scenarios, we use two EHR datasets: the MIMIC-III datasetFootnote 7 and the Chinese Medical Clinical (CMC) dataset. We focus on four representative modalities to learn multimodal topics from EHRs: Note, Disease, Symptom, and Medication. These modalities can be divided into two categories: unstructured texts and structured medical terms. Note, which contains unstructured texts, belongs to the unstructured modality, while Disease, Symptom, and Medication consist of standard terms and are considered structured modalities. Unstructured texts can be challenging to work with because of the presence of synonyms, abbreviations, and informal language, which often make the text more difficult to process. To preprocess the unstructured texts in EHRs, we use standard NLP techniques such as word segmentation, stop word removal, and abbreviation replacement.

The details of these two datasets are summarized in Table 2. In the MIMIC-III dataset, the disease modality contains ICD codes (Organization et al., 1978), while the symptom modality is composed of Human Phenotype Ontology (HPO) codes (Köhler et al., 2017). The explanations of ICDs and HPOs are fed into the pretrained language model BioBERT (Lee et al., 2020) to generate the corresponding embeddings of these codes. BioBERT, as a domain-specific language model, is also used to embed the Note and Medication in MIMIC-III. As for the CMC dataset, it collects EHRs from Chinese medicine clinics in Hong Kong,Footnote 8 and all information stored in CMC is in Chinese, motivating us to adopt ChineseBERT (Cui et al., 2021) for embedding.

Table 2 The summary of the two datasets
Full size table

4.1.2 Comparative methods

First, we assess the performance of ConMEHR by comparing it with the following topic modelling techniques.

  • LDA (Blei, 2012) is a conventional topic modelling approach that assumes each document is generated from a small number of latent topics, and each topic is explained by a distribution over words. We use the implementation in gensim,Footnote 9 a Python library.

  • NVDM (Miao et al., 2016) is a neural variational document model that employs a continuous stochastic document representation in combination with a generative model based on a variational auto-encoder. We use the PyTorch implementationFootnote 10 for comparison.

  • ProdLDA [31] is a topic modelling approach that utilizes the logistic normal distribution as the prior of topic proportion and adopts amortized variational inference. We use the original implementationFootnote 11 for comparison.

  • ETM (Dieng et al., 2020) is a document generative model that combines topic models with word embeddings. We replace the Word2Vec module in ETM with BioBERT (Lee et al., 2020) and ChineseBERT (Cui et al., 2021), which have been pre-trained on large corpora, to obtain more accurate semantic embeddings for MIMIC-III and CMC data, respectively. We use the implementation in.Footnote 12

  • ETM-multi is an extended version of ETM that has multiple parallel data encoding and reconstruction branches for different modalities. The latent representations of different modalities are integrated together to obtain the topic distribution. Each modality is reconstructed from the topic distribution via its own reconstruction module.

  • MCSE-TM integrates a contrastive multimodal objective, derived from the state-of-the-art MCSE model (Zhang et al., 2022), into ConMEHR’s topic modelling framework. This objective allows the integration of data from multiple modalities into a shared embedding space, thereby enhancing the overall performance of the model.

Then, we conduct ablation studies to investigate the impacts of the CL modules in our ConMEHR by removing CL modules:

  • ConMER\(^{-}\) is the ablated version of ConMEHR in which both the topic-level and modality-level CL modules are removed.

  • ConMEHR\(^{-}\)+ModalityCL keeps the modality-level CL module of ConMEHR while the topic-level CL module is discarded.

  • ConMEHR\(^{-}\)+TopicCL keeps the topic-level CL module of ConMEHR while the modality-level CL module is discarded.

4.1.3 Evaluation metrics

The performance of all comparative models is evaluated using the following four metrics:

Table 3 Two illustrative examples from the MIMIC-III dataset
Full size table
Fig. 5
figure 5

The values of Coherence, Diversity and Quality in the MIMIC-III dataset with different values of topic number K

Full size image
Fig. 6
figure 6

The values of Coherence, Diversity and Quality in the CMC dataset with different values of topic number K

Full size image

  • Coherence. Topic coherence measures the degree of similarity between the top words within each topic. It is calculated by averaging the pointwise mutual information of pairs of words (Dieng et al., 2020). Topic coherence is a measure of how closely related the top words in a given topic are, and it is typically quantified by computing the average pointwise mutual information between pairs of words within that topic. Essentially, this means that the higher the coherence score for a given topic, the more semantically similar and connected its top words are to each other.

    As mutual information measures the occurrence of a word pair, it emphasizes their semantic similarity. Table 3 provides an illustrative example. ‘EHR1’ and ‘EHR2’ are two samples in MIMIC-III containing the disease code ‘ICD5849’. The HPO code ‘HP:0001919’ from ‘EHR1’ and ‘HP:0003774’ from ‘EHR2’ have similar semantic meanings. When calculating the mutual information between ‘ICD5849’ and ‘HP:0001919’, only their co-occurred EHRs will be considered so that samples like ‘EHR2’ will be excluded.

    Hence, we adopt a new measure of coherence, which replaces mutual information with cosine similarity. Specifically, the Coherence score is calculated as:

    $$\begin{aligned} Coherence = \frac{1}{K}\sum _{k=1}^{K}\frac{1}{45}\sum _{i=1}^{10} \sum _{j=i+1}^{10}g_{c}(\mathbf{{w}}_{i}^{(k)},\mathbf{{w}}_{j}^{(k)}), \end{aligned}$$
    (16)

    where \(\{\mathbf{{w}}_{1}^{(k)},...,\mathbf{{w}}_{10}^{(k)}\}\) denotes the latent representations of the top 10 most likely words in the topic k, and \(g_{c}(\cdot ,\cdot )\) calculates the cosine similarity between two words. A large Coherence value indicates that the top 10 words of the same topic are quite likely to be semantically similar.

  • Diversity. It measures the degree of diversity among various latent topics. Following the definition in Dieng et al. (2020), it is calculated as the percentage of unique words in the top words of all topics. If the Diversity score approaches 1.0, then we can say that the learned topics have no significant overlaps.

  • Quality. It shows the quality of the BoW reconstruction of the topic model. For each document in the dataset, we obtain the document-to-topic distribution and use it to reconstruct the document. The Quality score is calculated as the average reconstruction loss of the dataset, as described in Eq. (9), following the definition proposed in Miao et al. (2016). Therefore, a lower Quality score means that the model can generate better document-to-topic distributions for document reconstruction.

  • Ratio. It reflects the relative contributions from the four modalities for learning latent consensus topics and is in the form of \(r_N/r_D/r_S/r_M\). To obtain the Ratio, we first select the 10 top-scoring words for each topic and then calculate the proportion of these \(K*10\) top words from different modalities. For example, \(r_N\) is the proportion of top words from Note modality. By examining the Ratio result, we can determine whether any modalities have dominated the topic learning process while some other modalities’ information is ignored due to imbalanced dimensionality across different modalities.

4.2 Hyperparameter selection

For all comparative models, the topic number (the number of subgroups) is a hyperparameter that needs to be predefined. Here, we investigate the influence of the topic number and select the optimal values for the following experiments. We choose the topic number from {5, 10, 15, 20, 25, 30, 35, 40}. Figures 5 and 6 show the changes in the Coherence, Diversity, and Quality scores for MIMIC-III and CMC, respectively, as the number of topics increases. By examining these figures, we choose the optimal topic number for each model to be the one that yields high Coherence and Diversity scores while maintaining a low Quality score.

Table 4 The selected topic number for each comparative method
Full size table

Table 4 contains the optimal K for all comparative models. Let us explain the choice of our ConMEHR model as an example. As shown in Fig. 5g, when K is less than 20, the Coherence and Diversity values are close to their maximum value of 1.0 while the Quality value remains high. When K is greater than 20, the Coherence and Diversity values show an overall downward trend, and the Quality curve slows down its decrease. Therefore, we choose K = 20 for ConMEHR in the MIMIC-III dataset. Figure 6g shows that for the CMC dataset, the Coherence and Diversity scores decrease after the number of topics is greater than 10, while the Quality score reaches its optimal value at K = 10. Therefore, we set K = 10 for the CMC dataset in the following experiments. We also choose the weights of various losses in Eq. (14) using the grid search method. Finally, we selected \(\lambda _{kl}\) = 100, \(\lambda _t\) = 10, and \(\lambda _m\) = 1 for the MIMIC-III dataset and \(\lambda _{kl}\) = 10, \(\lambda _t\) = 1, and \(\lambda _m\) = 1 for the CMC dataset.

Table 5 Comparison in topic modelling on the MIMIC-III and CMC datasets. \(\downarrow \) (or \(\uparrow \)) indicates smaller (or larger) values are preferred. The best results are highlighted in bold
Full size table

4.3 Quantitative evaluation

In this section, we employed hold-out cross-validation to ensure a fair comparison with other topic modelling methods. Although both multi-fold and hold-out cross-validation is commonly used, multi-fold is typically more suitable for small sample sizes. However, since our experimental section utilized the MIMIC-III and CMC datasets, which consist of 13,492 and 6,868 samples respectively, hold-out cross-validation was the appropriate choice for our study. Additionally, to reduce the impact of randomness, we trained all models five times with a fixed set of five different seeds and presented indicator performance and standard deviation as in Table 5.

First, we compare ConMEHR with models that only handle single-modality data, where we concatenate inputs from the four modalities into one sequence as the input. We observe that our ConMEHR model outperforms LDA, ProdLDA, NVDM, and ETM for both datasets, achieving the best Coherence and Diversity values. For the Quality score, ConMEHR has the second-smallest value for MIMIC-III and the smallest value for CMC. Although ProdLDA has the lowest Quality score for MIMIC-III, it has a highly imbalanced Ratio result of 0.61/0.20/0.03/0.17, indicating that Note modality has dominated the topic learning process.

The incorporation of multiple parallel structures to handle various modalities in the ETM-multi, MCSE-TM, and ConMEHR models led to excellent performance in achieving a balanced Ratio score across both datasets when compared to LDA, ProdLDA, NVDM, and ETM. These models used parallel branches to model data from different modalities separately, enabling the effective capture of data complexity and diversity, resulting in overall improved performance.

We further investigated multi-modal models with different CL strategies. When comparing MCSE-TM with ETM-multi, we observed that MCSE-TM, which integrates a contrastive multimodal objective that is absent in ETM-multi, exhibited superior performance in terms of Coherence, Diversity, and Quality metrics. The addition of a CL module facilitated the alignment of information from diverse modalities in the representation space, resulting in an overall enhancement of the topic model’s performance. Although both MCSE-TM and ConMEHR incorporate CL modules, our evaluation metrics revealed that ConMEHR outperformed MCSE-TM. This indicates that our proposed CL module, which operates at both the modality-level and topic-level, is more effective than existing CL modules. The respective roles of the modality-level and topic-level in topic modelling were further explored in our ablation study, as detailed in Sect. 4.5.

4.4 Qualitative evaluation

4.4.1 Distributions of words across different topics

In order to interpret the meaning of the learned topics, we examine the distributions of words across different topics and identify the most influential words for each topic. We set the number of topics to 20 in the MIMIC-III dataset, as discussed in Sect. 4.2, resulting in 20 different topics generated from our model. For illustrative purposes, we select T1, T2, T3 for further investigation.

Fig. 7
figure 7

The word clouds of the top words in different topics. a Topic T1; b Topic T2; c Topic T3

Full size image
Table 6 Top words of three most used topics from ConMEHR
Full size table

Figure 7 displays the top words in the topics using a word cloud (Wilson, 1912). We can observe that T1, T2, and T3 are related to heart, blood, and renal diseases, respectively. To further validate this finding, we list the topic words in Table 6 and categorize them based on the modalities they belong to. It is interesting to note that the top words within each topic are relevant. For instance, T1 is associated with two diseases, namely, ‘Atrial fibrillation’ and ‘Hypotension, unspecified’. Clinical studies have shown that heart disease is a significant factor that can cause hypotension (Gorelik et al., 2016). T2 includes ‘Sepsis’ and ‘Bacteremia’, where previous research has found that sepsis is a stage that occurs after the progression of bacteremia (Bone et al., 1997). Furthermore, we can observe that the top words from different modalities have connections. For example, the primary disease term in T3 is ‘Acute kidney failure, unspecified’, and the medication ‘Potassium Chloride’ in T3 can protect kidney function in patients with kidney disease (Saxena, 1989). These observations suggest that the learned topics are meaningful and can explain latent patient groups, comorbidities, associated symptoms, and potential medications

4.4.2 The characteristics of patients

In this subsection, we further explore the characteristics of patients to see whether their information is consistent with their associated topics. For each topic from T1, T2, T3, we select five relevant patients based on their topic distributions. Figure 8 visualizes patients’ topic distributions, where the left column indicates the corresponding admission self-reports that appeared in patients’ raw EHR. The admission self-report is the patients’ self-description of their main conditions upon admission. For example, we can see that patients in T1 usually present as ‘cardiac arrest’ and ‘atrial flutter’, which is consistent with the observation from previous experiments that T1 is associated with heart disease.

Fig. 8
figure 8

The topic distributions of the top 5 patients in selected disease topics. The heatmap on the right indicates patient mixture memberships for the topics. The columns on the left show the admission self-report for each patient

Full size image

From both quantitative and qualitative results, we conclude that ConMEHR is effective in learning the meaningful subgroup from multimodal EHRs.

4.5 Ablation study

In this section, we investigate the impact of two CL modules: the topic-level CL module, which separates topic embeddings, and the modality-level CL module, which assists in multimodal data aggregation. We evaluate three ablated versions of our model: ConMEHR\(^{-}\), which removes both CL modules; ConMEHR\(^{-}\)+ModalityCL, which removes the topic-level CL module; and ConMEHR\(^{-}\)+TopicCL, which removes the modality-level CL module. Table 7 presents the Coherence and Diversity scores of all ablated models for performance comparison. We observe that the full model achieves the best performance, while the ablated version without any CL modules has the lowest Coherence and Diversity scores. Removing either the modality-level or the topic-level CL module leads to a deterioration in performance.

In addition to quantitative evaluation, we also use visualizations to examine whether CL helps in learning separable representations of patients. We use the popular t-SNE method to project high-dimensional representations into low-dimensional vectors (Van der Maaten & Hinton, 2008). Figure 9 displays the t-SNE visualization results for the MIMIC-III dataset, where we adopt default settings. Figure 9a and b are the results from ConMEHR\(^{-}\) and ConMEHR, respectively. It is evident that ConMEHR generates more separable patient representations, demonstrating the effectiveness of CL.

Table 7 Effectiveness of CL modules. \(\downarrow \) (or \(\uparrow \)) indicates smaller (or larger) values are preferred
Full size table
Fig. 9
figure 9

The t-SNE visualization of patient embeddings of ConMEHR\(^{-}\) and ConMEHR models in the MIMIC-III datasets. a patient embeddings of ConMEHR\(^{-}\) in the MIMIC-III dataset; b patient embeddings of ConMEHR in the MIMIC-III dataset

Full size image

5 Discussion and conclusions

We develop the ConMEHR model to explain patient subgroups by extracting meaningful topics from multimodal EHRs. In ConMEHR, the modality-level and topic-level CL modules are adopted to obtain a unified latent space for multiple modalities and diversify patient subgroups, respectively. We compare the performance of ConMEHR with several topic modelling methods on two real-world EHR datasets and demonstrate that ConMEHR performs well in generating topics. The key aspects of our study are as follows.

Technical Status: To illustrate the technical status of our patient stratification approach, we highlight its differences from classification models such as Bagging CART and XGBoost-based methods (Affes & Hentati-Kaffel, 2019; Du Jardin, 2021; du Jardin, 2022) commonly used in decision support. Specifically, we emphasize two key aspects. On the one hand, patient stratification is the process of classifying patients into different groups based on their shared medical conditions. However, unlike decision tree methods such as Bagging CART and XGBoost, which primarily focus on predicting patients into pre-defined classes, our patient stratification approach goes beyond classification by explaining each subgroup based on common medical conditions among patients within the same group. This can help clinicians and researchers understand the underlying factors contributing to a specific patient’s condition and provide relevant prognoses. On the other hand, unlike our patient stratification algorithm, Bagging CART and XGBoost cannot incorporate semantic representation in their decision-making process. This means that our model considers the meaning and context of the data, as well as any relevant background knowledge, to generate more accurate and meaningful patient groupings. By incorporating semantic representation, it can also identify underlying patterns and correlations that may be missed by traditional statistical methods, leading to more precise and effective patient stratification.

Technical Contribution: There are two highlights in this research. The first one is the modality-level CL module in ConMEHR, which can learn a unified latent representation space for multiple modalities and integrate latent representations to obtain patient subgroups. The second highlight is the topic-level CL module, which guarantees diverse topic representations.

Practical Implementation: Our approach provides a powerful solution for patient stratification by enabling the identification of subgroups based on their medical conditions. The model does not require re-training unless new patients have medical conditions that are not already present in the training data. Considering the existence of a vast EHR corpus like 13,492 EHRs in MIMIC-III, it is highly likely that new patients’ medical conditions or terms will already be present within the corpus. Leveraging the large EHR corpus is key to enabling our approach to scale effectively and generate accurate patient classifications. In the uncommon event that new patients present with medical conditions that were not included in the training data, our approach offers a practical solution. Rather than starting from scratch, the existing model can be fine-tuned to incorporate the new conditions, reducing the need for a complete re-training of the model. This fine-tuning process leverages the existing knowledge and expertise captured in the model, enabling it to adapt to new scenarios more efficiently. This feature provides a highly adaptable and flexible solution for patient stratification, ensuring that the model can accommodate any new medical conditions that arise over time.

Applicable Significance: Our work can automatically detect subgroups of patients, which is an important task in the domain of precision medicine. This will have a direct impact on patients, healthcare providers, and researchers. Specifically, it would significantly accelerate the decision process and help to design treatments tailored to each subgroup. Such an impact will last since the research outcome as a new modelling approach could help achieve precision medicine by leveraging information from EHRs. Thus, it has the potential to be broadly applied in future healthcare.

Future Direction: In the future, we will also extend our model to incorporate more modalities and also medical knowledge graphs. We will develop a concrete system and provide it to clinicians. With our patient stratification system, clinicians would further accelerate their decision process. Patients will be, for example, diagnosed with diseases at early stages and assessed for disease subtypes and prospective risks. Meanwhile, we will develop models using not only EHR data but also other patients’ health-related data collected in daily life. By learning the trajectories of patients and monitoring their changes in subgroups, we can send alerts to patients and clinicians when some health states are changed. In this way, we believe our work would help to boost the development of precision medicine.