Health Status Prediction with Local-Global Heterogeneous Behavior Graph

Our task is to predict the health status based on personal behaviors with the multi-source sensor data collected in daily life. This task has important medical implications because it can provide early prevention of diseases and complement the clinical treatment in hospital. As we know, most existing health prediction methods can be divided into two categories: Electronic Health Record (EHR)-based methods [45, 48] and mobile sensor-data-based methods [9, 11]. The EHR-based methods are more relevant to medical studies since they use the record data collected during the hospital treatment process with professional medical equipment. However, people have little EHR data before they are detected with diseases, and thus this kind of method, however good, can only provide a piece of the puzzle. More kinds of data about the early-life experiences of patients should also be considered in treatment, which is actually what the second kind of method does. Sensor-data-based methods focus on the personal behavior in daily life and take advantage of mobile devices, which are more convenient to monitor people’s health and also provide additional useful information for clinical treatment. Below we will introduce these two categories in detail.

EHR data are collected during the hospital treatment process and contain nearly all the information of patients, such as diagnoses, medication prescriptions, and clinical notes. In early stages, expert-defined rules are adopted to identify disease based on the EHR data, such as type 2 diabetes [35] and cataracts [53]. Much work based on EHR data has been done with deep learning models, with the disease classification task most commonly. Cheng et al. [15] and Acharya et al. [4] train a CNN model to classify the normal, preictal, and seizure EEG signals. Che et al. [13] propose a multi-class classification task to predict the different stages of Parkinson’s disease with Recurrent Neural Network (RNN). Kam and Kim [34] do binary classification of sepsis by regarding the EHR data as input to a long short-term memory network (LSTM). In addition to the disease classification, the future event prediction is another task that has attracted much attention recently, which aims to predict future medical events according to the historical records. For example, Futoma et al. [20] and Rajkomar et al. [44] use the EHR data from the hospital to predict events such as mortality, readmission, length of stay, and discharge diagnoses with deep feed-forward networks and LSTMs. There has been much work on mental health prediction based on EHR data, among which T1-weighted imaging [45, 48] and functional magnetic resonance imaging (fMRI) [19, 28] are the most commonly used data to study brain structure, with other physiological signals such as electroencephalograms also playing an important role. Recently, machine learning has received more attention for its effect on improving the management of mental health. Costafreda et al. [16] do a depression classification task with SVM using the smoothed gray matter voxel-based intensity values. Rosa et al. [58] propose a sparse L1-norm SVM to predict depression with the feature of region-based functional connectivity. Cai et al. [10] collect the electroencephalogram (EEG) signals of participants and use four classification methods (SVM, KNN, DT, and ANN) to distinguish the depressed participants from normal controls.

Mobile devices provide another way for health status prediction, where diverse sensors can be used to catch various signals of people, thus making it easier to monitor daily behavior and predict health status. Machado et al. [43] calculate the signal magnitude area of the acceleration signal to recognize activities with several cluster algorithms (e.g., K-Means, Mean Shift). Koenig et al. [37] and Banhalmi et al. [7] use the camera in smartphones to monitor heart rate (HR) and heart rate variability (HRV), which are vital signs of cardiovascular health. Stafford et al. [61] and Goel et al. [27] detect the sound of breathing and cough by microphone in smartphones for assessing pulmonary health in a quick and efficient way. Besides the use of data from only one source, many researchers are devoted to taking advantage of information from various sources [31, 56, 71]. Asselbergs et al. [5] integrate accelerometer data, call history, and the short message service pattern to predict mood. Burns et al. [9] predict depression based on GPS, accelerometer, and light sensor data from smartphones. Nag et al. [49] estimate heart health status by combining sensor data from wearable devices and other factors, such as inherent genetic traits, circadian rhythm, and living environmental risks analyzed from cross-modal data, which provides better personalized health insight.

However, all of the above work cannot well explore local and global temporal characteristics of the daily behavior based on multi-source wearable sensors.

Recently, the emergence of structural data, especially structured graphs, has promoted the development of GNNs [23, 69, 72]. As the early work of GNNs, recurrent graph neural networks (RecGNNs) [21, 59] apply recurrent architectures to learn the node representation, where message passing is done constantly with nodes’ neighborhoods until the node representations are stable. Inspired by the success of Convolutional Neural Networks (CNNs), the convolution operation is also introduced to graph data in both spectural [17, 29, 36] and spatial ways [6, 24, 25]. The spectral approaches adapt the spectral graph theory to design a graph convolution. The spatial approaches inherit the message passing idea in RecGNNs but are different in getting node representations by stacking multiple convolutional layers. Besides RecGNNs and ConvGNNs, many other graph architectures have been developed to cope with different scenarios. For example, graph autoencoders (GAEs) [12, 65] are used to learn the graph embedding by reconstructing the structural information such as the adjacency matrix of graph. Spatial-temporal graph neural networks (STGNNs) [32, 42, 60] aim to model both the spatial and temporal dependency of data and learn the representation of the spatial-temporal graph, which have advantages in the related tasks, such as human action recognition.

Most of the existing GNNs focus on homogeneous graphs where nodes are in the same type and can be calculated in the same way. In comparison, heterogeneous graphs contain diverse types of nodes and edges, leading to a more complicated situation in calculation. On the one hand, different types of nodes may have different semantic meanings and different feature spaces. On the other hand, the heterogeneous graph represents both homogeneous and heterogeneous relations of data. Recently, some work has been done on heterogeneous graphs. Dong et al. [18] propose a path2vec method to learn heterogeneous graph embeddings with a meta-path-based random walk. Chen et al. [14] process different kinds of nodes with several projection matrices used to embed all the nodes into a same space and then do link prediction. Wang et al. [67] further introduce hierarchical attention to heterogeneous graphs to learn attentions for both nodes and meta-paths. Until now, the application of heterogeneous GNNs to individual behavior analysis and health status prediction is yet to be explored.

The individual behavior refers to the way that a person lives. Our purpose is to model the individual behavior in a period based on multi-modal data streams collected by wearable devices and then learn effective representations to predict the health status.

As shown in Figure 1, we take multi-source data streams as input and detect behavior-related middle-level concept sequences with pre-trained backbone models. Then, the behavior-related middle-level concept sequences are used to build the behavior graph, which consists of multiple local context sub-graphs and a global temporal sub-graph. Specifically, the concepts are regarded as different types of nodes to build local context sub-graphs. Each local context graph is regarded as a node in the global temporal sub-graph to catch temporal dependency. The representations of the local context sub-graph and global temporal sub-graph are learned by local context modeling and global temporal relation modeling, based on which the final representation of the behavior graph is learned and used to predict the health status.

We take three kinds of data streams (i.e., accelerometer, microphone, and WiFi) as input and detect behavior-related middle-level concept sequences with three pre-trained backbone models (i.e., activity, audio, location detectors), where each concept sequence and the denotes the specific concept class (e.g., walking detected by the activity detector). Meanwhile, we also obtain timestamp sequences , where and each timestamp is a 2-dimensional vector that represents the start and end time of the detected concept class in the corresponding data stream. More details of the pre-trained backbone models are introduced in Section 4.2, and the notations and their corresponding explanations are shown in Table 1.

To capture the temporal structure of the individual behavior, we need to build a behavior graph that contains both the local context information and the long-term relationships from the multimodal data stream. However, a huge densely connected network may increase the computation complexity and impact the performance. For this reason, we decompose the whole graph into two kinds of sub-graphs: local context graphs to explore the local information of individual behaviors in the short term, and a global temporal graph to capture temporal dependency in the long term. The local context graphs are regarded as nodes of the global temporal graph.

As introduced in Section 3.3.1, we have created a heterogeneous local context graph at each time step of multi-source data streams. Now we will introduce how to capture local context information of the short-term individual behavior with the heterogeneous graph neural network. The network contains m layers of node message passing modules and edge embedding learning modules. Here we only introduce these two kinds of modules for one layer. As shown in Figure 2, the node message passing module is used to learn the node embeddings and graph semantic representation, while the edge embedding learning module is used to learn the edge embeddings and graph structural representation. Then the final representation of the local context graph is obtained with the combination of the semantic and the structural representation. It is worth noting that all local context graphs share the same network parameters.

The self-attention network (SAN) is introduced in Transformer [64] for the first time, which has a sequence-to-sequence architecture and is popularly used in neural machine translation. Taking a token sequence as input, SAN calculates the attention scores between each token and other tokens with multiple attention heads. Then the token embeddings are updated with other token embeddings according to their attention scores. From this perspective, SAN can be regarded as a graph neural network with a token sequence as fully connected nodes, while the multi-head attention mechanism is a special message passing method. Inspired by this, we implement the global temporal relation modeling with the self-attention network.

Specifically, we can get a sequence of all local context graph representations as illustrated in Section 3.4. For the temporal information among different local graphs, we adopt position embeddings in [64] to encode the relative position of each local context graph, noted as . Therefore, the representation for the ith graph is the sum of and . Then the correlations between any two local context graphs can be calculated with the attention scheme:where and are learnable matrices.

The attention scores are scaled and normalized with a Softmax function, which are used to get an attended representation for each local context graph :where is a learnable projection matrix, is the dimension of , T is the number of local context graphs, and is the attended representation for the ith local context graph. Finally, we can get the structure-aware representation of the global temporal graph by adding the attended representations of all local context graphs.

The final loss function is written as the sum of a classification loss and a node variance constraint: where is a trade-off parameter.

Classification Loss: With the representation of the global temporal graph, we predict the health status label y by a fully connected layer with Softmax activation. Then we calculate the cross-entropy loss:where N is the number of instances used in the training process and is the ground-truth label of the health status.

Node Variance Loss: It is worth noting that many GNNs will face the problem of node homogenization after several epoches of node message passing since all nodes exchange information with their neighbors. In the local context graph modeling illustrated in Section 3.4, the message passing is applied not only on homogeneous nodes but also on heterogeneous nodes, which may make the representations of nodes similar. To alleviate this problem, we add a constraint to the node representations to control the variance of all nodes. Specifically, we concatenate all the node embeddings into a matrix noted as , where N is the number of nodes of all types and d is the dimension of the node embedding. Then we calculate the variance and get a vector , where each element represents the variance of the corresponding dimension in E. Finally, the node variance loss is defined as the average of the elements in the variance vector followed by a sigmoid function:

We evaluate the performance of our method on the StudentLife dataset [66], which is collected from 48 Dartmouth students over a 10-week term. It contains sensor data, Ecological Momentary Assessment (EMA) data (i.e. stress), pre- and post-survey responses (i.e., PHQ-9 depression scale), and educational data (GPA). During the 10 weeks, students carry their phones throughout the day. Data streams from multiple sensors, including accelerometer, microphone, GPS, Bluetooth, light sensor, phone charge, phone lock, and WiFi, are collected in real time by the mobile phone. Besides, students are asked to respond to various EMA questions and surveys, which are provided by psychologists to measure their mental health status. Educational performance data, such as the grades, are also collected.

In our experiment, we use data streams collected by three representative sensors (i.e., accelerometer, microphone, WiFi) as the input, since these three kinds of sensor data contain the most useful information to reflect the individual behaviors. The reason for collecting location information with WiFi instead of GPS is that most students’ activities are in an indoor environment, where the college’s WiFi AP deployment is more effective to accurately infer the location information than the GPS.

For the ground-truth annotations of the multi-source data streams, we use photographic affect meter (PAM) [55] values in EMA data. The PAM value practically represents a score between 1 and 16, which represents the Positive and Negative Affect Schedule (PANAS) [68] and reflects the instantaneous mental health status of users. The annotations are collected by a mobile application that captures users’ feelings according to users’ preference of specific photos. To keep with the conceptualization of PANAS, which ranges from low pleasure and low arousal to high pleasure and high arousal, the PAM score is further divided into four quadrants: negative valence and low arousal with score 1 to 4, negative valence and high arousal with score 5 to 8, positive valence and low arousal with score 9 to 12, and positive valence and high arousal with score 13 to 16. Following [55], we map the PAM value into the above four classes.

Finally, we use data streams of 30 students who have valid PAM annotations. Specifically, we get 912 samples in total, each sample consists of 3-day multi-modal data streams collected by the sensors of accelerometer, microphone, and WiFi. For each sample, the instantaneous PAM label of the last day is regarded as the ground-truth label of the whole data streams. The sample number for each student and the sample distribution on four classes are shown in Figure 3. Up to now, the mental health status prediction task in our experiment is practically a four-class classification problem. For the training and test, we split our datasets into 10 splits and build 10 tasks on them. Each task takes nine splits for training and the remaining one for test. We also show the average results of all 10 tasks.

Discussion on data selection. In this article, our aim is to model the personal behavior and predict the health status based on the multi-source sensor data collected from people’s daily lives. To the best of our knowledge, StudentLife is the only public dataset that satisfies our requirements. On the one hand, the data should be collected from both healthy and unhealthy people. On the other hand, the sensor data we take advantage of should be continuous and long term; i.e., the data are recorded as long as the people carry their wearable devices, such as mobile phones and smart wristbands. Some work has been done on this task. For example, [11] uses GPS data to predict depression with an SVM classifier. [9] predicts depression based on GPS, accelerometer, and light sensor data from smartphones. However, all these works do not release their data. Although some medical datasets, such as MIMIC-III [33] and ANDI [57], have been widely used to predict diseases [13] or other medical related events [20], they do not satisfy the need of our task. First, these medical datasets pay more attention to disease analysis, where the studied people are mainly patients. By contrast, our work focuses on personal health, where daily behaviors are considered for both healthy and unhealthy people. Second, these medical datasets are discretely collected from patients during the hospital treatment process with professional medical equipment, such as medical imaging, while the long-term sensor data we use is continuously collected from daily life.

As introduced in Section 3, to create the behavior graph that reflects the structure information contained in the multi-source data streams, we first need to detect the behavior-related concepts contained in the data streams. Here, three backbone models proposed in [39] are adopted to get the middle-level semantic concepts from raw sensor data. For the accelerometer, a decision tree model for activity classification is used with the features extracted from the accelerometer stream to infer the concept class (i.e., stationary, walking, running, and unknown). For the microphone, the audio data are classified into concept classes (i.e., silence, voice, noise, and other) with an HMM model. As for the WiFi, students’ WiFi scan logs are first recorded, and then the location concept classes, such as in[dana-library], are inferred according to the WiFi AP deployment information, which results in a number of 9,037 classes. The location classes are in a long-tail distribution with many classes appearing few times, and hence we choose the top 100 most frequent classes, which cover 93% of the location data.

After obtaining the three kinds of detected concepts, we cut the sequences into days and build the local context graph and the global temporal graph to predict mental health status with the method illustrated in Section 3. We use the metrics of accuracy, precision, recall, and F1-score to evaluate our model. Accuracy is the ratio of correctly predicted samples to total predicted samples. Precision, recall, and F1-score are first calculated in each class and then weighted by the sample number of each class.

Discussion on data synchronization. In existing health-related systems and methods for analyzing wearable sensor data, such as the risk situation recognition system [70], synchronization of different sensors is a very important issue. Specifically, the system always contains several devices to collect different kinds of sensor data as well as a smartphone to receive the data from sensors. An algorithm of data synchronization is necessary since the sensors are on different devices and there exist time differences between sensor data generated by sensors and received by smartphones. As for our work, the sensors we use here are all embedded in the same smartphone [66] and have a common reference time naturally and do not have the time error when sending and receiving the data, and thus the synchronization is not essential.

Since there are no previous works on PAM prediction on the same dataset, we compare our method with three popularly used conventional machine learning algorithms (i.e., RF [63], KNN [38], and SVM [8]) and two deep learning algorithms (i.e., DNN [40] and LSTM [30]). To apply these baselines on multi-source data streams collected from wearable devices, we compute the behavior feature by extracting a 108-dimensional feature vector to represent the duration time of three kinds of concept classes in 1 day. Specifically, the activity concept takes four elements of the vector, which represent the duration time of stationary, walking, running, unknown in 1 day. The audio concept takes four elements, which represent the duration time of silence, voice, noise, other in 1 day. The location concept takes 100 elements, which represent the duration time of 100 locations student staying in 1 day. The behavior feature contains the principal information of the individual life, such as where did the person go in that day and how long did he or she communicate with other people. We also compare our method with a recent GNN-based method (i.e., HAN [67]) that can capture the structural information of a graph with different kinds of nodes. Details of the compared baselines and the variants of our method are illustrated as follows.

To better illustrate the effectiveness of our model for learning representations from behavior-related data streams, we propose to apply our model on the grade prediction task. The grade annotation is the GPA, which indicates a student’s overall long-term academic performance in a range of 0 to 4.

First, we take our model, which is well trained on the health status prediction task, to extract the global representation for each student’s data stream and use KNN to do the grade prediction. Then we compare our model with a baseline, which takes the 108-dimensional feature introduced in Section 4.3 as the representation for each day, and add them to get the final feature. The same KNN is used to do the grade prediction.

We use the mean absolute errors (MAEs), the coefficient of determination (R²), and Pearson correlation as evaluation metrics for the grade prediction. We adopt the leave-one-out way to evaluate the performance. The average results are shown in Table 4.

As shown, our model outperforms the baseline on three metrics, which demonstrates that our model is effective in learning the structure-aware representation of the individual’s long-term behavior. Moreover, our model has good generalization ability and can be used to extract global behavior-related features in different tasks without model fine-tuning.