Continuous support for rehabilitation using machine learning

3.1.1 Learning settings

Standard problems are classification, where a set of predefined classes is given and one has to choose the correct one, or regression, where one predicts a continuous value. There is a vast amount of successful supervised machine learning algorithms for mentioned settings, such as Support Vector Machines (SVMs) [36], Random Forests [15] and recently (Deep-) Neural Networks (DNNs) [21], which could, in principal, be applied to the problem of recommending activities for patient rehabilitation.

The patient activity recommendation problem would then be modelled as standard (multi-class) classification, where an individual class refers to an individual activity and a good activity gets assigned label 1 (i. e. it should be chosen) and a bad activity label 0. Existing systems for pathway recommendation already successfully followed this approach by using SVMs and k-nearest-neighbours (KNN; a non-parametric supervised learning approach) [45], [44], as well as Bayesian networks (BNs) [23] to learn respective machine learning models to recommend activities within clinical pathways.

For patient A, all available feature-related information is modelled as features and all available activities as classes. It is sensible to model additional features, e. g. the average performance of patient A for individual activities in the past week, in order to find possible correlations. Assuming that labels for the feature set exist (where a physician potentially defined a mapping from preference-related information to labels for the past), a supervised classifier can be learned to recommend an activity for patient A.

While the individual models (i. e. SVM, KNN and BN) can deal with complex relationships among available patient-related features, the essential problem is that – within a clinical pathway – more than one activity might be beneficial for a patient and, additionally, they have more fine-grained utilities (i. e. continuous values such as in regression). In addition, if the activity recommended to patient A was not good, there is no natural way to provide an update to the classifier, as it is unclear which activity might be better.

A better solution would thus be to approach the activity recommendation problem as cost-sensitive (multi-class) classification [43] problem. Here, all classes are assigned a continuous value which refers to the cost of choosing the class (where, other than in standard classification, a value close to 0 would refer to a good class). The learning setting is strongly related to standard regression, more specifically one-against-all regression, reducing learning a good solution for it to learning an individual regressor for each class and choosing the output of the one with the lowest cost. As a consequence, all mentioned supervised machine learning algorithms can be directly used for learning the individual regression functions.

A cost-sensitive classifier for patient A can be trained with the same labelled corpus of data as a standard classifier. However, when a recommended activity turned out to be bad (or if two activities turn out to be good over time), the model can be directly updated.

While cost-sensitive classification is better suited to model utilities of activities compared to standard classifiers, the individual recommendations remain independent. However, choosing a particular activity might have severe impact on the utilities of subsequent activities. Such dependencies can be modelled within decision processes, which we cover in Sec. 3.2.

3.1.2 Learning representations

Independent of the learning setting one usually assumes a vectorised representation as input. Here, one (a) chooses all feature-related relevant information (in cooperation with physicians) for an individual set of alternative activity recommendation (e. g. proposing patient A to go for a 1 km run at an average speed of 9 km/h vs. to go for a 5 km walk at an average speed of 4 km/h), (b) discusses the domains of the chosen pieces of information to model them as features and (c) stipulates how the label is to be derived (e. g. solely based on the expertise of the expert) and scored (e. g. a 1 or 0 vs. continuous values for costs).

Step (a) is very important, as it requires a common understanding of the problem. Step (b) then entails to correctly format the data, i. e. distinguishing among categorical (e. g. smoker vs. non-smoker) and numerical features (e. g. heart rate) and correctly transforming them into features for the vector. Step (c) is also central for the eventual quality of the activity recommendation solution, as it has to carefully examined if the expert knowledge of the physician can sufficiently generalise to all patients. To this end, there will be activities which have to be actually performed and then labelled, which renders a problem for standard supervised machine learning.

There are approaches which directly deal with other representations, such as graphs. Here, one either (a) models dependencies among prior mentioned features (within the vector; resulting in a graph of individual features) as input to a probabilistic graphical model (PGM) [35], [16], (b) conducts random walks within the graph [13] to derive node and edge statistics and uses these as input to standard supervised machine learning or (c) chooses a subset of relevant types of nodes and relationships as input to, for example, a lifted PGM [18].

To this end, it is not clear from standard supervised machine learning (independent of the presented learning settings) how appropriate labels for cost-sensitive multi-class classification should be provided, as one can only recommend a single activity to a patient at a distinct point in time. While there are different practical methods to derive the label for the activity (e. g. by modelling expert rules based on a set of (clinical-) patient information or by asking the patient to give feedback), only feedback for a single activity is provided. The resulting problem is referred to as partial feedback which is now introduced.

3.2.1 Learning protocol

RL assumes an underlying decision process, which is referred to as Markov Decision Process (MDP) [30]. In a MDP, the world as modelled as states (which consist of all relevant information a decision needs to be taken; e. g. all patient data, the complete history of medical treatment and activities of a patient, and all sensor data) and actions (e. g. possible activities or dialogues with the patient) is assumed. A single action in a state can be chosen, after a novel state as well as a reward are observed. The latter is a numerical value which quantifies how good the action was in that state (i. e. it is the same as the reverse cost in cost-sensitive classification). The reward, however, is only of local utility, i. e. the action might be very good for a short time frame but could have detrimental effects on the future (e. g. eating a lot of ice cream when having the felt positive emotion as reward). The process is repeated over a number of episodes, which depict individual trajectories of the whole process (e. g. a day or week of a patient in rehabilitation, where states refer to individual times during the day). The challenge tackled by RL algorithms is now to actively choose which action to take in which state to maximise the so-called expected cumulative reward, i. e. the aggregated and averaged reward estimate over a number of episodes. The resulting function (i. e. the classifier for classification or regressor for regression) is referred to as policy, which recommends an action based on a state.

For a patient in rehabilitation, individual states refer to all relevant information for her/his well-being which might be useful for activity recommendation (e. g. the information for patient A plus information about activities chosen before on the same day). The action space of the MDP then refer to all possible activities with possible configurations, such as going for a 2 km walk. After an action was chosen and the respective activity has been recommended to the patient, a numerical reward value can be inferred based on diverse criteria, e. g. the Borg assessment of patient A. The learning task is then to strategically find out which activities yield large cumulative reward scores for individual days, by alternating between exploring relatively unknown activities and exploiting relatively known, good activities.

3.2.2 Learning approaches

In recent years, the problem of RL became increasingly popular, which led to a drastic increase in published approaches with high theoretical and empirical quality. The central application has been robotics, but the increase in popularity has to be attributed to learning to play (computer-) games with superhuman abilities – prominent examples are the game of Go and the general OpenAI^[1] platform which supports a variety of video games.

RL approaches are categorised into model-based [10], value-based [27] and policy search [37]. In model-based RL, learning the reward function to subsequently employ planning algorithms to get an estimate of the policy is aimed for. Value-based RL approaches aim to quantify the utility of an action in a state, which is less generalisable than learning the reward itself, but often converges faster to a solution. Finally, policy search approaches directly learn the policy. The latter are often employed in robotics, as the space of available actions (and resulting states) is usually extremely large, and such approaches can find fast solutions for approximate policies.

However, when using RL approaches for activity recommendation one central problem remains: a RL agent assumes to be able to randomly act in the beginning in order to gradually improve its behaviour, which is not possible in computer-assisted patient recommendation, as the patient’s well-being is the highest priority. We must not, for example, risk the health of patient A by recommending him/her to go for two runs on the same day.

As a consequence, methods to bring the RL agent to a sufficient level of expertise are needed until it takes over the activity recommendation decision process to personalise its behaviour for a particular patient. Such techniques fall under the category of Imitation Learning [34] when dealing with decision processes. In a MDP, it is required to have a set of so-called expert demonstrations, which consist of optimal sequential action selections for numerous states. More specifically, physicians need to manually annotate numerous activity recommendations for different patients, which have to be work well with respect to the eventual well-being of the patient. For patient A, a demonstration would comprise the current state (i. e. all mentioned information which are recorded) combined with the activity chosen by a physician.

However, such large sets of recommendations are costly and cumbersome to generate, as complete trajectories of the decision process have to be modelled, i. e. each consequence of a recommended activity for the duration of the process (e. g. the goal of 5000 steps within a day). One could exploit general knowledge of physicians by modelling generally applicable rules for individual activity recommendations [22]. Whenever a rule then applies to a patient’s current situation, one can choose among a predefined list of activities with known rewards. An agent can then use such rules to adapt its behaviour for a general patient, which can be adapted by standard RL for individual patients. Such a general rule for applicable to patient A could comprise a subset of the feature-related information we have, e. g. only physiological information, of a specific time during the day combined with the activity walk for 5 km. The rule is then applicable to other patients with the same physiological information as well, irrespective of their demographic information.

For computer-assisted patient rehabilitation, two aspects with respect to RL are distinguished. The first aspect simply deals with the fact that accurate claims about the possible performance of RL algorithms on real data without empirically evaluating them are not possible. Thus, while it makes sense to reuse State-of-the-Art RL approaches of similar problem settings, the resulting system sufficiently flexible has to be kept to possibly test different ones.

The second aspect is more important, as it deals with the general challenge of medical assistance systems and automated decision-making. The RL algorithms are adequate for the problem of learning to recommend activities, assuming that they can initially act randomly and have no constraints on choosing activities. First, it has to be ensured that all possible activities the system is allowed to choose are either completely safe (i. e. not critical in the least amount) or that the eventual decision to choose the recommended activity is taken by a medical expert (i. e. he or she is in full control). The set of safe activities for patient A after having been running would thus only comprise walking.

In summary, three parallel approaches for using RL for patient rehabilitation are required: (a) Defining, in close work with medical experts, which actions are allowed in which state, which actions need confirmation and which actions are to be ruled out completely. This serves as initial training for the system and can be achieved by modelling resulting constraints based on structured models. (b) Initialising the system using imitation learning approaches based on expert rule. These rules – similar to mentioned constraints – encapsulate what a physician would generally recommend to a patient with available profile and what available outcomes of a recommendation are good or bad. The resulting rules can then be used to simulate an appropriate learning environment for an agent to warm-start, where artificial reward values are generated in order to reflect general (i. e. non-personalised) feedback. (c) Exploiting each piece of feedback (i. e. each chosen action with reward), which could be enabled by an interactive machine learning process [38]. The latter requires a domain expert to actively train the system with respect to giving it a reward estimate as well as information about useful and not useful features. While reward feedback is standard in both supervised machine learning as well as RL, feature feedback [31] is powerful. It was shown that feature feedback is able to considerably improve the performance and decrease the time until convergence of machine learning and RL models [20].

3.3.1 Clustering algorithms

In the beginning of using a machine learning-based system, one often does not have sufficient amount of labelled data. This is also the case for computer-assisted patient rehabilitation before actual patients are using the system in order to provide feedback for the recommended actions. However, even at this stage important insights into the data can be gained, which can have tremendous impact on the supervised machine learning algorithms for activity recommendation. The practical (i. e. empirical) performance of a machine learning algorithm is often dependent on multiple design choices, which include available parameters (e. g. learning rate for individual updates) but also how specialised the machine learning models should be. This directly applies to RL algorithms (and thus to the activity recommendation problem), as states could be partitioned into individual clusters and then learn individual RL policies for each of these. A cluster, here, could be minimised to individual patients, where the result would be extremely specialised (i. e. personalised) policies, which, however, cannot be reused that easily for novel patients (but might perform better for the addressed patient).

Such design decisions can be supported via clustering algorithms, which partition the data into different sets (disjoint or overlapping based on the given approach) based on a chosen similarity metric. For computer-assisted patient rehabilitation, such a metric would either compare selected information about patients or, in later stages when activities have been chosen for actual patients, compare states of the modeled decision process of the pathway.

For the running example, this entails to use the feature-related information of patient A as well as of other patients, and to then apply a clustering algorithm on the resulting dataset. The same approach can be applied to states modeled for patient A and the residual patients, which would additionally represent a distribution of chosen activities (as the latter are part of the state features).

Two prominent classes of clustering algorithms are link- and density-based. A well-performing link-based algorithm is K-Means [12], [24], which takes a parameter to set the number of eventual cluster and iteratively re-assigns both data points to cluster (based on their distance to the individual centroids of the clusters) as well as the centroids of the cluster themselves. k-Means converges when no data point changes its assigned cluster (and thus getting disjoint clusters). For computer-assisted patient rehabilitation, it is firstly needed to define similarity metrics on the selected information about a patient (e. g. Cosine- or Euclidean similarity which are usually employed to compare vectors and/or their elements) and heuristically set the number of clusters one estimates useful.

In comparison, density-based clustering algorithms do not focus on links, but on dense areas depicting a cluster. DBScan [11], for example, takes as input the minimal amount of data points to form a cluster as well as the minimal similarity between pairs of data points such that a data point can be considered a core point, i. e. similar to a centroid in k-Means. The resulting algorithm creates possibly overlapping clusters, which have dense areas at their core. For computer-assisted patient rehabilitation, again similarity metrics have to be defined, but do not have to constrain the number of clusters. This is useful for initial exploration phases to get an early estimate of how the data is distributed. In addition, the clusters might have arbitrary shapes (other than in k-Means), which could be a better fit (depending on the patient distribution based on the chosen information).

As a consequence, both classes of clustering algorithms are useful to employ. After a number of different parameterised clustering have been generated, one has to examine them with medical experts to assess their meaningfulness. The eventual added value has to be determined in the application of recommending activities, which could yield better results when learning supervised machine learning models for chosen clusters.

3.3.2 Embeddings

An embedding for a word, image or element of a graph (e. g. a node) is a vector which is able to generalise the meaning of the respective modality. Taking a single word as an example, one could compare it with another word based on its characters (e. g. apple vs. orange) which would yield zero similarity. However, both words represent fruits which makes them similar to some extent. This could be captured by well-trained embeddings, which are learned on huge amounts of data (e. g. large corpora of text or images, or large knowledge graphs) for finding out in what contexts the respective modality instances occur. Resulting embeddings can also be directly used as representation for data points in mentioned clustering algorithms, as pairwise comparisons of vector embeddings can be conducted with the Cosine similarity.

There are numerous approaches to learn embeddings for text [1], [26] and images [40], usable throughout computer-assisted patient rehabilitation if textual medical reports or scans are incorporated. However, since using an ontology to structure available patient-related data is advantageous (see Sec. 3.4), another important modality are graphs.

For graphs, one class of approaches can be depicted as translational [3], [17], which generally assume that summing up subject vectors and predicate vectors yields object vectors (which in turn is used as optimisation criterion). Resulting approaches are time and space efficient, and yield good performance (in terms of generalisation when applied to actual tasks). Another class of approaches deals with random walks [14], [42] (analogously to mentioned learning representations), where the resulting walks are used as context to describe nodes and edges. There are numerous other graph embedding approaches, which, for example, are based on tensor factorisation [29] (i. e. relational learning).

In order to learn an embedding for patient A, the available feature-related information (or the available states) of all available patients need to be modeled as structured representation in the form of a knowledge graph (see Sec. 3.4). After an appropriate graph embedding technique has been applied to the resulting graph, the embedding for patient A can be retrieved and used for a supervised- (or reinforcement-) learning task.

As common in machine learning applications, it cannot be clear which embedding approach works best, as they have to be empirically evaluated for the recommendation task.

3.3.3 Explainabilty

Lastly, unsupervised learning can also be useful to explain the actual reasoning process of a machine learning algorithm (such as a RL algorithm). Here, one tries to uncover the actual subset of features or their simplified relationships in order to present the end-user (i. e. the patient or the physician in charge) more than the plain prediction (i. e. the recommended activity).

Explanation components are either completely unsupervised [6] by returning the most important features by repeatedly permuting individual features to uncover correlations or partially supervised [32], [33] by learning a simpler model based on available or previously gathered training data. Here, either a list of features is generated [6] or more advanced relationships among individual features are tried to be disclosed [32], [33].

For computer-assisted patient rehabilitation, it is important to personalise the explainability of the activity recommendation process. More specifically, it is important to assess if the end-user is receptive to explanations, how complex the explanations should be (in terms of the displayed feature relationships) and what kind of visualisation is preferred.

More specifically, the system could either display patient A a single list of which individual features have been used to recommend an activity (e. g. run 1 km because of features heart rate, weight and age) or a list of features with positive as well as negative influence on the decision respectively (e. g. run 1 km because of positive features heart rate and age, and negative feature weight).

3.4.1 Resource description framework & ontologies

RDF^[2] is a graph-based data model which represents information as a directed multigraph with nodes and edges with a single label, and is the central data model of the Semantic Web [2].

Nodes consist of IRIs (representing abstract things), literals (i. e. concrete data values) and blank nodes (i. e. dummy convenience nodes). A RDF graph is expressed as a set of (subject, predicate, object) triples, each interpreted as an edge labelled with predicate going from the subject node to the object node. RDF does not support any semantics on its own, other than those carried over from the XML datatype definition. Resulting RDF data can be retrieved using the SPARQL query language,^[3] a query language, protocol and API for requesting information from RDF graphs.

By modeling available knowledge as RDF graph one is able to interlink heterogeneous information, such as sensor data or patient-related information. To this end, it is sensible and recommended to reuse available ontologies, which model relevant concepts and properties. Prominent examples for relevant ontologies are IoT light,^[4] universAAL,^[5] WBAN (Wireless Body Area Network) [28], SmartBAN^[6] or SSN (Sensor Network Ontology).^[7]

Compared to other graph-based models, such as Labelled Property Graphs (LPGs)^[8][9] or Grakn.ai,^[10] RDF combined with advanced Semantic Web technologies (e. g. for querying structured data) consists of well-document, established technologies which offer sufficient expressivity paired with the possibility to extend and interlink the developed schema.

3.4.2 Learning with knowledge modelling

There are two general ways to exploit RDF-based data for machine learning. The first option is referred to as vocabulary-based semantics. Because you can identify relationships with URIs, you can define agreed-upon vocabularies such as schema.org.^[11] The second option is inference-based semantics. It is used to express what the data inherently means, to infer facts, to conduct reasoning (e. g. in OWL^[12]) or to check the consistency of the graph (e. g. in SHACL (Shapes Constraint Language)).^[13]

A combination of both approaches is thus strongly encouraged. Vocabulary-based semantics provide a starting point by enabling to integrate all information to make it queryable for appropriate machine learning technologies. Inference-based semantics are useful when the problem to be tackled by machine learning technologies is challenging (such as applying RL), especially because data about a specific patient has to be gathered over the course of rehabilitation, where no information about the utility of an activity for the patient is available in the beginning.

For the example patient, patient A, all available feature-related information need to be modeled in RDF by reusing available concepts and properties of mentioned ontologies. This enables to ensure that, for example, when using the weight of a patient as feature, it is known how to interpret the available value (e. g. when different units are used), as otherwise machine learning approaches might fail. In addition, the resulting knowledge graph enables to learn graph embeddings, as described in Sec. 3.3.2.

As the patient’s well-being is the utmost priority, it is not directly possible to employ machine learning technologies (as their recommendations are close to random behaviour). Therefore, modelling activity recommendation rules in close collaboration with medical experts would be possible, which define the initial behaviour of an agent (see Sec. 3.2.2). These rules can be defined based using inference-based semantics, where the extended state the rehabilitation (consisting of clinical state and personal state) is used to find the most appropriate rule. Such rules, which usually consist of implications only, can be defined by Notation3 (N3),^[14] an RDF-based vocabulary with appropriate classes and properties for rules, or SHACL, a constraint language for RDF.

SHACL enables to define so-called shapes on top of existing RDF graphs. For example, a patient shape can be defined and list of properties that a patient needs to have can be specified, which is similar to a database schema definition. Each property can be further constrained by a regular expression, cardinality or other more advanced means. Moreover, it is possible to define IF-THEN rules that are applied on specific parts of the graph.

For the running example, a SHACL rule expressing that certain physiological information (e. g. blood pressure smaller than 110) yield a recommendation of activity run 1 km could be modelled and then used to learn a baseline policy for patient A.

Using inference-based semantics in the form of SHACL shapes and rules is thus a powerful means to establish system safety and -trust, as one is able to control the degree the system is allowed to autonomously decide.