Triage nurse prediction as a covariate in a machine learning prediction algorithm for hospital admission from the emergency department

Dear Editors,

We write to respond to Hond et al.’s recent study entitled, “Machine learning for developing a prediction model of hospital admission of emergency department patients: Hype or hope?” [1]. In their study, they highlighted the benefits of reducing Emergency Department (ED) crowding, how prediction of admission can help to achieve this, and demonstrated that logistic regression and machine learning (ML) models are equally excellent predictors of hospital admission.

Hond et al. cite Afnan et al.’s meta-analysis [2] to suggest that triage nurse prediction is “not good enough to accurately predict the hospitalization of ED patients”. Whilst true that triage nurse predictions are not accurate enough on their own to allow one-to-one planning of admission, the meta-analysis showed that triage nurse predictions do significantly correlate with outcome [2]: when a nurse predicts admission, a patient’s chance of needing a hospital bed significantly increases from a baseline rate of 29–63%. Likewise, a nurse’s prediction of discharge decreases the chance of admission to 12%. Therefore, triage nurse prediction of disposition could constitute a covariate in an ML prediction model which only draws on early data (<30 min).

The model’s use of number of consultations as a component of complexity/comorbidity is based on a 2018 prospective cohort study which identified a number of factors, including number of consultations, which predicted length of stay (LOS) [3]. Even under the assumption that LOS is a reasonable proxy for comorbidity, the advent of COVID-19 since that study complicates use of “number of consultations” as a component. Many patients presenting to the ED with COVID-19 would have multiple comorbidities, but staff may wish to minimise contact with these patients and instead expedite their transfer to ITU/COVID wards. For EDs which see a high volume of COVID presentations, using “number of consultations” as a feature for model training, which may correlate positively with case complexity sometimes and inversely at others, could cause the model to overfit.

Hond et al. claim that uninterpretable ML models will outperform logistic regression when faced with complex data patterns, but this is not necessarily true, especially when the model’s features are meaningful to domain experts [4]. This applies to the ED, for example, a patient’s presenting complaint and disease severity score, which are meaningful to ED clinicians as predictors of needing hospital care, are features included in Hond et al.’s model [1]. This aligns with their results which showed no predictive benefit of their ML model over their logistic regression model. Several ethical and epistemic problems with uninterpretable ML algorithms have been highlighted in other healthcare domains [5], and many of these concerns generalise to ED admission prediction. For example, the algorithm may generalise poorly to different populations due to the presence of confounding variables in the initial evaluation (as has been the case for an ML algorithm designed to detect intracranial haemorrhage [6]); if this were the case in the context of admission prediction, patients from certain demographics may be less likely to receive a hospital bed promptly than others. For instance, patients with learning disabilities may have greater difficulty reporting pain [7], pulse oximetry is worse at detecting hypoxemia for Black people than Caucasians [8], and there may be many more confounders we would fail to consider. Furthermore, how practically would a healthcare professional convince a patient that they would need to stay in hospital based on an entirely inscrutable algorithm’s recommendation? Would healthcare professionals and bed managers themselves be inclined to trust such an algorithm? Interpretable models could mitigate these concerns because healthcare professionals can check a model’s reasoning process for plausibility, errors and confounders, which would also promote trust when using the tool.

We read Hond et al.’s study with great enthusiasm and hope they will consider creating and evaluating interpretable ML models for admission prediction, for which triage nurse prediction could be a covariate.