Abstract
Objective
To explore the relationship between novel, time-varying predictors for healthcare delivery strain (eg, counts of patient orders per hour) and imminent discharge and in-hospital mortality.
Materials and Methods
We conducted a retrospective cohort study using data from adults hospitalized at 21 Kaiser Permanente Northern California hospitals between November 1, 2015 and October 31, 2020 and the nurses caring for them. Patient data extracted included demographics, diagnoses, severity measures, occupancy metrics, and process of care metrics (eg, counts of intravenous drip orders per hour). We linked these data to individual registered nurse records and created multiple dynamic, time-varying predictors (eg, mean acute severity of illness for all patients cared for by a nurse during a given hour). All analyses were stratified by patients’ initial hospital unit (ward, stepdown unit, or intensive care unit). We used discrete-time hazard regression to assess the association between each novel time-varying predictor and the outcomes of discharge and mortality, separately.
Results
Our dataset consisted of 84 162 161 hourly records from 954 477 hospitalizations. Many novel time-varying predictors had strong associations with the 2 study outcomes. However, most of the predictors did not merely track patients’ severity of illness; instead, many of them only had weak correlations with severity, often with complex relationships over time.
Discussion
Increasing availability of process of care data from automated electronic health records will permit better quantification of healthcare delivery strain. This could result in enhanced prediction of adverse outcomes and service delays.
Conclusion
New conceptual models will be needed to use these new data elements.
Keywords: time-varying predictors, hospital occupancy, patient-nurse interactions, hospital strain, healthcare delivery strain
INTRODUCTION
Hospital strain is a construct that attempts to capture the effect of dynamic unit- and hospital-level factors that may impact patient care. Prior research has demonstrated that factors such as unit census, admissions, and average patient acuity (“capacity strain”) are associated with intensive care unit (ICU) mortality,1,2 ICU admission,3,4 ICU length of stay (LOS), and post-ICU discharge outcomes5; timing of life sustaining therapies and death6; and rehospitalization.7 Further, these associations persist after controlling for individual patient characteristics.
Although some studies have quantified the effects over 1 calendar day (eg, the day of ICU admission or discharge), Gabler et al1 and Wagner et al5 demonstrated that averaging strain across a period of 3 calendar days showed associations that were not present when only data from a single calendar day were used. Dynamic strain metrics that incorporate static and time-varying information are now feasible with the real-time availability of data within automated electronic health records (EHRs). Dynamic quantification of strain could improve research and add clinical value by enhancing clinicians’ situational awareness, predicting the occurrence of adverse outcomes or service delays, and/or forecasting likely LOS at the individual patient level or census at the hospital or unit level. Recent work by Rossetti et al8 demonstrated that models that include time-varying nursing note metadata (eg, number of heart rate measurements over the past 12 h) were predictive of a composite outcome defined by first occurrence of in-hospital mortality, cardiac arrest, unanticipated transfer to the ICU, rapid response, and sepsis.
In this report, we expand on prior literature that has focused on occupancy- and acuity-related strain variables. We examine an expanded set of healthcare delivery strain variables, now available from automated EHRs, that capture patient–nurse interactions. Treating them as time-varying exposures, we explore their association with hospital LOS (time to discharge) and inpatient mortality. We also assess the degree of independence of this association with respect to patients’ longitudinal comorbidity burden and acute physiology.
MATERIALS AND METHODS
Setting
The setting for our investigation consists of 21 hospitals owned by Kaiser Permanente Northern California (KPNC), an integrated healthcare delivery system. Under a mutual exclusivity agreement, 9500 salaried physicians of The Permanente Medical Group provide care for 4.3 million members of the Kaiser Foundation Health Plan (KFHP) at facilities owned by Kaiser Foundation Hospitals. Deployment of the Epic EHR in all KPNC hospitals and clinics was completed in 2010.
This project was approved by the KPNC Institutional Review Board for the Protection of Human Subjects, which has jurisdiction over the 21 study hospitals and waived the requirement for individual informed consent.
Study cohort and unit of analysis
Using previously described methods,9–12 we identified KPNC hospitalizations meeting the following criteria: the patient was ≥18 years of age; hospitalization type was inpatient or observation; and the admission date was from November 1, 2015 to October 31, 2020. We did not include day surgeries, admissions to the labor and delivery service, or hospitalizations where adults were placed outside the general medical-surgical ward (ward), stepdown unit (SDU, also known as transitional care unit), ICU, operating room (OR), or postanesthesia recovery unit (PAR). The unit of analysis was an individual hospitalization (ie, unlike our previous work, we did not concatenate hospitalizations that involved inter-hospital transport).
Unit and nurse data linkage
We linked individual registered nurse (RN) records from the KPNC nursing staffing database to individual patient records on an hourly basis. The nursing staffing database captures when an RN was on duty but does not indicate the exact unit. We linked individual RN identification numbers (IDs) to individual hourly patient records. Our algorithm employed combinations of RN hours on duty with the presence of a nurse’s ID on orders, medication administration records (MAR) transactions, shift assessments (instances where a nurse obtained vital signs from a patient), and (in those cases where no data provided a direct RN-patient linkage) probabilistic rules. Additional details on the linkage strategy are provided in the Supplementary Appendix.
Dependent variables
The dependent variables (outcomes) for our analyses were time to discharge and inpatient mortality. These outcomes were chosen because they can be easily measured and are important both clinically and operationally. To account for differential follow-up time and possible censoring, we partitioned the time frame into hourly intervals anchored at hospital admission. For each outcome and for each patient, we included all patient-hours until the outcome occurred or the follow-up ended; we defined a corresponding outcome status variable taking on value 1 if the outcome occurred in the patient-hour and 0 otherwise. We modeled time to discharge as the likelihood that, at any given hour, the patient would be discharged in the next hour, that is, the hazard of discharge. For inpatient mortality, we modeled the likelihood that, at any given hour, the patient would die during the hospital stay, that is, the hazard of death.
Previously described independent variables
We included the following predictors10–13 in our models: age, sex, admission venue (via the emergency department or not, from a skilled nursing facility or not), admission diagnosis, whether the admission was for observation or not according to the 2 midnight rule,11 acute severity of illness, longitudinal comorbidity burden, and admission care directive. As described previously, we grouped International Classification of Diseases codes into Primary Conditions. We assigned all patients a severity of illness score (the Laboratory-based Acute Physiology Score, version 2, LAPS2) based on laboratory data, vital signs, and neurological status at the time of admission (rooming in, with physiologic data obtained from the preceding 72 h) and then every hour (with physiologic data obtained from the preceding 24 h).10,12 We also assigned patients a longitudinal comorbidity score (the COmorbidity Point Score, version 2, COPS2) based on all patient diagnoses accrued in the 12 months preceding hospitalization, and, for comparison, a Charlson Comorbidity Index score (CCI).14,15 We captured patients’ hourly location (ward, SDU, ICU, OR, and PAR). Last, we extracted patients’ care directive for each hour, categorized as full code/not full code.
New time-varying EHR candidate strain variables
Table 1 details the new predictors we evaluated. All new strain variables were time-varying and patient-specific; at each hour, we constructed each variable utilizing information at or before that hour. We grouped patient orders for a given patient during a given hour into 6 types: administrative, blood transfusion, laboratory testing, dietary, imaging, and triggered communication (eg, an order to notify a physician in the event of X). In the Epic EHR, medication administration is tracked in the MAR module. We grouped MAR transactions into 6 types based on the medication administration route: intramuscular, inhalation, simple intravenous, intravenous requiring monitoring (eg, an insulin or dobutamine drip), miscellaneous, and oral. We also created 2 additional strain variables based on MAR transactions: the sum of all types of MAR transactions since admission and the sum of all types of MAR transactions in the last 4 hours. We also calculated 13 hourly unit-level variables (eg, % occupancy, mean LAPS2 of patients in the unit, patient deterioration defined as an increase of 40 LAPS2 points within a 1-h period). Last, we calculated 4 nurse hourly work intensity variables (eg, for a patient X cared for by a nurse who also cared for patients W, Y, and Z, the maximum LAPS2 of all 4 patients cared for by that nurse).
Table 1.
Candidate strain indicators evaluated in study
| Variable | Interval |
|---|---|
| Count of administrative ordersa | Hourly |
| Count of blood transfusion orders | Hourly |
| Count of laboratory orders | Hourly |
| Count of diet orders | Hourly |
| Count of imaging orders | Hourly |
| Count of nursing orders | Hourly |
| Count of intramuscular medicationsb | Hourly |
| Count of aerosol or inhalation medications | Hourly |
| Count of medications using simple intravenous administration | Hourly |
| Count of medications using intravenous drips | Hourly |
| Count of medications using other all other modalities | Hourly |
| Count of orally administered medications | Hourly |
| Sum of all orders and MAR transactions since admission | From admission |
| Sum of all orders and MAR transactions in last 4 h | Last 4 h |
| Percent occupancy for unit where patient wasc | Hourly |
| Number of patients in unit where patient was | Hourly |
| Number of new patients admitted to unit where patient was | Hourly |
| Number of discharges from unit where patient was | Hourly |
| Maximum percent occupancy in the last 4 h for unit where patient was | Last 4 h |
| Maximum range in percent occupancy in the last 4 h where patient was | Last 4 h |
| Mean age of patients in unit | Hourly |
| Mean LAPS2 of patients in unitd | Hourly |
| Mean COPS2 of patients in unite | Hourly |
| Maximum LAPS2 of patients in unit where patient was, excluding patient | Hourly |
| Number of patients with LAPS2 ≥ 110 in unit where patient was | Hourly |
| Number of deaths in unit where patient was | Last 4 h |
| Number of physiologic deteriorations in unit where patient wasf | Last 4 h |
| For this patient’s nurse, the maximum LAPS2 of patients under her/his care | Last 4 h |
| For this patient’s nurse, the % of patients under her/his care with a LAPS2 ≥ 110 | Last 4 h |
| For all nurses who cared for this patient, the maximum LAPS2 | Last 4 h |
| Count of other patients cared for by this patient’s nurse | Last 4 h |
Administrative orders are those involving the admission, discharge, and transfer process. Bloodwork orders are those involving actual blood draws from a patient. Laboratory orders include all other laboratory tests involving the patient. Imaging orders include ordinary roentgenograms, magnetic resonance imaging, computed tomography, and ultrasound. Nursing orders include those involving discrete physical actions by a nurse (eg, obtaining vital signs) as well as ad hoc physician orders (eg, “notify MD when patient’s spouse arrives on ward”).
The electronic health record uses a module known as the MAR to track administration of medications. Simple intravenous administration refers to medications whose administration does not require continuous monitoring (eg, most antibiotics); intravenous drips are those that require continuous monitoring (eg, insulin, vasopressors).
We calculated occupancy by first identifying the highest number of patients who stayed at a given unit at a given hospital during the study period; this was set to equal 100%. We then divided the actual number of patients in a given unit by this number.
The LAPS2 is assigned on the basis of a patient’s worst vital signs, pulse oximetry, neurologic status, and 16 laboratory test results in the preceding 72 hours. The univariate relationship of an admission LAPS2 with 30-day mortality is as follows: 0–59, 1.0%; 60–109, 5.0%; ≥110, 13.7%. After age, sex, admission diagnosis, and comorbid conditions are controlled for, the adjusted odds ratio for inpatient mortality for an increase in LAPS2 of 5 points is 1.134 (95% CI, 1.133–1.135). See Escobar et al10 for details.
The COPS2 is based on patient diagnoses accrued in the preceding 12 months and is assigned every month to all adults with a KPNC medical record number and has a range from 0 to 1010 (higher scores indicate worse mortality risk). The univariate relationship between the COPS2 and 1-year mortality is as follows: 0 to 39, 0.3%; 40–64, 5.3%; ≥ 65, 17.2%. See Escobar et al10 for details.
Physiologic deterioration is defined as an increase of 40 points in a patients’ LAPS2 score from 1 hour to the next.
Abbreviations: MAR: Medication Administration Record; LAPS2: Laboratory-based Acute Physiology Score, version 2; COPS2: COmorbidity Point Score, version 2; KPNC: Kaiser Permanente Northern California.
These variables were chosen based on clinician judgment and motivated by 2 potential drivers of adverse hospital outcomes. The first is the relationship between staffing and workload, which is driven by the total number of patients.1,5,6,16,17 This was captured through unit-level occupancy-related variables and the complexity of care of patients in the unit (eg, mean acuity score of patients in the unit, number of deaths in the unit where the patient was). The second is cognitive load, which in this context refers to the mental activity required to care for a patient. We used orders and MAR transactions as proxies for cognitive load.18–24 Since we were exploring the impact of these new variables on outcomes, we used measurements in each hour and explored summaries of these measurements over windows of time based on clinician judgment.
Statistical methods
All analyses were conducted in R version 4.0.2 and SAS version 9.04. We summarized patient-level demographic and baseline characteristics using counts and percentages with either means and standard deviations or quartiles. Each row in the analytic dataset corresponded to 1 patient-hour. We summarized hourly continuous exposures numerically using medians, means, and standard deviations. To compare variables across unit type, we used analysis of variance for continuous variables and chi-squared tests for proportions. We quantified linear associations between each of the novel hourly strain variables and hourly LAPS2 using Pearson correlations.
Because nursing staffing ratios are different for different units and because future detection systems might be unit-specific, we stratified all analyses by initial hospital unit. To explore bivariate associations between each novel time-varying strain variable and the outcome, we used discrete time (hazard) survival models with a complementary log-log link25; effect estimates are interpreted as risk (hazard) ratios. We included dummy variables for each day to allow the baseline risk (hazard) to vary by day. We adjusted for the following base set of potential confounders and covariates: age, sex, longitudinal comorbidity burden (COPS2), admission diagnosis (Primary Condition), admission venue (from a skilled nursing facility or not, admitted via the emergency department or not), whether or not the patient was transported in from another hospital, hospital, unit at each hour, care directive in effect at admission (full code or not), admission severity of illness (LAPS2), and hourly LAPS2. We presented point estimates and corresponding Wald-based 95% CIs. For analytic purposes, we censored records at 60 days (99.9th percentile of LOS) following hospital admission for time to discharge models.
RESULTS
We initially retrieved 1 004 558 adult hospitalizations with a total of 87 846 502 patient hourly records. We then removed 20 934 hospitalizations because of data quality issues (3 384 902 hourly records; details in Supplementary Appendix) and 29 147 hospitalizations (299 694 hourly records) where the patient was discharged home directly from a surgical area. The median LOS was 57.0 hours (lower and upper quartiles: 32.4 and 99.9 h). The characteristics of the 954 477 hospitalizations (84 162 161 hourly records) included in our cohort are summarized in Table 2 and a consort diagram is provided in the Supplementary Appendix. Patients whose first unit was the SDU had the highest longitudinal comorbidity burden (as measured by the COPS2), but patients admitted to the ICU were more acutely ill on admission (as measured by the LAPS2). Raw inpatient mortality was highest among patients initially admitted to the ICU (9.7%, range across hospitals, 5.3–12.2%), followed by the SDU (3.0%, 0.5–9.6%), and ward (1.9%, 1.4–2.3%). We examined the distribution of the dependent variables continuously as time to discharge and time to death. Discharge times were concentrated during daylight hours among survivors but were evenly distributed across the day among decedents (Supplementary Appendix S3).
Table 2.
Characteristics of hospitalizations included in study cohorta
| First unitb |
|||
|---|---|---|---|
| Ward | SDU | ICU | |
| Number of patients | 444 991 | 30 451 | 90 390 |
| Number of hospitalizations (% of total hospitalizations) | 804 670 (84.3) | 38 652 (4.0) | 111 155 (11.6) |
| Age (median, mean ± SD) | 68.0 (65.9 ± 17.7) | 70.0 (68.0 ± 16.8) | 66.0 (64.2 ± 17.3) |
| Sex (% male) | 48.2 | 52.9 | 55.0 |
| Inpatient (%)c | 72.7 | 80.0 | 95.9 |
| Observation (%) | 27.3 | 20.0 | 4.1 |
| Admission via emergency department (%) | 78.9 | 77.1 | 74.0 |
| Admission from skilled nursing facility (%) | 2.2 | 2.4 | 2.6 |
| Surgical patient (%)d | 16.9 | 13.2 | 18.3 |
| Transport ine | 7.4 | 7.8 | 14.9 |
| CCI score (median, mean ± SD)f | 3.0 (3.5 ± 3.1) | 3.0 (3.7 ± 3.1) | 3.0 (3.5 ± 3.1) |
| CCI score ≥ 4 (%) | 43.0 | 46.6 | 43.2 |
| COPS2 (median, mean ± SD)g | 31.0 (45.4 ± 41.2) | 39.0 (51.2 ± 41.5) | 37.0 (50.6 ± 42.8) |
| COPS2 ≥ 65 (%) | 27.0 | 31.4 | 30.5 |
| Admission LAPS2 (median, mean ± SD)h | 56.0 (59.2 ± 37.2) | 66.0 (68.0 ± 42.6) | 77.0 (82.4 ± 55.9) |
| Admission LAPS2 ≥ 110 (%) | 10.3 | 17.9 | 31.9 |
| Full code on admission (%)i | 85.3 | 83.5 | 90.6 |
| Admission diagnosis (% with)j | |||
| Sepsis | 7.4 | 6.2 | 11.3 |
| Community-acquired pneumonia | 1.9 | 1.7 | 0.9 |
| AMI | 1.7 | 3.1 | 4.3 |
| Congestive heart failure | 2.3 | 3.7 | 1.1 |
| Gastrointestinal bleeding | 3.1 | 2.2 | 3.0 |
| All other | 83.6 | 83.1 | 79.4 |
| Outcomes | |||
| LOS (median, mean ± SD) | 2.2 (3.4 ± 4.3) | 2.4 (3.6 ± 4.9) | 3.7 (5.8 ± 7.7) |
| LOS ≥ 4 days (% with) | 23.8 | 26.6 | 46.2 |
| Inpatient death (%) | 1.9 | 3.0 | 9.7 |
The unit of analysis is an individual hospitalization; these are stratified based on first nonsurgical unit entered by the patient. See Supplementary Appendix for consort diagram.
Hospitalizations where the first hospital unit was the OR or PAR unit are included based on the first nonsurgical unit (ward, SDU, and ICU) they entered. All statistical comparisons across units (eg, ward vs ICU) using analysis of variance were significant at the < .0001 level.
Inpatient hospitalizations are those where, at the time of admission, the admitting physician anticipated that the patient would remain in the hospital for at least 2 midnights; those hospitalizations not meeting this criterion are observation hospitalizations. See Escobar et al11 for additional details.
The patient was ever in the OR during the hospitalization.
Interhospital transfers are common in this integrated system. “Transport in” indicates that the patient was transferred from/to another hospital.
The CCI score ranges from 0 to 40; increasing values indicate worsening comorbidity burden; see Charlson et al14 and Deyo et al15 for details.
The COPS2 is based on patient diagnoses accrued in the preceding 12 months and is assigned every month to all adults with a KPNC medical record number and has a range from 0 to 1010 (higher scores indicate worse mortality risk). The univariate relationship between the COPS2 and 1-year mortality is as follows: 0 to 39, 0.3%; 40–64, 5.3%; ≥ 65, 17.2%. See Escobar et al10 for details.
The LAPS2 is assigned on the basis of a patient’s worst vital signs, pulse oximetry, neurologic status, and 16 laboratory test results in the preceding 72 hours for the admission score, and in the preceding 24 hours for subsequent hourly scores. The univariate relationship of an admission LAPS2 with 30-day mortality is as follows: 0–59, 1.0%; 60–109, 5.0%; ≥ 110, 13.7%. After age, sex, admission diagnosis, and comorbid conditions are controlled for, the adjusted odds ratio for inpatient mortality for an increase in admission LAPS2 of 5 points is 1.134 (95% CI, 1.133–1.135). See Escobar et al10–12 for additional detail.
Patients can have 1 of 4 care directives: full code, partial code, do not resuscitate, and comfort care only. For analytic purposes, we grouped all care directives into 2 categories (full code/not full code). See Escobar et al10 for additional detail.
See text and Escobar et al10 for details on how we grouped International Classification of Diseases codes into Primary Conditions.
Abbreviations: SDU: stepdown unit; ICU: intensive care unit; COPS2: COmorbidity Point Score, version 2; LAPS2: Laboratory-based Acute Physiology Score, version 2; LOS: length of stay; OR: operating room; KPNC: Kaiser Permanente Northern California, AMI: acute myocardial infarction.
Table 3 shows the distribution of the novel time-varying predictors we studied. The table, stratified by admission unit and admission severity of illness, also provides some aggregate metrics. Ward patients accounted for the largest proportion of hospitalizations as well as patient hours. At the aggregate level, the proportion of patient hours with no orders was highest among SDU patients and lowest among ICU patients, a pattern also observed with respect to hours with no MAR transactions. Hourly order counts generally reflected the intensity of care and severity of illness experienced by patients. For example, ICU patients with admission LAPS2 ≥ 110 averaged 0.50 laboratory orders per hour, whereas those with LAPS2 < 110 averaged 0.44; the corresponding numbers for SDU patients were 0.40 and 0.34, while those for ward patients were 0.37 and 0.33. Similarly, oral medication orders were more common among patients with LAPS2 < 110 whereas intravenous drips were more common among patients with LAPS2 ≥ 110. For example, among patients with LAPS2 < 110, ward, SDU, and ICU patients had 0.68, 0.70, and 0.75 oral medication orders per hour, respectively; the corresponding numbers for those with LAPS2 ≥ 110 were 0.66, 0.69, and 0.64. In contrast, the corresponding numbers for intravenous drips for patients with LAPS2 < 110 were 0.04, 0.06, and 0.25; for patients with LAPS2 ≥ 110, they were 0.06, 0.11, and 0.38. However, SDU patients (who only comprised 4% of hospitalizations) did not always have characteristics that were intermediate between ward and ICU patients. For example, they actually had fewer patient hours where patient deaths occurred in the unit, and they also had fewer hours where a patient deterioration occurred.
Table 3.
Candidate strain variables stratified by admission unit and admission illness severitya
| Ward |
SDU |
ICU |
||||
|---|---|---|---|---|---|---|
| LAPS2 < 110 | LAPS2 ≥ 110 | LAPS2 < 110 | LAPS2 ≥ 110 | LAPS2 < 110 | LAPS2 ≥ 110 | |
| Patients | 423 067 | 62 202 | 26 123 | 5985 | 65 010 | 30 633 |
| Hospitalizations | 722 007 (76%) | 82 663 (9%) | 31 741 (3%) | 6911 (1%) | 75 643 (8%) | 35 512 (4%) |
| Patient hours | 56 163 630 (67%) | 9 197 537 (11%) | 2 467 658 (3%) | 861 782 (1%) | 9 439 847 (11%) | 6 031 707 (7%) |
| Aggregated hourly variablesb | ||||||
| % of hours with no orders | 74.3 | 73.2 | 75.1 | 74.7 | 68.9 | 66.7 |
| % of hours with no MAR transactions | 53.1 | 53.0 | 53.7 | 49.6 | 45.6 | 40.4 |
| % of hours with occupancy <75% | 47.1 | 44.5 | 48.2 | 50.0 | 55.6 | 54.2 |
| % of hours with occupancy ≥85% | 19.2 | 21.1 | 24.8 | 22.3 | 17.1 | 18.4 |
| Individual hourly variablesc | ||||||
| Count of administrative orders | 0.09 | 0.07 | 0.11 | 0.08 | 0.09 | 0.07 |
| Count of blood transfusion orders | 0.01 | 0.01 | 0.00 | 0.01 | 0.01 | 0.01 |
| Count of laboratory orders | 0.33 | 0.37 | 0.34 | 0.40 | 0.44 | 0.50 |
| Count of diet orders | 0.04 | 0.03 | 0.04 | 0.03 | 0.04 | 0.03 |
| Count of imaging orders | 0.02 | 0.02 | 0.03 | 0.03 | 0.04 | 0.04 |
| Count of nursing orders | 0.45 | 0.34 | 0.43 | 0.30 | 0.51 | 0.43 |
| Count of intramuscular medications | 0.14 | 0.14 | 0.14 | 0.16 | 0.16 | 0.19 |
| Count of aerosol or inhalation medications | 0.05 | 0.09 | 0.08 | 0.14 | 0.07 | 0.12 |
| Count of medications using simple intravenous administration | 0.48 | 0.43 | 0.41 | 0.42 | 0.53 | 0.59 |
| Count of medications using intravenous drips | 0.04 | 0.06 | 0.06 | 0.11 | 0.25 | 0.38 |
| Count of medications using other all other modalities | 0.04 | 0.04 | 0.04 | 0.04 | 0.05 | 0.05 |
| Count of orally administered medications | 0.68 | 0.66 | 0.70 | 0.69 | 0.75 | 0.64 |
| Sum of all orders and MAR transactions since admission | 254.7 | 323.8 | 277.5 | 392.9 | 548.9 | 715.9 |
| Sum of all orders and MAR transactions in last 4 h | 9.4 | 8.9 | 9.4 | 9.5 | 11.7 | 12.1 |
| Percent occupancy for unit where patient was (%) | 73.9 | 75.1 | 73.2 | 73.1 | 70.1 | 70.7 |
| Number of patients in unit where patient was | 90.1 | 90.7 | 29.0 | 35.6 | 55.4 | 53.8 |
| Number of new patients admitted to unit where patient was | 1.3 | 1.3 | 0.5 | 0.6 | 0.8 | 0.8 |
| Number of discharges from unit where patient was | 1.3 | 1.3 | 0.5 | 0.5 | 0.8 | 0.8 |
| Maximum percent occupancy in the last 4 h for unit where patient was (%) | 75.4 | 76.5 | 75.0 | 74.8 | 71.6 | 72.2 |
| Maximum range in percent occupancy in the last 4 h where patient was (% difference) | 3.2 | 2.9 | 4.0 | 3.7 | 3.4 | 3.3 |
| Mean age of patients in unit | 66.8 | 67.1 | 67.9 | 69.8 | 65.6 | 66.1 |
| Mean LAPS2 of patients in unit | 60.7 | 64.6 | 69.7 | 79.8 | 83.0 | 90.0 |
| Mean COPS2 of patients in unit | 53.4 | 55.2 | 57.0 | 63.9 | 54.5 | 58.5 |
| Maximum LAPS2 of patients in unit where patient was, excluding patient | 157.4 | 162.2 | 142.4 | 154.7 | 170.7 | 176.5 |
| Number of patients with LAPS2 ≥ 110 in unit where patient was | 8.0 | 8.5 | 3.3 | 4.6 | 7.5 | 7.9 |
| Number of deaths in unit where patient was | 0.07 | 0.08 | 0.02 | 0.03 | 0.07 | 0.07 |
| Number of physiologic deteriorations in unit where patient was | 1.36 | 1.40 | 0.47 | 0.63 | 0.88 | 0.87 |
| For this patient’s nurse, the maximum LAPS2 of patients under her/his care | 98.0 | 120.1 | 102.4 | 127.1 | 114.4 | 135.5 |
| For this patient’s nurse, the % of patients under her/his care with a LAPS2 ≥ 110 (%) | 7.9 | 20.7 | 12.6 | 29.6 | 21.4 | 39.6 |
| For all nurses who cared for this patient, the maximum LAPS2 | 109.7 | 130.3 | 113.8 | 136.8 | 127.1 | 146.9 |
| Count of other patients cared for by this patient’s nursed | 4.0 | 3.9 | 3.1 | 3.2 | 2.9 | 2.9 |
See text and Table 1 for variable definitions. Admission acute severity of illness (LAPS2) was calculated at the time patients were roomed in. The number of patients adds up to more than 100% because patients can have stayed in multiple units; hospitalizations and total patient hours do add up to 100%.
Results are aggregated across all hours in the column.
Results shown are means for all hours within the column. All results shown are at the individual patient hour level: for example, sum of all orders and MAR transactions in last 4 hours was calculated for every patient for every hour based on data from the preceding 4 h.
Note that mean number of count of other patients cared for by this patient’s nurse is high among patients in the ICU (where minimum staffing ratio is 1 nurse to 2 patients) due to frequent cross coverage.
Abbreviations: SDU: stepdown unit; ICU: intensive care unit; COPS2: COmorbidity Point Score, version 2; LAPS2: Laboratory-based Acute Physiology Score, version 2; MAR: Medication Administration Record.
Table 4 shows selected candidate strain variables’ relationships with respect to longitudinal comorbidity risk (COPS2), LAPS2 24 hours into the hospital stay, time to discharge, and mortality risk. The table is stratified by admission unit, 2 variable categories (orders/MAR transactions and dynamic variables), and degree of correlation of the indicated variable with the LAPS2 at 24 hours (for each variable, we selected the variable with the most positive, closest to the null, and most negative correlation; results for all candidate variables are shown in the Supplementary Appendix). The table shows that relationships between our candidate strain variables and outcomes were not always intuitive. For example, in ward patients the count of medications administered via intravenous drips (eg, insulin), which usually involves sicker patients, and which had a low correlation (0.13) with severity of illness at 24 hours, had a hazard ratio of 0.83 (95% CI, 0.81–0.86) for mortality, indicating that patients receiving such medications were less likely to die. However, with respect to likely discharge, the hazard ratio was 0.39 (95% CI, 0.38–0.40), indicating that patients receiving drip medications would tend to have longer hospital stays. In all units, there was no association between mortality for a given patient and the maximum LAPS2 of the other patients cared for by that patient’s nurse.
Table 4.
Candidate strain variables’ relationship with time to discharge and mortality riska
| Pearson correlation with |
Hazard ratio (95% CI) |
Hazard ratio (95% CI) |
|||
|---|---|---|---|---|---|
| COPS2b | 24-h LAPS2 | Mortality | Time to discharge | ||
| Ward | |||||
| Orders and MAR transactions | |||||
| (Highest)c | Count of medications using intravenous drips | 0.02 | 0.13 | 0.83 (0.81–0.86) | 0.39 (0.38–0.40) |
| (Middle) | Count of diet orders | −0.01 | 0.00 | 0.30 (0.25–0.36) | 0.09 (0.09–0.09) |
| (Lowest) | Count of administrative orders | −0.05 | −0.02 | 0.84 (0.79–0.88) | 0.84 (0.83–0.84) |
| Dynamic variables | |||||
| (Highest) | For this patient’s nurse, the maximum LAPS2 of patients under her/his care | 0.19 | 0.46 | 1.00 (1.00–1.01) | 1.00 (1.00–1.00) |
| (Middle) | Number of deaths in unit where patient was | 0.01 | 0.01 | 1.02 (0.97–1.08) | 1.02 (1.01–1.03) |
| (Lowest) | Count of other patients cared for by this patient’s nurse | −0.06 | −0.11 | 0.96 (0.94–0.97) | 0.99 (0.99–1.00) |
| SDU | |||||
| Orders and MAR transactions | |||||
| (Highest) | Count of medications using intravenous drips | 0.03 | 0.19 | 0.48 (0.42–0.56) | 0.50 (0.47–0.54) |
| (Middle) | Count of administrative orders | −0.01 | 0.00 | 0.70 (0.54–0.89) | 0.75 (0.72–0.78) |
| (Lowest) | Count of orally administered medications | 0.03 | −0.01 | 0.67 (0.61–0.73) | 0.86 (0.85–0.86) |
| Dynamic variables | |||||
| (Highest) | For this patient’s nurse, the maximum LAPS2 of patients under her/his care | 0.26 | 0.59 | 1.00 (1.00–1.01) | 1.00 (1.00–1.00) |
| (Middle) | Count of other patients cared for by this patient’s nurse | 0.04 | 0.00 | 0.91 (0.84–0.97) | 1.06 (1.05–1.07) |
| (Lowest) | Percent occupancy for unit where patient was | −0.05 | −0.12 | 0.99 (0.99–1.00) | 1.00 (0.99–1.00) |
| ICU | |||||
| Orders and MAR transactions | |||||
| (Highest) | Count of medications using intravenous drips | 0.04 | 0.40 | 0.75 (0.74–0.77) | 0.64 (0.63–0.65) |
| (Middle) | Count of administrative orders | 0.00 | 0.00 | 1.01 (0.97–1.05) | 0.99 (0.98–1.01) |
| (Lowest) | Count of orally administered medications | 0.02 | −0.02 | 0.70 (0.68-0.72) | 0.87 (0.87–0.88) |
| Dynamic variables | |||||
| (Highest) | For this patient’s nurse, the maximum LAPS2 of patients under her/his care | 0.20 | 0.68 | 1.00 (1.00–1.00) | 1.00 (1.00–1.00) |
| (Middle) | Number of deaths in unit where patient was | 0.00 | 0.01 | 1.09 (1.02–1.17) | 1.04 (1.02–1.06) |
| (Lowest) | Count of other patients cared for by this patient’s nurse | −0.02 | −0.19 | 0.99 (0.96–1.01) | 1.02 (1.02–1.03) |
See text for description of statistical methodology and Tables 1 and 2 for description of listed variables. Hours beyond 1440 (60 days) were dropped from analyses; these came from 729 (0.08%) of the 954 477 hospitalizations.
COPS2 (see Table 2) is based on data from 12-month period preceding hospitalization. The LAPS2 (see Table 2) is that assigned after patient was in the hospital for 24 hours.
Variables are listed in descending order based on the Pearson correlation between the indicated variable at 24 hours and the hourly admission LAPS2 at 24 hours (top is the most positively correlated, middle is the variable that has correlation with LAPS2 at 24 hours that is closest to zero, bottom the one with the most negatively correlated).
Abbreviations: SDU: stepdown unit; ICU: intensive care unit; COPS2: COmorbidity Point Score, version 2; LAPS2: Laboratory-based Acute Physiology Score, version 2; MAR: Medication Administration Record.
To better understand trends in each of the novel hourly strain variables over time, we plotted the means of each variable at each hour, anchored at admission and, separately, either at the time of discharge or death. We plotted trends in mean hourly LAPS2 for comparison. For illustrative purposes, Figures 1–4 show hourly data from patients with 2 important commonly encountered diagnoses with different pathophysiology—sepsis and acute myocardial infarction (AMI). These figures highlight temporal relationships that are less evident when examining tabular data. They show hourly severity of illness scores and counts of laboratory orders for sepsis and AMI ward and ICU patients. Figures 1 and 2 show these counts viewed starting at the time of admission, whereas Figures 3 and 4 show these counts looking backward from the time of discharge. In the period immediately following admission, hourly severity scores increase slightly and then gradually decrease. Laboratory order counts per hour do not track severity scores—they have a sharp decrease followed by small oscillations that are more pronounced among decedents than among survivors. However, if one examines data prior to discharge, very different patterns are apparent. Among survivors, severity scores gradually decrease prior to discharge, while the opposite occurs among decedents. For both survivors and decedents, however, laboratory order counts per hour show multiple oscillations that do not track illness severity. Similar patterns were observed with other diagnoses.
Figure 1.
Laboratory orders per hour and acute physiology in sepsis patients, first 72 h after hospital admission. The top line (in red) and the left Y-axis are for the mean hourly LAPS2 (Laboratory-based Acute Physiology Score, version 2); the bottom line (in blue) and the right Y-axis are for the mean number of laboratory orders per patient per hour. The upper panels are for ward and intensive care unit (ICU) patients who survived to discharge; the lower panels are for decedents. The number of patient hours for ward survivors and decedents were 3 403 240 and 165 373, respectively; for ICU patients, the corresponding numbers were 683 667 and 134 420, respectively. See text and Table 2 for details on the LAPS2.
Figure 2.
Laboratory orders per hour and acute physiology in acute myocardial infarction patients, first 72 h after hospital admission. Color, axes, and hospital unit schema are the same as in Figure 1. The number of patient hours for ward survivors and decedents were 592 970 and 12 034, respectively; for ICU patients, the corresponding numbers were 238 283 and 13 053, respectively.
Figure 3.
Laboratory orders per hour and acute physiology in sepsis patients in the 72 h prior to discharge. Color, axes, and hospital unit schema are the same as in Figure 1, except that the X-axis looks backward from discharge (which is the time of death for decedents). The number of patient hours for ward survivors and decedents were 3 432 018 and 167 164, respectively; for ICU patients, the corresponding numbers were 691 802 and 135 829, respectively.
Figure 4.
Laboratory orders per hour and acute physiology in acute myocardial infarction patients in the 72 h prior to discharge. Schema are the same as Figure 3. The number of patient hours for ward survivors and decedents were 595 963 and 12 136, respectively; for ICU patients, the corresponding numbers were 239 700 and 13 179, respectively.
DISCUSSION
We have conducted a preliminary exploration of the relationship between novel time-varying hospital EHR variables and 2 important outcomes, mortality and imminent discharge. We found that it is feasible to extract and transform EHR data to permit quantifying some aspects of caregiver hourly as well as cumulative intensity of effort. It is also possible to quantify the environment around a patient (including occupancy as well as the workload of nurses caring for the patient). Importantly, our analyses showed that many of these variables do not have a consistent temporal relationship with patients’ longitudinal comorbidity burden or severity of illness (whether measured on admission or at an hourly level). The fact that these novel variables do not simply track patients’ acute physiology or longitudinal comorbidity burden highlights the importance of incorporating them as time-varying covariates as well as the need to develop new conceptual models for understanding relationships between strain and patient outcomes. Although we often assume that sicker patients need more care, some patients who aren’t acutely ill may still require a lot of care, as measured by order burden. This finding highlights an important nuance of strain and underscores why variables other than severity of illness should be assessed when studying and managing strain.
This work could be used in the future to optimize workflows and work burden within the hospital. For example, if 1 unit is experiencing significant strain, operational leaders could be notified and redistribute resources (unit assistants, patient care assistants) from a unit not experiencing strain. Supplementing resources to floors that experience strain would potentially mitigate the harmful effect of strain on patient outcomes, hospital cost, and healthcare worker burnout. These concepts are particularly relevant during the coronavirus disease 2019 pandemic when medical floors and ICUs have been operating under strain, whereas other units (surgical units) may have many fewer patients due to canceled surgeries. Further work is necessary to understand how the trends in dynamic variables studied here should translate to actions in the operational world.
Our findings need to be placed in context of other recent work with variables extracted from hospital EHRs. One area that has seen considerable success is in prediction of mortality, where multiple studies, including our own, have demonstrated that EHR data (laboratory tests, vital signs, and/or care directives) substantially improve prediction of in-hospital mortality as well as in-hospital deterioration (which is a major cause of in-hospital mortality).10,26–29 These advances have permitted more accurate risk adjustment, quantification of strain,3 and even prediction in real time.12,30,31
In contrast, the ability to predict hospital LOS (which can also be expressed as time to discharge) at an accuracy level suitable for use in real time has lagged considerably. Many studies, including our own, have failed to achieve accurate estimates of hospital LOS even after adding novel predictors and despite the use of modern machine learning and artificial intelligence techniques.32–38 It has been possible to enhance accuracy in prediction of dichotomized LOS (eg, “LOS > 7 days”) but that level of prediction accuracy is of limited operational utility. Although conceptualizing hospital factors using the strain construct provides some insights into how environmental and process of care factors affect LOS,39 the level of accuracy remains insufficient for direct operational use.
Consideration of our results suggests some possible reasons for the discrepancy between our ability to predict mortality and our ability to predict time to discharge. With respect to mortality, the role of diagnoses and acute physiology is very pronounced and very clear, particularly when enhanced by knowledge of a patient’s care directive. In contrast, our quantitative understanding of the process of hospital care is much more rudimentary. Our results show that relationships between patients’ biological factors (age, sex, longitudinal comorbidity burden, and acute physiology), time in the hospital, environmental factors (occupancy and adverse events in the unit where an individual patient is), and service provision (orders and MAR transactions) are very complex. Rendering these relationships intelligible using simple analytic constructs may not be possible at our current level of understanding.
Rossetti et al8,40 have examined the impact of time-varying nurses’ EHR interaction data on clinical outcomes, focusing on patient-specific nursing metadata (counts of vital signs and nursing notes taken for the patient). Although we have similarly explored patient-specific counts of orders and MAR transactions, we have also quantified the environment around the patient through characteristics of the unit that the patient is in and the cognitive burden of the nurse for caring for other patients.
Our analyses do have important limitations. Currently, data on physicians working in KPNC hospitals are not available in database format, so we could not explore the relationship between physicians, RNs, and the metrics we quantified. Because of data limitations, we also could not disaggregate data at the hospital subunit level, since most of these hospitals’ wards are subdivided and some hospitals have more than 1 ICU. We also lacked the resources to map orders and MAR transactions onto discrete disease states. Last, despite our best efforts, we could not always link all patients to their nurses at all times; this latter limitation may be overcome in the future as EHRs become more sophisticated.
Although this study focused on novel time-varying strain metrics (eg, environmental factors in the hospital that include occupancy and patient severity of illness) and workload measures (eg, based on orders and MAR transactions) on mortality and imminent discharge, future work should examine other strain metrics such as fall risk assessment and pressure injury risk. These nursing care plan-based assessments reflect nurse workload, as patients with limited mobility require multiple people to assist with toileting and harm prevention (falls with head bleed, deep pressure ulcers). These serious adverse events are likely associated with inpatient outcomes such as mortality and time to discharge.
Future progress in this area should take several directions, all of which are feasible given current computation capability and growing EHR data availability. One direction is to create better conceptual models that link the content of a given EHR marker (eg, an individual MAR transaction) to a patient’s actual disease state. This could be used to create ontologies for hospitalized patients’ care paths. Definition of such care paths could be enhanced through the use of natural language processing of patients’ free-text notes. Well defined care paths could then be used as analytic tools for more detailed mechanistic studies that could incorporate both quantification of strain metrics as well as their effects on LOS. Such progress is already underway with the work of Rossetti et al8 on a conceptual framework to phenotype clinician behaviors based on clinician interactions with clinical information systems and to interpret and leverage this information to predict patient outcomes.
A second direction would be to identify critical process markers for LOS. These would be individual indicators (eg, an order for X) or small discrete indicator patterns (eg, “order for W within X hours of order for Y among patients with diagnosis Z”) based on clinical content expertise and having a strong association with time to hospital discharge.
Full understanding of the effects of individual clinical indicators and aggregate indicators of hospital strain on LOS will also require capture of nonclinical data. Although it is clear that clinical and service factors play a role, many nonclinical factors are also important, particularly among patients with prolonged LOS. For example, a patient may be clinically stable but discharge may not be possible due to problems with placement (eg, difficulties with skilled nursing facility bed availability) or absence of caregivers. Some of these factors may be evident at discharge. Extraction of these particular data elements would require examination of nonclinician provider free-text progress notes using natural language processing. The discharge process is complex and may be influenced by factors not captured in the EHR, such as social factors (eg, healthcare provider- and caretaker-biases, social support structure, insurance coverage, etc.).
Finally, 1 other area that could be explored using the predictors we described here is that of graph or network theory. Cross coverage across nurses taking care of patients is common, and it may be that network science may provide the best way to visualize the relationships between nurses, physicians, patient clinical factors, orders, and medications. Incorporation of network features alongside more traditional metrics (comorbidity burden and acute physiology) and markers such as the ones described in this article may permit not just better quantification of strain and accurate prediction, but also deeper understanding of hospital processes of care.
FUNDING
This work was funded by the Permanente Medical Group, Inc., and Kaiser Foundation Hospitals, Inc. Dr. VXL was also supported by National Institutes of Health grant R35GM128672.
AUTHOR CONTRIBUTIONS
GJE and VXL conceptualized the work. CL, BLL, AS, and GJE were involved in data curation. CL, BLL, and GJE contributed to the statistical analysis plan. BLL implemented all analyses. CL and GJE drafted the original article. All authors interpreted the findings, revised the article for intellectual content, approved of the final article to be published, and have agreed to be accountable for all aspects of this work.
SUPPLEMENTARY MATERIAL
Supplementary material is available at Journal of the American Medical Informatics Association online.
Supplementary Material
ACKNOWLEDGMENTS
The funders did not play any role in the preparation of the article, interpretation of the findings, or the decision to submit the article. We wish to thank Mss. Robin Betts and Lisa Arellanes for administrative assistance, Ms. Nancy Cadet and Mr. Marc Flagg for assistance with mapping fields in the Epic EHR, the Division of Research Strategic Programming Group for creating the linked dataset, Dr. Elizabeth Scruth for her help in data quality audit, and Ms. Kathleen Daly for editing and formatting the article.
CONFLICT OF INTEREST STATEMENT
None declared.
DATA AVAILABILITY
The data used in this study contained Protected Health Information of KPNC members and cannot be shared.
Contributor Information
Catherine Lee, Division of Research, Kaiser Permanente, Oakland, California 94612, USA; Department of Health Systems Science, Kaiser Permanente Bernard J. Tyson School of Medicine, Pasadena, California 91101, USA.
Brian L Lawson, Division of Research, Kaiser Permanente, Oakland, California 94612, USA.
Ariana J Mann, Electrical Engineering, Stanford University, Stanford, California 94305, USA.
Vincent X Liu, Division of Research, Kaiser Permanente, Oakland, California 94612, USA; Intensive Care Unit, Kaiser Permanente Medical Center, Santa Clara, California 95051, USA.
Laura C Myers, Division of Research, Kaiser Permanente, Oakland, California 94612, USA; Intensive Care Unit, Kaiser Permanente Medical Center, Walnut Creek, California 94596, USA.
Alejandro Schuler, Center for Targeted Learning, School of Public Health, University of California, Berkeley, California 94704, USA.
Gabriel J Escobar, Division of Research, Kaiser Permanente, Oakland, California 94612, USA.
REFERENCES
- 1. Gabler NB, Ratcliffe SJ, Wagner J, et al. Mortality among patients admitted to strained intensive care units. Am J Respir Crit Care Med 2013; 188 (7): 800–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Anesi GL, Liu VX, Gabler NB, et al. Associations of ICU capacity strain with disposition and outcomes of patients with sepsis presenting to the emergency department. Ann Am Thorac Soc 2018; 15 (11): 1328–35. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Anesi GL, Chowdhury M, Small DS, et al. Association of a Novel Index of hospital capacity strain with admission to intensive care units. Ann Am Thorac Soc 2020; 17 (11): 1440–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Volchenboum SL, Mayampurath A, Göksu-Gürsoy G, Edelson DP, Howell MD, Churpek MM. Association between in-hospital critical illness events and outcomes in patients on the same ward. JAMA 2016; 316 (24): 2674–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Wagner J, Gabler NB, Ratcliffe SJ, Brown SE, Strom BL, Halpern SD. Outcomes among patients discharged from busy intensive care units. Ann Intern Med 2013; 159 (7): 447–55. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Hua M, Halpern SD, Gabler NB, Wunsch H. Effect of ICU strain on timing of limitations in life-sustaining therapy and on death. Intensive Care Med 2016; 42 (6): 987–94. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Kohn R, Harhay MO, Bayes B, et al. Ward capacity strain: a novel predictor of 30-day hospital readmissions. J Gen Intern Med 2018; 33 (11): 1851–3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Rossetti SC, Knaplund C, Albers D, et al. Healthcare process modeling to phenotype clinician behaviors for exploiting the signal gain of clinical expertise (HPM-ExpertSignals): development and evaluation of a conceptual framework. J Am Med Inform Assoc 2021; 28 (6): 1242–51. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Escobar GJ, Greene JD, Gardner MN, Marelich GP, Quick B, Kipnis P. Intra-hospital transfers to a higher level of care: contribution to total hospital and intensive care unit (ICU) mortality and length of stay (LOS). J Hosp Med 2011; 6 (2): 74–80. [DOI] [PubMed] [Google Scholar]
- 10. Escobar GJ, Gardner MN, Greene JD, Draper D, Kipnis P. Risk-adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system. Med Care 2013; 51 (5): 446–53. [DOI] [PubMed] [Google Scholar]
- 11. Escobar GJ, Plimier C, Greene JD, Liu V, Kipnis P. Multiyear rehospitalization rates and hospital outcomes in an integrated health care system. JAMA Netw Open 2019; 2 (12): e1916769. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Escobar GJ, Liu VX, Schuler A, Lawson B, Greene JD, Kipnis P. Automated identification of adults at risk for in-hospital clinical deterioration. N Engl J Med 2020; 383 (20): 1951–60. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Escobar GJ, Ragins A, Scheirer P, Liu V, Robles J, Kipnis P. Nonelective rehospitalizations and postdischarge mortality: predictive models suitable for use in real time. Med Care 2015; 53 (11): 916–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987; 40 (5): 373–83. [DOI] [PubMed] [Google Scholar]
- 15. Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol 1992; 45 (6): 613–9. [DOI] [PubMed] [Google Scholar]
- 16. Kerlin MP, Harhay MO, Vranas KC, Cooney E, Ratcliffe SJ, Halpern SD. Objective factors associated with physicians’ and nurses’ perceptions of intensive care unit capacity strain. Ann Am Thorac Soc 2014; 11 (2): 167–72. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Kim S-H, Chan CW, Olivares M, Escobar G. ICU admission control: an empirical study of capacity allocation and its implication for patient outcomes. Manage Sci 2015; 61 (1): 19–38. [Google Scholar]
- 18. Potter P, Wolf L, Boxerman S, et al. An analysis of nurses’ cognitive work: a new perspective for understanding medical errors. In: Henriksen K, Battles JB, Marks ES, Lewin DI, eds. Advances in Patient Safety: From Research to Implementation. Volume 1: Research Findings. Rockville, MD: Agency for Healthcare Research and Quality; 2005. https://www.ncbi.nlm.nih.gov/books/NBK20452/. [PubMed] [Google Scholar]
- 19. Potter P, Wolf L, Boxerman S, et al. Understanding the cognitive work of nursing in the acute care environment. J Nurs Adm 2005; 35 (7–8): 327–35. [PubMed] [Google Scholar]
- 20. Laxmisan A, Hakimzada F, Sayan OR, Green RA, Zhang J, Patel VL. The multitasking clinician: decision-making and cognitive demand during and after team handoffs in emergency care. Int J Med Inform 2007; 76 (11–12): 801–11. [DOI] [PubMed] [Google Scholar]
- 21. Redding DA, Robinson S. Interruptions and geographic challenges to nurses’ cognitive workload. J Nurs Care Qual 2009; 24 (3): 194–200; quiz 01–2 [DOI] [PubMed] [Google Scholar]
- 22. Kalisch BJ, Aebersold M. Interruptions and multitasking in nursing care. Jt Comm J Qual Patient Saf 2010; 36 (3): 126–32. [DOI] [PubMed] [Google Scholar]
- 23. Koch SH, Westenskow D, Weir C, et al. ICU nurses’ evaluations of integrated information displays on user satisfaction and perceived mental workload. Stud Health Technol Inform 2012; 180: 383–7. [PubMed] [Google Scholar]
- 24. Singh H, Spitzmueller C, Petersen NJ, Sawhney MK, Sittig DF. Information overload and missed test results in electronic health record-based settings. JAMA Intern Med 2013; 173 (8): 702–4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Allison P. Discrete-time methods for the analysis of event histories. Sociol Methodol 1982; 13: 61–98. [Google Scholar]
- 26. Tabak YP, Johannes RS, Silber JH, Kurtz SG. Should Do-Not-Resuscitate status be included as a mortality risk adjustor? The impact of DNR variations on performance reporting. Med Care 2005; 43 (7): 658–66. [DOI] [PubMed] [Google Scholar]
- 27. Tabak YP, Johannes RS, Silber JH. Using automated clinical data for risk adjustment: development and validation of six disease-specific mortality predictive models for pay-for-performance. Med Care 2007; 45 (8): 789–805. [DOI] [PubMed] [Google Scholar]
- 28. Pine M, Jordan HS, Elixhauser A, et al. Enhancement of claims data to improve risk adjustment of hospital mortality. JAMA 2007; 297 (1): 71–6. [DOI] [PubMed] [Google Scholar]
- 29. Escobar GJ, Greene JD, Scheirer P, Gardner MN, Draper D, Kipnis P. Risk-adjusting hospital inpatient mortality using automated inpatient, outpatient, and laboratory databases. Med Care 2008; 46 (3): 232–9. [DOI] [PubMed] [Google Scholar]
- 30. Churpek MM, Yuen TC, Winslow C, et al. Multicenter development and validation of a risk stratification tool for ward patients. Am J Respir Crit Care Med 2014; 190 (6): 649–55. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Kollef MH, Chen Y, Heard K, et al. A randomized trial of real-time automated clinical deterioration alerts sent to a rapid response team. J Hosp Med 2014; 9 (7): 424–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Liu V, Kipnis P, Gould MK, Escobar GJ. Length of stay predictions: improvements through the use of automated laboratory and comorbidity variables. Med Care 2010; 48 (8): 739–44. [DOI] [PubMed] [Google Scholar]
- 33. Pendharkar PC, Khurana H. Machine learning techniques for predicting hospital length of stay in pennsylvania federal and specialty hospitals. Int J Comput Sci Appl 2014; 11 (3): 45–56. [Google Scholar]
- 34. Lu MS, Lucyk K, Lorenzetti D, Quan H, Lorenzetti D, Quan H. Systematic review of risk adjustment models of hospital length of stay (LOS). Med Care 2015; 53 (4): 355–65. [DOI] [PubMed] [Google Scholar]
- 35. Cai XP-C, Coiera E, Martin-Sanchez F, Day R, Roffe D, Gallego B. Real-time prediction of mortality, readmission, and length of stay using electronic health record data. J Am Med Inform Assoc 2016; 23 (3): 533–61. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36. Barnes S, Hamrock E, Toerper M, Siddiqui S, Levin S. Real-time prediction of inpatient length of stay for discharge prioritization. J Am Med Inform Assoc 2016; 23 (e1): e2–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. Bacchi S, Gluck S, Tan Y, et al. Mixed-data deep learning in repeated predictions of general medicine length of stay: a derivation study. Intern Emerg Med 2021; 16 (6): 1613–7. [DOI] [PubMed] [Google Scholar]
- 38. Cohen BS, Liu J, Zachariah P, et al. Predicting healthcare-associated infections, length of stay, and mortality with the nursing intensity of care index. Infect Control Hosp Epidemiol 2021; 16: 1–8. [DOI] [PubMed] [Google Scholar]
- 39. Kohn RH, Weissman GE, Anesi GL, et al. Ward capacity strain: a novel predictor of delays in intensive care unit survivor throughput. Ann Am Thorac Soc 2019; 16 (3): 387–90. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. Rossetti SC, Knaplund C, Albers D, et al. Leveraging clinical expertise as a feature-not an outcome-of predictive models: evaluation of an early warning system use case. AMIA Annu Symp Proc 2020; 2019: 323–32. [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The data used in this study contained Protected Health Information of KPNC members and cannot be shared.