Economic evaluation and analyses of hospital-based electronic medical records (EMRs): a scoping review of international literature

The electronic medical record (EMR) industry is expanding rapidly, growing from nearly nonexistent in 2000 to being worth more than $31 billion a year in 2018¹. Globally, EMR adoption has been recognized as an important step toward healthcare modernization—necessary to improve patient outcomes, increase health system efficiency, and unlock extensive research potential in healthcare. For instance, the United States allocated $27 billion for hospitals demonstrating “meaningful usage” of EMR in 2009². EMR adoption is heavily dependent on each country’s socioeconomic policies, health-system organization and financing, and the integration of primary and acute care networks. While many high-income and upper–middle-income countries have reported varying degrees of EMR adoption, lower socioeconomic countries report drastically lower adoption rates, often citing lack of funding and inadequate existing infrastructure as the main obstacles³.

The impacts of EMR implementation vary but can be profound for hospital business⁴. These can include improvements in quality, safety, information integrity and completeness, ability to use the data collected in the EMR as part of routine care, and for data analytics to drive quality improvements and research activities. The majority of these impacts, while providing significant value, do not provide immediate financial returns. They are also difficult to quantify and value, and so are usually excluded from traditional, financial-based business cases for EMR investment.

Recently, Lau and Kuziemsky⁵ attempted to synthesize the research and empirical evidence over the past two decades to provide some guidelines to identify and measure the benefits of digital health implementation. The authors stated that the framework presented in the handbook aims to “provide a high-level conceptual scheme to guide eHealth evaluation efforts to be undertaken by the respective jurisdictions and investment programs in Canada.” The framework relies heavily on measuring the successful implementation of Information and Communications Technology (ICT) in different settings, a systematic review on the determinants of success in inpatient clinical information systems, and the synthesis of results from health-information system (HIS) evaluations. The three eHealth benefit domains identified in this framework include care quality, access (to care), and productivity. “Care quality” covers a wide range of categories and measures, including patient safety, appropriateness and effectiveness, and health outcomes. “Access” captures the ability of patients and providers to access services, and participation by patients and carers. “Productivity” includes efficiency, care coordination, and net cost.

The guidelines briefly discussed economic methods that are potentially applicable for eHealth benefit evaluation (Chapters 5 and 14 in⁵). These methods were informed by a scoping review of 33 economic evaluation studies applied for HIS, published between 2000 and 2013⁶. A wide range of scopes and types of HIS were considered in this study, ranging from sector systems (e.g., health information-exchange (HIE) networks, primary care EMRs, and immunization information systems) to organization (institutional information systems) and goal-specific systems (computerized provider order entry (CPOE), medication management, disease management, and clinical documentation systems). While informative, the wide and extremely heterogeneous scope provided limited practical guidance for an economic evaluation. EMR applications in primary care settings often have very different targets, and thus short- versus long-term impacts, compared to those in hospital settings. Additionally, the benefits of HIS do not entirely overlap with those generated from EMR for digital hospital implementations.

Owing to the perceived benefits and the “infrastructure” nature of EMR, governments have generally stepped up to provide incentives and subsidies for EMR adoption. However, the cost-effectiveness of this investment has been questioned due to mixed evidence of cost requirements and observed impacts⁷. This is due partly to many providers making only token attempts to satisfy the criteria for meaningful use without capturing the economic values of EMR investments. Additionally, the diversity of EMR implementations in the acute and primary settings complicates the assessment of economic impacts and fiscal significance. Increasingly, resource and fiscal constraints require both the public and private sectors to consider the benefit–cost balance of EMR investments versus other demands, both in and outside healthcare. For large-scale and medium-to-long-term investments and rollouts of EMR to be evaluated properly, robust conceptual frameworks that systematically identify and measure the economic impacts of EMR, backed by healthcare funding agencies, decision-makers, and service providers, are urgently needed.

Our project aims to contribute to this effort by investigating the available options, both in the academic literature and practice, to evaluate the economic values of EMR, and develop a cost–benefit analysis framework to identify, measure and value the impacts of EMR in the Australian hospital-care setting. This paper presents the first piece of work: a comprehensive scoping review examining how EMR implementations in the hospital setting have been evaluated using cost–benefit analysis approaches internationally.

A scoping review was conducted, following the methodology outlined by Arksey and O’Malley⁸ and Levac et al.⁹. While the main study aims to understand the scope of the literature on CBA of EMR in hospital settings, we were also interested in exploring themes that naturally emerged from the literature. The scoping review was conducted in five stages, as mapped in Fig. 1.

The process of conducting the scoping review.

Full size image

This review method was chosen, instead of a systematic literature review and meta-analysis, because we did not attempt to critically appraise or synthesize quantitative evidence even though CBA studies often reported quantitative results¹⁰. We propose that both the qualitative and quantitative results from CBA studies in the literature were not necessarily comparable or synthesisable. Those studies might be conducted in different countries, and/or the EMR implementation objectives and practices differ significantly. For theme exploration, a scoping review therefore is the most appropriate starting point, and a base for further systematic reviews with specific quantitative focuses.

Stage 1—research questions

We reviewed both peer-reviewed and gray literature to understand the evaluation methods, economic benefits and costs of EMR implementation in hospital settings. EMR systems in primary care or nonhospital settings were excluded. More specifically, we asked:

• Q1: How many studies can be classified as a (full) CBA, as opposed to a partial economic analysis? (That is, studies reporting both benefits and costs, not just benefits or costs only.) Did those studies clearly discuss benefits and costs incurred by different stakeholder groups, and the analytical perspectives?

• Q2: What were the benefit or cost items? Were they financial or economic? How were they measured and valued? What were the main assumptions or findings on the changes over time of those benefits or cost items, and across settings?

• Q3: What were the main findings regarding the net impacts (i.e., net benefits or net costs)? Did the literature provide some explanations on how and why such impacts were observed? Did the findings reflect some or all of the quadruple aims of healthcare delivery¹¹?

In this scoping review, a CBA study is defined as one that involves identifying, measuring and comparing the benefits and costs of an EMR implementation program or project. The costs and benefits need to be presented quantitatively (e.g., in dollars), based on financial and/or economic data, and summarized using net present values (NPVs) or internal rates of returns (IRRs) or returns on investment (ROIs). While studies calculating net financial cost (cost minus revenue) or incremental cost-effectiveness ratio (ICER) were not excluded from the review, they were not classified as CBAs. Additionally, the analyses should reflect an assessment of opportunity costs. That is, any EMR implementation inevitably presented a trade-off of resource uses: diverting scarce resources of labor, capital, and materials from alternative projects (e.g., other interventions or productions of healthcare services) to the EMR project that contributes to increasing volumes and/or quality of healthcare services in the future.

Stage 2—inclusion criteria and search strategy

The literature search was conducted in February 2021, following the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) guidelines¹², supplemented by the PRISMA checklist¹³ for scoping review (see Supplementary Table 2). Search terms are also listed in Supplementary Table 1.

The two main electronic databases used were MEDLINE and Econlit. Additionally, reference lists of identified publications, some key journals (e.g., the Journal of Benefit Cost Analysis) and relevant websites (organizations, conferences, and networks) were hand-searched for additional articles that might meet the inclusion criteria.

The inclusion criteria strictly followed the research questions identified in Stage 1. That is: studies that (i) evaluated EMR or EHR or eHealth implementation for hospitals, (ii) included both benefits and costs; qualitative discussions accepted as we anticipated that many studies could not quantify all the benefits or costs items deemed important, but (iii) reported at least some quantitative results, either costs or benefits; that is, studies discussing benefits or costs without reporting any quantitative values would be excluded.

Stage 3—study selection

All studies included from the literature search in Stage 2 were imported into Covidence, a platform to conduct literature review in teams (https://www.covidence.org/). The identification of included studies was undertaken by five independent reviewers. First, all duplicates were removed. Second, the first two reviewers (TC, LW) screened the titles and abstracts for eligibility. A study was included if both reviewers agreed that it met the inclusion criteria. Conflicts in reviewers’ conclusions were resolved by discussion and/or examining the full article. Third, full texts of all included studies were reviewed independently by two other reviewers (CW and DS). Reference lists of the included studies were screened by two reviewers (CW and DS) to identify further studies that might have been missed. Published systematic reviews and meta-analyses were examined to ensure that no eligible studies were missed. Both reviewers systematically and independently documented the properties of the included studies and developed their own data extraction tables and notes. Data extraction fields were then compared, discussed and finalized. Each reviewer completed the data extraction process independently, following the agreed data extraction format.

From steps 2 to 4, the screening quality was controlled by an independent reviewer (KHN), who randomly selected and screened papers independently for inclusion/exclusion. All reviewers of each round met, compared and discussed their independent decisions to reach final agreement.

Stage 4—data extraction from included studies

We documented characteristics and quantitative information in each study to synthesize findings. Quantitative information was sorted by key analytical features and themes. All data were entered onto a “data charting form” that recorded both general information about the study (author, year, country, and setting) and specific information related to CBA of interest, that is, analytical method and duration, benefit and cost items, analysis perspective, data type, sample size (if any), statistical methods used (if any), and quantitative main findings.

Stage 5—collation of data and findings

Using the charted data, we grouped the similar findings into common themes to answer the research questions formulated in Stage 1. All benefit and cost items documented in this literature were described and then grouped into categories. We did not limit the number of categories or the number of items to be included in each category in order to understand the diversity in benefit and cost definitions and measures. We reported precisely what were used, from the information available in the studies (including their appendices, if any), and allowed for the themes to emerge naturally.

While interested in benefit and cost values, we did not attempt to conduct meta-regression analysis or “quantitatively synthesize” those values. The quantitative results were, however, summarized and recorded for each study. This is consistent with the scoping review approach whereby quantitative and quality assessment was not the main objective. However, we provided some judgements on whether or not the literature provided robust or generalizable findings that can be useful for future CBA of EMR implementation in hospital settings.

Finally, we attempted to map the objectives and evaluation findings of all studies to the quadruple aims of healthcare delivery. That is, improve population health, enhance patient experience, reduce cost per patient, and improve work-life balance of the healthcare workforce¹¹. Since EMR, like any other healthcare infrastructure and investment, has been developed to achieve the ultimate outcomes of healthcare delivery, it is crucial to benchmark the EMR implementation against these aims.

Summary of included studies

The scoping review search resulted in 1184 unique articles. Through the abstract and full-text screening process, a final list of 28 articles were collated. A summary of all 28 included articles is included in Table 1, and the reviewed process is summarized in the PRISMA diagram (Fig. 2). All studies were numbered (see the numbering in Supplementary References) and referred to in the following sections.

The Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) flowchart.

Full size image

While the investigated EMR systems were varied, the majority (17/28)^{(ID#1-2,#6-7,#9,#11-14,#17,#19-22,#24-27)} were categorized under the EMR definition provided by Dranove, et al. (2014)¹⁴, which includes CPOE. A further eight studies being classified as EHR systems^{(ID#4-5,#10,#15,#16,#18,#23,#28)} and two EHR for clinical research (EHR4CR) ^(ID#3,#8) systems were also included within the scope of the literature review.

Studies were all published between 2010 and 2019, with an average of three studies published each year. Generally, there were fewer studies published in more recent years; most studies were published between 2010 and 2014 (17/28). Across the literature, the data used to estimate benefits and costs dated back from 1995 to 2016, with most data from 2000 to 2010. Twenty studies were undertaken in the US^{(ID#4-#6,#9-#11,#13-14,#16-25,#27-28)}, with the remaining studies from OECD countries (Canada^(ID#12), Korea^(ID#1), Israel ^(ID#15), and the European Union^(ID#3,#8)) and three from middle- and lower-income countries (China^(ID#2,#26), Malawi^(ID#7)).

Types of economic analysis or evaluation (Q1)

Studies were classified into four categories: cost–benefit analyses (CBA), cost studies (including financial analyses), impact evaluation (primarily focusing on hospital or patient outcomes), and effectiveness and efficiency analyses (including cost-effectiveness analyses, cost efficiency, and productivity analyses).

Despite a relatively rich literature on the impact of EMR implementation, only three studies^(ID#1-#3) were qualified as (full) CBA studies (following the definition in Stage 1). The unit of analysis is a hospital where EMR was implemented. These studies presented benefits and costs in dollar values and presented estimates of net benefit. Some benefit and cost items were valued using nonmarket prices (i.e., economic or shadow prices) as opposed to financial values (or market prices). A short description of these studies is presented in Box 1.

The second category—“cost studies”, including financial analyses—primarily examined the cost or financial expenses of EMR systems (7/28)^{(ID#5-8,#10,#13-14)}. Among those, some had a slight feature of a CBA by incorporating both aspects cost and revenue resulting from EMR implementation. The reported results were often in net cost or differences in cost between EMR-based hospitals and those without or pre EMR. These studies were conducted at both patient and hospital level, i.e., the unit of analysis was either hospitals or individual patients admitted to hospitals.

“Impact evaluation” studies examining the outcomes by stakeholders (either outcomes for patients, or staff, hospitals, society, or health system) resulting from the EMR implementation: 12 studies out of 28 ^{(ID#4,#16-17,#19-26,#28)}. Studies looking at “cost-savings” were included in this group when the cost-savings were described as a result of changed/improved (patient) outcomes or reduced wastes. Similar to cost studies, the unit of analysis was either individual patients or hospitals.

Last, we grouped “cost effectiveness” and “efficiency and productivity analysis” studies into the last category (6/28 studies). These studies considered both the cost and outcomes of EMR implementations (i.e., full evaluation) but did not quantify them in terms of monetary values (hence not CBA). Three cost-effectiveness analyses^{(ID#11–12,#15)} estimated costs, quality-adjusted life years (QALYs), and presented incremental cost-effectiveness ratio (ICERs) for a targeted hospital patient population. Two studies^(ID#9,#27) examined cost efficiency resulting from EMR implementation and one investigated labor (nurse) productivity improvement^(ID#18). The unit of analysis in these studies was hospitals in the United States.

Key information for all studies is presented in Tables 1 and 2, and findings of the three CBAs are summarized in Box 1.

Evaluation scope and measurements of benefits and costs

As shown in Table 2, the unit of analyses varied from services (lab tests, examinations) to patients, hospitals, and hospital sector. The variation naturally led to different choices of measurements and analytical methods. Indicators that measure the same impacts (e.g., clinical outcomes such as adverse events or mortality), were defined at both patient level (disaggregate) and organization/system level (aggregate).

Impacts could be sorted into six domains, which were then mapped to the quadruple aims of health care (see Table 3, and further discussion on the quadruple aims in the “Discussion” section): (1) patient outcomes and experience, (2) hospital outcomes, (3) health-system (sector) outcomes, (4) cost specific to EMR implementation, (5) hospital resource utilization and (6) productivity and efficiency measures. These six domains have some overlaps because many indicators can be used to measure outcomes at multiple levels: patient, organization/hospital, and health system/sector. As shown in Table 3, within each domain, multiple measures and instruments were used for the key outcomes of interest. For instance, death/mortality risk and adverse events are key outcomes for patients, hospitals, and population health. Specifically, within “patient outcomes” domain, the impact of EMR on adverse drug events (ADEs) was measured by the counts of ADEs in elderly patients or less complex patients; when examined at the hospital level (i.e., hospital outcomes domain), the impact of EMR on patient safety was then measured by the mortality rate after ADEs. Similarly, cost per episode of care can be measured by admission (patient level) or as average cost per patient (hospital level), or cost per casemix-adjusted episode of care (health-system level). The partial productivity measures (efficiency and productivity index) are special cases where both outcomes and costs were included in the measure, therefore, warranting its own domain.

Nonetheless, there are some clear demarcations between these domains. Patient outcomes were primarily observed through deaths (count of), adverse events (count of, or rate of), quality adjusted life years (QALYs), and outcomes at the organization level (hospital) focused on key performance indicators such as “length of stay” and “medication errors” (count of, or rate of) or “inefficiency score”. System-level outcomes (healthcare sector and society, overall) have been on population-based “mortality rates” (of mother, children, and older people). A large number of studies examined the costs of EMR implementation through initial upfront monetary costs (financial investment) and operating (recurrent) expenditure. These are mostly financial analyses of EMR impact. The domain of “hospital resource utilization”, which was associated with cost savings in a large number of studies, focused entirely on changes in tests, procedures, and consumable goods (including paper, medical supplies, and prescriptions). These studies might quantify the impact into cost, but the primary interested was the estimation of resource savings. However, it is noted that very few studies provide unambiguous definitions of the indicators used to measure impacts.

Findings on the net benefit or net cost (Q3)

The studies generally indicate that there is a positive overall impact from the EMR implementation (see summarized quantitative findings and outlook on EMR, Table 2). Of the 28 studies, 19 indicated positive outcomes from the implementation of EMR and 8 indicated mixed outcomes from EMR implementation, often citing a trade-off between increased quality of care and increasing costs. Furukawa, et al. (2010) posed the only study that indicated EMR implementation was not cost-effective. This study did not discuss any changes in the quality of care provided.

Most studies that focused on the domains relating to the costs of EMR implementation and resource savings indicated a “lag effect”: it often took approximately 2–3 years to clearly observe the benefits of EMR implementation^(ID#26). In addition, three studies indicated that the level of EMR benefits is reliant on the creation of a network effect, both within a hospital ^(ID#13) and across a region of hospitals ^(ID#5,#6). Notably, the benefits of EMR reported in this literature varied with the date of publication. During the period of 2010–2014, 10 out of 18 studies (56%) indicated that EMR systems might be beneficial for healthcare delivery. However, during the period of 2015–2019, 9 out of 10 studies (90%) were favorable toward the EMR implementation.

In this paper, we have reported findings from the first comprehensive scoping review examining the economic impacts of EMR implementation in the hospital setting using CBAs. We wanted to first understand the types of CBAs used to evaluate EMR implementation; second, to determine how comprehensive the analyses were, and how the benefit and cost items were identified, measured and valued; and third, to determine whether the literature found that the benefits of EMR implementation justify the investment costs.

The review showed that despite a relatively rich ten-year literature, only three studies^(ID#1–#3) qualified as CBA studies, the rest focused on impact evaluation, cost/financial analysis, benchmarking (e.g., efficiency analysis) or evaluation of interventions that were EMR-powered^{(ID#13–14,#17)}. More than two-thirds of the studies originated from the United States, following the Obama Care boost, and in favor of statistical analysis approaches rather than CBA. Only three studies originated in Europe, while none was from the United Kingdom. Of the three studies conducted in Europe, one of them was a CBA, despite its narrowed scope (impacts of hospital EMR on research and development). The other two CBA studies, in China and Korea, investigated a wider scope of impacts; however, the measurement and valuation of impacts were limited to financial impacts or cost savings from reduced adverse events. Patient outcomes and experience, changes in healthcare workforce as the results of EMR implementation were either not measured or valued. These limitations were acknowledged in all studies.

While most studies pointed toward a positive impact of EMR implementation, such impact is not quantitatively comparable across studies (as summarized and described in Tables 2 and 3). This is attributable to three main reasons: heterogeneity of impact measures (both benefits and costs), different implementation conditions (despite all in the hospital setting), and a large variety of analytical methods employed to understand the economic value of EMR, thus biases associated with those methods. As EMR adoption continues to spread, the reproducibility of findings that allow comparative analyses across settings and healthcare systems is critical to know what works and what does not, how, and why.

First, there were a wide range of analysis units, measures and indicators used to report on the impacts of EMR implementation, from patient to health system levels (as summarized in Tables 2 and 3). Comparable analyses will require increasing standardization of measures of impacts at all levels. Standardization of measures is particularly important as uncertainty existed within the literature as to what mechanisms EMR acts on to create better safety outcomes for patients. Encinosa et al. (2012), gave an analogy of EMR acting in the role of a “car airbag” that mitigates damage once an adverse event has occurred, and the CPOE systems act similarly to the role of a car’s “electronic stability controls” in reducing complications^(ID#10). While there is substantial evidence that an effective EMR system allows for reductions of medical errors and waste, the causal links from EMR implementation to ultimate patient outcomes such as reduced mortality rate and shorter length of stay are less well understood in the literature. It is proposed that causal links might be better understood if studies can use direct and more precise health outcomes such as vital signs (e.g., body temperature, pulse, and respiration rates) to mediate the impact of EMR on ultimate outcomes (e.g., mortality rates). In any evaluation (especially economic evaluation), understanding the direction and magnitude of change (quantitative) can be equally important as the mechanism of change (e.g., qualitative through understanding the theory of change). The quantitative measures, when measured with standardized instruments across time and implementation units, can highlight the success/failure for lessons learnt and/or further investigations. Additionally, due to current misalignment between evaluation focuses and healthcare aims, the impacts of EMR on quality of care, patient experience, and the healthcare workforce were largely omitted (see Table 3, and further discussion below).

Second, the review identified a very large degree of heterogeneity of implementation conditions and types of EMR systems within the hospital setting. Variance in the latter is especially prominent in studies using large-scale, hospital and panel data. Due to the rapid development of EMR, later adopters were likely to use more sophisticated technologies compared with early adopters, allowing for better capture of the impacts. Variance in the former (across settings/adoption situations) was also noted in the literature, mostly related to the geographical area of adoption. For instance, Dranove et al. (2014), demonstrated that the benefits accrued to EMR were reliant on the access to and use of ICT within the local community^(ID#6). On the other hand, developing nations and low-efficiency hospitals were shown to benefit from EMR implementation, despite their relatively low level of ICT (see further discussion in theme 6). These seemingly contradicting findings suggest that the large heterogeneity in impact measurement makes it challenging to compare all the results across all the study settings.

Third, studies employed a wide range of economic and statistical methods to understand the value of EMR implementation. These methods ranged from economic evaluation methods such as CBAs^(ID#1–3), cost-effectiveness analyses^{(ID#11–12,#15)}, and efficiency and productivity analyses^{(ID#9,#18,#27)} to econometric models such as difference-in-difference^(ID#24), a system of cost equations^(ID#5), or statistical analysis such as multivariate regressions^{(ID#6,#11,#12,#13,#16,#19,#20,#21)}, interrupted time series^(ID#4,#26), generalized linear models^(ID#10), and logistic regressions^(ID#18). These models included different sets of variables and measures, many of which were specific to the datasets used or the data-collection context. All studies acknowledged methodology limitations and the potential biases (on the impact estimates of EMR implementation) resulting from the modeling and data limitations. The main sources of biases include self-selection bias of hospitals implementing EMR, self-reporting of EMR usage, and “conservative” estimates of benefits.

Self-selection bias was often examined in studies that utilized longitudinal data, in which investigators attempted to control for the self-selection endogeneity through fixed-effect models. There was a wide range of hypotheses regarding the correlates of EMR adoption in hospitals. Zhivian et al. (2012) proposed that inefficient hospitals were more likely to adopt EMR as the potential improvements were greater than in hospitals that were already more efficient^(ID#27). Other studies linked EMR adoption with teaching hospital status^(ID#18) or whether the hospital belonged to a community fund-raising scheme^(ID#20). As such, it is unclear if the endogeneity would lead to the under- or over-estimation of the EMR impacts, resulting to limited ability to interpolate EMR as a causation for the measured outcomes^(ID#9).

EMR use was commonly self-reported both by clinicians and by the organizations collectively (i.e., hospitals)^(ID#16). Self-reported savings would inevitably lead to large uncertainty around the estimation of both EMR-related benefits and costs due to self-selection bias. Clinicians who engaged in cost-saving activities could potentially overlook other benefits or costs if they were not directly or clearly related to the cost-saving targets at hand. Conversely, it was believed that hospital response rates were dependent on the successes of EMR, leading to a potential overestimation of EMR benefits.

Many studies stated that they relied on “conservative estimates” when evaluating the benefits of EMR^{(ID#1–2,#7,#8)}. Conservative estimates were applied to the loss of productivity following the initial implementation of the EMR system, in two of the CBAs, with predictions for productivity loss higher than expected^(ID#1–2). Similarly, studies also used “conservative estimates” in deciding on the counterfactual with costs associated with the counterfactual underestimated to conservatively measure the financial impact from EMR implementation^(ID#7–8).

Outside the three main questions of the scoping review, we discovered three additional important topics that warrant future research (see Fig. 3). First, the maturity of EMR implementation in the past two decades and how it would affect future evaluation of EMR. Second, the evidence of economies of scale and scope with respect to the impacts of EMR implementation. Third, the (incomplete) alignment of selected evaluation approach/framework with the quadruple aims of healthcare delivery.

Emerging themes from the literature of economic evaluation and analysis of hospital-based electronic medical records.

Full size image

Maturity of EMR implementation

The literature overwhelmingly indicates that most EMR systems were still maturing^15,16,17. The full extent of the costs and benefits of EMR implementation could not be understood, measured, and valued using the current metrics or data-collection systems, both were outdated. While publication dates were within the last 11 years (since 2010), most data used for the economic evaluation came from the prior decades (late 1990s–2010).

Despite its relatively short time, a trend has emerged, that is, increasing benefits over time were observed as EMRs slowly matured. Initial studies with data sourced during the late 1990s found that the promised cost-savings from an administration perspective were not delivered through EMR implementation^(ID#6,#9). However, later evaluations that accounted for improvement in care quality, using evidence sourced from the 2000s, indicated that EMR provided quality increases at a competitive cost^{(ID#14,#25–26)}. In other words, the most recent literature found that EMR is cost-effective. Last, more recent CBAs, sourcing data from the period 2006–2012, indicated that the EMR implementation has become a solution that is financially viable, even when studies did not fully account for improved patient outcomes^(ID#1-3).

This change in viability can be attributed to five main factors. First, the capabilities of EMR technology have vastly developed throughout this timeframe. This is perhaps best reflected in the changing role of EMR. Initially, the system was developed and utilized as a system to keep track of resource utilization, costs, and outcomes^(ID#6); however, improvements in technology have enabled EMR to provide clinical support and hence provide greater utility to hospitals^(ID#22). In parallel, the system has been refined over time to improve the cost-saving utilization. ^(ID#4,#13). Second, clinicians have become more accepting of EMR implementation over time. Clinician uptake and technical ability have been marked as a key determinant in the success of EMRs. Third, the role of network effects has led to increasing benefits and decreasing costs associated with EMR^(ID#5) Fourth, there is a potential that EMR can contribute to developing improved procedures and workflow through increasing the amount of information available. Furthermore, there is the potential of EMR to be used as the building block for improving medical technologies through the emergence of predictive and prescriptive analytics driven by machine learning and artificial intelligent developments.

In addition to the natural progression of EMR maturity, developments have been made in the systems to support the EMR implementation, which together has led to increasing benefits over time. In the literature, this was often discussed as “workforce training” and “development of ICT services”. Although the perceptions of physicians towards the benefits of EMR remain widely predetermined¹⁸, larger productivity gains can be made through EMR, especially when physicians are incentivized by sharing in the benefits from the gains^(ID#18) However, EMR implementation has been consistently associated with an implementation lag where disbenefits (e.g., lower productivity) are accrued during the training phase^(ID#2,#6). The duration of the disbenefits in hospital efficiency studies has been linked with the level of general ICT adoption within a region, suggesting that the prevalence of complementary ICT skills within the community acts to both minimize system costs, and to increase benefits. As such, an increasing use of ICT systems in all operating aspects of society in the future is expected to further benefit the EMR systems.

Returns to scale and scope

Another common theme relating to returns to scale and scope is that present EMR technologies offer highly effective solutions for simple medical problems but show decreasing effectiveness for highly complex problems. In economic terms, studies found strong evidence of increasing returns to scale and scope of the EMR implementation in small and simple hospital systems but decreasing returns (approaching negative) in large and complex systems. The evidence was visible at both national health system and within the context of a hospital. For instance, great success has been observed in the implementation of the EMR in the Kumuzu Central hospital in Malawi^(ID#7). In these systems, with their relatively low stock of ICT capital and labor available, the marginal returns (e.g., improved patient processing and treatment efficiency) from the EMR adoption were significant. Similarly, a US study noted a phenomenon that EMR applications showed their highest effectiveness in the scenario of noncomplex cases^(ID#22).

The mechanism driving increasing returns to scale and scope for simple and small medical problems or systems is intuitive. For instance, if all patient information is well captured and linked in a unified EMR system (between departments within a hospital, or across hospitals within a health district/region), when a particular patient repeatedly presents at different locations, it requires less time spent on collecting past medical history and relevant personal information—examples include allergies, comorbidities and complexities. This allows faster and more effective treatment decisions while accruing much lower cost of acquiring patient information (i.e., cost of both medical staff and patient’s time and effort). However, the mechanism that drives decreasing (even negative) returns scale and scope, i.e., EMR showing minimal additional effectiveness in complex cases, is less certain. A common explanation in the literature centers around the lack of precision measures of outcomes in complex situations, such as the contribution of EMR implementation on mortality rate and length of stay^(ID#21–22). For example, medication errors that can be effectively reduced by an EMR system might contribute heavily to the mortality risk of patient of simple casemix; however, a patient with a complex casemix might have very high mortality risk, regardless of whether or not medication errors occur.

The alignment of EMR evaluation with the quadruple aims of healthcare delivery

Our review found that the current literature of economic evaluation of EMR did not sufficiently address the quadruple aim of healthcare delivery, that is, improve population health, enhance patient experience, reduce cost per patient, and improve work-life balance of the healthcare workforce¹¹.

Of the 28 studies included in the scoping review, no study explicitly addressed all aspects of the quadruple aim of health care, particularly the impacts of EMR implementation on the healthcare workforce (see Table 3). Considering the high levels of clinician burnout partially attributed to EMR often discussed in noneconomic studies, a gap in the economic literature exists: that is, quantifying the impact of EMR on the work-life balance of the healthcare workforce. Another important aim of healthcare delivery—patient experience—was addressed in only one study. Hydari et al. (2019)^(ID#24) observed qualitative benefits in “increased confidence in the health system” (by patients) after EMR adoption, no studies contemplated the patient experience beyond health outcomes or quality-adjusted life years (through the QALY estimates).

A potential explanation for this relatively narrow focus of EMR evaluation to date is the maturity of digital health across settings and countries. The economic development and project-management literatures have shown that piece-meal project implementations, even with best intentions in mind, but without proper alignment to overall goals and purposes of the sector, can result in high wastage and undesirable impacts, especially when projects compete for the same labor and capital resources to produce the same outputs^19,20.

Project implementations face constraints of time, budget (allocated funding and inputs), and set target outputs/results to achieve (number of staff trained, number of machines installed, or go live dates, etc.). If their success is defined by a narrow set of input–output metrics, rather than measures aligned with sector purposes (i.e., who will benefit, what will change systematically as a result of the project), and goals (i.e., broader sector goals of healthier population, improved patient and staff experience, and reduced cost per capita), many projects might achieve their targeted results (outputs) without contributing to achieving sector purposes and goals. Therefore, project design and implementation without proper attention to purposes and goals (and the verification indicators that allow them to measure the contribution of the project outputs to changes and impacts at a higher level) tend to fall into the situation of “crowding out”, that is, multiple projects competing for already-scarce resources (i.e., labor/workforce time, existing facility, hardware system, and so on). Such phenomenon of micro–macro paradox has been observed and studied in many economic sectors, including health²¹. That is, individual projects showing success: outputs produced as planned within the budget and timeline—but overall, there are no substantial changes toward the desired outcome—such as population health improvement, improved patient experience, etc.

It has been shown that using a systematic approach to project-cycle management that is tightly linked to sector purposes and goals and within sector-wide approach framework can result in desirable outcomes in the medium- and long term (i.e., sustainable)^{19,20,22,23,24}. While the sector-wide approach was developed specifically in relation to development practices, the concept (and its implementation tools) is translatable to other settings/practices. That is, all stakeholders within a sector (funders, lenders, providers, government agencies, and users) implement projects that collectively contribute to achieve the desirable sector goals and purposes, in order to harmonize the efforts, avoid overlapping and wastage, and more importantly magnify each other’s outputs or results. In the case of digital health, it is essential that the EMR implementation, within an organization or settings (hospital, acute care, primary or aged care), starts with the healthcare aims (sector goals) and purposes (operationalized goals).

While the review scope was limited by its focus, i.e., EMR implementation in hospital only rather than EMR/EHR for the whole healthcare system, the diversity of hospital-specific settings, research questions, and methodological analyses posed a challenge to synthesize and present all the information that might be of use for decision-makers and hospital managers. Additionally, the grouping of study types as well as impact domains is limited to our interpretation of the information presented in the original studies. Last, there might be studies qualified to be included, yet missed, in the process of review. A future update and additional in-depth examinations of specific themes identified in this review would undoubtedly progress the research frontier in EMR implementation and economic evaluation.

Electronic medical records and digital hospitals are becoming a crucial component of healthcare infrastructure and delivery. This trend is likely to accelerate as the technology improves, and digital and artificial intelligent-based technologies become more precise and sophisticated. More investments will be diverted toward the EMR implementation. Yet, an essential part of a valid and reliable business case—economic feasibility studies—remains underdeveloped, as shown in our comprehensive scoping review. Current literature demonstrates a lack of an appropriate economic framework to understand the aims and value of the digital hospital implementation; studies only managed to identify and measure minor benefits and cost. Business cases resting on those narrow analyses, understandably, are failing to gain traction in jurisdictions that still consider EMR rollouts under financial and economic constraints. Further to these concerns is the common observation that disbenefits occur in the short term after implementation; however, such concerns are short-sighted and akin to avoiding a new gym-exercise routine in fear of the inevitable muscle soreness in the days following the first sessions.