- Research article
- Open access
- Published:
Assessing measures of comorbidity and functional status for risk adjustment to compare hospital performance for colorectal cancer surgery: a retrospective data-linkage study
BMC Medical Informatics and Decision Making volume 15, Article number: 55 (2015)
Abstract
Background
Comparing outcomes between hospitals requires consideration of patient factors that could account for any observed differences. Adjusting for comorbid conditions is common when studying outcomes following cancer surgery, and a commonly used measure is the Charlson comorbidity index. Other measures of patient health include the ECOG performance status and the ASA physical status score. This study aimed to ascertain how frequently ECOG and ASA scores are recorded in population-based administrative data collections in New South Wales, Australia and to assess the contribution each makes in addition to the Charlson comorbidity index in risk adjustment models for comparative assessment of colorectal cancer surgery outcomes between hospitals.
Methods
We used linked administrative data to identify 6964 patients receiving surgery for colorectal cancer in 2007 and 2008. We summarised the frequency of missing data for Charlson comorbidity index, ECOG and ASA scores, and compared patient characteristics between those with and without these measures. The performance of ASA and ECOG in risk adjustment models that also included Charlson index was assessed for three binary outcomes: 12-month mortality, extended length of stay and 28-day readmission. Patient outcomes were compared between hospital peer groups using multilevel logistic regression analysis.
Results
The Charlson comorbidity index could be derived for all patients, ASA score was recorded for 78 % of patients and ECOG performance status recorded for only 24 % of eligible patients. Including ASA or ECOG improved the predictive ability of models, but there was no consistently best combination. The addition of ASA or ECOG did not substantially change parameter estimates for hospital peer group after adjusting for Charlson comorbidity index.
Conclusions
While predictive ability of regression models is maximised by inclusion of one or both of ASA score and ECOG performance status, there is little to be gained by adding ASA or ECOG to models containing the Charlson comorbidity index to address confounding. The Charlson comorbidity index has good performance and is an appropriate measure to use in risk adjustment to compare outcomes between hospitals.
Background
Comparing outcomes between clinicians, institutions or area health services requires consideration of potential confounding factors, such as patient age, health status or disease severity, that could account for any observed differences in outcomes. Risk adjustment is a statistical technique used to account for variations in outcome that arise from differences in the characteristics of patients, so as to provide a fair and more accurate comparison of health services. Patient age, sex, cancer site and cancer stage are commonly used in risk adjustment models as they have high levels of completeness and accuracy in routinely collected datasets such as cancer registries and hospital episode collections.
Adjustment for comorbid conditions is common when studying outcomes following cancer surgery [1]. The Charlson comorbidity index summarises information about the presence of a number of medical conditions, assigns a severity score to each, and sums the severity scores to create a single measure of comorbidity [2]. Algorithms have been derived to calculate the Charlson comorbidity index from hospitalisation records coded using the Ninth [3, 4] and Tenth [5, 6] Revisions of the International Classification of Diseases. The Charlson comorbidity index has been shown to be a good predictor of both short-term (in hospital and 30-days) and longer-term (12-month) mortality [7].
Functional status is an additional patient factor that may explain differences in outcome. One measure of functional status is the Eastern Cooperative Oncology Group (ECOG) Performance Status, [8] a score ranging from 0 (“fully active”) through 3 (”capable of only limited self-care”) to 5 (“dead”). An alternative measure of a patient’s general health is the American Society of Anesthesiologists (ASA) score [9]. This measure of physical status ranges from 1 for a normal healthy patient to 5 for a moribund patient who is not expected to survive surgery and is recorded routinely by anaesthetists prior to any surgical procedure. ECOG and ASA scores are available in some administrative data collections.
While recent studies have assessed the predictive capacity of comorbidity and health status measures in risk adjustment for outcomes of cancer surgery, [10, 11] they did not consider performance status. A study that compared comorbidity index, ASA and ECOG performance status [12] considered each measure separately, and did not assess the additional effect of health and performance status over comorbidity. In a recent study comparing ASA with ECOG, [13] no comparison was made with comorbidity index as it was not possible to derive a comorbidity index from the available data. Additionally, these studies used data extracted from patients’ medical records, not routinely collected data.
Therefore, we conducted this study to assess how frequently the ASA score and ECOG performance status can be obtained from population-based administrative data collections in New South Wales, the most populous state in Australia, and to assess the contributions of ASA score and ECOG performance status additional to the Charlson comorbidity index, in comparing outcomes of colorectal cancer surgery between hospital peer groups based on risk adjustment models.
Method
Data sources
Our study population comprised all people with an incident cancer of the colon or rectum registered in NSW, the most populous state in Australia. We used the first recorded tumour of the colon or recto-sigmoid junction (ICD-O-3 codes: C18, C19) or rectum (C20) registered on the NSW Central Cancer Registry (CCR) between 1 January 2000 and 31 December 2008. The NSW CCR, a population-based register of all cancers in NSW residents, receives notifications from public and private hospitals, departments of radiation oncology, nursing homes, pathology laboratories, outpatient departments and day procedure centres as a statutory requirement.
Cancer notifications were linked to the Admitted Patients’ Data Collection (APDC), the Clinical Cancer Registry (ClinCR) and the NSW Registry of Births, Deaths and Marriages (RBDM). The APDC contains demographic and episode-related information on all hospital separations for public and private hospitals in NSW, [14] while the ClinCR contains clinically-based information, including the ECOG performance status, for patients with cancer treated in public facilities within six area health services [15]. Hospital admissions with procedures defined as being for curative intent were used, [15] with these procedures based on clinical consultation and previously used procedure codes [16, 17]. Where patients had more than one hospital admission with a procedure of interest, the first admission was used in analysis. We used hospital records with separations in 2007 and 2008, as this was the period for which clinical cancer registry data were available. Fact of death was obtained from the RBDM for the period up to 2010.
Probabilistic data linkage was conducted by the NSW Centre for Health Record Linkage, and we received only anonymised data. Ethics approval for the study was given by the NSW Population and Health Services Research Ethics Committee (HREC/11/CIPHS/32).
Outcomes
We considered three binary outcomes. Mortality was assessed at 12 months, measured from date of procedure until death from any cause. Hospital stay following surgery was measured from the date of procedure until separation, accounting for transfers, and categorized based on exceeding the recommended length of stay for procedures based on the Australian Refined Diagnostic Related Group [18]. This allowed for length of stay to be assessed based on procedure complexity. Readmission within 28 days was derived from the date of separation until any subsequent readmission.
Peer group
Peer groups classify similar hospitals based on the number of separations, primary role and location (metropolitan or rural). Admitting hospitals were classified into one of six groups based on NSW Peer Group definitions. [19].
Measures of patient health
We calculated the Charlson comorbidity index using codes for comorbid conditions recorded during the index admission in the APDC. We excluded metastatic cancer from the calculation as in the context of cancer treatment, this is part of the condition of interest rather than a comorbidity. Charlson comorbidity indices were categorised into values of 0, 1 and 2 or more in risk adjustment models as very few patients (8 %) had scores of two or higher.
The ASA score was obtained from the suffix of ICD procedure codes indicating administration of an anaesthetic (see Additional File 1). Possible ASA scores are 1 (normal healthy patient); 2 (mild systemic disease); 3 (severe systemic disease); 4 (severe systemic disease that is a constant threat to life) or 5 (moribund patient who is not expected to survive) [9]. We grouped ASA scores of 4 or 5 in risk adjustment models due to the sparseness of higher scores.
ECOG performance status was obtained from ClinCR records. We used the ECOG performance status recorded closest in time on or before the data of procedure. Possible ECOG scores are 0 (fully active, no performance restriction); 1 (restricted in physically strenuous activity but ambulatory, able to carry out work of a light or sedentary nature); 2 (ambulatory and capable of all self-care but unable to carry out any work activities. Up and about more than 50 % of waking hours); 3 (capable of only limited self-care, confined to bed or chair more than 50 % of waking hours); 4 (Completely disabled, cannot carry out any self-care. Totally confined to bed or chair) or 5 (dead) [8]. We grouped ECOG performance status of 2 or higher in risk adjustment models due to the sparseness of higher scores.
Other patient factors
Extent of tumour spread, obtained from the CCR, summarises the most aggressive extent of the disease based on diagnostic and therapeutic evidence within four months of diagnosis [20]. Patient sex, age and emergency status of admission were obtained from the APDC, with age calculated at date of procedure. Country of birth, also obtained from APDC, was classified as Australian versus other.
Statistical analysis
Descriptive statistics were used to compare patient characteristics for patients with non-missing values of Charlson comorbidity index, ASA score and ECOG performance status. Associations between the three patient health measures were summarised using Spearman’s rank correlation coefficients.
Risk adjustment models were constructed using two-level multilevel logistic regression models to compare the binary outcome measures across the peer groups, with patients (level-1) clustered within hospitals (level-2). Random-intercept models were fitted, with hospital included as a random effect. Only patients with non-missing values of Charlson comorbidity index, ASA score and ECOG performance status were used so that risk adjustment models could be compared on the same group of patients. Hence we used only records from patients treated in public hospitals. As such, only the four peer groups representing public hospitals were considered: Principal Referral A, Principal Referral B, Major Metropolitan and Major Non-Metropolitan. Major Metropolitan and Non-Metropolitan peer groups were grouped together due to the low patient numbers in Major Non-Metropolitan hospitals.
We evaluated the utility of ASA score and ECOG performance status by fitting all combinations of these over and above a base model assessing the association between hospital peer group and each of the three study outcomes (12-month mortality, extended length of stay and readmission within 28 days) adjusted for patient age, sex, extent of disease, emergency status of admission and Charlson comorbidity index. Model performance was assessed using Akaike’s information criterion (AIC), with lower values indicating a better fitting model. We also calculated the C statistic, summarising the concordance between observed and predicted events. Concordance ranges between 0.5 (a model with no predictive ability) and 1.0 (a model with perfect predictive ability) [21]. Finally, we compared estimated logistic regression parameters for hospital type to assess the extent of confounding explained by the inclusion of ASA score and ECOG performance status.
Data processing and descriptive analyses were conducted using Version 9.3 of the SAS System for Windows (Cary, NC, USA), with multilevel logistic modelling conducted using adaptive quadrature via the xtmelogit command of Stata Version 12 for Windows (College Station, TX: StataCorp LP).
Results
We obtained records from 6,964 individuals who were operated in 2007 or 2008, and their characteristics are presented in Table 1. The Charlson comorbidity index could be calculated for the entire sample, as it is based on comorbid conditions recorded during the index admission. ASA score was recorded for 78.0 % of these individuals, and while they were similar to those without ASA score recorded for most characteristics, those with ASA score recorded were more likely to have been operated on in a public hospital.
There were 3,848 individuals who had a potentially curative surgical procedure at a public hospital in 2007 or 2008. Of these, 3,079 individuals had a record in the Clinical Cancer Registry and 748 (24.3 %) had an ECOG performance status recorded before their cancer surgery. Patients with a recorded ECOG performance status were more likely to be male, have a non-Australian country of birth, and have a more advanced extent of disease compared to those without ECOG performance status.
The three measures of patient health status were not highly correlated with one another (rASA,Charlson = 0.29, rASA,ECOG = 0.15, rECOG,Charlson = 0.11) (Table 2).
There were 575 records with non-missing measures of health status and other covariates that could be used in risk adjustment comparisons. The majority had Charlson scores of 0 (n = 502, 87.5 %) with 25 (4.4 %), 38 (6.6 %), 3 (0.5 %) and 6 (1.0 %) scoring 1,2, 3 and 4 respectively. There were no Charlson scores higher than 4. Results of model performance appear in Table 3. For each outcome, the estimated intracluster correlation coefficient was zero for all models.
Including ASA score and ECOG performance status reduced the AIC for each outcome, but the effect was not consistent. The AIC statistic was minimised by: the inclusion of both measures for 365-day mortality; the inclusion of ASA score for extended length of stay and the inclusion of ECOG performance status for 28-day readmission. Concordance was maximised by the inclusion of both measures for all outcomes. However the maximum concordance for each outcome was similar to that of the best performing models based on AIC statistics.
The effect of including ASA score and ECOG performance status on the logistic regression parameter estimates for hospital-type is summarised in Table 4. Changes in parameter estimates due to adjusting for different measures of health status were small, relative to the parameter estimates’ standard errors, and there was no consistent pattern in the confounding effect of ASA score and ECOG performance status.
Discussion
Our study reports on the utility ASA score and ECOG performance status for risk adjustment over and above Charlson comorbidity index using routinely collected, population-based data for NSW. The correlation between the three measures was not strong, suggesting that the measures represent independent attributes of patient well-being. Our study found that, for patients with non-missing Charlson comorbidity index, ASA score and ECOG performance status, there was no clear optimal measure of patient-health for risk adjustment. However, there was no clearly deficient measure. We also found that, after including Charlson comorbidity index, there was little to be gained by adding ASA score or ECOG performance status when comparing outcomes between hospital types.
Within the NSW hospital episode statistics (the APDC), it is standard coding practice that only comorbid conditions which have an impact on the admission of interest are coded. While this means that the Charlson scores derived from the APDC diagnosis codes are based on conditions likely to impact outcome, it leads to under-estimation of the prevalence of comorbid conditions in the patient cohort [22]. We have previously investigated the impact of including additional comorbidity information from previous hospital admissions in risk adjustment models to compare hospital cancer outcomes [13]. This previous work demonstrated that although Charlson scores were higher when more sources of comorbidity information were included, there was little change in the performance of Charlson score in the hospital risk adjustment models. Thus, for comparison of hospital cancer outcomes, calculating the Charlson score from comorbidity information within the index admission was the most efficient approach, and the approach used in this study. We acknowledge however that additional comorbidity information from other sources could improve risk adjustment models in other contexts, for example to compare individual patient outcomes.
While ASA score is reported more commonly in public hospitals than private, patients with ASA score have similar characteristics to those with no recorded ASA score. The reporting of ECOG performance status however, is limited. Only a quarter of patients whose cancer surgery was conducted in a public hospital had an ECOG performance status recorded before their cancer surgery. While we used the ECOG performance measured closest in time prior to cancer surgery, the ideal would be to record ECOG close as possible to surgery, possibly in a preoperative setting. As the analyses in this study were necessarily restricted to a subset of patients with colorectal cancer in NSW who were treated in a public hospital and had complete data for Charlson, ECOG and ASA, it is possible that these findings do not generalise to other patient groups. This should be further investigated in future studies.
The findings from this study regarding predictive ability agree with those from other published work. While comorbidity (as measured by either ASA or Charlson comorbidity index) has been demonstrated as a risk factor for poor outcomes in studies of colorectal cancer surgery, different comorbidity indices added little to the predictive ability regression models, [10] and yielded strikingly similar results [11]. Similar performance of ASA, Charlson comorbidity index and ECOG performance status was seen in a study that modelled90-day post-surgery mortality for carcinoma of the bladder [12]. It should be noted though, that these studies were based on data extracted from primary medical records. Recently, cancer-specific measures of comorbidity (‘C3 indices’) developed using administrative data have been found to have slightly improved performance compared with the Charlson index for colorectal cancer [23]. However, the developers conclude that the differences are so small as to be insignificant, and that any measure of comorbidity will suffice in most circumstances [23]. Consistent with these findings, our study suggests that alternative measures of patient health derived from administrative data collections provide similar predictive ability.
While our study is based on population-level data from high quality data collections, there are a number of limitations. There were only two years of data with the potential for ECOG performance status to be measured, and only a small proportion of patients had ECOG performance status recorded. We could not include patients who received surgery in private hospitals, as ECOG performance status was recorded in the Clinical Cancer Registry which does not yet include private hospitals. Those with non-missing ECOG performance status differed systematically from the entire cohort, and it is not clear whether the same conclusions would be reached in a more representative population. While multiple imputation methods could be used to include patients with missing data, the high proportion and likely non-random nature of the missing ECOG data in our study were against this approach [24]. We could not assess thirty-day mortality, arguably a more important indicator of surgical quality, in the risk adjustment models due to the low numbers of deaths. Finally, the levels of completeness of data fields likely vary in different settings such as the public and private sector, and within and between countries, so our findings may not generalise to other contexts.
Conclusion
ECOG performance status has some use in risk adjustment, but its routine use in population-based comparisons of hospital outcomes cannot be recommended due to limited coverage. Coverage of ASA scores is greater than for ECOG performance status, but there are still substantial issues of non-reporting, particularly in private hospitals.
While predictive ability of regression models is maximised by inclusion of one or both of ASA score and ECOG performance status, there is little to be gained by adding ASA or ECOG to models containing the Charlson comorbidity index to address confounding. The Charlson comorbidity index is an appropriate measure to use in risk adjustment to compare outcomes between hospitals.
Abbreviations
- ECOG:
-
Eastern Cooperative Oncology Group
- ASA:
-
American Society of Anesthesiologists
- NSW:
-
New South Wales
- CCR:
-
Central Cancer Registry
- ClinCR:
-
Clinical Cancer Registry
- APDC:
-
Admitted Patients’ Data Collection
- RBDM:
-
NSW Registry of Births, Deaths and Marriages
- AIC:
-
Akaike Information Criterion
References
Sarfati D, Hill S, Blakely T, Robson B, Purdie G, Dennett E, et al. The effect of comorbidity on the use of adjuvant chemotherapy and survival from colon cancer: a retrospective cohort study. BMC Cancer. 2009;9:116.
Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40:373–83.
Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol. 1992;45:613–9.
Romano PS, Roos LL, Jollis JG. Adapting a clinical comorbidity index for use with ICD-9-CM administrative data: differing perspectives. J Clin Epidemiol. 1993;46:1075–9.
Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi J, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005;43:1130–9.
Sundararajan V, Henderson T, Perry C, Muggivan A, Quan H, Ghali WA. New ICD-10 version of the Charlson comorbidity index predicted in-hospital mortality. J Clin Epidemiol. 2004;57:1288–94.
Quan H, Li B, Couris C, Fushimi K, Graham P, Hider P, et al. Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am J Epidemiol. 2011;173:676–82.
Oken M, Creech R, Tormey D, Horton J, Davis T, McFadden E, et al. Toxicity and response criteria of the Eastern Cooperative Oncology Group. Am J Clin Oncol. 1982;5:649–55.
Owens WD, Felts JA, Spitznagel Jr EL. ASA physical status classifications: a study of consistency of ratings. Anesthesiology. 1978;49:239–43.
Dekker J, Gooiker G, van der Geest L, Kolfschoten N, Struikmans H, Putter H, et al. Use of different comorbidity scores for risk-adjustment in the evaluation of quality of colorectal cancer surgery: does it matter? Eur J Surg Oncol. 2012;38:1071–8.
Hines RB, Chatla C, Bumpers HL, Waterbor JW, McGwin G, Funkhouser E, et al. Predictive capacity of three comorbidity indices in estimating mortality after surgery for colon cancer. J Clin Oncol. 2009;27:4339–45.
Mayr R, May M, Martini T, Lodde M, Pycha A, Comploj E, et al. Predictive capacity of four comorbidity indices estimating perioperative mortality after radical cystectomy for urothelial carcinoma of the bladder. BJU Int. 2012;6 Pt B:E222–7.
Young J, Badgery-Parker T, Dobbins T, Jorgensen M, Gibbs P, Faragher I, et al. Comparison of ECOG/WHO Performance Status and ASA Score as a Measure of Functional Status. J Pain Symptom Manage. 2015;49:258–64.
Population and Public Health Division. New South Wales Admitted Patient Report. Sydney: NSW Ministry of Health; 2012.
Young JM, Jorgensen M, Dobbins T, Solomon MJ: CESR Technical Report 1: the Quality and Usefulness of the NSW Clinical Cancer Registry Minimum Dataset and Colorectal Dataset Extension for Colorectal Cancer Services Research. 2012 28 October 2013]; Available from: http://hdl.handle.net/2123/8936.
Semmens JB, Platell C, Threlfall TJ, Holman CD. A population-based study of the incidence, mortality and outcomes in patients following surgery for colorectal cancer in Western Australia. Aust N Z J Surg. 2000;70:11–8.
Thompson BS, Coory MD, Lumley JW. National trends in the uptake of laparoscopic resection for colorectal cancer, 2000–2008. Med J Aust. 2011;194:443–7.
Casemix Policy Unit: Costs of Care Standards 2009/10 - NSW Health. [Policy] 2011 2013/03/18/00:23:29 18 March 2013]; Available from: http://www0.health.nsw.gov.au/policies/gl/2011/GL2011_007.html.
Demand and Performance Evaluation Branch. NSW Health: NSW Health Services Comparison Data Book 2008/2009. Australia: NSW Health: NSW; 2010.
Barraclough H, Morrell S, Arcorace M, McElroy HJ, Baker DF. Degree-of-spread artefact in the New South Wales Central Cancer Registry. Aust N Z J Public Health. 2008;32:414–6.
Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29–36.
Goldsbury DE, Armstrong K, Simonella L, Armstrong BK, O'Connell DL. Using administrative health data to describe colorectal and lung cancer care in New South Wales, Australia: a validation study. BMC Health Serv Res. 2012;12:387.
Sarfati D, Gurney J, Stanley J, Salmond C, Crampton P, Dennett E, et al. Cancer-specific administrative data-based comorbidity indices provided valid alternative to Charlson and NHI indices. J Clin Epidemiol. 2014;67:586–95.
Sterne JA, White IR, Carlin JB, Spratt M, Royston P, Kenward MG, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009;338:b2393.
Acknowledgements
We thank the data custodians for data preparation and provision, and the NSW Centre for Health Record Linkage for performing data linkage. This project was funded by a Cancer Epidemiology Linkage Grant from the Cancer Institute NSW. Jane Young is supported by a Cancer Institute NSW Academic Leader in Cancer Epidemiology award.
Author information
Authors and Affiliations
Corresponding author
Additional information
Competing interests
The authors declare they have no competing interests.
Authors’ contributions
DC, TD and JY conceived of the study, and participated in its design and coordination. TD and TBP performed the statistical analyses. All authors helped to draft the manuscript. All authors read and approved the final manuscript.
Timothy A Dobbins and Jane M Young contributed equally to this work.
Additional file
Additional file 1:
ICD Procedure codes used to obtain ASA score.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/.
The Creative Commons Public Domain Dedication waiver (https://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Dobbins, T.A., Badgery-Parker, T., Currow, D.C. et al. Assessing measures of comorbidity and functional status for risk adjustment to compare hospital performance for colorectal cancer surgery: a retrospective data-linkage study. BMC Med Inform Decis Mak 15, 55 (2015). https://doi.org/10.1186/s12911-015-0175-1
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12911-015-0175-1