The Central Role of Noise in Evaluating Interventions that Use Test Scores to Rank Schools

Kenneth Chay, Patrick McEwan and Miguel Urquiola

No 10118, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Several countries have implemented programs that use test scores to rank schools, and to reward or penalize them based on their students' average performance. Recently, Kane and Staiger (2002) have warned that imprecision in the measurement of school-level test scores could impede these efforts. There is little evidence, however, on how seriously noise hinders the evaluation of the impact of these interventions. We examine these issues in the context of Chile's P-900 program a country-wide intervention in which resources were allocated based on cutoffs in schools' mean test scores. We show that transitory noise in average scores and mean reversion lead conventional estimation approaches to greatly overstate the impacts of such programs. We then show how a regression discontinuity design that utilizes the discrete nature of the selection rule can be used to control for reversion biases. While the RD analysis provides convincing evidence that the P-900 program had significant effects on test score gains, these effects are much smaller than is widely believed.

JEL-codes: I2 (search for similar items in EconPapers)
Date: 2003-11
New Economics Papers: this item is included in nep-edu and nep-ure
Note: ED LS
References: Add references at CitEc
Citations: View citations in EconPapers (29)

Published as Chay, Kenneth Y., Patrick J. McEwan and Miguel Urquiola. "The Central Role Of Noise In Evaluating Interventions That Use Test Score To Rank Schools," American Economic Review, 2005, v95(4,Sep), 1237-1258.

Downloads: (external link)
http://www.nber.org/papers/w10118.pdf (application/pdf)

Related works:
Journal Article: The Central Role of Noise in Evaluating Interventions That Use Test Scores to Rank Schools (2005)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:10118

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w10118

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().