[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article

Effects of an asynchronous online data literacy intervention on pre-service and in-service educators’ beliefs, self-efficacy, and practices

Published: 01 July 2019 Publication History

Abstract

Today's teachers are inundated with data, and their capacity to use data productively and responsibly is a salient but complex skillset. In this context, the present experimental study (N = 107) investigated the efficacy of an asynchronous online data literacy intervention intended for pre-service and in-service educators. The facilitated, interactive, and highly-structured intervention engaged participants in asking and answering four different kinds of questions (e.g., achievement status and growth, strengths and weaknesses, instructional implications) at five different student levels (e.g., individual, subgroup, school) with external, standardized assessment data presented in tables, charts, and score reports. Findings indicate medium-to-large intervention effects on participants' data-driven decision making self-efficacy and anxiety (ds ranged from .44 to 1.39) and in-school implementation of data use practices (d = 0.68). Impact patterns varied somewhat by population, though all sub-populations indicated highly favorable perceptions of the intervention.

Highlights

Experimental efficacy study of online data literacy intervention for teachers.
Intervention targeted capacity to use external, standardized assessment data.
Effects on participants' data-driven decision making self-efficacy and anxiety.
Effects also observed for teacher classroom practices related to data use.
Participants indicated highly favorable perceptions of the intervention's impact.

References

[1]
L. Ajayi, How asynchronous discussion boards mediate learning literacy methods courses to enrich alternative-licensed teachers' learning experiences, Journal of Research on Technology in Education 43 (1) (2010) 1–28.
[2]
S.Z. Athanases, L.H. Bennett, J.M. Wahleithner, Fostering data literacy through preservice teacher inquiry in English language arts, The Teacher Educator 48 (1) (2013) 8–28.
[3]
Y.C. Aydin, E. Uzuntiryaki, B. Demirdogen, Interplay of motivational and cognitive strategies in predicting self-efficacy and anxiety, Educational Psychology 31 (2011) 55–66.
[4]
A. Bandura, Self-efficacy conception of anxiety, Anxiety Research 1 (1988) 77–98.
[5]
A. Bandura, Self-efficacy: The exercise of control, Freeman, New York, 1997.
[6]
D.C. Berliner, The development of expertise in pedagogy, American Association of Colleges for Teacher Education, Washington, DC, 1988.
[7]
C. Bocala, K.P. Boudett, Teaching educators habits of mind for using data wisely, Teachers College Record 117 (4) (2015) 1–20.
[8]
C. Bocala, S.F. Henry, S. Mundry, C. Morgan, Practitioner data use in schools: Workshop toolkit, U.S. Department of Education, Washington, DC, 2014.
[9]
G.T. Brown, Teachers' conceptions of assessment: Validation of an abridged version, Psychological Reports 99 (1) (2006) 166–170.
[10]
D. Carlson, G. Borman, M. Robinson, A multistate district-level cluster randomized trial of the impact of data-driven reform on reading and mathematics achievement, Educational Evaluation and Policy Analysis 33 (3) (2011) 378–398.
[11]
Cavalluzzo, L.; Geraghty, T.M.; Steele, J.L.; Holian, L.; Jenkin, F.; Alexander, J.M.; et al. (2014): Using data to inform decisions: How teachers use data to inform practice and improve student performance in mathematics. Retrieved from https://files.eric.ed.gov/fulltext/ED555557.pdf.
[12]
Y. Chen, N.S. Chen, C.C. Tsai, The use of online synchronous discussion for web-based professional development for teachers, Computers & Education 53 (4) (2009) 1155–1166.
[13]
H. Chick, R. Pierce, The statistical literacy needed to interpret school assessment data, Mathematics Teacher Education and Development 15 (2) (2013) 5–26.
[14]
C. Chou, Interactivity and interactive functions in web‐based learning systems: A technical framework for designers, British Journal of Educational Technology 34 (3) (2003) 265–279.
[15]
C.E. Coburn, E.O. Turner, Research on data use: A framework and analysis, Measurement: Interdisciplinary Research & Perspectives 9 (4) (2011) 173–206.
[16]
J. Cohen, Statistical power analysis for the behavioral sciences, Lawrence Erlbaum Associates, Hillsdale, NJ, 1988.
[17]
B. Cowie, B. Cooper, Exploring the challenge of developing student teacher data literacy, Assessment in Education: Principles, Policy & Practice 24 (2) (2016) 147–163.
[18]
A. Datnow, L. Hubbard, Teacher capacity for and beliefs about data-driven decision making: A literature review of international research, Journal of Educational Change 17 (1) (2016) 7–28.
[19]
A. Datnow, V. Park, B. Kennedy-Lewis, Affordances and constraints in the context of teacher collaboration for the purpose of data use, Journal of Educational Administration 51 (3) (2013) 341–362.
[20]
K. Dunlap, J.S. Piro, Diving into data: Developing the capacity for data literacy in teacher education, Cogent Education 3 (1) (2016).
[21]
K.E. Dunn, Educational psychology's instructional challenge: Pre-service teacher concerns regarding classroom-level data-driven decision-making, Psychology Learning and Teaching 15 (1) (2016) 31–43.
[22]
K.E. Dunn, D.T. Airola, W.J. Lo, M. Garrison, What teachers think about what they can do with data: Development and validation of the data driven decision-making efficacy and anxiety inventory, Contemporary Educational Psychology 38 (1) (2013) 87–98.
[23]
J. Ebbeler, C.L. Poortman, K. Schildkamp, J.M. Pieters, Effects of a data use intervention on educators' use of knowledge and skills, Studies In Educational Evaluation 48 (2016) 19–31.
[24]
C.C. Farrell, J.A. Marsh, Contributing conditions: A qualitative comparative analysis of teachers' instructional responses to data, Teaching and Teacher Education 60 (2016) 398–412.
[25]
M. Finster, A. Milanowski, Teacher perceptions of a new performance evaluation system and their influence on practice: A within-and between-school level analysis, Education Policy Analysis Archives 26 (2018) 41.
[26]
L.S. Fuchs, D. Fuchs, Effects of systematic formative evaluation: A meta-analysis, Exceptional Children 53 (3) (1986) 199–208.
[27]
M. van Geel, T. Keuning, A.J. Visscher, J.P. Fox, Assessing the effects of a school-wide data-based decision-making intervention on student achievement growth in primary schools, American Educational Research Journal 53 (2) (2016) 360–394.
[28]
G. Gelderblom, K. Schildkamp, J. Pieters, M. Ehren, Data-based decision making for instructional improvement in primary education, International Journal of Educational Research 80 (2016) 1–14.
[29]
J.L. Green, S. Schmitt-Wilson, T. Versland, L. Kelting-Gibson, G.E. Nollmeyer, Teachers and data literacy: A blueprint for professional development to foster data driven decision making, Journal of Continuing Education and Professional Development 3 (1) (2016) 14–32.
[30]
E.S. Gummer, E.B. Mandinach, Building a conceptual framework for data literacy, Teachers College Record 117 (4) (2015) 1–22.
[31]
C.N. Gunawardena, C.A. Lowe, T. Anderson, Analysis of a global online debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing, Journal of Educational Computing Research 17 (4) (1997) 397–431.
[32]
L. Hamilton, R. Halverson, S. Jackson, E. Mandinach, J. Supovitz, J. Wayman, Using student achievement data to support instructional decision making, U.S. Department of Education, Washington, DC, 2009.
[33]
J.E. Henning, Teacher leaders at work: Analyzing standardized achievement data to improve instruction, Education 126 (4) (2006) 729–737.
[34]
D.C. Hillman, D.J. Willis, C.N. Gunawardena, Learner‐interface interaction in distance education: An extension of contemporary models and strategies for practitioners, American Journal of Distance Education 8 (2) (1994) 30–42.
[35]
I. Hoogland, K. Schildkamp, F. van der Kleij, M. Heitink, W. Kippers, B. Veldkamp, et al., Prerequisites for data-based decision making in the classroom: Research evidence and practical illustrations, Teaching and Teacher Education 60 (2016) 377–386.
[36]
N.R. Hoover, L.M. Abrams, Teachers' instructional use of summative student assessment data, Applied Measurement in Education 26 (2013) 219–231.
[37]
S.G. Huber, G. Skedsmo, Data use—a key to improve teaching and learning?, Educational Assessment, Evaluation and Accountability 28 (1) (2016) 1–3.
[38]
M.D. Hubers, C.L. Poortman, K. Schildkamp, J.M. Pieters, A. Handelzalts, Opening the black box: Knowledge creation in data teams, Journal of Professional Capital and Community 1 (1) (2016) 41–68.
[39]
A. Huguet, J.A. Marsh, C.C. Farrell, Building teachers' data-use capacity: Insights from strong and developing coaches, Education Policy Analysis Archives 22 (2014).
[40]
G.S. Ikemoto, J.A. Marsh, Cutting through the “data-driven” mantra: Different conceptions of data-driven decision making, The Yearbook of the National Society for the Study of Education 106 (1) (2007) 105–131.
[41]
D. Ingram, K.S. Louis, R.G. Schroeder, Accountability policies and teacher decision making: Barriers to the use of data to improve practice, Teachers College Record 106 (6) (2004) 1258–1287.
[42]
J. Jacobs, A. Gregory, D. Hoppey, D. Yendol-Hoppey, Data literacy: Understanding teachers' data use in a context of accountability and response to intervention, Action in Teacher Education 31 (3) (2009) 41–55.
[43]
J.B. Jimerson, M.N. Choate, L.K. Dietz, Supporting data-informed practice among early career teachers: The role of mentors, Leadership and Policy in Schools 14 (2015) 204–232.
[44]
J.B. Jimerson, V. Cho, J.C. Wayman, Student-involved data use: Teacher practices and considerations for professional learning, Teaching and Teacher Education 60 (2016) 413–424.
[45]
J.B. Jimerson, J.C. Wayman, Professional learning for using data: Examining teacher needs and supports, Teachers College Record 117 (4) (2015).
[46]
C. Kent, E. Laslo, S. Rafaeli, Interactivity in online discussions and learning outcomes, Computers & Education 97 (2016) 116–128.
[47]
K.A. Kerr, J.A. Marsh, G.S. Ikemoto, H. Darilek, H. Barney, Strategies to promote data use for instructional improvement: Actions, outcomes, and lessons from three urban districts, American Journal of Education 112 (4) (2006) 496–520.
[48]
N. Kingston, B. Nash, Formative assessment: A meta‐analysis and a call for research, Educational Measurement: Issues and Practice 30 (4) (2011) 28–37.
[49]
S. Konstantopoulos, W. Li, S.R. Miller, A. van der Ploeg, Effects of interim assessments across the achievement distribution: Evidence from an experiment, Educational and Psychological Measurement 76 (4) (2016) 587–608.
[50]
E. Kyndt, D. Gijbels, I. Grosemans, V. Donche, Teachers' everyday professional development: Mapping informal learning activities, antecedents, and learning outcomes, Review of Educational Research 86 (4) (2016) 1111–1150.
[51]
M.A. Lachat, S. Smith, Practices that support data use in urban high schools, Journal of Education for Students Placed at Risk 10 (3) (2005) 333–349.
[52]
M.K. Lai, S. McNaughton, The impact of data use professional development on student achievement, Teaching and Teacher Education 60 (2016) 434–443.
[53]
R.J. Little, A test of missing completely at random for multivariate data with missing values, Journal of the American Statistical Association 83 (404) (1988) 1198–1202.
[54]
N. Love, K.E. Stiles, S. Mundry, K. DiRanna, The data coach's guide to improving learning for all students: Unleashing the power of collaborative inquiry, Corwin Press, Thousand Oaks, CA, 2001.
[55]
E.B. Mandinach, E.S. Gummer, What does it mean for teachers to be data literate: Laying out the skills, knowledge, and dispositions, Teaching and Teacher Education 60 (2016) 366–376.
[56]
E.B. Mandinach, J.B. Jimerson, Teachers learning how to use data: A synthesis of the issues and what is known, Teaching and Teacher Education 60 (2016) 452–457.
[57]
J.A. Marsh, Interventions promoting educators' use of data: Research insights and gaps, Teachers College Record 114 (11) (2012) 1–48.
[58]
J.A. Marsh, M. Bertrand, A. Huguet, Using data to alter instructional practice: The mediating role of coaches and professional learning communities, Teachers College Record 117 (2015) 1–40.
[59]
J.A. Marsh, C.C. Farrell, How leaders can support teachers with data-driven decision making: A framework for understanding capacity building, Educational Management Administration & Leadership 43 (2) (2015) 269–289.
[60]
J.A. Marsh, J.S. McCombs, F. Martorell, How instructional coaches support data-driven decision making: Policy implementation and effects in Florida middle schools, Educational Policy 24 (872) (2010) 872–907.
[61]
B. Means, E. Chen, A. DeBarger, C. Padilla, Teachers' ability to use data to inform instruction: Challenges and supports, U.S. Department of Education, Washington, DC, 2011.
[62]
Means, B.; Gallagher, L.; Padilla, C. (2007): Teachers' use of student data systems to improve instruction. Retrieved August 15, 2017 from http://files.eric.ed.gov/fulltext/ED501547.pdf.
[63]
M. Molenda, J.A. Pershing, Improving performance, in: A. Januszewski, M. Molenda (Eds.), Educational technology: A definition with commentary, Lawrence Erlbaum Associates, New York, 2008, pp. 241–258.
[64]
M. Moore, Three types of interaction, American Journal of Distance Education 3 (2) (1989) 1–6.
[65]
L.N. Oláh, N.R. Lawrence, M. Riggan, Learning to learn from benchmark assessment data: How teachers analyze results, Peabody Journal of Education 85 (2) (2010) 226–245.
[66]
A.J. Onwuegbuzie, Academic procrastination and statistics anxiety, Assessment & Evaluation in Higher Education 29 (2004) 3–19.
[67]
M. Orland, Research and policy perspectives on data-based decision making in education, Teachers College Record 117 (4) (2015).
[68]
M.F. Pajares, Teachers' beliefs and educational research: Cleaning up a messy construct, Review of Educational Research 62 (3) (1992) 307–332.
[69]
W.R. Penuel, L.A. Shepard, Assessment and teaching, in: D. H., Gitomer C.A. Bell (Eds.), Handbook of research on teaching, American Educational Research Association, Washington, DC, 2016, pp. 787–850.
[70]
R. Pierce, H. Chick, Workplace statistical literacy for teachers: Interpreting box plots, Mathematics Education Research Journal 25 (2) (2013) 189–205.
[71]
R. Pierce, H. Chick, I. Gordon, Teachers' perceptions of the factors influencing their engagement with statistical reports on student achievement data, Australian Journal of Education 57 (3) (2013) 237–255.
[72]
R. Pierce, H. Chick, J. Watson, M. Les, M. Dalton, A statistical literacy hierarchy for interpreting educational system data, Australian Journal of Education 58 (2) (2014) 195–217.
[73]
J.S. Piro, K. Dunlap, T. Shutt, A collaborative data chat: Teaching summative assessment data use in pre-service teacher education, Cogent Education 1 (1) (2014).
[74]
C.L. Poortman, K. Schildkamp, Solving student achievement problems with a data use intervention for teachers, Teaching and Teacher Education 60 (2016) 425–433.
[75]
R. Prenger, K. Schildkamp, Data-based decision making for teacher and student learning: A psychological perspective on the role of the teacher, Educational Psychology. Advance online publication (2018),.
[76]
T.D. Reeves, J.L. Chiang, Building pre-service teacher capacity to use external assessment data: An intervention study, The Teacher Educator 52 (2) (2017) 155–172.
[77]
T.D. Reeves, J.L. Chiang, Online interventions to promote teacher data-driven decision making: Optimizing design to maximize impact, Studies in Educational Evaluation 59 (2018) 256–269.
[78]
T.D. Reeves, S.L. Honig, A classroom assessment data literacy intervention for pre-service teachers, Teaching and Teacher Education 50 (2015) 90–101.
[79]
T.D. Reeves, K.H. Summers, E. Grove, Examining the landscape of teacher learning for data use: The case of Illinois, Cogent Education 3 (1) (2016).
[80]
T.D. Reeves, A.A. Tawfik, F. Msilu, I. Simsek, What’s in it for me? Incentives, learning, and completion in massive open online courses, Journal of Research on Technology in Education 49 (3-4) (2017) 245–259.
[81]
L.G.R. Rolando, D.F. Salvador, A.H.S. Souza, M.R. Luz, Learning with their peers: Using a virtual learning community to improve an in-service Biology teacher education program in Brazil, Teaching and Teacher Education 44 (2014) 44–55.
[82]
M.A. Ruiz‐Primo, R.J. Shavelson, L. Hamilton, S. Klein, On the evaluation of systemic science education reform: Searching for instructional sensitivity, Journal of Research in Science Teaching 39 (5) (2002) 369–393.
[83]
J.L. Schafer, Multiple imputation: A primer, Statistical Methods in Medical Research 8 (1) (1999) 3–15.
[84]
E.A. van der Scheer, A.J. Visscher, Effects of an intensive data-based decision making intervention on teacher efficacy, Teaching and Teacher Education 60 (2016) 34–43.
[85]
K. Schildkamp, L. Karbautzki, J. Vanhoof, Exploring data use practices around Europe: Identifying enablers and barriers, Studies In Educational Evaluation 42 (2014) 15–24.
[86]
K. Schildkamp, C.L. Poortman, A. Handelzalts, Data teams for school improvement, School Effectiveness and School Improvement 27 (2) (2016) 228–254.
[87]
G.W. Selnow, Using interactive computer to communicate scientific information, American Behavioral Scientist 32 (2) (1988) 124–135.
[88]
W.R. Shadish, T.D. Cook, D.T. Campbell, Experimental and quasi-experimental designs for generalized causal inference, Wadsworth Cengage Learning, Belmont, CA, 2002.
[89]
L. Shulman, Knowledge and teaching: Foundations of the new reform, Harvard Educational Review 57 (1) (1987) 1–23.
[90]
V. Snodgrass Rangel, C. Monroy, E. Bell, Science teachers' data use practices: A descriptive analysis, Education Policy Analysis Archives 24 (2016).
[91]
J.P. Spillane, Data in practice: Conceptualizing the data-based decision-making phenomena, American Journal of Education 118 (2) (2012) 113–141.
[92]
J. Sun, R. Przybylski, B.J. Johnson, A review of research on teachers' use of student data: From the perspective of school leadership, Educational Assessment, Evaluation and Accountability 28 (1) (2016) 5–33.
[93]
S. Taie, R. Goldring, Characteristics of public elementary and secondary school teachers in the United States: Results from the 2015–16 National Teacher and Principal Survey first look (NCES 2017-072), U.S. Department of Education, National Center for Education Statistics, Washington, DC, 2017.
[94]
M.K. Tallent-Runnels, J.A. Thomas, W.Y. Lan, S. Cooper, T.C. Ahern, S.M. Shaw, et al., Teaching courses online: A review of the research, Review of Educational Research 76 (1) (2006) 93–135.
[95]
P. Tsiotakis, A. Jimoyiannis, Critical factors towards analysing teachers' presence in on-line learning communities, The Internet and Higher Education 28 (2016) 45–58.
[96]
E.O. Turner, C.E. Coburn, The practice of data use: An introduction, American Journal of Education 118 (2) (2012) 99–111.
[97]
R. Van Gasse, K. Vanlommel, J. Vanhoof, P. Van Petegem, Teacher collaboration on the use of pupil learning outcome data: A rich environment for professional learning?, Teaching and Teacher Education 60 (2016) 387–397.
[98]
J. Vanhoof, K. Schildkamp, From ‘professional development for data use’ to ‘data use for professional development’, Studies In Educational Evaluation 42 (2014) 1–4.
[99]
H.T.G. Van den Hurk, A.A.M. Houtveen, W.J.C.M. Van de Grift, Fostering effective teaching behavior through the use of data-feedback, Teaching and Teacher Education 60 (2016) 444–451.
[100]
K. Vanlommel, J. Vanhoof, P. Van Petegem, Data use by teachers: The impact of motivation, decision-making style, supportive relationships and reflective capacity, Educational Studies 42 (1) (2016) 36–53.
[101]
L. Volante, X. Fazio, Exploring teacher candidates' assessment literacy: Implications for teacher education reform and professional development, Canadian Journal of Education 30 (3) (2007) 749–770.
[102]
D.A. Walker, T.D. Reeves, T.J. Smith, Confirmation of the Data Driven Decision-Making Efficacy and Anxiety Inventory’s (3D-MEA) score factor structure among teachers, Journal of Psychoeducational Assessment 36 (5) (2018) 477–491.
[103]
J.C. Wayman, J.B. Jimerson, Teacher needs for data-related professional learning, Studies In Educational Evaluation 42 (2014) 25–34.
[104]
J.C. Wayman, S. Shaw, V. Cho, Longitudinal effects of teacher use of a computer data system on student achievement, AERA Open 3 (1) (2017) 1–18.
[105]
B. Wellman, L. Lipton, Data-driven dialogue: A facilitator's guide to collaborative inquiry, MiraVia, Sherman, CT, 2004.
[106]
D.B. Wilson, Effect size determination program (version 2.0) [excel macro application], University of Maryland, College Park, MD, 2001.
[107]
K.S. Yoon, T. Duncan, S.W.-Y. Lee, B. Scarloss, K. Shapley, Reviewing the evidence on how teacher professional development affects student achievement, U.S. Department of Education, Washington, DC, 2007.
[108]
D. Zapata-Rivera, R. Zwick, M. Vezzu, Exploring the effectiveness of a measurement error tutorial in helping teachers understand score report results, Educational Assessment 21 (3) (2016) 215–229.

Cited By

View all
  • (2024)Health Data Visualization Literacy Skills of Young Adults with Down Syndrome and the Barriers to Inference-makingACM Transactions on Accessible Computing10.1145/364862117:1(1-1)Online publication date: 24-Feb-2024
  • (2024)Validating a novel digital performance-based assessment of data literacy: Psychometric and eye-tracking analysesEducation and Information Technologies10.1007/s10639-023-12177-729:8(9417-9444)Online publication date: 1-Jun-2024
  • (2023)Cognitive Overload, Anxiety, Cognitive Fatigue, Avoidance Behavior and Data Literacy in Big Data environmentsInformation Processing and Management: an International Journal10.1016/j.ipm.2023.10348260:6Online publication date: 1-Nov-2023
  • Show More Cited By

Index Terms

  1. Effects of an asynchronous online data literacy intervention on pre-service and in-service educators’ beliefs, self-efficacy, and practices
            Index terms have been assigned to the content through auto-classification.

            Recommendations

            Comments

            Please enable JavaScript to view thecomments powered by Disqus.

            Information & Contributors

            Information

            Published In

            cover image Computers & Education
            Computers & Education  Volume 136, Issue C
            Jul 2019
            152 pages

            Publisher

            Elsevier Science Ltd.

            United Kingdom

            Publication History

            Published: 01 July 2019

            Author Tags

            1. Improving classroom teaching
            2. Interactive learning environments
            3. Pedagogical issues
            4. Teaching/learning strategies

            Qualifiers

            • Research-article

            Contributors

            Other Metrics

            Bibliometrics & Citations

            Bibliometrics

            Article Metrics

            • Downloads (Last 12 months)0
            • Downloads (Last 6 weeks)0
            Reflects downloads up to 28 Dec 2024

            Other Metrics

            Citations

            Cited By

            View all
            • (2024)Health Data Visualization Literacy Skills of Young Adults with Down Syndrome and the Barriers to Inference-makingACM Transactions on Accessible Computing10.1145/364862117:1(1-1)Online publication date: 24-Feb-2024
            • (2024)Validating a novel digital performance-based assessment of data literacy: Psychometric and eye-tracking analysesEducation and Information Technologies10.1007/s10639-023-12177-729:8(9417-9444)Online publication date: 1-Jun-2024
            • (2023)Cognitive Overload, Anxiety, Cognitive Fatigue, Avoidance Behavior and Data Literacy in Big Data environmentsInformation Processing and Management: an International Journal10.1016/j.ipm.2023.10348260:6Online publication date: 1-Nov-2023
            • (2022)Characterization of Self-Efficacy Model Using Social Cognitive Theory for Students’ Learning Abilities and Decision-Making Based on IoTWireless Communications & Mobile Computing10.1155/2022/52652392022Online publication date: 1-Jan-2022

            View Options

            View options

            Media

            Figures

            Other

            Tables

            Share

            Share

            Share this Publication link

            Share on social media