Launching Registered Report Replications in Computer Science Education Research

Background and Context. The quality of a research field is greatly determined by the quality of its publications. However, in many research fields this quality is endangered because of biases towards publishing positive results, which have led to questionable research practices that negatively impact the reliability of the reported findings.

Objectives. We explored two approaches that could guard against these questionable research practices for computer science education research: 1. The replication of prior studies to confirm (or refute) previous findings and 2. requiring authors to submit a registered report, containing hypotheses, methods, and analytic procedures, before conducting the studies.

Method. Over the span of 18 months, we organized a special issue of the Computer Science Education journal that only accepted registered reports that replicated a previous computer science education research study. Registered reports involve peer review and approval at the research design stage, before data collection begins. The editorial process was thus modified to accommodate multiple rounds of review and a longer time period between original and final submissions. We believe this is the first use of registered reports in computer science education research. A questionnaire gathering feedback on the new process was also administered to the authors of the accepted reports.

Findings. We found seven author teams willing to submit a manuscript for the special issue. Out of this pool, preregistered reports from five teams were accepted to be taken forwards. One team then withdrew because the ethics procedure at their institution exceeded the special issue timeline. The remaining four author teams conducted their studies and resubmitted full papers that were accepted, pending some final corrections. Authors’ feedback about the registered reports process was positive.

Implications. We demonstrated that it is possible to attract interest for registered report replications in a computer science education journal issue, and to successfully conduct the necessary editorial steps. Future efforts should attend to challenges related to modified research and journal submission timelines and consider adding a second review cycle for the first stage of registered reports. While the procedures used in this special issue may be suitable for many research approaches, further discussion is warranted on how they can be combined with exploratory research (as opposed to hypothesis testing research) and how they can be adapted to non-positivist research.