Ph.D. Course: Crowdsourcing Science
The results of many published studies across many scientific domains are not easily reproduced by independent laboratories. For example, an initiative by Bayer Healthcare to replicate 67 pre-clinical studies led to a reproducibility rate of 20-25% (Prinz et al., 2011), and researchers at Amgen were only able to replicate 6 of 53 influential cancer biology studies (Begley & Ellis, 2012). Similar replication failures have been reported in social and cognitive psychology (Ebersole et al., 2015; Klein et al., 2014; Open Science Collaboration, 2015). This PhD Boot camp introduces PhD students in the behavioral sciences to 1) the ongoing “crisis of confidence” in science, 2) typical methodological challenges of conducting replications, 3) the philosophy of science and statistical background of replications, 4) highly collaborative approaches to replication, in which findings are replicated in independent laboratories before (rather than after) they are published. As part of the boot camp, students will be organized into replication teams and take part in a crowdsourced pre-publication independent replication project on which they will be credited as co-authors. Participation in the bootcamp is free, the replications will be funded by a grant from INSEAD, and the infrastructure for data collection is already in place. Crowdsourcing research involves recruiting numerous scientific teams to achieve large-scale projects no single team could feasibly carry out. Leveraging crowds of researchers increases the statistical power and generalizability of research designs, reduces investigator error and bias, and enhances scientific transparency. Actively participating in a large-scale replication effort provides an opportunity for students to experience the power of a crowd of researchers firsthand. Lecture topics will include the scientific crisis caused by high-profile replication failures, publication bias, questionable research practices, the open data movement, and crowdsourced replication efforts, among others.
Pre-Publication Independent Replication (PPIR) Team Project
As their course project, teams of graduate students will carry out a Pre-Publication Independent Replication (PPIR) of an unpublished study from researchers who have nominated their own findings for replication. Each team’s final PowerPoint presentation and report of their replication results will be the only assignment for the class. Further graduate methods courses at partner universities will likewise participate in the PPIR initiative, whose results will be reported in a scientific article in which student-replicators are co-authors.
Grading
The team presentation of their Pre-Publication Independent Replication will count for 50% of the final grade, and individual class participation for 50%.
Lecture 1: The Crisis of Confidence in Science
Today management, psychology, political science, economics, medicine, and other fields are grappling with a crisis of confidence not only in our research findings, but also the effectiveness of the tools to use to discover knowledge. This opening session will cover perverse incentives in science, publication bias, questionable research practices, and statistical analyses that suggest many published findings are not reliable. We will further discuss the costs and benefits of proposed approaches to increasing the robustness of scientific research. These include mandatory disclosure of data exclusions and stopping rules, pre-registration of analysis plans, increasing sample sizes to achieve high statistical power, adversarial collaborations, and registered replication reports, among others.
Lecture 2: The Replication Revolution
This lecture will cover the results of mass replication initiatives, as well as recent efforts to independently replicate findings before (rather than after) they are published.
Lecture 3: The Open Data Movement
One proposal to improve the reliability of our science is to make data from published research articles publicly available on the internet. This raises challenging issues regarding intellectual property and subject confidentiality.
Closing Session
In a final session to be scheduled after all the replications are completed, each team of students will present the results of their Pre-Publication Independent Replications (PPIR) project.
