Graduate Students Should Be Able to Specialize In Replication
Now that the need for more replication has forced its way onto the scientific agenda we should begin thinking about how to build systems to support its growth and institutionalization. New publications and conferences are all good steps, but we should go beyond relying on a loosely organized group of scientists who dedicate time to something that has collective utility but little individual utility.
I think one solution is to create well-defined graduate programs that focus on replication. The programs would vary depending on the fields they are designed to police. For example, the psychology replication program would train students in methods for replicating psychology studies and identifying methodological weaknesses that might make a study a good candidate for replication. The programs would effectively be mechanisms for credentialing official science cops.
The programs would accomplish a number of things relative to current initiatives:
1. Create the sense of an unbiased authority. The current system relies on researchers policing the “competition” or policing themselves. Both situations incentivize skewed results. With “Replicators” there would be fewer conflicts of interest — even if they conducted some of their own original research — because official credentials would establish them as unquestioned authorities and put their work under the microscope. Replicators would also make it harder for people to deem replication attempts illegitimate and launch a war in the court of public opinion (see: Bargh, John).
2. Support the innovation of replication best practices. If Replicators do choose to do “original” research a good portion of it could focus on improving the methods for judging the validity of a study. (For example, Gregory Francis has a new paper on using effect sizes to test for publication bias.) A lot this research would overlap with statistics research, but there are also qualitative advances that can be made.
3. Create a self-sustaining system. The ponzi scheme structure of academia is designed so that fields can grow exponentially. Making replication a part of the academic system will create a future supply of Replicators.
The one downside is that by concentrating official responsibility in the hands of a few you exempt everyone else from doing better science. Unfortunately, until changes to the tenure process alter publishing incentives I’m skeptical that a loosely organized system of individual responsibility would be better. Of course there’s no reason we can’t have journals, conferences, individual efforts, and academic programs. As every presidential candidate says, we need an “all of the above” approach.
Francis, G. (2012). Publication bias and the failure of replication in experimental psychology Psychonomic Bulletin & Review DOI: 10.3758/s13423-012-0322-y