There’s a forceful rebuttal, to our previous rant, here: http://blogs.discovermagazine.com/neuroskeptic/2013/04/29/preregistration-problem/#more-3894
A few final points and then a suggestion – something that I hope everyone can agree with.
First, I think Neuroskeptic, and others are making a fundamental error:
“The trouble with the current system is that planned and exploratory analyses are confused. All registration does, at its core, is make the distinction clear”. But this is simply wrong. There are plenty of examples of planned science that will be classified as exploratory because they don’t involve pre-registration. The following frequently happens: I have a prior hypothesis about the brain (derived before I’ve looked at any data), I test it using an existing dataset (e.g., 1000 connectomes dataset). This is planned science, not exploratory – I repeat, it’s not exploratory. Fundamentally, this sort of experiment shouldn’t be penalised, shouldn’t have to declare that it’s exploratory, or suffer any problems from reviewers complaining that it’s post-hoc. It’s not post-hoc, not from the point of view of the hypothesis, which is what matters for dodgy science.
Second, Neuroskeptic side-steps my point about preregistration and unintended consequences, such as the potential for increasing fraud: “Neurorant’s third point is that like an AAA credit rating, preregistration could be exploited by fraudsters. Well, I’m sure that will happen – unfortunately, fraudsters exploit systems.”
Not all systems are equally exploitable. My point was that it’s possible that a system such as pre-registration becomes easier to exploit by fraudsters than the status quo. Just because fraud will happen under all systems doesn’t mean that some don’t encourage/allow more fraud than others.
Or alternatively way of putting it: can you be sure that there won’t be unintended negative consequences from pre-registration? (There are many examples of institutions and structures where a regulation, even when implemented with the best of intentions, has unexpected, bad side effects).
Finally. We could argue back and forth about these points ad nauseum. It turns out, we have different feelings (gut instincts? hunches?) about what we think is limiting scientific progress. I think that these pre-registration proposals will carry a significant cost in time and resources and make it harder to do creative science that pushes the field forward, for possibly little gain. Plenty of other people, not least Neuroskeptic, think otherwise. However, without some experimental evidence one way or the other it’s just an interventional hypothesis in need of testing in the complex maelstrom of the real world. No amount of arguments can change that. There are plenty of examples of interventions based on good, well-argued hypotheses that haven’t survived testing – have a look at the epic fails in the Alzheimer’s literature, or any neurological or psychiatric disorder, for that matter.
So, with that in mind. Why don’t we do something that we all agree on? Why not, as a community, conduct a randomised trial to test whether pre-registration improves the quality of neuroscientific/psychological research? If there’s a nice juicy effect of pre-registration with no irritating side effects, then let’s go for it, with both barrels. Otherwise, not.