Neuromag May 2017 - Page 17

firmatory. That is a big problem and one of the major precedents causing the replication crisis and the growing public mistrust of science. How can we ensure that we clearly distinguish ex- ploratory and confirmatory analyses, both for ourselves and others? Report exploratory research honestly If you know you are doing exploratory research, then report it as such! There should be big red flags in your discus- sion saying things like “these findings are exploratory; a confirmatory study is needed before firm conclusions can be drawn”. There is nothing wrong with exploratory research – it must con- tinue to be published – but we need more confirmatory experiments in the literature (particularly for surprising or weak effects). And in my opinion, the media should be barred from report- ing on anything in the experimental / biological sciences until it has been in- dependently replicated. But as I discussed above, it can be all too easy to fool yourself (“oh, of course I was going to normalise that way all along”). How can you ensure you are doing what you said you would do before you saw the data? Pre-registration Pre-registering an experiment means writing down your experimental hy- potheses, data collection plan, experi- mental design, outcome measures, data preprocessing, exclusion criteria, analyses, and the final test of your hy- pothesis all before having begun data collection. You consider everything you can think of (greatly helped by run- ning a pilot experiment) as explicitly as you can. Having analysis scripts pre- pared ahead of time would be the gold standard. Once you have collected the data you perform the analysis and re- port exactly that result. As someone who has been doing ex- periments for years without pre-reg- istration, I can tell you that it is hard. I am used to having some theoretical hypothesis and concrete experimen- tal idea, then rushing in and testing it, then sorting out the details later (in the same dataset). Pre-registration means trying to specify lots of things that I would have sorted out on the way. Nevertheless, it is a rewarding way to guard yourself against the gar- den of forking paths. What happens when (not if) you realise you really should do that normalisa- tion or change the analysis, after you have collected the data? That is fine: just report it as such (along with the results of the original analysis). Pre- registration is a mechanism to help you be honest about what is confirm- atory and what is exploratory. Readers of your paper can decide how much the deviations from what you had said you would do matter. So how do you actually pre-register an experiment? This can be as simple as a text document on your comput- er. However, an even better idea is to pre-register the experiment online in a time stamped repository (e.g. Github or the Open Science Framework). Then, when you write up the study, you can point readers to a link with your pre- registration as evidence that you are really trying to delineate exploratory and confirmatory analyses. For this purpose I really like This site has a 9-question template for pre-registration, which encourages a short and structured document. At a later time, you can choose to make your preregistration pdf publicly avail- able with a static URL link, which could be included in the methods of your pa- per. Registered reports At the next level in scientific transpar- ency are registered reports. A scien- tific publication in a registered report format is one in which (a) the authors submit a manuscript consisting of an introduction, methods, and an analysis plan; (b) reviewers critique the meth- ods and planned analyses, suggesting changes; (c) if the authors and review- ers agree on a protocol, then the paper is in-principle accepted at the journal; (d) the authors go collect and analyse the data; (e) the journal publishes the results. There are now over 20 jour- nals that offer this format, including Cortex, Attention Perception and Psycho- physics, and Nature Human Behavior [7]. In my opinion, this format is excel- lent for guarding against researcher degrees of freedom and is particu- larly suited for either direct replica- tion studies or for studies whose out- come you expect will be contentious. Registered reports could allow you to sidestep fighting with reviewers who ultimately simply do not like your con- clusions. After all, if they agreed on your experimental protocol, then data could presumably have gone their way. The data could have confirmed their theory. However, the time-scale required for registered reports make them unsuit- able for use in something like a mas- ter’s project or lab rotation. By the time you have received comments from reviewers and adjusted your protocols, your time in the lab is likely up. Self-pre-registration on the other hand will also improve your ability to discriminate exploratory and confirm- atory research and will typically take you only a few hours. Open data and materials Scientists should want to make it as easy as possible for others to inde- pendently check our results and at- tempt replic ations. To facilitate these outcomes and to promote transpar- ency in the research process more generally, it is becoming increasingly standard to make raw data and study materials (such as stimuli or analysis code) publicly available at the time of publication. This not only helps others to verify your work, but can also help you. Proper archiving of study mate- rial is a requirement of most research funding and this way your materials and data are archived as part of the publication process. There are a number of ways to share data and materials online. My favour- ite resource for this is It is a freely available EC-funded research- sharing framework capable of hosting tens of gigabytes of data (terabytes available upon request), and it is host- ed on the same infrastructure as the LHC data from CERN. Every upload gets a DOI (digital object identifier), meaning you can cite your materials in the text of your paper with a reference that is never lost (no more self- or publisher-hosted content going miss- ing after a few years). What if you expect to get several pa- pers from the same dataset? Justifi- May 2017 | NEUROMAG | 17