By M.H. Beals, Loughborough University, J. H, Nielsen, UCL, B. A. Laken, UCL and M. Antonioletti, University of Edinburgh.

A speed blog from the Collaborations Workshop 2016 (CW16).

The importance and credit associated with publishing negative results.

As researchers, the majority our experiments and explorations do not always pan out. When this occurs, pressure prompts us to move on to the next idea, looking for that big result that will make our name and build our reputation. What are the knock-on effects of doing this? By not reporting our failures, are we cursing others to repeat them? Does our tendency to curate our results slow our progress and, if so, can we change this?

The negative and the null

First, let us distinguish between null results and making a refutations of established work. The latter is an expected, and arguably moral, response to incomplete or unreplicable results. Critically examining published work is a part of a researcher's daily activities: if you find fault, you should publish in some format, be it reply papers, rapid communication or roundtables, to ensure that any issues become known and can be corrected by the originators or their peers. Thus, refutation is a core part of the scientific method’s auto-corrective machinery. Null results, on the other hand, are the absence of a significant finding. Like refutations, their value is in their corrective potential. Questions that require creating or sifting through huge datasets (or archives) are particularly vulnerable to null results—your princess is in another castle. For example, is there oil under a particular sites? an astronomical object in a patch of sky? a Higgs-boson in an energy range? or a linguistic trend in a corpus? The dissemination of these failures may save others time and money by ruling out a particular problem space, provided that the documentation is robust.  In other cases, explaining how a methodology failed to produce the desired results may inspire new methods or research questions within an existing dataset—a dataset documented by and attributed to the author of the null results publication.

A home for null results

Hackneyed though it may be, the pressure to 'publish or perish' exists across academia, and with important qualifications. The expectation that you will regularly publish novel (positive) findings discourages wasted effort on writing, reviewing and disseminating null results, even for journals that profess a neutral stance. Despite this general bias against refutations, the traditional journal article may well be the best form for problematising existing research. For null results, however, meaningful dissemination may take other forms. The most important consideration is making your research accessible to an audience that will find it useful. Submitting to publications that will rank highly in online aggregators, rather than rely upon curation by specialty journals, can help researchers discover them without regularly consulting a central repository of null results. Conversely, the traditional role played by notes and queries can be expanded to include null results, normalising them within a journal's regular discourse. Data journals are another possible home. By providing extensive data, and the methodology for obtaining and interpreting this data, publication of null results may prevent unnecessary and costly duplication of effort. Documenting annotations in linked data provides a model for how such findings might be embedded. If all these efforts fail, a stealth approach may be appropriate: explicitly exploring null results when discussing new or redeployed methodologies.

A culture of positivity?

However helpful null results may be, there is undoubtedly a bias—amongst authors and journals—towards publishing positive results. Prestige and career advancement has traditionally come from an association with a particular discovery rather than painstaking work associated with identifying, redirecting or narrowing a problem space. Implicit practices may be as much to blame as explicit pressures. There is little tradition for citing negative or null results, even if they helped you narrow down your search or informed your research. Acknowledging those who impacted our practices, who redirected us away from flawed methods or whose "failures" contributed to our success, would be more honest and would lead to a more productive debate. Academic acculturation, the piecemeal process by which junior scholars learn whose published work has been overturned or who's behind-the-scenes efforts should be known, would be vastly truncated, Likewise, a cultural shift towards dynamic research outputs and post-publication review could reposition these results with wider debates and investigations, allowing for unexpected collaborations and the pulling of resource. By openly valuing these hidden layers of research, regardless of the 'sexiness' of the end result, we place our emphasis on the work rather than the reveal.