Posted by j.laird on 7 June 2021 - 2:00pm

Photo by Emily Morter on Unsplash

By Dr Philip Grylls.

The interplay between reproducible research and early career researchers (ECRs) is often challenging. Starting a career in research, nobody intends to produce irreplaceable science and methods. Why is it then that there is a replicability crisis being uncovered across scientific domains and how do ECRs succumb to such practices? 

To understand the challenges to the adoption of reproducible research practices one must consider the pace of reproducible research and how this conflicts with the reward structure in academia and career progression for ECRs.

Pressure for results

The first step on the academic ladder is the PhD, a time-pressured experience where although not strictly required, a positive and novel result or method will see the award come much easier. This is followed by short-term postdoctoral research contracts where anything less than exceptional will not lead to fellowships and long-term academic positions. During these positions the measure of success is rapid high-impact publications of positive results. However, research is by its nature at the edge of our understanding so a positive result is not guaranteed. For example, the percentage of null findings in pre-registered research, where the declaration of a study is made prior to conducting the research, was 60% where as in traditional literature null findings are as low as 10% [Allen and Mehler, 2019]. An ECR’s career progression is more secure if they maintain the status quo of 'fast science' instead of slower paced and higher risk reproducible research.

Reproducible research comes with more overheads and time investment than the type of research that has become the norm. Additionally, as mentioned reproducible research includes the requirement to spend time publishing negative results. The benefits of reproducible research tend to be general and not individual-specific; improving the research landscape by increasing faith in the field, increasing the openness of methods and data, and improving robustness of results through thorough peer review. The commonly stated benefit to the individual is an investment in ones future, however, given the incentive structures don’t guarantee this future it can seem a hollow gain.

Incentives

The current incentive structures encourage exploratory analysis and science, and upon finding an interesting result, to build the narrative backwards making a publication read as though this was the intent. A requirement for reproducible research is to enable proper reporting of such exploratory research where a conclusion that follow-up confirmatory research is required is given proper recognition. To achieve this a tonal shift is required in the diversity of journal articles, which will see a benefit to ECRs who will be able to more honestly publish exploratory science which tends to be faster than a pre-registered study. The provision of robust open data-sets designed and collected for exploratory research must become the norm thus enabling ECRs to have access to quality data. Late career researchers and large grant holders who have the stability to create such a dataset need to be incentivised to create and maintain these. With this ECRs will be able to conduct exploratory research where the publication of both positive and negative explorations will enrich the data.

The length of an average ECR position puts them at a disadvantage as a properly constructed pre-registered study will often be longer than a position. ECRs need recognition and reward for collaborating on studies at multiple stages. ECR contracts must always contain multiple facets designed to grant experience with pre-registered studies as well as time to conduct exploratory research. To become eligible for fellowships and permanent positions the onus must not be on successful studies, or number of studies, but the quality of contribution.

The creation and maintenance of open datasets is something we identify as beneficial to both reproducible science and to ECRs. However, without significant investment in longevity these projects are doomed to fail as the contracts of ECRs who have the skills to maintain them end, and eventually the grants that fund them, end. This problem is accentuated by a low standard in software sustainability. Writing code, especially sustainable code, takes time and training that may not be granted to ECRs. Additionally there are limited career advantages to writing good code as it is not often considered a research output. The benefits to sharing software are therefore weighted toward the community and not the individual who invests the time to cultivate coding skills and actually writes software.

Need for recognition

At present the system actively pushes ECRs, as well as established researchers, away from reproducible research due to the time investments and risk associated with ‘unpublishable‘ negative results. A coordinated effort from established researchers, employers, and journals will be required to break the cycle. Without buy-in at all levels, most ECRs who choose reproducible research will be out-competed by those who do not. In addition roles for non-traditional researchers need career incentives and funding. One such role is that of 'research software engineer', promoted by the SSI and SocRSE. This role defines and rewards people who produce research software.

I suggest that for a research environment that is committed to reproducibility more recognition needs to be given to ‘research adjacent’ (or more truthfully research critical) roles such as Research Software Engineers, Research Data Scientists, Research Statistics Practitioners etc. At present an ECR is supposed to embody all of these roles but a system that rewards each aspect individually will produce a wider variety of researchers with generalists and specialists alike who work collaboratively to produce higher quality reproducible research.

References

Christopher Allen and David M A Mehler. Open science challenges, benefits and tips in early career and beyond. 17, 2019. doi: 10.1371/journal.pbio.3000246.