By Sarah Alhozaimy, University of Manchester, David Mawdsley, University of Manchester, Doug Mulholland, University of Waterloo, Thor Wikfeldt, KTH.

Ensuring reproducibility of research has been identified as one of the challenges in scientific research. While reproducibility of results is a concern in all fields of science, the emphasis of this group is in the area of computer software reuse and the reproduction of results. The availability of complete descriptions, ideally including program source code, documentation and archives of all necessary components and input datasets would be a major step to resolving research reproducibility concerns.

One of the barriers to reproducible research is the issue of training for researchers. Many of the tools used in reproducible research, such as version control, unit testing and automation are seen (by non-software engineers) as being of interest only to professional coders. “Selling” the benefits of an engineering approach to scholarly publication will be an important part of encouraging reproducible research; training programmes such as the Software and Data Carpentries, and Code Refinery have an important role to play in introducing these ideas to researchers, ideally early in their careers. They present a compelling case to the learner of the benefits of, e.g., automation in improving the speed and efficiency of the research process, and reducing the risk of error. The tools and concepts they teach are an important part of the reproducible research ecosystem.

The increasing awareness of the importance of reproducibility in research is leading to increasing adoption of software development practices such as version control systems. Although such systems have been in use for a long time in industrial software development, they are being increasingly adopted to maintaining research software. This is good for reproducibility the uptake of other industry-standard best practices, such as automated testing and continuous integration should be encouraged.

At the same time, an increasing number of new technological platforms and tools are being developed. These tools range from article writing platforms, such as Authorea and Overleaf, to tools intended for interactive data exploration, such as Jupyter notebooks and knitr for R. Platforms for sharing data are also being developed, such as Zenodo and figshare, and Authorea allows authors to upload their data together with their articles.

Both R and Python feature tools that assist in the production of reproducible research papers. The R package Knitr allows chunks of R code to be added to LaTeX or markdown documents. These allow documents that look like “regular” scientific papers to be produced in a completely reproducible way, given access to the underlying data and the source LaTeX or markdown file.

R notebooks, which were introduced in RStudio, offer an environment similar to Python’s Jupyter notebooks. In both R and Jupyter notebooks, chunks of text and code can be interleaved into a sort of interactive lab workbook; here the emphasis is on producing a working document; although a notebook may be the result of a research project, it will typically be more focussed on the technical implementation of the project, rather than containing background information (such as a literature review), that one might expect to see in an academic article.

Even with good development practice, issues such as lack of documentation can be a barrier to reuse of research software. It is unclear how the production of high-quality documentation for such software could be encouraged or enforced. The practicality of this in a broad research context is constrained by factors such as commercialisation concerns; however, small steps in one or more dimensions of the problem are still progress overall.

The tools for reproducible research exist; although many tools remain domain specific or require specialised training, which leads to their being little used.  The use of reproducible research tools may be increased as these tools become more domain agnostic. Providing incentives for researchers to do their work in a reproducible way will need to involve both showing them the benefits of working reproducibly, in terms making their work easier to maintain and expand, and persuading scholarly publishers’ that producing reproducible research is in their interest. This would increase the uptake of these tools; particularly if the pressure came from high-profile journals.