Credit: Pexels.com

By Frances Cooper (editor), Aleks Nenadic, Anna-Maria Sichani, Yasemin Turkyilmaz, Pablo Bernabeu and Steve Crouch

This post is part of the CW20 speed blog posts series.

Software is increasingly becoming recognised as fundamental to research. In a 2014 survey of UK researchers undertaken by the Institute, 7 out of 10 researchers supported the view that it would be impossible to conduct research without software. As software continues to underpin more research activities, we must engage a variety of stakeholders to incentivise the uptake of best practice in software development to ensure the quality of research software keeps pace with the research it supports.

The rise of open research - particularly when it applies to software - only serves to highlight the increasing awareness of the importance of software in research, and the necessity of reproducible software. Not only does software need to report correct results for publication, software must also be accessible and verifiable by others.

Examples of irreproducible research due to incorrect software continue to mount. Between 2001 and 2010, Geoffrey Chang published a series of highly-cited papers on multidrug resistance transporters. Unfortunately, due to a fault with an internal software utility that erroneously flipped two columns of data, the results could not be reproduced which led to five paper retractions. An example from last year was the discovery of problems with "Willoughby-Hoye" Python scripts when attempting to verify an experiment involving cyanobacteria. Essentially, different operating systems report the ordering of files within a directory differently depending on their implementation, which led to fundamentally different results. If such research misadventures are to be avoided, better practices in research software development must become more prevalent. To achieve this, we need to explore ways to highlight and incentivise the uptake of such practices from several perspectives: grassroots, funders, publishers and university departments.

Grassroots

Individual members of the research community can change the culture. Often moving beyond their own disciplines, researchers engage in promoting the improvement of the scientific landscape, in areas such as reproducibility, best practices in programming and collaborative working. Many grassroots communities (e.g., UK Reproducibility Network, Peer Reviewers’ Openness Initiative) have emerged, often through channels such as Twitter. When grassroots communities grow enough, they begin influencing institutions, at which point the distinction between grassroots and institution begins to blur. This is indeed necessary, because a grassroots movement alone can not redraw the allocation of time and resources. The following scenarios demonstrate this challenge. 

For many PhD students, being able to code can directly improve the sustainability and quality of their work. Yet, many bachelor’s and master’s degrees haven’t yet incorporated training in these skills. In these situations, only students who have the time and resources can extend their training with extracurricular courses. Next, consider the situation of academics applying for grants. They may face the choice between quality and quantity of research, feeling that the system incentivises publishing as many papers as possible in high impact journals over any efforts to implement reproducible methods. Fortunately, policy change can help in these scenarios, as we argue below.

Funders

Funders have a lot to gain from incentivising researchers to adopt open research practices. It provides increased opportunities for the quality and reusability of software to be improved, and also potentially raises the impact and visibility of that software across the community. This not only makes research funding more efficient and productive by increasing adoption and reducing the wasted effort of re-inventing the wheel, but also increases the opportunities for collaborations between research groups. Since funders are in key positions of authority to initiate the change, the scale of change - and economic savings - have the potential to be significant.

One method of incentivisation for funders would be to launch calls with ring-fenced software development funding as the norm, ending the traditional approach of achieving development implicitly within projects, and bringing it out into the open. In addition, by encouraging open publication of software outputs from granted projects and in the proposal stage rewarding those who openly publish their software, software is put on a more equal footing with publications that depend on it, and the quality and productivity of research will improve as a result. These initiatives would help create an environment of innovation and adoption of best practices in cutting-edge research. Aspirationally, cohesive and concrete agreement across the whole of UKRI for what this would mean in practice would engender consistency of expectations whenever software development is planned within proposals - of particular benefit to interdisciplinary projects - further reinforcing the message that best practice is to be encouraged.

Publishers

Publishers are ideally placed to help reinforce the use of best practices for open research. This will not only ensure that these best practices reach a large audience but will also gain the publishers more traffic. They can incentivise researchers to adhere to best practices for open research by requiring data and/or code to be submitted along with publications, whenever possible. Doing so would level the playing field in the “rush to publication” for the researchers who are not following best practices and the others who are already abiding by them, and push academia further along the road of open, verifiable and more robust research. At the same time, the number of retracted papers due to errors in the scientific process and questionable research practices would be reduced, maintaining and strengthening the publishers’ academic standing and reputation. Additionally, if publishers initiate and support community peer review of software used to generate results for publications, the adoption of best practices in research software development becomes incentivised and can help software be placed on an equal footing with publications. 

University departments

How can university departments help incentivise open research practices? Departments are uniquely placed to bring about large changes in this area. They are in control over which researchers get hired and what is taught to the next generation of researchers. In terms of policy, hiring committees (both for internal and external positions) could require potential candidates to demonstrate how they intend to conduct open and reproducible research, and even to commit to this practice if hired. Job listings could also be required to explicitly state that good software practices and reproducible research are part of the job. Additionally, institutions can provide opportunities for members to learn more about good software practices. This includes adapting the curricula of their BSc and MSc degree programs, and providing continuous training to researchers, including postgraduate students and academics, on how to do good open research.

Recently, major progress has been made with the introduction of research software engineers (RSEs) to university departments, who fill skill gaps and reduce the need for current faculty to train and adapt themselves. While RSEs tend to fulfil a technical role, the success of this model may be extended to education. Indeed, in recent years there has been an increasing demand for software training across all levels of higher education. The engineer and educator roles could be combined or separate. In either case, considerable integration with existing lecturers (e.g. in courses such as statistics) is likely to be necessary. The procedures and materials used by researchers may be just as important as the final product (often a publication). Therefore, these intermediary stages may require proportionate levels of attention.