Photo by Janko Ferlič

The Open Evidence Bank (OEB) is a curated collection of accessible articles and data that contribute to our understanding of the research software landscape. Our aims for the Open Evidence Bank are to:

• Create an open registry of relevant research, 
• Ensure that research is easily discoverable and accessible by the community,
• Provide evidence to underpin policy and best practice.

What are the inclusion criteria?

We include outputs in a diversity of forms, including peer reviewed papers and preprints. Where possible, data sets and analysis code used for the publication should be open. We make exceptions for this where data is considered sensitive. The research must be open to all, so where a publication is behind a paywall, there must be an option to read an associated preprint or report that describes the research in full. The publication must have a DOI.

OEB entries

We have classified the entries into the following categories: People, Infrastructure and Policy based on which area of research software landscape they are relevant to. 

People 

Research Software Engineers: State of the Nation Report 2017

• Synopsis: The report describes the new expert community that has flourished in UK research, details the successes that have been achieved, and the barriers that prevent further progress.
• Citation: Brett, Alys, Croucher, Michael, Haines, Robert, Hettrick, Simon, Hetherington, James, Stillwell, Mark, & Wyatt, Claire. (2017, April 6). Research Software Engineers: State of the Nation Report 2017. Zenodo. http://doi.org/10.5281/zenodo.495360
• Data
• Code
• Blog post
• Tags: Research Software Engineers, career paths

A brief overview of the Research Software Engineering landscape in the Netherlands and several other countries

• Synopsis: An overview of the Research Software Engineering landscape in the Netherlands. It includes a summary of the results from a survey held in December 2017 in the Netherlands and also several other countries.
• Citation: B. van Werkhoven, T. Bakker, O. Philippe and S. Hettrick, "Survey on Research Software Engineering in the Netherlands," 2018 IEEE 14th International Conference on e-Science (e-Science), Amsterdam, 2018, pp. 38-39.  https://doi.org/10.1109/eScience.2018.00017
• Data
• Code
• Tags: Research Software Engineers, career paths

International survey of RSEs

• Synopsis: Survey of RSEs starting from the UK in 2016, and adding more countries (Canada, Germany, the Netherlands, South Africa United States, Australia and New Zealand) over 2017 and 2018.
• Citation: Olivier Philippe; Martin Hammitzsch; Stephan Janosch; Anelda van der Walt; Ben van Werkhoven; Simon Hettrick; Daniel S. Katz; Katrin Leinweber; Sandra Gesing; Stephan Druskat; Scott Henwood; Nicholas R. May; Nooriyah P. Lohani; Manodeep Sinha. (2019, March 6). softwaresaved/international-survey: Public release for 2018 results (Version 2018-v.1.0.2). Zenodo. http://doi.org/10.5281/zenodo.2585783
• Data
• Code
• Blog post
• Tags: Research Software Engineers, software sustainability, career paths

Raising the status of software in research: A survey-based evaluation of the Software Sustainability Institute Fellowship Programme

• Synopsis: The study is focusing on understanding the benefits that the fellowship has afforded its recipients and raising the status of software in research.
• Citation: Sufi S and Jay C. Raising the status of software in research: A survey-based evaluation of the Software Sustainability Institute Fellowship Programme [version 1; peer review: 3 approved with reservations]. F1000Research 2018, 7:1599. https://doi.org/10.12688/f1000research.16231.1
• Data
• Code: This is a qualitative study so there are no associated code.
• Tags: career paths, software sustainability

Infrastructure

An investigation of software-related research outcomes recorded in ResearchFish

• Synopsis: This analysis investigates software-related research outcomes recorded in ResearchFish.
• Citation: Simon Hettrick, & Oliph. (2017, January 18). softwaresaved/ResearchFish: 1.1 release of software to investigate research outcomes (Version v1.1). Zenodo. http://doi.org/10.5281/zenodo.250494
• Data
• Code 
• Blog post
• Tags: software tools, research output

Survey of software in research from 2014

• Synopsis: Survey of software used in research conducted in 2014.
• Citation: Simon Hettrick. (2018, February 23). softwaresaved/software_in_research_survey_2014: Software in research survey (Version 1.0). Zenodo. http://doi.org/10.5281/zenodo.1183562
• Data
• Code
• Blog post
• Tags: software tools

Survey of software used in research from 2014-2016

• Synopsis: The report contains the combined results from 5 surveys on software used in research run between 2014 and 2016. 
• Citation: Hettrick, S., Philippe, O., Chue Hong, N., Sufi, S., Silva, R., & Peru, G. (2016). Software used in research based on combined surveys [Data set]. Zenodo. http://doi.org/10.5281/zenodo.60276
• Data
• Code
• Blog post
• Tags: software tools

Forking as a Tool for Software Sustainability — An Empirical Study

• Synopsis: Forking—the process of cloning a repository, allowing development to progress separately to the original project— has become an important part of source control functionality. As forking enables software reuse, it has the potential to play a role in software sustainability, which aims to improve the longevity of software. We studied the relationship between forking and the sustainment, or active life, of a software project. An examination of 9,118 projects hosted on GitHub shows a significant relationship between forking and software sustainment, with projects that have forks being, on average, sustained for longer than those that do not, a phenomenon that is true when considering both the length of the original project, and the length of the original project extended by any forks. The results provide evidence that maintaining software via forking is a sustainable software practice, and making software open source improves its sustainability through enabling reuse via forking.
• Citation: S. Alhozaimy, R. Haines and C. Jay (2017). Forking as a Tool for Software Sustainability — An Empirical Study. Proceedings of the Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE5.1). http://dx.doi.org/10.6084/m9.figshare.5328796.v1
• Data
• Code
• Tags: software tools, Research Software Engineers

Policy

A study of software mentions in grants

• Synopsis: A study of software mentions in grants found in UKRI "Gateway To Research" repository to tease out how much money do the Research Councils invest into research that relies on software?
• Citation: Simon Hettrick, & Steve Crouch. (2019, January 30). softwaresaved/software_in_grants_GTR: Release for publication (Version 1.0). Zenodo. https://zenodo.org/record/2553194
• Data 
• Code
• Blog post
• Tags: software tools, grants

What Makes Research Software Sustainable? An Interview Study With Research Software Engineers.

Open access version: https://arxiv.org/abs/1903.06039. 

• Synopsis: Software is now a vital scientific instrument, providing the tools for data collection and analysis across disciplines from bioinformatics and computational physics, to the humanities. The software used in research is often home-grown and bespoke: it is constructed for a particular project, and rarely maintained beyond this, leading to rapid decay, and frequent ‘reinvention of the wheel’. Understanding how to develop sustainable research software, such that it is suitable for future reuse, is therefore of interest to both researchers and funders, but how to achieve this remains an open question. Here we report the results of an interview study examining how research software engineers – the people actively developing software in an academic research environment – subjectively define software sustainability. Thematic analysis of the data reveals two interacting dimensions: intrinsic sustainability, which relates to internal qualities of software, such as modularity, encapsulation and testability, and extrinsic sustainability, concerning cultural and organisational factors, including how software is resourced, supported and shared. Research software engineers believe an increased focus on quality and discoverability are key factors in increasing the sustainability of academic research software.
• Citation: De Souza, M. R., Haines, R., Vigo, M., & Jay, C. (2019). What Makes Research Software Sustainable? An Interview Study With Research Software Engineers. In 12th International Workshop on Cooperative and Human Aspects of Software Engineering, Montreal, Canada, 25/05/19. https://doi.org/10.1109/CHASE.2019.00039
• Data: We do not have permission to share the original transcripts, which contain identifying information.
• Code: This is a qualitative study so there are no associated code.
• Tags: Research Software Engineers, software sustainability, career paths