The following guidance on good practice in submitting software outputs to REF 2021 has been developed by the UK Software Sustainability Institute in consultation with REF 2021 Sub-panel 11. It should be read in conjunction with the information already provided in the Guidance on Submissions (REF 2019/01, Annex K) and Panel Criteria and Working Methods (REF 2019/02, Paragraph 254 and Annex B).
The submission should be in the form of a PDF that provides key information on the software (no length limit, within reason). To ensure the sub panel has enough information to assess the submission, it should include the following components
Access Details
DOI/URL pointing to a code repository or deposit in a digital repository containing:
- source code;
- compiled code and/or instructions on how to compile or install the software;
- user documentation;
- test suite and/or quality assurance procedures.
Where the submitted software is part of a larger code base (e.g. a library contributed to an open source project) it must be clear which part the submission refers to. Ideally, the DOI/URL should reference a specific version of the software.
Publication Details
Evidence to show when and how the submitted software was first brought into the public domain (which must be within the REF 2021 publication period). If the submission was an enhancement of pre-existing software, the date when the enhancement was published should be given, and the remainder of the submission should make a clear distinction between the enhancement and the existing software (which may need to be described to provide context).
Authorship Details
Where the software is the work of multiple authors, they should be listed where possible, whether from the submitting unit or not. In every case the role of the individual to whom the output is attached should be made clear.
Research Process
A summary of the research process of which the software is the output and the insights it embodies, where research is defined for REF purposes as a process of investigation leading to new insights, effectively shared (see Guidance on Submissions, Annex C for an expanded definition).
[NB the Criteria and Working Methods (paragraph 254) allow for the submission of a separate description of the research process and content, ‘where this is not evident within the output’. If this is covered in the software submission itself, as suggested here and in the Guidance on Submissions, Annex K, the separate statement is not necessary].
Description of the Software
A description of the purpose of the software, the context in which it is used, and its functionality (300 words maximum).
Supporting Evidence
Numbered list of other publicly accessible sources of supporting evidence. This might include:
- DOI for a companion software paper;
- DOIs for research outputs resulting from use of the software;
- Usage metrics.
Addressing the REF Assessment Criteria
REF2021 will assess outputs on the basis of their Originality, Significance and Rigour, as defined in Panel Criteria and Working Methods paragraphs 191-193. The following subsections provide further suggestions for how these assessment criteria might be evidenced for software outputs.
Originality
The REF definition of originality is the extent to which an output makes an important and innovative contribution to understanding and knowledge in the field. For software this might include, but is not limited to:
- implementation of a novel algorithm;
- development of innovative research methods;
- a solution to a problem that has not been solved before in practice;
- interpretation of or engagement with novel forms or scales of data;
- improvement in the scale, resolution or accuracy at which research can be performed.
Significance
The REF definition of significance is the extent to which the work has influenced, or has the capacity to influence, knowledge and scholarly thought, or the development and understanding of policy and/or practice. For software this might include, but is not limited to:
- enabling or accelerating significant research advances;
- enabling research or thinking that was not previously possible;
- influencing policy, practice or users/audiences;
- being used by a large number of researchers, or within a diverse range of disciplines.
Rigour
The REF definition of rigour is the extent to which the work demonstrates intellectual coherence and integrity, and adopts robust and appropriate concepts, analyses, sources, theories and/or methodologies. For software this might include, but is not limited to:
- being used to produce peer-reviewed research outputs;
- clear documentation that explains:
- the software purpose and intended users, including the research context;
- how the software works, including the research process it enables;
- architecture and design rationale, to provide confidence in the appropriateness of the approach;
- practices to improve reproducibility of results, e.g. automation, recording of provenance, use of standard data formats;
- a clear strategy for internal software quality assurance, e.g. a unit test suite to provide confidence in research results, and consistent coding standards to ensure readability of code;
- formal verification;
- adoption of community-accepted standards;
- certification;
- code and documentation held in version control;
- regular releases;
- structured end-user evaluation;
- quality-based peer review (e.g. publishing a software paper in a specialist software journal, or receiving an ACM artefact badge).