The Reproducible Research Vibe from the Fellows 2014 summer meeting

Posted by s.sufi on 8 July 2014 - 10:15am

Nine of this year’s Fellows met in sunny Southampton on June 23rd-24th 2014 to discuss various aspects of reproducible research and how it would shape future engagement with their research domains.

From the discussions that took place over the following two days, it became clear that the UK research community is focussing on Open Access, Open Science and Open Data, and that the time is ripe to build on these endeavours and promote the necessity and benefits of reproducible, computationally derived results. 

To do this, the Fellows agreed that targeting researchers who were unaware of reproducible research should be a priority for the Institute. This, of course, will require simple and convincing messages about the benefits and need of reproducibility for often time pressed but influential people such as Principal Investigators.

As one fellow, Olexandr Konovalov, stated, “if there is no code, there is no paper”, which expressed the sentiment of the Computational Statistics community, and has implications for the wider research community too. This requirement to make code available should be food for thought for many domains that have not yet developed their own levels of reproducibility.

There was a healthy discussion on the tools and techniques that people use to support reproducibility. These were seen as necessary but Stephen Eglen argued that the research mind-set needed to change in order for true sufficiency to be achieved. As he said, researchers joining his team are told to “assume that one day their code will be shared with others”, thus effectively baking in reproducibility into their day-to-day work. If other groups adopted this model, there would be a significant move towards opening up research software for all.

There were some in-depth discussions about spreadsheets and GUIs and the role they can play in obscuring the transparent approach to research that is required for reproducibility. Even though these tools may be a good start point (and may well be mandated), the broad consensus was that making your analysis scriptable was essential for proper reproducibility. Of course, this has the additional benefit that script-based systems scale better when processing bigger data sets. In other words, it’s time that computational research moved beyond the spreadsheet.

Something we discuss a lot at the Institute is the definition of “software”, and the fact that many researchers discount their own coding as not being “proper software”. Some of the Fellow were surprised that, as far as the Institute is concerned, they were "doing software" even though they were only using a few modules of a statistical package and some additional scripts. There are many more people writing software than many researchers realise, which is why we are keen to develop best practice sustaining research software.

There was a general agreement that training and education were of particular importance to reproducibility. There was a lot of interest in the latest offering from Software Carpentry, currently known as Data Carpentry, which teaches data handling skills. The Fellows were also keen for training in how to teach computational skills, assess the impact of training (through questionnaire design), effectively advertise courses, and how to organise the volunteers who are keen to help in these efforts. The Fellows felt that learning these skills would help them better serve their communities and the push for reproducibility.

The Fellows meeting could be summarised as two days of great ideas, fantastic feedback and scintillating conversation. It provided the Institute with many new avenues to explore, and confirmed in my mind that our Fellowship has successfully identified a group of researchers who are leading the research community into a new era of cross-disciplinary and reproducible research.

Share this page