Posted by a.hay on 25 February 2015 - 10:00am

By Steve Crouch, Research Software Group Leader, and Mark Basham, SSI Fellow and Senior Software Scientist at Diamond Light Source.

January 30th 2015 saw the latest Institute-sponsored Hackathon at Diamond Light Source, bringing together top coding talent from across the STFC Rutherford Appleton Laboratory and beyond.

Participants were able to propose interesting scientific and software development projects related to Python, and work on them in groups. One constraint, however, was that these groups be formed of individuals that don't normally work together, and this led to some surprising results.

Run as part of their Python community of practice events, the Diamond Hackathon attracted 30 scientists, researchers and engineers from ISIS, the Central Laser Facility and Diamond on the Harwell Campus, and also from the Culham Centre for Fusion Energy. Inspired by the Institute’s Collaboration Workshops, the event allowed attendees to submit ideas for projects before the event and vote on the best ones early on in the day.

This diverse set of participants, together with a healthy dose of motivation and caffeine, ensured a lively pitching of project ideas to attract enough votes to form development groups for candidate topics. It was great to see such enthusiasm sustained throughout the day, and this was reflected in the very positive event feedback. Asked how much they enjoyed the event, the average across all respondents was a very high 4.35 out of 5, with 93% indicating they would like to do another hackathon.

The projects undertaken were as diverse as the participants, from a database-backed distributed logging framework, which allows a developer to embed binary objects within a log entry for debugging later, to a particularly ambitious project to build a Raspberry PI-based system to manage video feeds from a multitude of sources.

The latter was to satisfy a use case emerging from the Central Laser Facility, where they have around 50 observational cameras per laser and where manually cabling these video streams can prove - to say the least - difficult. This highlights the attention the projects gave to addressing big challenges faced by facility scientists.

The high quality of the projects made judging them particularly difficult. However, two joint winners were eventually selected. The first winner was an automated help generation system for the CCTBX crystallography library, addressing its widely acknowledged lack of documentation.

By building a set of HTML help files automatically through pulling out the embedded help text, code examples and unit tests from the source tree, a developer can easily find the help they need. As a kind of meta-project to assist developers, this could benefit the entire CCTBX developer community.

The second winning project aimed to speed up data reads from netCDF 3D environmental data files by parallelising the investigative reading across multiple processes. NEMO is used by approximately 200 oceanographers and the approach could also benefit users in tomography and even Grid compute accounting through the APEL accounting tool. The most impressive aspect of this project was its interdisciplinarity.

In any case, all of the projects will continue their work after the hackathon, and it will be interesting to see how far they get.

Being an Institute-sponsored event, it was reassuring to see such a high standard of organisation and development practices across the groups, especially in a hackathon. When discussing their projects, the groups were clearly making good use of requirements and skills brought to each team, and it's always heartening to see revision control systems - in this case, Git - being used from the outset.

One team even employed test-driven development, and nearly all participants reported the event had helped them to improve their software engineering and Python programming skills to some degree. A particularly impressive outcome from the event was the extent that participants’ knowledge of Python libraries and projects was improved, with respondents reporting an average 4.07/5 for this aspect. Such feedback reflects the high degree of knowledge and skills transfer that occurs at hackathons - particularly this one.