ProtoMS passes the test: developing a test suite for biomolecular simulation software

Posted by s.aragon on 20 July 2016 - 4:56pm

DevasenablogJuly.jpgBy Devasena Inupakutika and Steve Crouch, Software Sustainability Institute, and Richard Bradshaw, University of Southampton.

As a part of their research, Jonathan Essex’s Research Group developed ProtoMS, a biomolecular simulation software that allows the simple development of methods for the calculation of relative protein/ligand binding free energies. The Software Sustainability Institute worked with them as part of an Open Call project to develop a test strategy and Python test suite, and to verify the operation of the ProtoMS software as an overall product. The great news is that the latest release now includes the test suite and has already found some interesting issues which have been resolved.

A firm and stable unit test suite is crucial for ongoing development in large projects. Writing unit tests adds value to a project while reducing the cost of code changes. With our aim to explore the software for its accessibility and usability and how to adopt a decentralised approach that can reduce strain on further development, we examined each unit of the ProtoMS Python code separately.

The first thing on Google’s “What we believe” list of Ten things we know to be true is: Focus on the user and all else will follow. Thus, unit tests that focus on real user applications scenarios make sense. Hence, a test strategy was devised after conducting the development review of the software from the user and developer perspective.

Developing the test suite

Based at Southampton, the ProtoMS team put together sets of reference data for test cases and documented the overall setup, simulation and analysis testing strategy along with the end-to-end information flow in the software for development of the test suite. During the whole test suite development process, we ran different applications and use-cases ensuring that those units work together at every step.

The team also supports Intel compilers with ProtoMS. We checked for the consistency and correctness of the test suite results with both Intel and GNU compilers, with the hope that eventually we have a set of tests that the ProtoMS team is confident with and that are passed with both sets of compilers and the default compilation options. During the development of these aspects of the test suite, a couple of compiler-specific bugs and formatting issues with Intel compilers popped up which were readily fixed.

Richard asked us to include some profiling of the routines touched by the developed test suite. We included the record of the coverage of the test suite at the end of the project. Details of the coverage have helped them find gaps or routines that haven’t been tested yet and can be used as information by future developers in identifying other uses that might fill these gaps.

Tests to understand behaviour

The suite of the unit tests helped the ProtoMS team in understanding the behaviour of the existing features, fixing bugs, and further documenting their design. It also made it easier to expand their code and continue to ensure that changes to the code will not break existing features. They said that “by running through the tests, we identified a few old bugs in the ProtoMS Fortran code”. Richard also commented that “I’m pleased to say everything now passes, across multiple platforms, and is ready for release!”

The ProtoMS team was very engaged in the collaboration, and helped me in getting a clear view of ProtoMS architecture, tools, functionality, and applications at every step throughout the review and test suite implementation. Issues highlighted and the recommendations made during the review have already been addressed and made publicly available. The test suite was incorporated into the last public version of ProtoMS. They also say that work has already begun to extend the test suite to cover new developments in the ProtoMS code base, and, of course, we look forward to hearing about the ProtoMS team’s experiences with the test suite in the future.

Image courtesy of Richard Bradshaw.

.