Warning: please be aware that Travis CI, which is mentioned in this article, has a security issue with its Free Tier service. By design, “secret” data such as access credentials are exposed within historical clear-text logs which are accessible by anyone via the Travis CI API. Please see this article for more information.

By Mike Jackson, Software Sustainability Institute, Raniere Silva, Software Sustainability Institute.

Continuous integration ensures that your software is built and tested regularly. It can help you to demonstrate that your software does what it claims to do, and that it does so correctly. It also helps you to rapidly release bug-fixes and more functional versions of your software. Continuous integration can also be used to automate experiments that run using software.

Why write this guide?

We wrote this guide to complement our guide on testing your software and provide a more in-depth overview of continuous integration. We also thought it would be of interest to consider how continuous integration servers could be used to automate the running of certain experiments and the reporting of their results.

Why are automated build and test important?

In our guide on testing your software we explain the importance of automated build and test. As a refresher...

An automated build helps you to easily and rapidly build your software. Whether your automated build is implemented using Make, ANT, NAnt, Maven, or just some shell scripts - invoking a few, simple, automated build commands is faster, easier and less error-prone than having to invoke numerous complex commands via a command-line or a GUI.

Automated tests are important - if your software is incorrect then this may affect the quality of the results it produces. In the worst case, you may find that your results are wrong and be forced to publish embarrassing errata or even retract your papers. Automated tests help you to ensure that your software behaves as you intend, produces the correct results, and behaves well if things goes wrong, e.g. if incorrect data is input.

Automated tests also allow you to fix or improve your software, secure in the knowledge that you'll detect any new bugs that you've introduced. Nothing is worse than fixing a bug only to discover that it's been replaced with a new one!

The benefit of continuous integration for automated build and test

Continuous integration is a way of ensuring that software is tested regularly. A continuous integration server automatically gets the current version of the software, rebuilds the software, and runs the tests. It then notifies the developers about the success or failure of the build and tests.

It is possible to implement automated build-and-test using something like a simple CRON job on Linux/UNIX. However, continuous integration servers offer a number of advantages. They publish build and test results within a structured, web-based dashboard to make it easy to see the status of the build and tests, the successes, the failures and reasons for these. They can also present information on builds and tests in progress, and aggregate build-and-test runs from multiple developers. Continuous integration servers can also support various forms of notifications, for example, emails or RSS feeds. The continuous integration server runs on its own machine so the developer can continue to work on his own machine while the test are under way.

The most significant advantage of a continuous integration server is its ability to monitor changes to source code held under revision control. If the code changes, then the server can automatically spawn a new build-and-test job. This means that the software is rebuilt and tested every time the code is changed. Typically, continuous integration servers will also allow build-and-test jobs to run at specific times, so a CRON-like, nightly-build-and-test, can be done, as well as a build-and-test job run on-demand.

This is why continuous integration helps your software to always be releasable: tests are run in response to changes to the code, and you are notified quickly when tests fails so that you can correct the reason for the failure  It is easier to fix a bug in something you wrote a few minutes ago, than something you wrote yesterday (or last week, or last month).

Some servers, for example TeamCity, run tests before new code is committed to the repository. They intercept changes to the repository, build the updated code, run the tests, and only allow the code to be stored in the repository if the test succeeds. This means that test-failing code that will break the build will not be added to the repository. Though this barrier to committing code is useful, it can prove frustrating to developers.

Which research projects use continuous integration?

There are a number of research projects in the UK that use continuous integration. Many use Travis, a well-established open-source hosted service that works with projects hosted on GitHub and free to use for open-source projects, or Jenkins, a well-established open-source continuous integration server that projects host, install and configure themselves. For example, the SEEK platform uses Travis, and the Taverna scientific workflow management system uses Jenkins to avoid bugs when their web-based services go live. The results of SEEK's and Taverna's build-and-test jobs are made publicly available, respectively, at seek on Travis and Taverna's instance of Jenkins. Other users of continuous integration include:

• The Orbital research data management project at The University of Lincoln. They have blogged about their motivations for using Jenkins.
• OSCAR (Open Source Chemistry Analysis Routines) an open-source extensible system for the automated annotation of chemistry in scientific articles at the Department of Chemistry at The University of Cambridge. OSCAR's dashboard is publicly available.
• GAP, a open-source System for Computational Discrete Algebra, now developed and maintained by the Centre for Interdisciplinary Research in Computational Algebra at the University of St Andrews and others. GAP's dashboard is publicly available.

Travis is based on Ubuntu virtual machines. For projects that want Windows-based testing, AppVeyor offers similar functionality, again free for open-source projects. GAP and recipy both use AppVeyor to test their source code under Windows.

Popular continuous integration products

Besides Travis, Jenkins and AppVeyor, other popular continuous integration products include GitLab CI, TeamCity, Hudson and CruiseControl. However, there are a whole range of products available which differ in everything from whether they are open source or closed source; their support for parallel job execution; integration with clouds; how they publish build and test results; their support for publishing test reports (e.g. JUnit reports formatted as HTML); which revision control tools they can monitor; which build tools they can invoke; how tests are triggered; and how they perform notification. Wikipedia has a comprehensive list of products and their key features.

Before selecting a product to use, consult our guide on choosing the right open-source software for your project - much of this advice applies to closed-source software too.

If instead of hosting, installing and configuring your own continuous integration service you decide to go with a hosted service check out our blog post on hosted continuous integration servers.

Could I use a continuous integration server for running experiments?

Yes!

As a researcher you might use software to run experiments, such as running a simulation or performing data analysis. At its simplest, a continuous integration server just runs stuff and publishes the results in a structured way. If a continuous integration server allows you to run shell scripts then it allows you to run your experiments and view the results. Many continuous integration servers can be triggered to run a job at regular intervals or on-demand - allowing you to configure them to run experiments regularly if you wish. The advantage of a continuous integration server over a CRON job is the support for revision control, build and scripting tools, a results dashboard and notifications.