By Mike Jackson, Software Architect, Iain Bethune, EPCC, The University of Edinburgh​, Lennon Ó Náraigh, School of Mathematical Sciences, University College Dublin, and Prashant Valluri, Institute of Materials and Processes, School of Engineering, The University of Edinburgh.

Mathematical modelling of complex fluid flows has practical application within many industrial sectors including energy, the environment and health. Flow modelling can include oil and gas flows in long-distance pipelines or refinery distillation columns, liquid cooling of micro-electronic devices, carbon capture and cleaning processes, water treatment plants, blood flows in arteries, and enzyme interactions. Multi-phase flow modelling models flows consisting of gases, fluids and solids within a single system e.g. steam and water or oil and gas within a pipe, or coal dust within the air.

Simulations of this sort are highly computationally intensive, so high-performance computing (HPC) resources are required. However, current commercial computational fluid dynamics (CFD) codes are limited by a lack of efficient multi-phase models, poor numerical resolution and inefficient parallelisation features. This severely restricts their application within both academia and industry. Industry, for example, continues to rely on empirical modelling and trial-and-error pilot-scale runs, which incurs significant capital cost investments and delays before commissioning.

TPLS (Two-Phase Level Set) is a CFD code developed by Prashant Valluri of the Institute of Materials and Processes, School of Engineering at The University of Edinburgh and Lennon Ó Náraigh of the School of Mathematical Sciences, University College Dublin. TPLS uses an ultra-high 3D Direct Numerical Simulation approach combined with the Level-Set method for tracking the developing interface between phases. TPLS employs a 2D message passing interface (MPI) process decomposition coupled with a hybrid OpenMP parallelisation scheme to allow scaling to 1000s of CPU cores. TPLS has been optimised for HECToR, the UK's former national supercomputer. This performance and scalability work was undertaken in collaboration with Iain Bethune and David Scott of EPCC and funded by HECToR dCSE and EPSRC grants. TPLS is designed to address the limitations of commercial CFD codes and provide a simulation capability that is unrivalled in computational efficiency and numerical accuracy.

In May 2013, TPLS was released on SourceForge under a BSD open source licence. The code, in FORTRAN-90, is held within a Subversion repository. However, there are a number of researcher-specific versions of TPLS under development. Whether or not these are held under revision control is up to an individual researcher. These versions may contain algorithms and/or configuration parameters that form part of the researcher's work and which they may not want to make public until they have published the results of their work.

To date, the TPLS team have focused on code development, optimization, testing and rigorous validation. Improving the usability of TPLS was not a priority. Consequently, TPLS has not yet been adopted by users outwith the TPLS team's own research groups to the extent that was hoped. The main reasons for the lack of dissemination appears to be the limited documentation and the difficulties users have both in using TPLS and modifying it for their own research.

The recent launch of ARCHER, the UK's new National Supercomputing Service, and HECToRs successor, also saw the launch of a complementary eCSE funding scheme. The TPLS team have been awarded eCSE funding to work upon ARCHER to continue the parallelisation and optimisation of TPLS to further improve its robustness, flexibility and performance. Coincident with this, the team applied to us for consultancy in software development and open source best practice as part of the Institute's open call. We are working with TPLS to help to improve the usability, maintainability and sustainability of TPLS. The Institute's support is for 2.5 months and the key activities include preparing documentation, improving usability, code quality and maintainability and helping to organise the user community.

Completing these activities will deliver a number of benefits. Users will be able to get started with TPLS, understand its capabilities, deploy TPLS as a binary package and apply TPLS within their own research, all without having to modify any code, or understand FORTRAN-90. This will allow TPLS to be used by non-developers. There will be no need to strip out researcher-specific configuration parameters from the code before updating the version of TPLS on SourceForge. TPLS will be more configurable and modular, reducing the need for researchers to develop their own versions of TPLS. Rather, they will be able to develop components that plug-in to TPLS. TPLS developers will be able to test TPLS to ensure that any changes they have made have not introduced any bugs. Users interested in extending, modifying, or fixing TPLS will be able to understand the architecture of TPLS, to implement their changes, and know how contribute these changes to TPLS, if they wish to do so. The TPLS team will be able to manage and integrate, contributions from users and from their own PhD researchers, as well as requests for help and support in an open and systematic way.

These complementary, eCSE and open call, projects will help to increase the number of users and developers of TPLS, within both academia and industry, by improving both the ease with which users can adopt TPLS and the effectiveness with which TPLS can scale up to exploit powerful HPC resources like ARCHER. Together, it is hoped these will allow TPLS provide a simulation capability that is unrivalled in computational efficiency and numerical accuracy.