Posted by m.jackson on 22 July 2014 - 1:51pm

By Mike Jackson, Software Architect.

As part of our open call collaboration with TPLS I was to develop a suite of unit tests. TPLS is written in FORTRAN and while there are de-facto standard unit test frameworks for Java (JUnit) or Python (PyUnit), for FORTRAN there are none. In this blog post I look at the test frameworks that are available for FORTRAN, compare two, FRUIT and pFUnit, and explain why I opted to use FRUIT for TPLS.

There are a few test frameworks available for FORTRAN listed on Wikipedia and Fortran Wiki. These include:

Name	Implementation	Licence	Active
FORTRAN Unit Test Framework (FRUIT)	FORTRAN with Ruby extensions	BSD-style	2006-2014
pFUnit	FORTRAN with Python extensions	NASA open source licence 1.3	2014
ObjecxxFTK - Objecxx Fortran ToolKit	FORTRAN with Python extensions	Commercial licence for a "modest licensing fee"	2014
flibs	FORTRAN with Tcl Tk extensions	BSD-style	2005-2008

 

 

 

 

 

 

 

 

 

 

 

 

Two others, FUnit (last active 2009) and FortUnit (last active 2004) had no code available and seem to be dead projects.

Out of the four available, I decided to look at FRUIT and pFUnit in more detail, as they are both actively maintained, unlike flibs, and are available under an open source licence, unlike ObjecxxFTK (See our guide on "Choosing the right open source software for your project" for guidance on choosing between alternative offerings.)

I'll discuss FRUIT and pFUnit shortly, after a digression to comment on...

Why are Ruby, Python or Tcl/Tk needed?

Unit test frameworks for Java and Python require only Java or Python to compile and run unit tests written to use them. So why do the FORTRAN test frameworks require these additional languages?

Java and Python support reflection, by which code can introspect upon itself to find functions or classes at runtime. Test frameworks such as JUnit or PyUnit use reflection to automatically find test classes or methods to run. They might look for functions named in a special way (e.g. prefixed by "test_"), methods that are members of a class that sub-classes a unit test class provided by the framework, or functions that are marked up by a framework-specific annotation (e.g. "@test").

For languages that do not support reflection, such as FORTRAN, or C, test frameworks require a developer to write test "driver" functions to explicitly call each of their test functions. This incurs a maintenance overhead - every time a developer writes a new test function, they need to remember to add it to a test driver function too.

Test framework developers write extensions in Ruby, Python or Tcl/Tk to pre-process FORTRAN code and identify test functions that are marked up according to a framework-specific convention. They then automatically generate FORTRAN code that implements these driver functions.

FORTRAN Unit Test Framework (FRUIT)

Here, I provide an overview of FRUIT, based on my experiences with FRUIT 3.3.4 downloaded on 02/06/2014.

FRUIT consists of two FORTRAN files which contain functions can be used to write tests. These contain "assert" sub-routines which can be used for different types of test e.g.

call assert_true(1 == 1, "Boolean test") call assert_false(1 == 1, "Boolean test fails") call assert_equals(123, 123, "Integer equality") call assert_equals(1.23, 1.23, "Double equality") call assert_equals(1.23, 1.3, 0.1, "Double equality within a tolerance") call assert_equals("abc", "abc", "String equality") call assert_equals(a, b, 3, 2, d, "2D array equality within a tolerance")

Test functions, which use these assert sub-routines, can be placed within a module e.g.

module twophase_io_fruit_test use fruit subroutine test_assert_examples() ... end subroutine test_assert_examples subroutine test_row_to_grid() ... end subroutine test_row_to_grid   subroutine test_row_to_grid_phi() ... end subroutine test_row_to_grid_phi end module twophase_io_fruit_test

Developers can write their own test drivers e.g.:

program manual_fruit_driver use fruit use twophase_io_fruit_test call init_fruit call test_assert_examples() call test_row_to_grid() call test_row_to_grid_phi() call fruit_summary end program manual_fruit_driver

When a test driver is compiled along with the test modules and the two FRUIT FORTRAN files, a stand-alone executable is produced. This standalone executable, when run, executes the tests and summarises the results:

$ ./test_fruit_driver Test module initialized . : successful assert, F : failed assert .F.F.F.F.F.F.F........................................................ Start of FRUIT summary: Some tests failed! -- Failed assertion messages: [_not_set_]:Expected [T], Got [F]; User message: [True fails] [_not_set_]:Expected Not [T], Got [T]; User message: [False fails] [_not_set_]:Expected [123], Got [321]; User message: [Equals fails] [_not_set_]:Expected [1.2300000], Got [3.3199999]; User message: [Equals fails] [_not_set_]:Expected [1.2300000], Got [1.3000000]; User message: [Fudge equals fails] [_not_set_]:Expected [abc], Got [cba]; User message: [Equals fails] [_not_set_]:Expected [4.0000000000000000], Got [123.00000000000000]; User message: [2d array has difference, Equals fails] -- end of failed assertion messages. Total asserts : 446 Successful : 439 Failed : 7 Successful rate: 98.43% Successful asserts / total asserts : [ 439 / 446 ] Successful cases / total cases : [ 0 / 0 ] -- end of FRUIT summary

Set-up and tear-down functions can be used to perform initialisation and finalisation operations common to all tests within a module:

module twophase_io_fruit_test use fruit subroutine setup() ... end subroutine setup subroutine teardown() ... end subroutine teardown subroutine test_assert_examples() ... end subroutine test_assert_examples subroutine test_row_to_grid() ... end subroutine test_row_to_grid subroutine test_row_to_grid_phi() ... end subroutine test_row_to_grid_phi end module twophase_io_fruit_test

To invoke these requires introducing a new module, termed a "basket" in FRUIT, that invokes the set-up and tear down functions, as well as the test functions themselves e.g.

module manual_fruit_basket use fruit contains subroutine twophase_io_fruit_test_all_tests use twophase_io_fruit_test call setup write(*,'(/A)') " ..running test: test_assert_examples" call set_unit_name('test_assert_examples') call run_test_case(test_assert_examples, "test_assert_examples") if (.not. is_case_passed()) then call case_failed_xml("test_assert_examples", "twophase_io_fruit_test") else call case_passed_xml("test_assert_examples", "twophase_io_fruit_test") end if call teardown ... end subroutine test_twophase_io_fruit_all_tests subroutine fruit_basket call twophase_io_fruit_test_all_tests end subroutine fruit_basket end module manual_fruit_basket

If using a FRUIT basket approach, the driver invokes the functions of the basket e.g.

program manual_fruit_basket_driver use fruit use manual_fruit_basket call init_fruit call init_fruit_xml   call fruit_basket   call fruit_summary call fruit_summary_xml call fruit_finalize end program manual_fruit_basket_driver

The above driver also contains additional calls that allow an XML report of the tests run and the successes and failures to be output. The XML report conforms to a standard XML schema used by a number of unit test frameworks including JUnit, Python's nosetests or CUnit.

FRUIT can create the FORTRAN test drivers automatically. This requires Ruby, Ruby's Gem package manager, and the Ruby rake build tool. FRUIT uses its Ruby pre-processor to parse a FORTRAN file containing test functions and auto-generate both basket ("fruit_basket_gen.F90") and driver ("fruit_driver_gen.F90") files. The constraints on each test file is that it:

• Ends in the suffix "_test.f90" e.g. "twophase_io_fruit_test.f90".
• Declares a module that ends in the suffix "_test" e.g. "twophase_io_fruit_test".
• Contains subroutines that take zero arguments and have the prefix "test_" e.g. "test_assert_examples".

The auto-generation can be done via running a simple Ruby script within the same directory as the test files. The Ruby script is quite concise, for example:

require 'rubygems' require 'fruit_processor' fp = FruitProcessor.new fp.pre_process

Compiling the tests in this scenario needs:

• The file with the tests e.g. "twophase_io_fruit_test.f90".
• Any files needed by these e.g. the code being tested.
• The auto-generated basket and driver files e.g. "fruit_basket_gen.f90" and "fruit_driver_gen.f90".
• FRUIT's two FORTRAN files.

The same test files can be used with the Ruby pre-processor or as part of the simple standalone FORTRAN approach described earlier. The advantage of using the pre-processor is it removes the need to maintain the basket and driver files and keep them up to date.

Some of the issues with FRUIT are that:

• A failed "assert" call does not cause a test function to exit immediately. The remaining lines of the test function are executed. This differs from other test frameworks which stop execution of a test function as soon as an assertion fails.
• A "." is printed when each "assert" call succeeds which can lead to an overwhelming number of "."s being displayed. Other test frameworks print a "." when each test function successfully completes. As FRUIT is open source, commenting out one line can suppress this!
• A large amount of boiler-plate code must be written to support "setup" and "teardown" if there is a desire to avoid using Ruby.
• FRUIT's own examples use Ruby Rake build files, not only to auto-generate the FORTRAN driver code but also to compile all source and test code. I think it is unreasonable to expect existing projects to migrate to a new build tool just to use FRUIT. However, as described, a simple Ruby script can be used to manage the auto-generation of driver code. This can then be run, manually or via a Makefile, for example.

pFUnit

Here, I provide an overview of pFUnit, based on my experiences with pFUnit 2.2 downloaded on 02/06/2014 due to issues with more recent releases and platforms available, described below.

With pFUnit, tests are written in FORTRAN-style ".pf" files with set-up, tear-down and test functions annotated e.g.

module twophase_io_pfunit_test use twophase_initialisation_wave use twophase_io use pfunit_mod implicit none contains @Before subroutine setup() ... end subroutine setup @After subroutine teardown() ... end subroutine teardown @Test subroutine test_assert_example_fail() ... end subroutine test_assert_example_fail @Test subroutine test_assert_fudge_fail() ... end subroutine test_assert_fudge_fail @Test subroutine test_assert_examples() ... end subroutine test_assert_examples end module twophase_io_pfunit_test

Test assertions are also expressed via annotations e.g.

@assertTrue(1 == 1, "Boolean test") @assertFalse(1 == 2, "Boolean test") @assertEqual(123, 123, "Integer equality") @assertEqual(1.23, 1.23, "Double equality") @assertEqual(1.23, 1.3, "Double equality within a tolerance", 0.1) @assertEqual("abc", "abc", "String equality") @assertEqual(a, b, "2D array equality within a tolerance", d) @assertEqual(a, b, "3D array equality within a tolerance", d)

The ".pf" files are pre-processed into FORTRAN files using pFUnit's Python pre-processor. The auto-generated FORTRAN code does the following:

#include "testSuites.inc"

So, developers must also write this "testSuites.inc" file listing the test modules, each with a "_suite" suffix e.g.

ADD_TEST_SUITE(twophase_io_pfunit_test_suite)

Compiling the tests in this scenario needs:

• The FORTRAN file auto-generated from the ".pf" file e.g. "twophase_io_pfunitTests_gen.F90".
• Any files needed by these e.g. the code being tested.
• The directory that contains "testSuites.inc".
• pFUnit's own source files, compiled libraries and other files.

Some of the issues with pFUnit are that:

• pFUnit needs a current version of the gfortran compiler, ideally 4.8.3+. Attempting to build the current release of pFUnit on a Scientific Linux 6 machine supporting gfortran 4.4.7 fails.
• However, attempting to use gfortran 4.8.2 gives an execution error also, but this is a known problem.
• pFUnit needs to be built and installed so that its libraries and include files are available. FRUIT, by contrast has no such requirement.
• Developers need to write and maintain not only FORTRAN files but also pFUnit-specific ".pf" and ".inc" files.

Picking FRUIT

For test frameworks based on languages that do not support reflection there is a trade-off between:

• Explicitly writing and maintaining test driver functions, ensuring these call all the test functions and updating these as new test functions are added.
• Using dependencies (e.g. Ruby or Python) outwith the language in which the code and associated tests are implemented.

The former can be a deterrent to writing tests due to the time and effort involved every time a test is written. The latter can prove a deterrent to writing tests in the first place due to the time to set up the additional dependencies required as well as having to learn additional tools, languages or file formats. This is on top of starting to write the tests.

For a software development project, involving software developers, I would recommend pFUnit as it offers a wider range of in-built test functions and generally feels like a more comprehensive and polished product.

However, for TPLS, a research project, where the developers are primarily researchers who may not be used to writing tests I picked FRUIT. FRUIT does not require any non-FORTRAN languages (whether these by Ruby, Python or pFUnit's own syntax) to be learned. It has a lower barrier to uptake.

FRUIT's core functionality is held within its two FORTRAN files, and there is no need to build and install any separate components. This, and its licensing under a BSD-style licence, allows them to be held within TPLS's Subversion repository so that all researchers have access to them when downloading TPLS. Though, I have recommended that TPLS developers keep an eye on FRUIT's repository for updates to its FORTRAN core and the copies in TPLS be updated accordingly, so TPLS can benefit from bug fixes or enhancements to FRUIT.

Code samples of both FRUIT and pFUnit are in the ssi branch of the TPLS SourceForge repository.

If you have any advice, hints and tips concerning unit testing in FORTRAN then please feel free to share these below.