Posted by j.laird on 26 August 2021 - 10:00am

Image by Rajesh Balouria from Pixabay 

By SSI Fellow Sorrel Harriet.

In 2019, I made it the focus of my SSI Fellowship to explore the landscape of research software engineering in terms of the people and processes underpinning it. The result was a small-scale study which sought to evaluate the impact of management and process on academic research software [1]. One of the more interesting insights to emerge from the project was the idea that research software projects are, in a sense, successful by default, (I’ll explain what I mean by that shortly). The project also touched upon some of the challenges faced by research software developers and how these align/differ from those of their commercial counterparts. 

Since completing the study my attention has moved even further towards software development as a collaborative activity, helped in part by my recent move to a cloud native consultancy where I’m constantly reminded about the socio-behavioural dimension of software development.

To cut a long story short, this all prompted me to begin exploring the fuzzy world of agile team coaching[2] as a means of supporting collaboration and improving team performance. Which raises the question, how much of it is applicable and useful to research software engineering?

In this article I will look at research software engineering from an agile team coaching perspective and suggest some lessons it could take from modern software engineering. The term ‘agile’ is something of a red herring here, in that I’m not looking at this in terms of specific agile methodologies. I’m looking at it more broadly and holistically, focusing mainly on the outer layers of Simon Power’s ‘Agile Onion’[3].

Image from Adventures with Agile

Lesson 1: Adopt a broader definition of success

The nature of academic research funding models means that research software projects are invariably very plan-driven, wherein success is defined in quite narrow terms. The software needs to support whatever high-level research goals it was intended for, and it needs to be delivered (roughly) on time and within a (relatively small) budget. If it manages to do that, it can be called a success. And why shouldn’t it? It’s a side-hustle for many researchers, a means to an end. It only needs to be good enough, then it can either be discarded or it becomes someone else’s problem. And yet… 

In adopting a narrow definition of success on research software projects, we may be ignoring the value that lies in failure. Put another way, if we concern ourselves solely with output, and not with the processes preceding it, we are missing out on valuable opportunities to learn. 

Agile (change-driven) delivery looks beyond a checklist of deliverables to consider the human success factors which contribute to the overall success of the project, and seeks to optimise the processes, culture and team composition accordingly. It’s a white box approach to software delivery, and seeing inside the system requires us to reframe success.

Lesson 2: Surface the pips

The phenomenon where things appear to be green on the outside (e.g. the project is reaching its milestones), but are actually red on the inside (e.g. there are serious unreported issues), is known by agile coaches as 'the watermelon effect’. Inside the watermelon we find pips (aka the small problems which didn’t seem worth surfacing). A single pip may seem insignificant, but group all the pips together and you have a gaping black hole. 

An effective coach won’t tell a team how they should be fixing themselves; they’ll enable the team to self-evaluate, and to challenge both the formal and informal processes that are generating the pips. Fundamentally this is about increasing communication and creating an environment in which it is safe to fail.

“Ok, so there were imperfections in our delivery processes  -  so what? We’ve done what we needed to do, and we’re all moving onto different projects now. Why should we care about a few pips?”  It’s a fair point, and maybe there are times when 'good enough' really is good enough, but there are other times when the pips left behind by one person become a major problem for the next person, leading to unnecessary rework or the creation of Frankenstein’s monster in code form. Even if you can’t eliminate them, acknowledging the pips is a step in the right direction and an important precursor to Lesson 3...

Lesson 3: Create opportunities to learn together

Once you have redefined success and started to expose your pips, the next stage is to start learning together. Learn what exactly? How to work better together; how to build quality in; how to reduce waste; how to build software and systems that can evolve over time…the list goes on. Achieving such feats on anything other than very small code projects requires extreme collaboration, which in turn requires a shift in mindset and potentially very different ways of working. (For example, by using pair and mob programming to achieve shared understanding and improve code quality [4]). 

The truth is, modern software engineering isn’t a lone developer hunched over a keyboard for days at a time; it’s a social activity that teaches us to collaborate. While progress can be slow at first and the feeling of exposure very uncomfortable, developing software together optimises for learning, creates a shared understanding of the work being done, and ultimately results in better quality software. Of course, learning is an incremental process and increments will often be very small but, like the pips in a watermelon, their cumulative effect can be transformative.

Lesson 4: Borrow the useful bits from Agile and DevOps

It doesn’t get more waterfall than most research projects, which might seem a little strange given that research, by definition, is fraught with uncertainty, but there’s a good reason for that. The end-to-end process of research needs to stand up to scrutiny. You can’t pivot half-way through a series of carefully controlled experiments without disqualifying half your data. In many ways, academic research seems incompatible with whatever’s trending in the commercial software engineering sphere, and maybe that’s why words like ‘agile’ and ‘DevOps’ don’t tend to get bandied around very much in academia  -  at least, not without provoking a few wry smiles.

But there is more to agile than iterative development and pointless metrics. There is the culture surrounding it, and research has shown that generative cultures which seek to optimise for collective learning are associated with higher performing software teams [5]. 

Moreover, in the execution of many a meticulously planned research project are the micro decisions which happen at ground level, on the fly. These micro decisions, often undocumented and potentially made by one developer without wider consultation, are rarely subjected to the same degree of scrutiny as those happening at the project level, yet the consequences for the research can be significant.

With all of that in mind, it is worth looking past the frameworks and jargon of agile to the underlying practices which foster collective learning, shared ownership and collaborative decision making, and identifying those which can transcend the commercial context. 

Somewhat ironically, one of the first things we could do in academia is to re-examine our approaches to learning software development. Developers in academia are overwhelmingly self-taught. What training does exist is often boot-camp like, undertaken outside of the learner’s real-world context, and targeted at individuals rather than whole teams. Meanwhile, industry is experimenting with faster and longer-lasting ways of transforming teams, often borrowing from sociocultural theories of learning. Take the dojo style of learning as one example, in which coaches and subject matter experts are embedded within the fabric of the team, taking them from guided practice to being full participants in a community of practice[6].

Of course, this kind of transformative learning doesn’t come cheap, but there are other, less expensive practices that research software developers could start using immediately to accelerate their collective learning. For example, they can borrow core practices from XP such as paired programming, coding standard and code refactoring to drive up quality[7]; they can create fast feedback loops in the form of weekly retrospectives, planning sessions and demos [8]. Coaches can support the learning and embedding of these practices, but individuals and teams can also self-coach, and they can start doing it today by heeding these four lessons: redefine success; surface pips; learn together; and borrow the useful bits from agile and DevOps.

References and further reading

1. Harriet, Groen, Crouch (2021) On the Impact of Management and Process on Research Software Development
2. Sorrel Harriet (2020) Could Coaching Contribute to More Sustainable Research Software?
3. Simon Powers (2018) What is Agile?
4. Donal Spring (2018) The Last Unicorn Developer | Pairing and Mobbing
5. Puppet Labs (2014) State of DevOps Report 2014
6. Marcio Sete (2018) The Learning Dojo
7. Ron Jefferies (2011) What is Extreme Programming?
8. Billie Thompson (2021) How to Agile dead good