By James Acris, Patrick Austin, Eli Chadwick, Ivan Finch, and Kyle Pidgeon.

This is part 1 of a 2 parts blog post. The second part can be found here.

Introduction

Carpentries workshops provide significant benefits to early career employees at STFC. In 2022, our team ran a series of hybrid Software Carpentry workshops across October and November, building on previous experience running in a hybrid format in 2021 and fully online in 2020.

Our hybrid format featured a roughly 50/50 mixture of online participants using Zoom and in-person attendees spread across two rooms on the same site, totalling just under 40 attendees. All instructors broadcast their teaching over Zoom, even if they were in one of the rooms on-site, meaning the pacing was the same for everyone and any attendee could ask questions regardless of location.

This blog post summarises our experiences with organising and running these workshops, including attendee feedback.

Organisation / Administration

Significant extra effort is required to run a hybrid workshop, compared to either an in-person or an online one. This extra work comes in at the “bridging” point where the two formats meet.

During the workshop, effective live communication between team members at different locations is essential. A dedicated helper/organiser is needed on the day to manage rooms, communicate with helpers, and support the instructor in making sure every attendee is able to follow the course. We referred to this role as the “host.” It is also crucial to ensure organisers are attending in both formats, as this prevents any attendee’s needs from being overlooked.

Another administrative challenge with the hybrid format is that technical issues can cause everyone in the workshop to lose time, even if they only affect a subset of attendees. For example, if a physical room with 12 attendees loses connection, the whole workshop of 40 people must wait for them, or the instructors may decide to move on and leave those attendees behind. Neither option is ideal.

One alternative is to deliver the online and in-person components of the workshop completely separately – even on different days. This saves bridging work and can prevent technical issues from affecting too many people, but it does require twice as much instructor time, which is often a very limited resource.

Flexibility

The hybrid format provided a great deal of flexibility for the attendees. It is notable that even though most attendees work on-site at least a few days per week, their preference was split roughly 50/50 on attending online or in person.

One bonus of the format is that it is adaptable to last-minute changes. For example, when a burst water main meant buses couldn’t reach the site, affected helpers and attendees could switch from in-person to online attendance on the day of the workshop. Less extreme reasons for changes also included sickness and childcare.

Our post-workshop surveys showed that both online and in-person attendees were happy with this flexibility, as we received only neutral or positive feedback for the workshop format (see below).

Engagement

One big difference between the physical and online breakout rooms was the amount of interaction between attendees. A common observation relayed from other workshops, which aligns with our own experience, is that when everyone is in the room together, there is a lot of conversation and attendees will often help each other. However, if everyone is online and placed into breakout rooms of 5-6 people, those rooms can be very quiet, with attendees’ video and microphone off unless they are asking for help. We even found that many attendees stopped joining the breakout rooms as time went on.

There was also next to no uptake on the Zoom reactions for feedback, and this, combined with the quiet breakout rooms, made it difficult for helpers and instructors to know if online attendees were keeping up.

As we ran the new Intermediate Research Software Development (IRSD) course after the regular Unix, Git and Python curriculum, we noticed that online engagement decreased as the material became more advanced. This may be partly due to the nature of the IRSD course – we ran it in an instructor-led format, but the course is “intended primarily for self-learning” and has fewer opportunities for quick multiple-choice style questions. However, in-person engagement remained good throughout, indicating that the course material was not the sole problem.

Perhaps more fundamentally, not all the attendees seemed to be coding along with the instructors. This had a more significant impact in the IRSD where the content heavily depends on previous code having been written.

Pacing

The pace of delivery is always a challenge due to the variability in how attendees learn, regardless of the workshop format. We found that the hybrid nature of our workshops, alongside the large attendance, exacerbated this problem and made general solutions hard to find.

One issue we found was that if in-person groups of attendees began discussions about exercises whilst completing them, this was hard to communicate back to the instructor. Moving on with the workshop then left those in-person behind.

Another pacing issue is that it takes more time for online attendees to type a question in the Zoom chat compared to an in-person attendee who can raise a hand immediately. This can mean that a request for a command to be repeated can be communicated too late after the instructor has already moved on. At this point, it is more challenging to catch up, creating a negative feedback loop. We used the Etherpad to compensate for this issue where possible (see below).

Tools

We used several online tools to share material and engage with participants, including Etherpad, Zoom, and Slido.

The tool that got the most use was Etherpad, which we planned to use to share sign-in sheets, important links, and some exercise material. However, it quickly became clear that there was a demand amongst participants to have instructors’ code and command history to hand, so our Etherpad morphed into a copy-paste feed for that, managed by a helper/host. This was an okay solution, but over time, the document became very long, and it was not easy to go back and find a particular command in amongst all the text. It also required a lot of work and communication between the host and instructor to ensure that all the code/command history made its way into Etherpad.

Attendee Feedback

At the end of the Unix and Git course, we asked attendees: Was the hybrid format a bonus, a detriment, or neutral to your workshop experience?

In 2021, from 39 responses:

In 2022, from 27 responses:

Most attendees found the hybrid format neutral to their experience – it did not affect their learning positively or negatively. More attendees found it positive than negative, and this was much more pronounced for online attendees than for in-person attendees. In 2021, almost 25% of in-person attendees found the format detrimental to their learning - but in 2022, no attendees at all reported that the format was detrimental.

In 2022 we also received written feedback on the hybrid format:

• ‘Good hybrid system, helpers were helpful and well-knowledged, enjoyable work structure, nice to meet new people’
• ‘Great teaching, hybrid was done very well - helpers always ready to support. Everything was well explained. Would have liked to have been able to attend in person’
• ‘the helpers and instructors were very valuable and great teachers. The online and in-person hybrid format of the workshops worked well’
• ‘hybrid worked well, it was good to practice myself while seeing what I was doing/copying on the big screen’

The feedback we received that directly refers to the hybrid format is very positive, suggesting that hybrid workshops are well worth considering despite their organisational challenges. However, we received mixed feedback on engagement and pacing, both factors that were impacted by the format in ways that weren’t obvious to attendees.

In 2022, our Net Promoter Score (NPS) (a proxy for the question “would you recommend this workshop?”) was in the 90s after Unix and Git and settled at 74 after all the workshops were completed. This indicates that our delivery of the IRSD workshop in particular could do with some improvement, but also that hybrid workshops with the core Software Carpentry curriculum can be very successful.

Conclusion

Running hybrid workshops is harder for organisers, but provides significant benefits to attendees.

It is critical to have organisers attending in both formats, to prevent attendees’ needs from being overlooked.

Attendees appreciate the ability to choose the format that suits them best, as well as the freedom to switch formats in an emergency.

Engaging online attendees can be challenging, and this makes it harder to know if the pacing is working for them. For more advanced courses where opportunities to check in with attendees are less regular, the disparity between in-person and online attendee engagement becomes stronger.