On-Call Reprised and Rejuvenated
In a recent informal survey about the training provided by companies for those going On-Call: 25% reported no training at all, 30% indicated training was available as a wiki (with maybe a one-time overview session), 40% reported all of the above with a "shadow shift" on top, and only 5% said they performed some kind of scenario or tabletop practice outside of incidents.
Even with a small sample set, it is clear that many places treat incidents as extra work with no dedicated training. There has been good research done on the importance of training responders how to take action in ambiguous situations. For instance, arctic rescuers have shown the critical importance of relying on improvisation and practicing it as a team. With this in mind, we set out to reboot our On-Call process at Blameless in the Spring of 2022. Not only would we switch from a monolithic rotation to service team ownership, we also developed a set of procedures and documentation that we worked on together.
In fact, shaping how we respond to incidents is a never-ending process, one of continuous improvement and reflection. This talk is about that improvement: the struggles we had with our rotation before, the challenges we endured as we faced organizational change, the pressures that pushed and pulled at the successful implementation of the new paradigm, and most importantly the approach to education and training that would carry the program forward and make it last.