A former chaos engineer offers 5 tips for handling online disasters remotely

Kolton Andrus Contributor

Kolton is co-founder and CEO of Gremlin, the chaos engineering company helping the world build a more reliable internet.

I recently had a scheduled video conference call with a Fortune 100 company.

Everything on my end was ready to go; my presentation was prepared and well-practiced. I was set to talk to 30 business leaders who were ready to learn more about how they could become more resilient to major outages.

Unfortunately, their side hadn’t set up the proper permissions in Zoom to add new people to a trusted domain, so I wasn’t able to share my slides. We scrambled to find a workaround at the last minute while the assembled VPs and CTOs sat around waiting. I ended up emailing my presentation to their coordinator, calling in from my mobile and verbally indicating to the coordinator when the next slide needed to be brought up. Needless to say, it wasted a lot of time and wasn’t the most effective way to present.

At the end of the meeting, I said pointedly that if there was one thing they should walk away with, it’s that they had a vital need to run an online fire drill with their engineering team as soon as possible. Because if a team is used to working together in an office — with access to tools and proper permissions in place — it can be quite a shock to find out in the middle of a major outage that they can’t respond quickly and adequately. Issues like these can turn a brief outage into one that lasts for hours.

Quick context about me: I carried a pager for a decade at Amazon and Netflix, and what I can tell you is that when either of these services went down, a lot of people were unhappy. There were many nights where I had to spring out of bed at 2 a.m., rub the sleep from my eyes and work with my team to quickly identify the problem. I can also tell you that working remotely makes the entire process more complicated if teams are not accustomed to it.

There are many articles about best practices aimed at a general audience, but engineering teams have specific challenges as the ones responsible for keeping online services up and running. And while leading tech companies already have sophisticated IT teams and operations in place, what about financial institutions and hospitals and other industries where IT is a tool, but not a primary focus? It’s often the small things that can make all the difference when working remotely; things that seem obvious in the moment, but may have been overlooked.

So here are some tips for managing incidents remotely:

There were many nights where I had to spring out of bed at 2 a.m., rub the sleep from my eyes and work with my team to quickly identify the problem… working remotely makes the entire process more complicated if teams are not accustomed to it.