After a week of outages that put IT resiliency to the test, CIOs are already preparing for the next disruption

URL has been copied successfully!

It all started on Thursday when a Microsoft outage disrupted workplace services including Outlook, Teams, and Copilot. One day later, the Federal Aviation Administration issued a ground stop that halted departures across the nation, upending travel plans for more than 1.5 million passengers. 

Even more disruptions have cascaded throughout this week: yet another outage affected banking websites and apps run by financial giants including Wells Fargo and Capital One, then at United Airlines, Apple, and a downed internal communications network at the National Security Agency.

There is no evidence that these systems disruptions affecting banking, airlines, tech giants, and the government are connected. But the heightened attention on disruptions that have affected major technology systems that most take for granted puts more attention on the reliability and resiliency of IT; how to communicate the latest developments of an outage to employees, customers, executives, and the board; and already has begun the process to develop post-mortem reports to assess how a system disruption was handled and what parts of their business continuity plans could be improved.

“Our approach is that resilience doesn’t really come from a document or a backup system,” says Jordan Ruch, chief information and digital officer at New Jersey-based health system AtlantiCare. “It is a muscle organizations build and it comes from planning with operations before something happens.”

Across corporate America, the biggest disruption to business operations this past week was the outage at Microsoft, which affected AtlantiCare. Ruch delineates different tiers of criticality for his technology systems. Clinical systems and core infrastructure have up to three areas of redundancy. In the event that a piece of equipment or network for those critical systems goes down, there’s always at least two backups.

Microsoft’s tools aren’t deemed that critical, so when that outage occurred, it slowed down “our ability to keep up with the demands of the health system,” according to Ruch. Because employees have been relying more on tools like Copilot, this event has spurred conversations internally at AtlantiCare to look at building more redundancy for those applications.

Marsha Stein, CIO at Ropes & Gray, says that the law firm had built in layers of redundancy that included an emergency notification system to communicate with employees when Teams was having issues last week and a secondary email system when Outlook was knocked out. “You can’t just say to a client, ‘Sorry, we don’t have any email, we can’t help you,’ as it turned out, for 18 hours,” says Stein. “It was really important that we had those systems in place so our lawyers could reach out to clients.”

By Tuesday morning, Stein spoke with Ropes & Gray’s governing managing committee for what she says will be an ongoing dialogue about what happened during the Microsoft outage and what could have gone better. “Nothing’s ever perfect,” says Stein, adding that leaders need to “think outside the box to think about what’s the next system that could be affected that we need to be prepared for.”

Shreyas Mysore, CIO at Lockton, says one challenge that the insurance brokerage and others faced last week was that Microsoft was slow to inform clients that its outage could have global implications. “I think that is one thing that Microsoft could have done better, giving us a heads-up of saying, ‘Look, there’s some critical infrastructure that’s starting to fail,’” says Mysore.

Communication is key when these events occur, and at Lockton, Mysore says the primary focus is first on encouraging brokers to reach out to their clients to share proactive updates, while also focusing internally on sharing information with leadership. The board can frequently be informed later—in this event, Lockton says he can wait to share the full debrief at that group’s next meeting in early December.

Mysore says the IT team will also produce a Microsoft outage incident report that looks at how the company was affected, the severity of the outage, and an internal checklist that assesses how executives were engaged, internal communications practices, and areas for improvement.

“I hold my vendors accountable for reliability, I hold myself accountable for resilience,” says Mysore.

Cybersecurity firm Fenix24 is able to move client conversations to alternative channels like Signal and WhatsApp when email systems are down. It also operates on a hybrid model, relying on Microsoft 365 but also an on-premises internal communications system. Heath Renfrow, chief information security officer and co-founder at Fenix24, says it is critical for enterprises to establish those alternative communications channels with employee groups and vital customer-client relationships before an outage occurs.

The Microsoft outage may also lead some firms to consider investing in more redundant communications systems like Google, especially for key executives. “That’s definitely a possibility and something you need to look at from a cost perspective, but it’s not a bad idea,” says Renfrow.

James Kretchmar, senior vice president and chief technology officer at cybersecurity company Akamai’s cloud technology group, says both vendors and customers need to focus on clear and consistent communication when outages occur. “It’s really important to get good communications out so that people who are relying on a service can understand how to best adapt their situation to what’s going on,” says Kretchmar. “You have to do a lot to get good, practical information out, but also without causing more problems.”

Microsoft, the FAA, and other organizations that have confronted outages over the past several days have to balance sharing realistic expectations for when their systems will be back online. Both under communicating and overpromising can hurt an organization’s reputation, according to Chuck Kesler, CISO at software company Pendo.

“I definitely have seen organizations sit on information too long, and that erodes trust,” says Kesler. But, he says if a system fix is anticipated to take two hours, it would perhaps be more ideal to share with key stakeholders that it may take four hours to resolve. “In my experience, things do usually end up taking just a bit longer than what you expect them to from a recovery perspective,” adds Kesler.

John Kell

This story was originally featured on Fortune.com

Share this story:
Share on Facebook
Share by Email
X (Twitter)
Follow Me
Tweet
Whatsapp
LinkedIn
Copy link

This post was originally published here.