Investigations for Software Failures – Cloudflare Outage

On November 18, you may have noticed that some of your websites were down for a few hours. A Cloudflare outage affected a significant number of major sites, including Twitter (yes, I still call it Twitter) and ChatGPT. The Cloudflare service is a global network service that performs security and caching duties for websites. It’s all in the background, but when it has an issue, it affects all of its client sites.

I saw that Cloudflare had released the “root cause” of the outage a few minutes after restoration:

“Many of Cloudflare’s services experienced a significant outage today beginning around 11:20 UTC. It was fully resolved at 14:30 UTC. The root cause of the outage was a configuration file that is automatically generated to manage threat traffic. The file grew beyond an expected size of entries and triggered a crash in the software system that handles traffic for a number of Cloudflare’s services.”

They further noted that it was not an external attack; it was an internal issue.

I find the term “root cause” is used here pretty loosely, which is how most companies use it when they are giving a quick explanation. I hope that they’ll look deeper at what in their processes and systems allowed this to happen, but I will hazard a guess (based on personal experience) that they will address Causal Factors only. That’s the fairly superficial practice I tend to see across the board when companies try to do “root cause analysis” without the benefit of built-in human performance expertise. I assume they’ll probably train their network folks and put out a policy on how to avoid this in the future. But perhaps they are an outlier (in a good way).

I would also guess that, looking at this issue, they are going to find it would have been pretty easy to see this coming. A file that exceeds a certain size and causes a global outage seems to this non-IT guy as something you should have safeguards already in place against. Proactive checks were likely going undone.

TapRooT® Root Cause Analysis should not be limited to safety, or quality, or any other department at your company. Anywhere you have humans, you can use TapRooT® to help. IT departments have lots of humans, so they are a prime opportunity for human performance improvement. Don’t limit your improvement opportunities to just the “classic” investigation areas. Look around your departments with a fresh set of eyes and ask yourself: “Where are my vulnerabilities, and what can I do to head them off early?” Don’t wait until your global customers are affected and demanding answers.

To schedule a free executive briefing about what it means to learn the TapRooT® System, visit our executive portal here.