In a rare moment of corporate culpability, Cloudflare CTO Dane Knecht has taken full responsibility for yesterday’s outage that took large swathes of the internet dark, knocking out some of the world’s most popular sites and apps including Spotify, X (Twitter) and ChatGPT.
Posting on Twitter, once it was up and running again, Knecht admitted his company had let customers and “The broader internet” down, apologised to users, and promised transparency about what had gone wrong. The CTO also confirmed that the meltdown was not due to an attack, at least one silver lining for a company that is heavily involved in protecting customers from cyberattacks, and good news in a more general sense following massive confirmed cyberattacks on household names including Marks & Spencers, Jaguar Landrover and Co-op, as well as last month’s high-profile outage at another behind the scenes online giant, AWS.
His statement in full read: “I won’t mince words: earlier today we failed our customers and the broader Internet when a problem in @Cloudflare network impacted large amounts of traffic that rely on us. The sites, businesses, and organizations that rely on Cloudflare depend on us being available and I apologize for the impact that we caused.
“Transparency about what happened matters, and we plan to share a breakdown with more details in a few hours. In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack.
“That issue, impact it caused, and time to resolution is unacceptable. Work is already underway to make sure it does not happen again, but I know it caused real pain today. The trust our customers place in us is what we value the most and we are going to do what it takes to earn that back.”
Issues seemed to begin for users in the early hours of Tuesday (November 18) UK time, with Spotify among the first sites to come under scrutiny by users unable to get online. As the morning wore on, more and more sites which relied on Cloudflare infrastructure began to drop offline, including internet monitoring service Down Detector, which made it tough to keep up to date on developments, and Prolific North itself, which made it tough to report them. Services were returning to normal by yesterday mid-afternoon.
Stuart Morris, chief technology and product officer at Leeds online security specialist SmartSearch, told Prolific North: “The irony of Cloudflare – one of several major key internet infrastructure service providers – being hit by outages is not lost on the digital community. Cloudflare services include “edge computing” and tools to protect websites from cyberattacks which help them stay online during distributed denial of service (DDOS) attacks and heavy traffic periods.
“In the past 12 months, internet services – from AWS to Crowdstrike – seem to be succumbing to ‘outages’ and perhaps, attacks from bad actors. Trust in these types of services is hard fought and regularly taken for granted by end users, and it is imperative for the sake of the brands who keep us safe every day that more is done to support them when things go wrong. In periods of political turmoil and in an increasingly digitalised world, more needs to be done to verify identities and fortify security. Digital identity should be an area of particular concern, especially as identity is increasingly commoditised by bad actors to conduct financial crime.
“Other than just being an inconvenience, the Cloudflare outage serves to remind us of our continued vulnerability online.”
