Cloudflare blames database error for 20% internet outage incident

Cloudflare, a major internet infrastructure and security provider, knocked down roughly 20% of the internet on Tuesday morning following a technical failure. Major websites, including ChatGPT, X, Coinbase, Spotify, and Toncoin, went offline. 

The internet security provider powers traffic for approximately one-third of the top 10,000 websites. The firm initially investigated the root cause of the issue as a potential cyberattack, but later confirmed that no malicious activity was involved.

Cloudflare has detailed the cause of the disruption, attributing it to a latent bug in its Bot Management System.

The outage sparks discussions about the need for decentralized internet systems 

According to the detailed report presented by Cloudflare, the failure was triggered by a change in one of their database systems’ permissions, which caused the database to output multiple entries into a “feature file” used by the Bot Management System.

The bot is designed to manage automated bot traffic and protect websites from distributed denial-of-service (DDoS) attacks, which have doubled in size and were then propagated to all the machines that make up the network, causing the software to crash.  

Dane Knecht, Cloudflare’s chief technology officer, issued a public statement apologizing for the disruption, stating that the company had failed its customers and the broader internet ecosystem. He reiterated the cause of the glitch, noting that a security breach was not the issue, but rather a configuration problem that had gone undetected during testing. 

The outage affected approximately 20% of all websites and lasted for about three hours, starting at 6:20 a.m. ET to around 9:30 a.m. ET when multiple services began resuming normal operations.  The service outage comes barely a month after a similar incident occurred at Amazon Web Services. At least 10,000 websites and apps were affected, raising concerns across the internet community about single points of failure in centralized internet systems. 

Alp Toker, director of NetBlocks, described the incident as a catastrophic disruption to Cloudflare’s infrastructure, highlighting the significant dependence of the internet on the infrastructure to protect against DDoS attacks and maintain uptime. 

According to Jack Moore, a global cybersecurity advisor at ESET, companies often have few choices but to rely on Cloudflare, Microsoft Azure, and AWS for hosting, which makes the scale of outages more impactful.

EthStorgae, a decentralized blockchain web stack, echoed the view shared by Ethereum co-founder Vitalik Buterin that it is necessary to preserve trustless and decentralized systems in a “Trustless Manifesto,” warning that centralized systems introduce a single point of failure. 

Cloudflare’s stock slips 3% following the incident

Cloudflare has since implemented a fix and is monitoring the system to ensure complete stability. According to Knecht, the company’s CTO, they are conducting further checks to prevent similar incidents and will provide a full, in-depth report later. 

The stock price of Cloudflare dropped by approximately 3% following the incident, raising concerns about the reliability of its services. So far, the stock is still down 2.83% over the past 24 hours, trading at $196.53 on the NYSE. 

The company confirmed that it has data centers in 330 cities and approximately 13,000 networks directly connected to it, including major ISPs, cloud providers, and enterprises. Based on the firm’s report, the following services were affected, including the Core CDN and security services, which gave the HTTP 5xx status codes.

HTTP status code triggered by Cloudflare’s outage. Source: Cloudflare

Other services that failed included Turnstile, Workers KV, Dashboard, email security, and Access.
The incident raised questions about internet resilience across the industry, with multiple observers noting that while these services provide significant help, the outages recorded last month at AWS and Cloudflare demonstrate the fragility of the infrastructure and its potential to cascade disruptions from a single provider.

Cloudflare has so far emphasized that the outage is unacceptable and has promised to enhance its monitoring and testing procedures. 

Get up to $30,050 in trading rewards when you join Bybit today