Happened as I started my workday and was resolved around then too. Reckon they detected it around 11PM last night and resolved it 7AM-ish in the western USA.
Cloudflare: “A Cloudflare spokesperson said the “root cause” of the outage was an automatically generated configuration file used to manage threat traffic that “grew beyond an expected size of entries,” which triggered a crash in the software system that handles traffic for several of its services.
What can one infer from “grew beyond an expected size of entries,” ? Asking as a complete cloud idiot and nothing else. What does this mean?
The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was triggered by a change to one of our database systems’ permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.
The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.
After we initially wrongly suspected the symptoms we were seeing were caused by a hyper-scale DDoS attack, we correctly identified the core issue and were able to stop the propagation of the larger-than-expected feature file and replace it with an earlier version of the file. Core traffic was largely flowing as normal by 14:30. We worked over the next few hours to mitigate increased load on various parts of our network as traffic rushed back online. As of 17:06 all systems at Cloudflare were functioning as normal.
thanks I was not chasing conspiracies or weird sh*t just the Tech Talk.
Mistook a huge DDOS for a permissions problem? seems a whole lot weirder than reason #1.
“which caused the database to output multiple entries into a “feature file” used by our Bot Management system. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.”
I’m lost here ^. Relaxed permissions mistakenly given to the file that allowed it to accumulate quadruple the info the original permissions were set for? Do I have that right? And AI was involved
I don’t get it. Am walking backwards slowly. Thanks for the official followup info.
A file the systems reference for bot management, was larger than what is supported by that system.
That file then got copied across the Cloudflare network causing service outages.
DDoS attack mitigation is one of Cloudflare’s key features, so I can understand why that might be their knee-jerk reaction. It’s a part of the product description And security is quite often the first concern that comes to mind in IT.
But as is also typical in IT, the issue is actually more likely to be a typo somewhere that caused an otherwise inconspicuous log file to blow out of proportion
What? They are not telling the truth. It was because Hillary and Bill Clinton did not want us to know about the truth of the Epstein files. Now you see no one is talking about Epstein files anymore. They are all focused on this. I am telling you the deep state conspired along with Hillary and Bill.
this is absolutely not the cra-cra rabbit hole I was going down, but lovely satire nonetheless since deflection and distraction are the du jour claims of both sides anymore…
“it’s in the trees/Look out!!”—opening line, Kate Bush, Hounds of Love
Well I was going to go with the latest craze, i.e. 3I/Atlas interstellar comet, before going with lock her up crowd. On how 3I/Atlas is an alien craft and how it heralds our doom. In a way it does portend our extinction but well that is a different story.
Have to really hand it to these theory folks. They can be so creative and imaginative. But that is the issue. To make progress and break the boundaries one has to be a bit crazy or to put it politely, deviate from the expected norm.
you like to butt heads with yourself as much as I do. I respect that as this challenges beliefs as well.
sometimes the ‘conspiracy people’ have it all 100% documented and are constantly proven right (this is an apolitical observation that serves neither side as I am independent and not a partisan). this is very common and under-reported/exposed.
other ‘conspiracy people’ are batsh** cra with no foundation or shred but taken by others as gospel. Also apolitical as their is no good guys/ bad guys paradigm.
The long game is to keep the waters as muddy as possible so nothing can be ascertained except through gut. Maybe this is my own conspiracy theory?
And security is quite often the first concern that comes to mind in IT.
