Technology

76918 readers

3409 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related news or articles.
Be excellent to each other!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
Check for duplicates before posting, duplicates may be removed
Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago

MODERATORS

L3s@lemmy.world

enu@lemmy.world

technopagan@lemmy.world

L4s@lemmy.world

L3s@hackingne.ws

L4s@hackingne.ws

276

Cloudflare blames massive internet outage on 'latent bug' (techcrunch.com)

submitted 1 day ago by tonytins@pawb.social to c/technology@lemmy.world

31 comments fedilink hide all child comments

Around the same time, Cloudflare’s chief technology officer Dane Knecht explained that a latent bug was responsible in an apologetic X post.

“In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack,” Knecht wrote, referring to a bug that went undetected in testing and has not caused a failure.

you are viewing a single comment's thread
view the rest of the comments

[–] fushuan@lemmy.blahaj.zone 2 points 1 day ago (1 children)

They probably mean that they did a change in a config file that is uploaded in their weekly or bi-weekly change window, and that that file was malformed for whichever reason that made the process that reads it crash. The main process depends on said process, and all the chain failed.

Things to improve:

make the pipeline more resilient, if you have a "bot detection module" that expects a file,and that file is malformed, it shouldn't crash the whole thing: if the bot detection module crahses, control it, fire an alert but accept the request until fixed.
Have a control of updated files to ensure that nothing outside of expected values and form is uploaded: this file does not comply with the expected format, upload fails and prod environment doesn't crash.
Have proper validation of updated config files to ensure that if something is amiss, nothing crashes and the program makes a controlled decision: if file is wrong, instead of crashing the module return an informed value and let the main program decide if keep going or not.

I'm sure they have several of these and sometimes shit happens, but for something as critical as CloudFlare to not have automated integration tests in a testing environment before anything touches prod is pretty bad.

[–] groet@feddit.org 5 points 1 day ago (1 children)

it shouldn't crash the whole thing: if the bot detection module crahses, control it, fire an alert but accept the request until fixed.

Fail open vs fail closed. Bot detection is a security feature. If the security feature fails, do you disable it and allow unchecked access to the client data? Or do you value Integrity over Availability

Imagine the opposite: they disable the feature and during that timeframe some customers get hacked. The hacks could have been prevented by the Bot detection (that the customer is paying for).

Yes, bot detection is not the most critical security feature and probably not the reason someone gets hacked but having "fail closed" as the default for all security features is absolutely a valid policy. Changing this policy should not be the lesson from this disasters.

[–] fushuan@lemmy.blahaj.zone 1 points 1 day ago (1 children)

You don't get hacking protection from bots, you get protection from DDoS attacks. Yeah some customers would have gone down, instead everyone went down... I said that instead of crashing the system they should have something that takes an intentional decision and informs properly about what's happening. That decision might have been to clo

You can keep the policy and inform everyone much better about what's happening. Half a day is a wild amount of downtime if it were properly managed.

Yes, bot detection is not the most critical....

So you agree that if this were controlled instead of open crahsing everything them being able to make an informed decision and opening or closing things, with the suggestion of opening in the case of not detection is the correct approach. What's the point of your complaint if you do agree? C'mon.

[–] groet@feddit.org 1 points 1 day ago

You don't get hacking protection from bots

I disagree. I don't know the details of cloudflares bot detecion, but there are many automated vulnerability scanners that this could protect against.

I said that instead of crashing the system they should have something that takes an intentional decision and informs properly about what's happening.

I agree. Every crash is a failure by the designers. Instead it should be caught by the program and result in a useful error state. They probably have something like that but it didn't work because the crash was to severe.

What's the point of your complaint if you do agree?

I am not complaining. I am informing you that you are missing an angle in your consideration. You can never prevent every crash ever. So when designing your product you have to consider what should happen if every safeguard fails and you get an uncontrolled crash. In that case you have to design for "fail open" or "fail closed". Cloudflare fucked up. The crash should not have happened and if it did it should have been caught. They didn't. They fucked up. But, i agree with the result of the fuck up causing a fail closed state.