•

How One Bad CrowdStrike Update Crashed the World’s Computers

https://www.wired.com/story/crowdstrike-outage-update-windows/

A defective CrowdStrike kernel driver sent computers around the globe into a reboot death spiral, taking down air travel, hospitals, banks, and more with it. Here’s how that’s possible.

How One Bad CrowdStrike Update Crashed the World’s Computers

https://web.archive.org/web/20240719155854/https://www.wired.com/story/crowdstrike-outage-update-windows/

"CrowdStrike is far from the only security firm to trigger Windows crashes with a driver update. Updates to Kaspersky and even Windows’ own built-in antivirus software Windows Defender have caused similar Blue Screen of Death crashes in years past."

"'People may now demand changes in this operating model,' says Jake Williams, vice president of research and development at the cybersecurity consultancy Hunter Strategy. 'For better or worse, CrowdStrike has just shown why pushing updates without IT intervention is unsustainable.'"

25 comments

floofloof

•

How did the update get through testing, if the bug has an immediately obvious catastrophic effect?

Telorand

•

Agreed, this seems like a pretty obvious failed smoke test.

Three options seem likely to me: the build was untested, the final package got corrupted after testing, the test environment has some kind of abberant config that hid the defect.

jlh

•

Kernel drivers are "reviewed" and signed by Microsoft for exactly this reason. It's a security risk if any program an administrator runs could load malicious kernel drivers into windows

Rolder

•

Something I have heard (take with a grain of salt) is that there was a new windows update that went out just before the crowdstrike update. And the issue happened with the new windows update.

EncryptKeeper

•

Not the case. I have dozens of servers last updated in May that crashed.

radivojevic

•

Pretty good test to see how easy it would be to shut the world down. Uninstall CrowdStrike.

floofloof

•

I'm sure Russia is taking note. Its computers were unaffected due to having no Crowdstrike installations. China too, apparently.

radivojevic

•

I’m sure they have their own solution for that, but yes, it would be unwise for a government to install software maintained by a foreign country. Kind of like voting booths.

jlh

•

China and Russia are switching to Linux, too.

lnxtx

•

Read-only friday, right? Right...?
Poor sysadmins.

MeekerThanBeaker

•

And this happens the same week that Kaspersky left the U.S.

They are laughing internally right now.

greybeard

•

Kaspersky has caused BSODs because of updates in the past as well. Hardly an AV maker hasn't. The problem here is that Crowd Strike has captured the enterprise market in a large portion of the globe.

MeekerThanBeaker

•

Oh I'm well aware. I hated deploying Kaspersky. But we switched to Crowdstrike last year and now this happened. Just a funny coincidence.

Luckily, we're a small company and a third use Macs. The others, well, I had three PC laptop and one virtual server issues. Not too bad. We're on the West Coast so glad I was aware of it last night when Australia got issues.

paraphrand

•

Every time our IT tells PC users to just leave their computer on for X hours so it gets updates, I wonder how that can be a great system.

LodeMike

•

No it crashed shitty systems run on Windows lol. Actual computers are fine.

Guest_User

•

Two quick points, given the massive impact of this eveny it is clear to say many critical systems run windows. Meaning them being windows doesn't make them any less "actual computers".

Also, the OS in this event is irrelevant. They could have botched an update to their Linux version and crashes all the Linux boxes leaving windows untouched. This was not a result of an issue of any OS but a bad update.

daddy32

•

They are less of an actual computers in a sense that they are not running stuff under their owner / operator control. This would happen in Linux with much lower chances, because there are no side update channels to such a critical component of the system used there.

However, to take back what I just wrote :) - I am sure rightly motivated engineers would be able to build such a security hole into Linux too, under enough pressure from bad corporate decisions.

bamboo

•

What do you mean “no side update channels”? There are lots of software that update outside of a distro repo and lots of software that pulls metadata from the internet that could cause an error in the parser.

LodeMike

•

Linux stuff generally doesn't crash if a file gets deleted. It'll just fail to boot.

Guest_User

•

Neither does window. A file deletion did not cause this. A human at Crowdstrike uploaded a bug to production. Bugs in production can happen on any OS, this is just a terrible, terrible look for Crowdstrike because they seriously messed up

Rolder

•

I mean, the end result would be the same: Large tracts of infrastructure not loading and causing hell

flambonkscious

•

Have you read anything about this? A file deletion is the workaround for affected hosts, silly!

LodeMike

•

flambonkscious

•

I was just trying to point out that you implied a file deletion is what's causing this, and Linux wouldn't crash. This fault is fixed by deleting a file, ironically

Ozymandias88

•

https://www.neowin.net/news/crowdstrike-broke-debian-and-rocky-linux-months-ago-but-no-one-noticed/

Faulty drivers don't discriminate by o/s.