The incident involves the blue screens of death for Windows machines that interrupts normal operation with a message: "Recovery: It looks like Windows didn't load correctly."

A driver update from CrowdStrike relating to their Falcon Sensor security software has been identified as the root cause of the issue.

CrowdStrike published a workaround which involves rebooting a Windows computer in safe mode and deleting the culprit driver file[s].

Oh Joy!

https://en.wikipedia.org/wiki/July_2024_global_cyber_outages
2/n

Microsoft/CrowdStrike issue workaround from CrowdStrike -

- Boot Windows into Safe Mode or the Windows Recovery Environment
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Locate the file(s) matching “C-00000291*.sys”, and delete it.
- Boot the host normally.
🛠️
https://theconversation.com/massive-global-it-outage-hits-banks-airports-supermarkets-and-a-single-software-update-is-likely-to-blame-235107
#Outage #Microsoft
3/n

Massive global IT outage hits banks, airports, supermarkets – and a single software update is likely to blame

Enterprises across Australia and the world are struggling to cope with a huge software outage – but your home computer is probably safe.

^{The Conversation}

It's not like any of us have ever introduced a bug in our software that has affected customers and systems ... worldwide 😬

OTOH we follow the mantra - test, test, test 😎
4/n

George Kurtz, President & CEO CrowdStrike, tweeted about 2 hours ago that -
"The issue has been identified, isolated and a fix has been deployed."

Wonder what "deployed" means. How does one deploy the fix (delete certain driver files) to remote devices that cannot boot normally? 🤔

So, are planes flying again?

#Outage #Microsoft #CrowdStrike
5/n

The offending software is called the "CrowdStrike Falcon Sensor software."

R U ready for some AI-powered upgrades?

"CrowdStrike Falcon® Complete Next-Gen MDR utilizes AI-native technology and world-class expertise to stop breaches across the entire enterprise attack surface."

https://www.crowdstrike.com/blog/crowdstrike-unifies-threat-data-and-ai-for-mdr/

#Outage #Microsoft #CrowdStrike
6/n

Microsoft recommends restoring Windows from backups. Easier said than done.

For Virtual Machines running Windows Client and Windows Server, VM restarts (as many as 15 may be required) seem to be effective??

https://status.cloud.microsoft/
#Outage #Microsoft #CrowdStrike
7/n

Steps for public cloud or similar environment including Virtual Machines:

Detach the OS disk volume from the impacted virtual server
Create a snapshot or backup of the disk volume as a precaution
Attach/mount the volume to to a new virtual server
Navigate to the C:\Windows\System32\drivers\CrowdStrike dir
Delete files “C-00000291*.sys”
Detach the volume from the new virtual server
Reattach the fixed volume to the impacted virtual server

More at https://www.eye.security/blog/crowdstrike-falcon-blue-screen-issue-updates
#Outage #CrowdStrike
8/n

CrowdStrike Falcon and Microsoft blue screen issue updates

Stay updated on the BSOD issue caused by the CrowdStrike Falcon and Microsoft update.

^{Piet Kerkhofs (Eye Security)}

Snapshot of some outage graphs from https://downdetector.com/

Not the kind of rising lines one wants to see on Friday or any other day.

The disruption is widespread, far beyond the more visible ones in the airline industry.

#Outage #CrowdStrike
9/n

My guesses for the cause of this CrowdStrike driver update worldwide snafu -

1. A last minute "trivial" change to the software after all testing was completed
2. Something went wrong in the packaging or delivery or installation of the software update.
3. Some AI-generated code segment 😜

What else can you think of?

#Outage #CrowdStrike
10/n

Another lesson perhaps for organizations running mission-critical services - do not auto-update all your servers and clients is one fell-swoop.

Stage them. With some soak time in between.

This is quite standard practice when pushing our own custom software into our own distributed network products.

#Outage #CrowdStrike
11/n

xkcd comic for today 😅

Title text: We were going to try swordfighting, but all my compiling is on hold.

Touché.

Source and explanation: https://www.explainxkcd.com/wiki/index.php/Main_Page
#Outage #CrowdStrike #xkcd
12/n