😂 TechCrunch: CrowdStrike offers a $10 apology gift card to say sorry for outage

https://techcrunch.com/2024/07/24/crowdstrike-offers-a-10-apology-gift-card-to-say-sorry-for-outage/ #crowdstrike #outage

Southwest Airlines was spared by #CrowdStrike outage because... they're running Windows 3.1 🤯

Resilience through obsolescence 🤷

https://www.tomshardware.com/software/windows/windows-31-saves-the-day-during-crowdstrike-outage

#ITOutage #outage

Windows 3.1 saves the day during CrowdStrike outage — Southwest Airlines scrapes by with archaic OS

Windows 3.1 and Windows 95 save the day.

^{Jowi Morales (Tom's Hardware)}

Another lesson perhaps for organizations running mission-critical services - do not auto-update all your servers and clients is one fell-swoop.

Stage them. With some soak time in between.

This is quite standard practice when pushing our own custom software into our own distributed network products.

#Outage #CrowdStrike
11/n

My guesses for the cause of this CrowdStrike driver update worldwide snafu -

1. A last minute "trivial" change to the software after all testing was completed
2. Something went wrong in the packaging or delivery or installation of the software update.
3. Some AI-generated code segment 😜

What else can you think of?

#Outage #CrowdStrike
10/n

Snapshot of some outage graphs from https://downdetector.com/

Not the kind of rising lines one wants to see on Friday or any other day.

The disruption is widespread, far beyond the more visible ones in the airline industry.

#Outage #CrowdStrike
9/n

Steps for public cloud or similar environment including Virtual Machines:

Detach the OS disk volume from the impacted virtual server
Create a snapshot or backup of the disk volume as a precaution
Attach/mount the volume to to a new virtual server
Navigate to the C:\Windows\System32\drivers\CrowdStrike dir
Delete files “C-00000291*.sys”
Detach the volume from the new virtual server
Reattach the fixed volume to the impacted virtual server

More at https://www.eye.security/blog/crowdstrike-falcon-blue-screen-issue-updates
#Outage #CrowdStrike
8/n

CrowdStrike Falcon and Microsoft blue screen issue updates

Stay updated on the BSOD issue caused by the CrowdStrike Falcon and Microsoft update.

^{Piet Kerkhofs (Eye Security)}

Microsoft recommends restoring Windows from backups. Easier said than done.

For Virtual Machines running Windows Client and Windows Server, VM restarts (as many as 15 may be required) seem to be effective??

https://status.cloud.microsoft/
#Outage #Microsoft #CrowdStrike
7/n

The offending software is called the "CrowdStrike Falcon Sensor software."

R U ready for some AI-powered upgrades?

"CrowdStrike Falcon® Complete Next-Gen MDR utilizes AI-native technology and world-class expertise to stop breaches across the entire enterprise attack surface."

https://www.crowdstrike.com/blog/crowdstrike-unifies-threat-data-and-ai-for-mdr/

#Outage #Microsoft #CrowdStrike
6/n

George Kurtz, President & CEO CrowdStrike, tweeted about 2 hours ago that -
"The issue has been identified, isolated and a fix has been deployed."

Wonder what "deployed" means. How does one deploy the fix (delete certain driver files) to remote devices that cannot boot normally? 🤔

So, are planes flying again?

#Outage #Microsoft #CrowdStrike
5/n

Microsoft/CrowdStrike issue workaround from CrowdStrike -

- Boot Windows into Safe Mode or the Windows Recovery Environment
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Locate the file(s) matching “C-00000291*.sys”, and delete it.
- Boot the host normally.
🛠️
https://theconversation.com/massive-global-it-outage-hits-banks-airports-supermarkets-and-a-single-software-update-is-likely-to-blame-235107
#Outage #Microsoft
3/n

Massive global IT outage hits banks, airports, supermarkets – and a single software update is likely to blame

Enterprises across Australia and the world are struggling to cope with a huge software outage – but your home computer is probably safe.

^{The Conversation}

Global Windows outage causing major disruption around the world.

Linked to a software update from cybersecurity giant ​​CrowdStrike.

Banks, airports, TV stations, hotels, and many other businesses are affected.

US airlines United, Delta, and American Airlines issued a “global ground stop” on all flights.

https://apnews.com/article/microsoft-crowdstrike-outage-australia-internet-banks-media-0a5f792b6571b37a35181d64028fefc4
#Outage #Microsoft
1/n

xkcd comic for today 😅

Title text: We were going to try swordfighting, but all my compiling is on hold.

Touché.

Source and explanation: https://www.explainxkcd.com/wiki/index.php/Main_Page
#Outage #CrowdStrike #xkcd
12/n

@Sou
some may ask why they didn't use ai 😂

"Whiteboards being used at
@belfastairport
as all the screens are blue. Passenger Martin McElroy took this, and told us that it's like “the dark ages, but to be fair they’re making it work, which you have to give them credit for.”

#outage #crowdstrike #microsoft