- Microsoft outage last night
- CrowdStrike outage today

TWO DIFFERENT THINGS.

Having said that, I will apologize preemptively for the following meme. #crowdstrike

Another lesson perhaps for organizations running mission-critical services - do not auto-update all your servers and clients is one fell-swoop.

Stage them. With some soak time in between.

This is quite standard practice when pushing our own custom software into our own distributed network products.

#Outage #CrowdStrike
11/n

My guesses for the cause of this CrowdStrike driver update worldwide snafu -

1. A last minute "trivial" change to the software after all testing was completed
2. Something went wrong in the packaging or delivery or installation of the software update.
3. Some AI-generated code segment 😜

What else can you think of?

#Outage #CrowdStrike
10/n

Snapshot of some outage graphs from https://downdetector.com/

Not the kind of rising lines one wants to see on Friday or any other day.

The disruption is widespread, far beyond the more visible ones in the airline industry.

#Outage #CrowdStrike
9/n

Steps for public cloud or similar environment including Virtual Machines:

Detach the OS disk volume from the impacted virtual server
Create a snapshot or backup of the disk volume as a precaution
Attach/mount the volume to to a new virtual server
Navigate to the C:\Windows\System32\drivers\CrowdStrike dir
Delete files “C-00000291*.sys”
Detach the volume from the new virtual server
Reattach the fixed volume to the impacted virtual server

More at https://www.eye.security/blog/crowdstrike-falcon-blue-screen-issue-updates
#Outage #CrowdStrike
8/n

CrowdStrike Falcon and Microsoft blue screen issue updates

Stay updated on the BSOD issue caused by the CrowdStrike Falcon and Microsoft update.

^{Piet Kerkhofs (Eye Security)}

Microsoft recommends restoring Windows from backups. Easier said than done.

For Virtual Machines running Windows Client and Windows Server, VM restarts (as many as 15 may be required) seem to be effective??

https://status.cloud.microsoft/
#Outage #Microsoft #CrowdStrike
7/n

The offending software is called the "CrowdStrike Falcon Sensor software."

R U ready for some AI-powered upgrades?

"CrowdStrike Falcon® Complete Next-Gen MDR utilizes AI-native technology and world-class expertise to stop breaches across the entire enterprise attack surface."

https://www.crowdstrike.com/blog/crowdstrike-unifies-threat-data-and-ai-for-mdr/

#Outage #Microsoft #CrowdStrike
6/n

George Kurtz, President & CEO CrowdStrike, tweeted about 2 hours ago that -
"The issue has been identified, isolated and a fix has been deployed."

Wonder what "deployed" means. How does one deploy the fix (delete certain driver files) to remote devices that cannot boot normally? 🤔

So, are planes flying again?

#Outage #Microsoft #CrowdStrike
5/n

xkcd comic for today 😅

Title text: We were going to try swordfighting, but all my compiling is on hold.

Touché.

Source and explanation: https://www.explainxkcd.com/wiki/index.php/Main_Page
#Outage #CrowdStrike #xkcd
12/n

Remember when I mentioned hours ago that plenty of malicious entities were going to be paying attention to today just to gather information?

This.

This right here.

They're paying attention to this.

https://www.digitaltrends.com/computing/southwest-cloudstrike-windows-3-1/

#crowdstrike

A Windows version from 1992 is saving Southwest’s butt right now

Southwest hasn't been impacted by the CloudStrike outage, and that's reportedly because it's still running Windows 3.1.

^{Jacob Roach (Digital Trends)}

#crowdstrike

https://www.instagram.com/reel/C9nIRomoQIj/?igsh=MzRlODBiNWFlZA==

Fckboy Problem.s on Instagram: "Follow (us) @FUCKBOYPROBLEM.S for more relatable reels 😂🍿💚 The accuracy 😭 A massive IT outage for Windows users knocked out important health-care, business and transportation systems worldwide starting Thursday evening an

287K likes, 873 comments - fuckboyproblem.s on July 19, 2024: "Follow (us) @FUCKBOYPROBLEM.S for more relatable reels 😂🍿💚 The accuracy 😭 A massive IT outage for Windows users knocked out important health-care, business and transportation systems wo…

^Instagram

In an unexpected turn of events, a sensible take on #Crowdstrike from the Orange Site.

Source: https://news.ycombinator.com/item?id=41004184

Windows security be like...

#CrowdStrike

I'm very thankful to #Microsoft and #Crowdstrike for this timely demonstration to make it clear, that the budget cuts to the #NGI by the @EUCommission endanger Europe's #digitalsovereignty

https://www.theregister.com/2024/07/17/foss_funding_vanishes_from_eus/

FOSS funding vanishes from EU's 2025 Horizon program plans

Elimination of most Next Generation Internet funding 'incomprehensible,' says OW2 CEO Pierre-Yves Gibello

^{Brandon Vigliarolo (The Register)}

#CrowdStrike is a different scale, but it reminds me of Patch Tuesday causing Skype to collapse: https://www.wired.com/2007/08/microsofts-patc/

Skype used to run a hybrid p2p network that bootstrapped off two supernodes that were run by Skype. But the network was so stable they switched those off, and kind of forgot how to use them.

1/2

What it looks like in the air when a closed-source security company breaks a software update.

This is a 12 hour timelapse of Delta, American Airlines, and United flights.

#crowdstrike

"I'm no security expert but [biting and insightful question that will receive a longer, more deflective and less insightful answer than it deserves]"

Sky News: "Global IT outage: 'We're deeply sorry', CrowdStrike CEO George Kurtz says"

https://farside.link/invidious/xySnsTQLqB8

https://youtu.be/xySnsTQLqB8

#crowdstrike

Welp at least the temporary fix sounds simple. #CrowdStrike

https://mashable.com/article/windows-bsod-crash-crowdstrike-update-worldwide-outage

CrowdStrike CEO is not having a good day.

#crowdstrike

Too funny: In 2010 McAffe caused a global IT meltdown due to a faulty update. CTO at this time was George Kurtz. Now he is CEO of #crowdstrike

https://www.zdnet.com/article/defective-mcafee-update-causes-worldwide-meltdown-of-xp-pcs/

Defective McAfee update causes worldwide meltdown of XP PCs

Oops, they did it again. Early this morning, McAfee released an update to its antivirus definitions for corporate customers that mistakenly deleted a crucial Windows XP file, sending systems into a reboot loop and requiring tedious manual repairs.

^{Ed Bott (ZDNET)}

Welcome today as we witness the largest IT outage of all time.

#windows #microsoft #crowdstrike #cybersecurity

edit: source of the picture is https://www.crowdstrike.com/en-us/

@Sou
some may ask why they didn't use ai 😂

"Whiteboards being used at
@belfastairport
as all the screens are blue. Passenger Martin McElroy took this, and told us that it's like “the dark ages, but to be fair they’re making it work, which you have to give them credit for.”

#outage #crowdstrike #microsoft

From https://x.com/AttilaBubby/status/1814216589559861673

#crowdstrike

I've posted (manual) recovery instructions for the CrowdStrike SNAFU: https://pomeroy.me/2024/07/crowdstrike-bricking-how-to-boot-into-windows/

#CrowdStrike #BSOD #recovery

CrowdStrike bricking – how to boot into Windows

It’s a bad day for CrowdStrike. Machines are being bricked globally, due to a faulty CrowdStrike update. Here’s a recovery process. You’ll need physical access to this machine for…

^{Rob Pomeroy}