Crowdstrikepocalypse

I find it alarming that this company that can afford to sponsor F1 teams etc apparently couldn't afford even a minimal prerelease test environment for their automatically-installed updates for their AV/IDS/EDR software, thus disabling a good chunk of the world's computers with what they claim was only a botched definition-file update. It would be very odd for a definition file update alone to do this, so either they also updated the low-level kernel module, or that kernel module loads content from the definition file in an unsafe way. It's not a simple fix to get an affected computer to boot normally again, and it seems that if it had Bitlocker enabled with no recovery key stored, the computer will need a wipe and OS fresh-install. So this might actually lead to data loss as well.

I also hear that the Mercedes F1 team found all their pit computers wrecked at the Hungary GP this morning, but with some heroic IT work they managed to get them all running again before the race.

I posted this in the minor complaints thread, but I'll put it here too:

how-to-fix-crowdstrike-bsods-in-three-minutes-fix-requires-manual-changes-but-they-are-simple

This has you deleting some Crowdstrike related driver files that were updated.  This was not a botched definition file.  But as you mentioned, lower level updates that crashed the windows kernel.

It's the definition of Silicon Valley, isn't it?  Move fast, break things?

I will bet you that nothing significant will change after this.  Some small immediate changes, but widespread change?  Nah.

Worst-case price list for a minimal prerelease test environment that would've prevented this:

1 slightly beefy Linux server to host the crash test dummies as VMs, say $5k max if bought new. This should afford a server that can run all of them at once and also host the prerelease update server, although the dummy VMs could take turns to save resources. But if they want to get updates out fast, parallel testing will save time.

$3k should cover licenses for one of each supported Windows variant.

It's probably possible to hack any supported version of MacOS to run in a VM as well, but let's assume it can't be done for software or more likely licensing reasons and that you need a bunch of hardware Macs as well, another $5k to get basic models with each supported OS variant (which I think is just a few different major versions plus the latest on an alternate architecture).

So we're up to $13k at most which is probably what the CEO would be paid if he took some time during his work day to expel a particularly voluminous fart.

Crowdstrike's CEO right now:

A thought I keep having:

Decades ago, people used to watch movies like Star Wars Ep. 1 where robot armies or other huge systems are taken out by blowing up a single facility and say things like "that's ridiculous, why would anyone make a huge system like that with a single central point of failure?"

Apparently the answer is "because it's a very profitable way to half-ass something."

CRWD is down 20% percent since Monday. Was this problem becoming known earlier in the week? Maybe doesn't matter, but could be a good buy next week.

I have such negative thoughts about Crowdstrike that I can't even express them right now without shutting down this thread. All I can say is emails and Seth Rich.

In reply to AngryCorvair (Forum Supporter) :

I always try to remember that the market can remain irrational longer than most of us can remain solvent even when I see the most tempting opportunities to buy or short, but I think Crowdstrike has reached "too big to fail" status.

Gary said:

I have such negative thoughts about Crowdstrike that I can't even express them right now without shutting down this thread. All I can say is emails and Seth Rich.

All I have to say about that is Rolling Stone, September 2019.

Funny that we have to be evasive. (But I don't trust Rolling Stone.)

In reply to Gary :

You don't have to trust publications if you can check facts.

In reply to GameboyRMH :

Yes.

Back in 2010 McAfee software had a bug that affected Windows XP so severely it took down a big chunk of the web. It compromised the company badly enough that they had to sell out to Intel to remain a going concern.

Guess who McAfee's CTO was then and what he's up to now?

Part of my day

I don't know how to make this a non-social media thing, but its a good visual of the impact on the travel industry

https://www.instagram.com/p/C9niVhFyAuH/
 

A friend of mine was heading to NZ today.  He never made it out of ATL, and now they are telling him the next available flight is Thursday.  (6 day delay)

In reply to prodarwin :

Let's see if the embed function does it:

Edit: Only partially, still opens the link to play...

My work laptop is still a brick as of right now. Pitchforks and torches would be appropriate.

My thoughts;

There will be a lot of people with Crowdstrike on their resume looking for work monday

Should we all be changing our PC OS to a LINUX type system?

In reply to porschenut :

Crowdstrike has agents that run on Linux and MacOS as well. The only reason those weren't affected this time apparently was because the broken update was for the Windows specific agent.

How did you spend your Clownstrike Day? After twiddling my thumbs for two hours at the office, here's what I got up to.

GameboyRMH said:

Gary said:

I have such negative thoughts about Crowdstrike that I can't even express them right now without shutting down this thread. All I can say is emails and Seth Rich.

All I have to say about that is Rolling Stone, September 2019.

Don't be obtuse and tell us.

I want to know

Fresh on the heels of the CDK hijacking, I sure had other thoughts about what was happening.

Wonder how many dealerships got hit with this as well...

Yesterday I flew from MCO-PHX on Southwest. Getting through the check-in gates at MCO was madness: I legitimately thought that the Spirit Airlines check-in gate was going to have a fistfight and/or riot. There were news crews covering it, even. Of course, part of that could just be that Spirit hires the worst possible people to work for them and they're not exactly known for their ability to defuse situations.

The crowds were absolutely insane. The knock-on effects of this outage are going to be severe; I wonder if the global economic impact will reach the trillions of dollars. 

z31maniac said:

GameboyRMH said:

Gary said:

I have such negative thoughts about Crowdstrike that I can't even express them right now without shutting down this thread. All I can say is emails and Seth Rich.

All I have to say about that is Rolling Stone, September 2019.

Don't be obtuse and tell us.

my grammar teacher would respond with the question "how does telling us make him obtuse?"