When I first heard the news, I thought “Crowdstrike” was a hacker group or bunch of protestors, e.g. “Anonymous”. Then I thought the same; who names their product “crowdstrike” without it being some sort of malicious malware joke?
The biggest issue I ever caused was an update that was suppose to notify us any time our site went down, but a difference between the development environments and the live server took the organization's website offline and generated thousands of email messages about the site being down to the developer's group mail. It was only offline for a couple minutes, but our email server took so long to process the volume of mail generated in that time I was still receiving emails about it for hours.
Original aurhor of the quote: Bill Vaughan, April 2, 1969, although surprisingly he may have been riffing on Agatha Christie ( ‘To err is human’ but a human error is nothing to what a computer can do if it tries.")
I've been up all night and am still working this issue. We had to cancel all elective surgeries today. When you delegate responsibility for your infrastructure to a vendor in the cloud, you suffer the consequences of their mistakes. When the systems are encrypted and won't boot, you tell me how easy that is to fix.
The CEO of CrowdStrike is George Kurtz. In 2010 he was CTO of McAfee when a virus database update wiped out svchost.exe on Windows XP PCs, causing blue screens all over the world. That incident, according to Wikipedia, is what inspired him to leave McAfee and found CrowdStrike.
@@jamesphillips2285 oh boy do I got a wild ride for you, McAfee has always been quite the company - look at the wiki for John McAfee and then scroll down to legal issues. His life took quite the plot twist.
as an IT guy for 40+ years, i can say that many times the systems and applications that you put in place to prevent problems are in fact the things that cause the problems. this is due to the level of access that these systems require and the amount of resources they consume both locally and on the network. we used to call it "monitoring your systems to death". but i am retired now and can go to bed on time and sleep all night and get up in the morning and walk outside and sit on the porch in the sun and drink my coffee while reading about all the chaos that i no longer need to deal with. hell yeah.
I too am retired from IT. I used to do patching and we always did pilot testing with all of the app owners before pushing anything to production. Did they get lazy you think? I would not go back to IT for anything.
@@stevehoward3981 i think their build/test/deploy processes are broken if this happened the way it seems. i worked for OEM's, both hw and sw and the best ones had a "eat our own dogfood" process before we deployed to GA. we would push whatever it was to our own systems and run it before we would even push it to pre-GA. i know Falcon Sensor is a bit of a different animal (i used it) but the principle should be the same. especially if what you are pushing to Prod could cause a BSOD. imagine this scenario; you are a 100% Windows shop, AD/DNS, Storage, File Services, Remote Access, Office Automation, CSM, backups and Virtualization. this is very common. and you are required by some HIPAA or other requirement to run Bit Locker and your key management is kind of iffy. Again common, because i've seen it. now every friggen Windows server has a BSOD and the only way to delete that corrupted file requires the BL keys. you are now out of business. this is the kind of shit that kept me up at night because in a lot of places IT is the redheaded step child when it comes to budget and severely understaffed and running by the seat of their pants. retiring from that shit show was like being reborn and i would never go back either. and i fucking hate Windows.
Over here in Aussie, it smashed us in our Friday afternoon peak! In our IT serious incident chat I could see people posting the blue screens of death that various machines were showing and my housemate burst into my office and told me that it was Crowdstrike - their company (finance) was down too. So I had to inform the chat. We couldn’t get flight plans uploaded so that resulted in a ground stop. A goddamn mess. We have a charter business (as well as regular RPT) that flies into many REMOTE airports for mining companies and related which now have stuffed computers, it will take DAYS to get IT support people into these places to get the computers fixed. 😬 At least our big hub ports are back up and running but now of course we have a massive backlog of upset and disrupted passengers. It was somewhat amusing watching all the euros and yanks waking up and being all “what’s going on?” 😮 My flatmate was up quite late restoring their company’s services. Bad day to be a windows admin! Luckily all my systems run on Linux/Kubernetes.
Greetings all. I worked in IT security infrastructure for decades. The industry has been replacing human analysis from the equation for a while now with cloud based technology (now marketed as 'AI' enabled for an investment sound bite). It all started with overseas outsourcing. It was bad crossing the communication divide with language but trust in 'AI' is absolute rubbish. This level of fail should never be possible no matter the excuse...well intentioned, accidental or malicious.
The company where I'm working was not affected, but one year ago would have been because we were in that time using a security solution that had CrowdStrike software on it
I’m getting tired of UA-cam auto deleting my perfectly reasonable post. I really hate this AI moderator they’ve got is a complete mess. It can’t tell what’s spam and what isn’t. There has been a huge amount of embarrassing cockups at these corporations where they suffered a data breach and it turns out that there was a patch for the hole that the nefarious party gained entry that was not applied or not applied correctly. My thinking is that a decision was made to just auto push these things through as soon as they are available to get them patched and deal with whatever fallout comes from incompatibility or whatever that would be better than dealing with the public relations disaster of a preventable breach. But I don’t think anyone anticipated that it would go this bad.
Much like the "Swiss cheese" model discussed on this channel, this failure represents much more than a single error by a single person. The fact that a company with such an enormous reach lacked the process to detect this problem before rolling it out and rolled it out globally all at once represents a lack of organizational maturity that is just astounding.
@@ericfielding2540 I don't believe the updates were optional. Crowdstrike Falcon automatically retrieves its own updates. This was not a Windows update. It was the actual Crowdstrike software that crashed, but since it ran in a privileged mode, the crash was fatal for the operating system as well.
It starts much earlier. Companies using such a buggy and insecure OS that it even needs extra software for this. Then buying snakeoil like this to "fix" the issues. Then the problems with the snakeoil product itself.
@@ericfielding2540 With this one, CrowdStrike bypassed the usual “staging” environment that IT departments use to test updates. IT departments thought they had the option to review and block CrowdStrike updates but in fact did not.
Fun fact-o-the day... The blancolirio affiliate office in Tucumcari offers guided tours on Tuesdays and Thursdays. ALWAYS opt for the Thurs reservation! Same $29.95 cost, but at the end where you see Juan's private parking spot they let you reach in a jar and grab a free pickled pig foot!
@@hamsterama That's incredible. Next I'm gonna find out he is a decorated fighter pilot with 12 tours... or something. No, but in all seriousness, this man is a societal role model if you ask me.
@@IncorrigibleBigotry You may not be too far off LOL! Before his airline career, Juan flew planes in the Air Force. He's dropped anecdotes here and there in some of his videos about his time flying in the military. I agree, he's a great role model! And he's got a very interesting life story.
We have the same situation here in Australia as well & we where sitting on the Tarmac in Perth waiting to see what our next move is, Push Back & Start Up or Flight Cancelled, Already some 3 Hours behind Scheduled Departure time, Had already been in the Cockpit for just over 4 Hours without even having moved so much as an inch. I decided to Deplane those Passengers who are on board For their own Comfort as it seemed the ATC Guys & Girls were having Problems on Access Data Relating to Ground Movements which was not Related to this Outage apparently they advised All this happened to myself yesterday Friday 19/07 around Midday & we finally departed PTH at around 10:00 Pm for a Flight cross Country to BNE. We should have departed at around 12:00 Pm Midday yesterday Friday 19/07. 787-9 Flight.
We've known for a long time that these tech companies were more focused on making money than building bulletproof products, indeed, the focus on profit MUST come at the expense of security -- it's not a guess, it is reality! The speed with which Crowdstrike patched the problem is a problem in its own right given that every hacker, nation state or otherwise, now knows this critical company put a quickly assembled patch up and I would bet some of them will be probing for weaknesses.
2:06 we have a term for this in the IT. We call this "Friday afternoon patch" :) But great to hear that Crowdstrike also subscribes to the idea to "I don't care about testing, just ship it!".
@@paulsherman51 great idea, let's make it a unit test. import unittest class TestTitle(unittest.TestCase): def test_title_is_string_and_length_four(self): title = "Test" # Example value for title self.assertIsInstance(title, str, "Title is not a string") self.assertEqual(len(title), 4, "Title does not have exactly four characters") if __name__ == '__main__': unittest.main()
I was flying in the North East this morning. Reporting in to New York Center got the "route change, advise ready to copy". Ready to copy the inevitable 12 point routing scipt, I received present position direct LAAYK then direct destination ! Really must of been no traffic!
Thanks for the astounding visual. 👍 I just finally got home after a verrrrry long Friday and a Saturday cleaning up the mess this made at my job. Unfortunately, most every computer needed to be touched, and there are some stragglers that we'll have to get on Monday because some areas aren't accessible on the weekend. Because of how early on in the boot process it crashes Windows, the chance of it being able to fix itself is extremely low.
McAfee rolled out a bad client update about a decade ago that also caused major disruptions. But McAfee had a much smaller customer base than Crowdstrike. I've spent all day handing out BitLocker recovery keys so folks can boot into safe mode and fix the problem. Busiest Friday I've had in years.
I left at 1432 westerly on 92. Sensors showed a clear path so I maintained speed at 55mph on mains and 35 in arterials. Upon arrival at my destination data suggests my mpg was underscore so my projected travel fuel amount was deemed sufficient. 😂
The major gas station chain in our area(Speedway) uses Crowdstrike in their system too so I had to drive all over town looking for a station that was actually open; the one that was had raised their prices by a good 10 cents and there was a huge line. This really demonstrates how easily it could be to cripple the computer backbone that runs the world in 2024.
It's foolhardy to have electricity to pump fuels and not be able to because of anything computer related. Find a workaround there will surely be a need in the future.
I flew Delta from LAX to Charleston, yesterday, with a layover in Atlanta. Got to CHS at midnight, but my bags didn't make it. At Atlanta I saw a line with what seemed like 1000 people in it. Didn't pay much attention, and carried on to my flight. Found out this morning about the outage. Just my luck. I was also stranded in Manchester, England the morning of the Continental/United merge. Their systems were down from that and had to stay an extra day.
I'm a Charlotte based first officer with a regional Airline. We had a 3:20 p.m. Charlotte/Chattanooga turn and then we went up to Charlottesville, Virginia for the night. The first turn went off surprisingly smoothly and on or ahead of time. We actually landed ahead of time in Charlotte, but we had an extremely extended taxi that wound up putting us about 15 minutes late. There was an incredible amount of aircraft on the taxiway trying to get into the ramp. Then we had an aircraft change for who knows why and wound up being about an hour late getting out for our last leg of the day. There were an awful lot of dissatisfied people meandering around the Charlotte airport.
This reminds me of the AP Radio News satellite receiver software update pushed to all of the receivers which bricked every single one of them. Somebody at headquarters clicked on the wrong file to send on the uplink to install for all receivers. That left radio stations without live national news, special audio clips and all of the text pushed news services. The solution was to send a big group of people all around the country to open up each satellite receiver and install a permanent daughter board that put the receiver back in service.
Which is why best practice is to always have a product ID and HW version map tagged in any firmware release. So when someone goofs, then x thousand devices may download the file. But then rejects it because it's for a different product or for a different hardware revision. Bandwidth consumed, but no bricked units.
I looked at the radar this morning around Chicago, 1 plane coming in to ORD from Mexico and a LOT of GA probably going to Oshkosh! Nice haircut! Thanks.
CI/CD has been a hot buzzword for the last 5 years or so. I expect IT companies to reverse the trend of firing QA people, having developers do development, testing, deployment and hot-fixing. That'll teach them. Or not.
@@marklnz You still generally avoid making changes that are going to affect the company on a Friday, because then you're stuck with people scrambling on a Friday to try and fix things.
My facility has had a handful of false-alarm deluge system activations over a few decades, causing millions of dollars of damage to aircraft and ground equipment. In that time there have been no hangar fires. Our mitigations are sometimes more hazardous than the hazards themselves.
In the 1980's, we discovered a central mainframe computer with remote terminals wasn't practical (everything went down with the mainframe). So we went to stand alone computers- but now- due to mission creep- we are getting more and more controlled by "them". We are all sitting ducks! I just went through "telephone tree- hold" hell for 3 hours at ATT, Verizon, and BOA. I must have heard their propaganda and "marketing education" message enough to throw up or throw a tantrum! We need to fight against "the man"!!!!
My work team and I had a merry day remediating Crowdstrike. Other teams elsewhere in the company had similar joy. The fix itself was easy, delete 1 file from a specific directory path. The hardship was the labor to do it. And, naturally, extra challenges when confronted with petulant servers. What people might not think about risk, is today’s issue is still outweighed by the many, many product updates that have had zero issues. Most CIO’s are very comfortable with having “now” security, as opposed to a program of being X number of days behind patching for vetting. No CIO wants to be pilloried for being security lax. Mileage may vary of course, as some IT systems, like enterprise databases, need careful planning and testing.
I, coincidentally, spent my part of my afternoon helping a friend remediate a bit of malware the claimed to be ransomeware. His AV hadn’t been kept up to date.
as an IT senior developer ... all I can say is any "defect" in the code/process should have been caught in the testing phase of the project. All developers make mistakes, that is a given ... but the good ones don't let them deploy to PRODuction ... If I was running the show that whole project Team would be on some seriously shaky ground right about now ... I'd be asking them 'have you given any consideration to another profession?" ....
And as an "IT senior developer" you should have experienced enough in your career to know that even with significant QA processes and procedures in place, it's still possible for things to sneak through. No process is ever foolproof. If you haven't learnt that yet, then I feel compelled to reflect *your* question back to you: 'have you given any consideration to another profession?"
@@marklnz I nominate you for political office in Washington, D.C. Your kind of thinking is what makes America the greatest in the world. Enough with that obsession to detail and "failure is not an option" thinking. Why, political office is not sufficient for you. I would like you to be the head of N.A.S.A. In 1986, even! P.S. - I have DONE major corporate software testing. The process is idiotically faulty.
@@marklnzIn response to your multiple snarky remarks made to several commentators…it was a major screw up by an “IT” company and their developers with final results yet to be determined but not good! I go back to the days of IBM Systems 36 and 38 where DP managers lived in fear of the next software “update”! If Crowdstrike screwed up by cutting corners on testing, then they should suffer the penalties…ask Boeing how that’s working out. My recommendation is that you stick to your IT work and leave the commentary to those with a bigger picture! 🤨
@@Bill_Woo From your tone you seem to apparently have some issue with what I said - which is funny because your last sentence effectively reiterates exactly what I said.
Appreciate this post - this brings to the forefront the need for both IT software vendors *and their customers* to have a validation and verification process in place, using test platforms, to verify software updates and upgrades before they are pushed into production.
@@chris-hayes Agreed. My perspective is that the investment in a validation program along with the systems, software and personnel is far less than omitting such a program when the well known substance hits the electrical convenience. This is a business management failure as much as a failure on the part of the systems and software technology end users and providers. It will be interesting to see what changes occur going forward.
You can never test all the possibilities to fail. I know ive been involved in 5 ERP system implementations, with a LOT of testing. After Go-live you spend month's fixing problems
@@larsbr4519 That's correct, nor is the goal to test everything. A validation program focuses on testing the critical aspects of a platform or software that can cause issues like this. If this defect eluded the business and test validation processes in place at customer institutions and the software vendor, the next step is root cause analysis and closing the gap so this does not occur again.
@@larsbr4519 Hats off to the institutions that caught this defect in testing, or perhaps a quarantine program for software updates, thus avoiding the impact to their customers. Perhaps some of them reported the problem to both Microsoft and the software vendor. It would be educational to learn about the business practices, polices and procedures as well as the validation program so other organizations can adopt these practices.
Work with real computers and not toys and work might suck less... Well you're probably stuck with the same ignorant leadership but that is just how things roll.
It took me about an hour to do the patches for the computers at my job. It makes me appreciate the good old days, when I did my job without any computers running Microsoft. Back then there was no internet.
Just read that in a survey most users are fed up with software companies continually rolling out new features that users didn't ask for and don't use. It's far past time tech companies stopped the endless pursuit of growth and focused on (long-term) value. (As I typed this comment, the Chrome browser interrupted me with a popup informing me of some new feature that I could care less about.... good thing no software execs were near me or one of them would be lying on the floor with a fat lip, lol.)
I don't know for sure but I've been assuming that this wasn't a software update - it's an anti-virus/malware program so I assume it was an updated virus definitions file or something.
People and X Spurts have ridiculed me for blocking updates in Windows and Android systems. I've yet to get bitten by crap like this. And I store MY data locally, Three copies physically reposing in different locations miles apart. "Its safe - it's in The Cloud", my wrinly old ass.
Thanks Juan. We are leaving from Sacramento to Seattle on Monday morning to catch a boat to Alaska.. Hopefully, a majority of the issues will be resolved by then.
My husband, who was an IT manager for a large company said he had some experience with "Crowdstrike". Said they were "ARROGANT ASSWIPES". Strong handed sales tactics. Experts at intimidation & coercion. Little to NO customer service & claimed their "SHITE DON'T STINK". Didn't PUBLISH their pricing. He learned that was a RED FLAG. Karma, it seems, has had a nice LUNCH with them.
I was with a large international carrier's IT department for 35 years. Just before Xmas many years ago, a "glitch" literally brought our computer system, which also directly supported a number of of other carriers, and related services, to its knees. At one point, it appeared the system had "dumped" literally tens of thousands of PNR's (Passenger Name Records) into thin air. But in our case, we had a brilliant group of coverage programmers who were able to restore the entire system in a few hours. To my company's credit, it (at the time) paid its IT staff generously and treated them with respect. It no doubt made the difference in the recovery. Now almost 30 years later, my former company has "outsourced" its most critical system coverage to India. In the case of CrowdStrike, it begs the question of what caliber of employees were involved in this debacle? It should have never happened and illustrates how a few lines of code can directly impact millions of MS Windows users. Our IT systems, and especially those related to the aviation industry, are particularly vulnerable.
That workaround is all well and good, providing you can actually get access to the drive from the BSOD screen via command prompt and if one happens to have bitlocker turned on (as I hear, it will be the default option for Win11 24H2), it may cause problems accessing the drive. If it asks for your unlock key, well that is stored on the microsoft account if you use one to login to windows, otherwise you'd need to have manually recorded it down somewhere. Office365 being affected means one may not be able to login to thier MS account on another machine to get the key. So that fix may on some systems prove to be harder to implement, depends on the company or system if boot from USB is allowed, if bitlocker is in use, and if the user themselves actually has the knowledge and confidence to go poking around inside the System32 folder of Windows.
I used to work for Bayer UK. An employee sent her holidays snaps (+ a virus?) to "all" , problem was, she sent them to all in the entire company in the entire world, not just all in her dept. Bayer worldwide was down for 3 days. Each computer had to get fixed.
I got one for you... someone sent a "greeting card" (+ worm) to a secretary at one of my customers' plants. She clicked the fatal link and within seconds every HMI/SCADA system on every plant floor in the company was infected and blue-screened. This naturally brought all production to a halt. After this incident they finally listened to me: "air-gap the manufacturing systems from the office networks".
2:07 Imagine being that programmer, imagine being that quality control person, imagine being the managers over these people, imagine being the manager responsible for pushing the update out, imagine …
Thanks Juan. I go to your channel for accurate information. Amazing how most of the media couldn't get the story straight. Having been a software engineer, I really sympathize with the bugger that made this change.
I had to deal with this, this morning. I came to work with a BSOD on my work station. The patch is currently fixed and reboots can remedy but there are fixes that an admin has to do, especially if Bitlocker is enabled. Earned my pay this morning. Others, well, I can see meetings coming next week on alternatives to this patch/security system that one file breaks the line. (hope some sleuths find either MSFT changed something, or get this, someone the day before on Reddit criticized CRWD being overvalued...hmmmm, plausible)
Interestingly enough this issue also affected City of Phoenix Fire Department computerized dispatch system. As a work around they went back tot he old school method of manually dispatching the calls. Also had impacts on the two large hospital networks Banner and Dignity in the Phoenix metro area.
We had a Kaspersky antivirus update do something like this in 2004. I was the first one in the office on a Monday morning and my desktop was ‘blue screened’. Every PC I checked was the same. I woke up one of the IT guys and he opened up his laptop, only to be greeted by the same blue screen. Uh-oh ….
I'm an IT manager - luckily only 2 out of our 40 servers were affected, and simply rebooting them brought them online. We did have one employee who's laptop was affected, and rebooting did not resolve the issue. Trying to walk a non-technical person through rebooting into safe mode to delete a system file is... not fun. Hopefully it was just the one - I can't imagine having to deal with hundreds of these issues without having hands on access to the machine.
We had servers running Crowdstrike at work. Luckily no workstations, so it was an impact that we could cope with. I feel sorry for all corporations that had Crowdstrike everywhere. Some rumor allegedly from Microsoft says that rebooting your computer 15 times will solve the problem.
Computers probably don't contact the Crowd Strike servers every time they restart, plus when it does contact Crowd Strike there are lots of requests so may not get a fast response. Hence it can take quite a few tries to get the patch. So 15 is a reasonable number of tries to attempt before assuming the fix isn't going to work.
Very good report, Juan, on a very important issue: companies that are too big to fail and service providers supplying them who are too widely and deeply embedded across the industrial and national spectrum to have such broad failures, impacting those user communities. There is a UA-cam channel BrenTech that seems to provide sensible and frequent updates on these types of IT related current events.
That thing got the company I work for. It was simple to recover from but took time. They provide internet security (as you noted) and their update was fubar and BSO'd our computers. I'm not much of a fan of cloud anything but that's what we are moving to.
Juan, I'm surprised you didn't let people know that the CrowdStrike doesn't affect commercial aircraft, so you don't have to worry about boarding an aircraft. NOW, it does affect both ticketing software and flight management software that the airlines use to run everything. So the challenge is to get on a flight that has not been canceled. But you're fine once your on the aircraft. As far as Air Traffic Control goes, I don't think the US is vulnerable, as I believe the ATC software being used is Linux/Unix based and does not have/use Falcon software. (I could be wrong about this.)
As a dev, I always find that amusing seeing airport displays, atms, etc using Windows-based software. Outside IT, if you're talking about all the biggest software companies in the world, their production systems are 90% Linux. Why does that matter? When over 90% of modern software is written for Linux, companies with Windows systems will have fewer and less supported options.
Man, that sucks. Take a look, Protocol Labs is hiring. Maybe something interesting in the listing. Otherwise, in Germany, we're desperately searching for skilled workers. You can basically start working anywhere you want. Everyone is searching. :)
@@marcellkovacs5452 well, the point here is that not individuals did click on "update" but the IT of all the companies shipped the update as soon as available, downing all computers at the same time.
A common use of Crowdstrike Falcon is EDR (Endpoint detection and response). It is a class of tools that allows monitoring behaviour associated with malware, detecing malware-like patterns and remotely responding to incidents (e.g. by blocking problematic actions by suspected malware). The update was an updated driver (which runs deep in the Windows kernel) for monitoring some channels used by some malware and that had some kind of bug, causing the blue-screens... Affected computers were running Crowdstrike Falcon (which apparently is quite popular...)
It affected systems at FDA as well. I was able to access our systems and put in a full workday after around 9 am, but many of my colleagues were unable to log on for most of the day.
Great Video ! The TV News headlined Crowdstrike and Microsoft... The News talked about SWA flying out of Midway Airport ! Flew out of Midway Airport when they were the Worlds Busiest Airport... First time i heard of Crowdstrike ! Miss the Pan Am Shuttle flights to LEO to make the connecting flight to the Lunar Shuttle... tjl
The irony of the fact that the name of the company responsible for this mayhem is "Crowdstrike", is astounding.
I just wonder, if Congress is waking up ....
SouthwestAirlines didn't have problems.
Pre programming people perhaps.... Lol
When I first heard the news, I thought “Crowdstrike” was a hacker group or
bunch of protestors, e.g. “Anonymous”.
Then I thought the same; who names their product “crowdstrike” without it being some sort of malicious malware joke?
I thought that 'crowdstrike' is a generic term for a crowdsourced DDoS attack.
To err is human. To really F things up you need computers.
The biggest issue I ever caused was an update that was suppose to notify us any time our site went down, but a difference between the development environments and the live server took the organization's website offline and generated thousands of email messages about the site being down to the developer's group mail. It was only offline for a couple minutes, but our email server took so long to process the volume of mail generated in that time I was still receiving emails about it for hours.
It was an error in transfer that corrupted the file and zeroed all data, making it an file consiting of only zeros, an empty file if you will.
Poetry. ❤
❤❤❤ Thanks so good.
Original aurhor of the quote: Bill Vaughan, April 2, 1969, although surprisingly he may have been riffing on Agatha Christie ( ‘To err is human’ but a human error is nothing to what a computer can do if it tries.")
We finally got to experience what everyone thought Y2K was going to be like.
Not really - this is a drop in the bucket compared to that
Not even a drop in the bucket!
The reason Y2K wasn't a disaster is that people prepared beforehand
pretty much
disaster mitigation is the most thankless of tasks: the more successful you are, the more people wonder what all the fuss was about!
I've been up all night and am still working this issue. We had to cancel all elective surgeries today. When you delegate responsibility for your infrastructure to a vendor in the cloud, you suffer the consequences of their mistakes. When the systems are encrypted and won't boot, you tell me how easy that is to fix.
The CEO of CrowdStrike is George Kurtz. In 2010 he was CTO of McAfee when a virus database update wiped out svchost.exe on Windows XP PCs, causing blue screens all over the world. That incident, according to Wikipedia, is what inspired him to leave McAfee and found CrowdStrike.
Shades of Thomas Midgley Jr. who invented leaded gasoline and followed up that top hit with the invention of Freon, also known as CFC gas.
That narrative sounds too good to be true (and likely a joke).
🤔
Don’t forget that crowdstrike helped the DNC cover up their 2016 email leak by Seth Rich (it wasn’t a hack). Kurtz…every single time
@@jamesphillips2285 oh boy do I got a wild ride for you, McAfee has always been quite the company - look at the wiki for John McAfee and then scroll down to legal issues. His life took quite the plot twist.
I didn't know Homer Simpson had moved from the nuclear power industry to the computer field.
Good one. 🙂
Homer retired. That was Bart!☺
I remember an episode where he starts a software company then gets “bought out” by Bill Gates.
D'oh!
Wheres the any key?
Stupid bird!......
as an IT guy for 40+ years, i can say that many times the systems and applications that you put in place to prevent problems are in fact the things that cause the problems. this is due to the level of access that these systems require and the amount of resources they consume both locally and on the network. we used to call it "monitoring your systems to death". but i am retired now and can go to bed on time and sleep all night and get up in the morning and walk outside and sit on the porch in the sun and drink my coffee while reading about all the chaos that i no longer need to deal with. hell yeah.
Your mornings sound like mine after retiring following 45 years of mayhem 😊.
Until whatever device you used to post this bricks after an ill-conceived, untested autoupdate.
I too am retired from IT. I used to do patching and we always did pilot testing with all of the app owners before pushing anything to production. Did they get lazy you think? I would not go back to IT for anything.
@@stevehoward3981 i think their build/test/deploy processes are broken if this happened the way it seems. i worked for OEM's, both hw and sw and the best ones had a "eat our own dogfood" process before we deployed to GA. we would push whatever it was to our own systems and run it before we would even push it to pre-GA. i know Falcon Sensor is a bit of a different animal (i used it) but the principle should be the same. especially if what you are pushing to Prod could cause a BSOD. imagine this scenario; you are a 100% Windows shop, AD/DNS, Storage, File Services, Remote Access, Office Automation, CSM, backups and Virtualization. this is very common. and you are required by some HIPAA or other requirement to run Bit Locker and your key management is kind of iffy. Again common, because i've seen it. now every friggen Windows server has a BSOD and the only way to delete that corrupted file requires the BL keys. you are now out of business. this is the kind of shit that kept me up at night because in a lot of places IT is the redheaded step child when it comes to budget and severely understaffed and running by the seat of their pants. retiring from that shit show was like being reborn and i would never go back either. and i fucking hate Windows.
Been there and done that but retired from the business in '06. Haven't missed a day of it.
Over here in Aussie, it smashed us in our Friday afternoon peak! In our IT serious incident chat I could see people posting the blue screens of death that various machines were showing and my housemate burst into my office and told me that it was Crowdstrike - their company (finance) was down too. So I had to inform the chat. We couldn’t get flight plans uploaded so that resulted in a ground stop. A goddamn mess. We have a charter business (as well as regular RPT) that flies into many REMOTE airports for mining companies and related which now have stuffed computers, it will take DAYS to get IT support people into these places to get the computers fixed. 😬 At least our big hub ports are back up and running but now of course we have a massive backlog of upset and disrupted passengers.
It was somewhat amusing watching all the euros and yanks waking up and being all “what’s going on?” 😮
My flatmate was up quite late restoring their company’s services.
Bad day to be a windows admin! Luckily all my systems run on Linux/Kubernetes.
"Hold at Bovington." I retired from AAL in 2002 - I see that's one thing that hasn't changed going into Heathrow!
Greetings all. I worked in IT security infrastructure for decades. The industry has been replacing human analysis from the equation for a while now with cloud based technology (now marketed as 'AI' enabled for an investment sound bite). It all started with overseas outsourcing. It was bad crossing the communication divide with language but trust in 'AI' is absolute rubbish. This level of fail should never be possible no matter the excuse...well intentioned, accidental or malicious.
Thanks Juan for all the energy and effort you exert to keep us informed.
The problem with so many companies and industries around the world relying on the same product for their "cyber security".
Well, the software worked, didn't it? You can't hack a computer that won't even boot!
No they're not. It's got plenty of competition around the world. There's just not headlines about "ABC Corporation's Computers Are Working Today"
@@stephenj4937 this! most likely a test. Crash every computer instantly!
The company where I'm working was not affected, but one year ago would have been because we were in that time using a security solution that had CrowdStrike software on it
I’m getting tired of UA-cam auto deleting my perfectly reasonable post. I really hate this AI moderator they’ve got is a complete mess. It can’t tell what’s spam and what isn’t.
There has been a huge amount of embarrassing cockups at these corporations where they suffered a data breach and it turns out that there was a patch for the hole that the nefarious party gained entry that was not applied or not applied correctly.
My thinking is that a decision was made to just auto push these things through as soon as they are available to get them patched and deal with whatever fallout comes from incompatibility or whatever that would be better than dealing with the public relations disaster of a preventable breach.
But I don’t think anyone anticipated that it would go this bad.
Much like the "Swiss cheese" model discussed on this channel, this failure represents much more than a single error by a single person. The fact that a company with such an enormous reach lacked the process to detect this problem before rolling it out and rolled it out globally all at once represents a lack of organizational maturity that is just astounding.
The other failure is that so many other companies allowed their computers to be automatically updated by the Crowdstrike company.
@@ericfielding2540 I don't believe the updates were optional. Crowdstrike Falcon automatically retrieves its own updates. This was not a Windows update. It was the actual Crowdstrike software that crashed, but since it ran in a privileged mode, the crash was fatal for the operating system as well.
It starts much earlier. Companies using such a buggy and insecure OS that it even needs extra software for this. Then buying snakeoil like this to "fix" the issues. Then the problems with the snakeoil product itself.
@@ericfielding2540 With this one, CrowdStrike bypassed the usual “staging” environment that IT departments use to test updates. IT departments thought they had the option to review and block CrowdStrike updates but in fact did not.
Fun fact-o-the day... The blancolirio affiliate office in Tucumcari offers guided tours on Tuesdays and Thursdays. ALWAYS opt for the Thurs reservation! Same $29.95 cost, but at the end where you see Juan's private parking spot they let you reach in a jar and grab a free pickled pig foot!
Should be a free taco.
Yet another amazing, timely and accurate video explanation! Thank you, Juan, for your dedication to public awareness and safety.
Thank you Juan. As always a neat short no fluff news article. Thank you.
I've been subscribed to this channel for around half a year - I had no idea Juan was an airline pilot. This man just keeps on impressing me.
JB is Edward R. Murrow award material, and it's about high time he be bestowed with that great honor to public service.
Someone can correct me if I'm wrong, but I believe that he works for American Airlines.
777 pilot for American
@@hamsterama That's incredible. Next I'm gonna find out he is a decorated fighter pilot with 12 tours... or something. No, but in all seriousness, this man is a societal role model if you ask me.
@@IncorrigibleBigotry You may not be too far off LOL! Before his airline career, Juan flew planes in the Air Force. He's dropped anecdotes here and there in some of his videos about his time flying in the military. I agree, he's a great role model! And he's got a very interesting life story.
We have the same situation here in Australia as well & we where sitting on the Tarmac in Perth waiting to see what our next move is, Push Back & Start Up or Flight Cancelled, Already some 3 Hours behind Scheduled Departure time, Had already been in the Cockpit for just over 4 Hours without even having moved so much as an inch.
I decided to Deplane those Passengers who are on board For their own Comfort as it seemed the ATC Guys & Girls were having Problems on Access Data Relating to Ground Movements which was not Related to this Outage apparently they advised
All this happened to myself yesterday Friday 19/07 around Midday & we finally departed PTH at around 10:00 Pm for a Flight cross Country to BNE.
We should have departed at around 12:00 Pm Midday yesterday Friday 19/07.
787-9 Flight.
Society is so damn vulnerable. The tech geniuses have built a tower of cards
"Too big to fail."
We've known for a long time that these tech companies were more focused on making money than building bulletproof products, indeed, the focus on profit MUST come at the expense of security -- it's not a guess, it is reality! The speed with which Crowdstrike patched the problem is a problem in its own right given that every hacker, nation state or otherwise, now knows this critical company put a quickly assembled patch up and I would bet some of them will be probing for weaknesses.
As one of the "tech geniuses", I respectfully question why you're using the internet, if that's your attitude.
It's not really tech geniuses making these dumb decisions as much as business idiots
Just wait until AI takes over
2:06 we have a term for this in the IT. We call this "Friday afternoon patch" :)
But great to hear that Crowdstrike also subscribes to the idea to "I don't care about testing, just ship it!".
Ouch.
'Test' is a four-letter word.
Critical Drinker's Bad Plot quote fits so well here: Nah! It'll be fine.
@@paulsherman51 great idea, let's make it a unit test.
import unittest
class TestTitle(unittest.TestCase):
def test_title_is_string_and_length_four(self):
title = "Test" # Example value for title
self.assertIsInstance(title, str, "Title is not a string")
self.assertEqual(len(title), 4, "Title does not have exactly four characters")
if __name__ == '__main__':
unittest.main()
“Read-only Fridays”
I was flying in the North East this morning. Reporting in to New York Center got the "route change, advise ready to copy". Ready to copy the inevitable 12 point routing scipt, I received present position direct LAAYK then direct destination ! Really must of been no traffic!
Juan Browne coming in with the most factual and punchy report ive heard on the issue all day! Keep em coming sir!
Amen, brother.
Thanks for the astounding visual. 👍 I just finally got home after a verrrrry long Friday and a Saturday cleaning up the mess this made at my job. Unfortunately, most every computer needed to be touched, and there are some stragglers that we'll have to get on Monday because some areas aren't accessible on the weekend. Because of how early on in the boot process it crashes Windows, the chance of it being able to fix itself is extremely low.
McAfee rolled out a bad client update about a decade ago that also caused major disruptions. But McAfee had a much smaller customer base than Crowdstrike. I've spent all day handing out BitLocker recovery keys so folks can boot into safe mode and fix the problem. Busiest Friday I've had in years.
It was the SAME Crowdstrike CEO that time at McAfee! I think he was CTO then. Not kidding…
I took off this morning at 5am - CE525. Radio was quiet, and I got direct destination once turned over to center. Nice.
Nice!
I left at 1432 westerly on 92. Sensors showed a clear path so I maintained speed at 55mph on mains and 35 in arterials. Upon arrival at my destination data suggests my mpg was underscore so my projected travel fuel amount was deemed sufficient. 😂
The major gas station chain in our area(Speedway) uses Crowdstrike in their system too so I had to drive all over town looking for a station that was actually open; the one that was had raised their prices by a good 10 cents and there was a huge line. This really demonstrates how easily it could be to cripple the computer backbone that runs the world in 2024.
It's foolhardy to have electricity to pump fuels and not be able to because of anything computer related. Find a workaround there will surely be a need in the future.
Manual backup systems should be maintained not removed as obsolete (my peeve with bean counters over riding reliabilty and security).
I flew Delta from LAX to Charleston, yesterday, with a layover in Atlanta. Got to CHS at midnight, but my bags didn't make it. At Atlanta I saw a line with what seemed like 1000 people in it. Didn't pay much attention, and carried on to my flight. Found out this morning about the outage.
Just my luck. I was also stranded in Manchester, England the morning of the Continental/United merge. Their systems were down from that and had to stay an extra day.
I'm a Charlotte based first officer with a regional Airline. We had a 3:20 p.m. Charlotte/Chattanooga turn and then we went up to Charlottesville, Virginia for the night. The first turn went off surprisingly smoothly and on or ahead of time. We actually landed ahead of time in Charlotte, but we had an extremely extended taxi that wound up putting us about 15 minutes late. There was an incredible amount of aircraft on the taxiway trying to get into the ramp. Then we had an aircraft change for who knows why and wound up being about an hour late getting out for our last leg of the day. There were an awful lot of dissatisfied people meandering around the Charlotte airport.
Crowdstrike hit, it is very effectiv!
So you Circled over the " Tank Museum" Bovington ! Really cool.
Crowd STRIKE is a great name for their company apparently.
"...they brought me in here to do a job, they asked me to stir the damn tanks and I stirred the tanks!"
“What was that gauge reading before you hit the switch?”
@@cwhitty05 Don't tell me how to push out a security update
This reminds me of the AP Radio News satellite receiver software update pushed to all of the receivers which bricked every single one of them. Somebody at headquarters clicked on the wrong file to send on the uplink to install for all receivers. That left radio stations without live national news, special audio clips and all of the text pushed news services. The solution was to send a big group of people all around the country to open up each satellite receiver and install a permanent daughter board that put the receiver back in service.
Which is why best practice is to always have a product ID and HW version map tagged in any firmware release. So when someone goofs, then x thousand devices may download the file. But then rejects it because it's for a different product or for a different hardware revision. Bandwidth consumed, but no bricked units.
I looked at the radar this morning around Chicago, 1 plane coming in to ORD from Mexico and a LOT of GA probably going to Oshkosh!
Nice haircut!
Thanks.
I saw this, I didnt even Google it just waited for Juan’s update. Thank you.
The connected world until it isn't.
Thx Juan
I had a great day at work! Just checked my computer every 20 minutes or so to make sure it's still blue.
Local grocery stores closed here. Nuts!
My grandmother said to never put all your eggs in one basket.
Hi Juan, a very warm welcome to London. If we had known you were coming we would have baked you a cake.
“What are you doing, Dave?”
CROWDSTRIKE - "Let's push the patch straight into production without testing on a Thursday night (fixed) to a hundred thousand systems. "
Great plan!
Seriously poor testing.
If by "Friday morning" you mean 6pm US Eastern Time on a Thursday, then sure.
CI/CD has been a hot buzzword for the last 5 years or so. I expect IT companies to reverse the trend of firing QA people, having developers do development, testing, deployment and hot-fixing. That'll teach them. Or not.
@@marklnz You still generally avoid making changes that are going to affect the company on a Friday, because then you're stuck with people scrambling on a Friday to try and fix things.
or was it a test to see how we would react?
The more I learn about our 'joined-up-modern-tech World', the more I like independent magnetos : )
Makes the old mainframe systems seem like a good thing.
Two is one. One is none.
@@danielayers - First law of redundancy : )
My facility has had a handful of false-alarm deluge system activations over a few decades, causing millions of dollars of damage to aircraft and ground equipment.
In that time there have been no hangar fires.
Our mitigations are sometimes more hazardous than the hazards themselves.
Lmao who pays, the alarm company?
Also, is it AFFF that drops?
@@FeistyFalcon10x sorry I don't know that level of detail.
Hope you have a good stop over here in the UK
What a nice day to be Retired IT!
In the 1980's, we discovered a central mainframe computer with remote terminals wasn't practical (everything went down with the mainframe). So we went to stand alone computers- but now- due to mission creep- we are getting more and more controlled by "them". We are all sitting ducks! I just went through "telephone tree- hold" hell for 3 hours at ATT, Verizon, and BOA. I must have heard their propaganda and "marketing education" message enough to throw up or throw a tantrum! We need to fight against "the man"!!!!
This is one of the many reasons I use Linux exclusively.
Thanks for the update JB. You are always in the loop 🤙🏻✈️
My work team and I had a merry day remediating Crowdstrike. Other teams elsewhere in the company had similar joy. The fix itself was easy, delete 1 file from a specific directory path. The hardship was the labor to do it. And, naturally, extra challenges when confronted with petulant servers.
What people might not think about risk, is today’s issue is still outweighed by the many, many product updates that have had zero issues. Most CIO’s are very comfortable with having “now” security, as opposed to a program of being X number of days behind patching for vetting. No CIO wants to be pilloried for being security lax.
Mileage may vary of course, as some IT systems, like enterprise databases, need careful planning and testing.
DUMB POST AWARD WINNER
I, coincidentally, spent my part of my afternoon helping a friend remediate a bit of malware the claimed to be ransomeware. His AV hadn’t been kept up to date.
Captain 👨✈️ Browne, so happy that you are safe and had only a short wait at the gate. Safe travels back home.🙏
as an IT senior developer ... all I can say is any "defect" in the code/process should have been caught in the testing phase of the project. All developers make mistakes, that is a given ... but the good ones don't let them deploy to PRODuction ... If I was running the show that whole project Team would be on some seriously shaky ground right about now ... I'd be asking them 'have you given any consideration to another profession?" ....
I work in software qa and I can't imagine how this was missed. Sounds like some of the test /release process was skipped.
And as an "IT senior developer" you should have experienced enough in your career to know that even with significant QA processes and procedures in place, it's still possible for things to sneak through. No process is ever foolproof. If you haven't learnt that yet, then I feel compelled to reflect *your* question back to you: 'have you given any consideration to another profession?"
@@marklnz I nominate you for political office in Washington, D.C. Your kind of thinking is what makes America the greatest in the world. Enough with that obsession to detail and "failure is not an option" thinking. Why, political office is not sufficient for you. I would like you to be the head of N.A.S.A. In 1986, even!
P.S. - I have DONE major corporate software testing. The process is idiotically faulty.
@@marklnzIn response to your multiple snarky remarks made to several commentators…it was a major screw up by an “IT” company and their developers with final results yet to be determined but not good! I go back to the days of IBM Systems 36 and 38 where DP managers lived in fear of the next software “update”! If Crowdstrike screwed up by cutting corners on testing, then they should suffer the penalties…ask Boeing how that’s working out. My recommendation is that you stick to your IT work and leave the commentary to those with a bigger picture! 🤨
@@Bill_Woo From your tone you seem to apparently have some issue with what I said - which is funny because your last sentence effectively reiterates exactly what I said.
Appreciate this post - this brings to the forefront the need for both IT software vendors *and their customers* to have a validation and verification process in place, using test platforms, to verify software updates and upgrades before they are pushed into production.
You'd think for such a massive software product, they would have those systems in place. Deploy and listen for screams I guess.
@@chris-hayes Agreed. My perspective is that the investment in a validation program along with the systems, software and personnel is far less than omitting such a program when the well known substance hits the electrical convenience. This is a business management failure as much as a failure on the part of the systems and software technology end users and providers. It will be interesting to see what changes occur going forward.
You can never test all the possibilities to fail. I know ive been involved in 5 ERP system implementations, with a LOT of testing. After Go-live you spend month's fixing problems
@@larsbr4519 That's correct, nor is the goal to test everything. A validation program focuses on testing the critical aspects of a platform or software that can cause issues like this. If this defect eluded the business and test validation processes in place at customer institutions and the software vendor, the next step is root cause analysis and closing the gap so this does not occur again.
@@larsbr4519 Hats off to the institutions that caught this defect in testing, or perhaps a quarantine program for software updates, thus avoiding the impact to their customers. Perhaps some of them reported the problem to both Microsoft and the software vendor. It would be educational to learn about the business practices, polices and procedures as well as the validation program so other organizations can adopt these practices.
We cancelled 1000s of doctors appointments today in our small area. No tech, no service. Widespread chaos!
Love it..affiliate office, here in Adelaide Oz. We stood down planes and cancelled flights.
Our niece and her hubby was on the way to Tokyo from Bangkok. The ticket counters were down in Bangkok.
I appreciate your positive approach.
So furious with Crowdstrike. Work today sucked thanks to them 😅
@@jay1373 Oh sh it! Lol
Work with real computers and not toys and work might suck less... Well you're probably stuck with the same ignorant leadership but that is just how things roll.
It took me about an hour to do the patches for the computers at my job. It makes me appreciate the good old days, when I did my job without any computers running Microsoft. Back then there was no internet.
How many machines do you have to patch? Wondering how much work it's gonna be. Cant imagine doing this is co has big remote workforce.
AS400
You must be getting old if you "appreciate the good old days" of no internet. 🙂That was 40 years ago.
@@PaulLoveless-Cincinnatiyep psun days of old.
@@phillipzx3754 Imagine thinking that 40 is "old". 😛
Just read that in a survey most users are fed up with software companies continually rolling out new features that users didn't ask for and don't use. It's far past time tech companies stopped the endless pursuit of growth and focused on (long-term) value.
(As I typed this comment, the Chrome browser interrupted me with a popup informing me of some new feature that I could care less about.... good thing no software execs were near me or one of them would be lying on the floor with a fat lip, lol.)
I don't know for sure but I've been assuming that this wasn't a software update - it's an anti-virus/malware program so I assume it was an updated virus definitions file or something.
People and X Spurts have ridiculed me for blocking updates in Windows and Android systems. I've yet to get bitten by crap like this. And I store MY data locally, Three copies physically reposing in different locations miles apart. "Its safe - it's in The Cloud", my wrinly old ass.
How about switching to Linux and Firefox, mate? :)
@@RubenKelevra can't. I'm a photographer so I have to run Windows or iOS.
SAS. Software as a service.
Thanks Juan. We are leaving from Sacramento to Seattle on Monday morning to catch a boat to Alaska.. Hopefully, a majority of the issues will be resolved by then.
They couldn't deliver my milk and I'm having to ration my tea. I'm British and unfortunately this may cause me to riot.
If you hadn't colonized India you wouldn't have gotten adicted to tea.
@@malcolm20091000 My daddy was a Gurkah. indigenous American are you?
My husband, who was an IT manager for a large company said he had some experience with "Crowdstrike". Said they were "ARROGANT ASSWIPES". Strong handed sales tactics. Experts at intimidation & coercion. Little to NO customer service & claimed their "SHITE DON'T STINK". Didn't PUBLISH their pricing. He learned that was a RED FLAG. Karma, it seems, has had a nice LUNCH with them.
LA to London flights go right over our house in Rifle Colorado.
Saw and heard,a very large A350 one night last week heading east.
Happy Trails ✌️
Strange to hear you talk about the Bovingdon hold... that's where I flew my first solo in 1968, before RAF Bovingdon closed a couple of years later!
Thanks Juan.
I was with a large international carrier's IT department for 35 years. Just before Xmas many years ago, a "glitch" literally brought our computer system, which also directly supported a number of of other carriers, and related services, to its knees. At one point, it appeared the system had "dumped" literally tens of thousands of PNR's (Passenger Name Records) into thin air. But in our case, we had a brilliant group of coverage programmers who were able to restore the entire system in a few hours. To my company's credit, it (at the time) paid its IT staff generously and treated them with respect. It no doubt made the difference in the recovery. Now almost 30 years later, my former company has "outsourced" its most critical system coverage to India. In the case of CrowdStrike, it begs the question of what caliber of employees were involved in this debacle? It should have never happened and illustrates how a few lines of code can directly impact millions of MS Windows users. Our IT systems, and especially those related to the aviation industry, are particularly vulnerable.
That workaround is all well and good, providing you can actually get access to the drive from the BSOD screen via command prompt and if one happens to have bitlocker turned on (as I hear, it will be the default option for Win11 24H2), it may cause problems accessing the drive. If it asks for your unlock key, well that is stored on the microsoft account if you use one to login to windows, otherwise you'd need to have manually recorded it down somewhere. Office365 being affected means one may not be able to login to thier MS account on another machine to get the key.
So that fix may on some systems prove to be harder to implement, depends on the company or system if boot from USB is allowed, if bitlocker is in use, and if the user themselves actually has the knowledge and confidence to go poking around inside the System32 folder of Windows.
I used to work for Bayer UK. An employee sent her holidays snaps (+ a virus?) to "all" , problem was, she sent them to all in the entire company in the entire world, not just all in her dept. Bayer worldwide was down for 3 days. Each computer had to get fixed.
I got one for you... someone sent a "greeting card" (+ worm) to a secretary at one of my customers' plants. She clicked the fatal link and within seconds every HMI/SCADA system on every plant floor in the company was infected and blue-screened. This naturally brought all production to a halt. After this incident they finally listened to me: "air-gap the manufacturing systems from the office networks".
And it's not like it's the first time this happens. Anyone remember the AT&T frame relay upgrade disaster from 1999?
Crowdstrike: "They can't hack your computer if it can't start up."
You can't hack into a crashed, down computer! Success!
Not only airlines, here in Australia we had banks, ATMs and supermarket self-checkouts go down. CASH IS KING BABY.
I used to live near the Bovingdon hold and it was always cool watching heavies circling in the stack on a clear morning.
Our Bungalow below the Bovingdon hold has seen a quite day today . Enjoy the good weather, we don't get much of it
2:07 Imagine being that programmer, imagine being that quality control person, imagine being the managers over these people, imagine being the manager responsible for pushing the update out, imagine …
Thanks Juan. I go to your channel for accurate information. Amazing how most of the media couldn't get the story straight. Having been a software engineer, I really sympathize with the bugger that made this change.
On the bright side, I don't own any crowdstrike stock.
I had to deal with this, this morning. I came to work with a BSOD on my work station. The patch is currently fixed and reboots can remedy but there are fixes that an admin has to do, especially if Bitlocker is enabled. Earned my pay this morning. Others, well, I can see meetings coming next week on alternatives to this patch/security system that one file breaks the line. (hope some sleuths find either MSFT changed something, or get this, someone the day before on Reddit criticized CRWD being overvalued...hmmmm, plausible)
Interestingly enough this issue also affected City of Phoenix Fire Department computerized dispatch system. As a work around they went back tot he old school method of manually dispatching the calls. Also had impacts on the two large hospital networks Banner and Dignity in the Phoenix metro area.
We had a Kaspersky antivirus update do something like this in 2004. I was the first one in the office on a Monday morning and my desktop was ‘blue screened’. Every PC I checked was the same. I woke up one of the IT guys and he opened up his laptop, only to be greeted by the same blue screen. Uh-oh ….
I'm an IT manager - luckily only 2 out of our 40 servers were affected, and simply rebooting them brought them online. We did have one employee who's laptop was affected, and rebooting did not resolve the issue. Trying to walk a non-technical person through rebooting into safe mode to delete a system file is... not fun. Hopefully it was just the one - I can't imagine having to deal with hundreds of these issues without having hands on access to the machine.
We had servers running Crowdstrike at work. Luckily no workstations, so it was an impact that we could cope with.
I feel sorry for all corporations that had Crowdstrike everywhere.
Some rumor allegedly from Microsoft says that rebooting your computer 15 times will solve the problem.
Our IT department is also saying the issue can be cleared with multiple reboots.
Computers probably don't contact the Crowd Strike servers every time they restart, plus when it does contact Crowd Strike there are lots of requests so may not get a fast response. Hence it can take quite a few tries to get the patch. So 15 is a reasonable number of tries to attempt before assuming the fix isn't going to work.
This reboot kluge I can verify as true at a large city hospital. Interesting to see nurses turned IT support.
Rebooting as a security incident response is laughable. Don't run mission critical systems on Microsoft's toys.
@@haqvor This wasn't a Microsoft problem. Cloudstrike is a third party security tool that many companies have installed on their systems.
Very good report, Juan, on a very important issue: companies that are too big to fail and service providers supplying them who are too widely and deeply embedded across the industrial and national spectrum to have such broad failures, impacting those user communities.
There is a UA-cam channel BrenTech that seems to provide sensible and frequent updates on these types of IT related current events.
I couldn't electronically sign for a UPS package here. They just had me sign on paper, but I bet that was a real problem for them.
Wow. I can only sign with my finger now. I'd be in a real pinch.
That thing got the company I work for. It was simple to recover from but took time. They provide internet security (as you noted) and their update was fubar and BSO'd our computers. I'm not much of a fan of cloud anything but that's what we are moving to.
Thanks!
Juan, I'm surprised you didn't let people know that the CrowdStrike doesn't affect commercial aircraft, so you don't have to worry about boarding an aircraft. NOW, it does affect both ticketing software and flight management software that the airlines use to run everything. So the challenge is to get on a flight that has not been canceled. But you're fine once your on the aircraft.
As far as Air Traffic Control goes, I don't think the US is vulnerable, as I believe the ATC software being used is Linux/Unix based and does not have/use Falcon software. (I could be wrong about this.)
Reservation systems used to use the AS/400 ... darn great beast
@@paulsherman51 They're great systems right up till you type "CICSPRDA INACT" and forget the printer name.
This impacted all industries.
The most terrifying thing is how much critical infrastructure ultimately relies on windows. I work in tech and know better 🙃
Also Hi from the UK!
Thaks for sharing . excellent explanation to clear the actual problem.
Very interesting Juan I didn't understand that anyway so you've kind of helped out a little bit stay safe.
Now we know where all those McAfee developers go when looking for a new career path where they can do more damage. 😂
Welcome back to London.
I hope that you feel at home here.
As a dev, I always find that amusing seeing airport displays, atms, etc using Windows-based software. Outside IT, if you're talking about all the biggest software companies in the world, their production systems are 90% Linux. Why does that matter? When over 90% of modern software is written for Linux, companies with Windows systems will have fewer and less supported options.
Who knew that centralized IT would be a problem? 😅 I did after being replaced by it and struggling to find a job 🤣🤷🏼♂️
What is "centralized IT" even? Not one big computer went down, thousands of unrelated machines went down.
Man, that sucks. Take a look, Protocol Labs is hiring. Maybe something interesting in the listing.
Otherwise, in Germany, we're desperately searching for skilled workers. You can basically start working anywhere you want. Everyone is searching. :)
@@marcellkovacs5452 well, the point here is that not individuals did click on "update" but the IT of all the companies shipped the update as soon as available, downing all computers at the same time.
Crowdstrike is having massive layoffs. They got rid of employees to cut corners, and other companies are doing the same.
@@marcellkovacs5452As the adage goes, "Don't put all your eggs in one basket" 😅
Yep, I work for a US Govt. Agency in IT, and we had to suspend operations and send employees home.
I would loooove to see the expression on the persons face that pushed the enter button on the patch. 😮
Mr. Bill?
I work at hospital in Michigan. This crippled 75% of our computers in our large system, as well as multiple systems. Still not fully functional.
A common use of Crowdstrike Falcon is EDR (Endpoint detection and response). It is a class of tools that allows monitoring behaviour associated with malware, detecing malware-like patterns and remotely responding to incidents (e.g. by blocking problematic actions by suspected malware).
The update was an updated driver (which runs deep in the Windows kernel) for monitoring some channels used by some malware and that had some kind of bug, causing the blue-screens...
Affected computers were running Crowdstrike Falcon (which apparently is quite popular...)
It affected systems at FDA as well. I was able to access our systems and put in a full workday after around 9 am, but many of my colleagues were unable to log on for most of the day.
Great Video ! The TV News headlined Crowdstrike and Microsoft... The News talked about SWA flying out of Midway Airport ! Flew out of Midway Airport when they were the Worlds Busiest Airport... First time i heard of Crowdstrike ! Miss the Pan Am Shuttle flights to LEO to make the connecting flight to the Lunar Shuttle... tjl