Microsoft made it somewhat easier for Cloudstrike to make major mistakes, but it was Cloudstrike that neglected to not do the appropriate testing and phased distribution that should be done for software with such low level access.
All due respect. But a bad business decision by microsoft did not deploy faulty code without staging around the globe. Crowdstrike neglected any - and i mean any, even the most basic, care in deploying, creating and testing that update. And they alsow managed to kill debian linux earlyer this year...so one could make a case for them beeing incompetent. And rightfully so. If a product is bad on microsoft windows because microsoft does not allow you to access the kernel in a smart way, sue them (class action), or limit its functionality. But they decided to backdoor their stupid written and lazy ass tested software, intentionally circumventing WHQL. Which js far bejind neglectance....thats intend. I would not be surprised if they get sued to oblivion over at least one of those points.
Crowdstrike is to blame for not being able to stage their updates. If we let an OS start without a critical component like an EDR, I think we should not operate on such machines
true - but even with or without staged updates, it is blaringly obvious this software 'patch' WAS NOT tested at all and should NEVER have been released without good through testing. Crowdstrike is an untrustworthy company because 1) it does not test EVERY patch it released to its' kernel drivers and 2) it does not practice staged updating in every instance - if at all.
They were not parsing signature file for valid structure. They trusted the input. For many years. So it is not political or anything. Crowdstrike just pushed shitty code to production, that exploded with shitty Crowdstrike signature update. They have not fuzzy tested themselves...
Not to mention them bypassing the Microsoft signature requirements for drivers by not updating the driver itself, but a config file the driver read. Which means they can change how the driver operates without updating it and getting another signature, despite being made a critical startup driver. Kinda defeats the spirit of signature required software @@tomstoob
Agreed! Over the years I have been amazed in how more "business applications" were being managed much like Smartphone Applications! Everyone should recognize that MANY Smartphone Applications are updated on a WEEKLY to DAILY basis! This means that only small amount of testing to executed before pushing out to production! Why? Current view is that any "bugs found" would be fixed in the next release! In a perfect world of not mistakes, it is awesome for all machines/systems get the latest important updates! However, as the CrowdStrike situation proved, we still need to protect ourselves for complete shutdown of outages due to bad updates! Staggered updates give to to learn that there are issues and the latest update should be removed from production ASAP! Wow! 🍀🙀🌟 😇
It’s so clear that so many of you are commenting without having listened to, or processed the contents of this extremely complex and informative video. I’m a Mac and Linux guy who works with Windows managers and I’m learning more from this video than I do working full time with certified Windows engineers! Really great work.
Anybody who can't separate data from code is too incompetent to be trusted to run anything at kernel level. On top of that no code running at kernel level should be able to auto-update itself without the authorization of the IT administrator.
Wow! This is the best video about this topic. When I heard about the Crowdstrike outage I immediately wanted your thoughts. Thank you so much for the historical details that helped explain why this happened. I'm not a programmer but I understood your explanation. Fantastic video. My big takeaway - Deploying without fully testing was Crowdstrike's error but Microsoft must take part of the blame for helping create the environment in which the error became so disruptive. I'm eagerly awaiting future videos from your channel!
Loved the insight! Calming voice! The only thing that would make the video even better for me is to have links to the sources in the video description. I know it's possible to manually find them all but it would be a nice touch.
I've been inspired to try my hand at cybersecurity thanks to you! I've been developing software for a while but this is a whole different ballgame lol. I've known of you since 2017 as most of us did, but just now found out you have a youtube channel. I have to say, it's an honor to be able to listen to a history maker like yourself.
Yep, no deployment strategy (deploy on small number of machines first to see if it works properly, then the rest), or the deployment strategy was skipped for some reason.
@@x_ph1l Yes, they did not have a tiered deployment strategy, but it's actually worse than that. They didn't perform a sanity check test on a single Windows box. The Falcon parser running in the kernel was sent code updates multiple times each day. These aren't just definition files, they contain actual code that is run by the Falcon parser within the kernel. These 'channel files' weren't securely signed. The Falcon parser didn't perform a hash check. The Falcon parser didn't even look for a file header. There appears to have been no checks at all. Crowdstrike's kernel parser just accepted and ran whatever file it was sent. If even one of those validation steps had been in place (validation on a windows box, tiered deployment, signed channel file, file header in the channel file, hash check of the channel file), this would never have happened.
worse than that, the CEO said it was company policy due to the patches being so urgent that it would take too long to check them. i think he meant to say that it would take to long to go through the microsoft validation, but if not, he has no business having power over technical issues because he just does not know enough to make those calls. basically, if you don't have time to do it right and test it, you really don't have time to do it wrong and have to fix it.
Read the report which came out yesterday/today. They definitely did... and it was definitely inadequate. Like, power-cycling your test machines seems a pretty important part of the process, because it takes way less effort vs the rest of the test suite & will at least catch egregious errors like this one. Then you move on. Bonus: when code fails a unit test, maybe don't respond by fixing the unit test so the code passes unless you're very sure the test is broken.
A lot of history was covered and Marcus provided a retelling of it from his personal experience and added his opinion. I found it interesting and enlightening. Based on his very publicly documented expertise in this field, his views carry a lot of credence with me and his insights are valuable. Hence, my comment of gratitude. I'm afraid I know nothing of you or your expertise in this matter and therefore your unqualified assertion of fact carries very little credence with me whether I agree with you or not.
We have quite a similar background 😊 though I chose the more predictable route of a job managing a team of penetration testers. I like the rational, cool-headed presentation Marcus. New sub gained.
as a rookie aspiring pentester I didn't understand 80% of this but that fact motivates me to keep learning as much as I can because these clearly aren't simple endeavors good god
He actually fixed a worldwide virus that cost people around 4 billion USD in damages, so when he got caught for his EX malware production he got very minimal punishment a UA-camr named "CrumbRS" made a really good video about it
@@zipz7622 Was just jibbing on it cos he said "was" an "ex", which means not an ex in present tense, ergo, he's a malware developer again today. It's not funny now I have to explain it.
It is nice to finally see someone from the US willing to look deeper than Microsoft's whining about the EU. MS tried to stack the deck in their favour and the EU called them out for it. Forcing someone to play on an even playing field is not forcing them to make their system vulnerable. Good move to call out Daves Garage and others for their skewed perspective. Trying to protect MS while blaming the EU. Apple standing in the background wondering what all the fuss is about is just the killing blow to the MS fanboys. Just insane.
can you give your thoughts on Apple's changes to the filesystem and generally how they're moving everything that's kernel related from any access points from userland?
KB5028997 and KB5034441 plus Windows allowing the system to be locked from repair is why I’ve been giving CrowdStrike the benefit of the doubt. Microsoft now saying the EU caused Windows to be this far broken is bonkers. Microsoft has had over a decade to get this right and it’s only been getting worse.
When Microsoft introduced a warning system that you are about to do something on your system intentionally or unintentionally, everyone disabled it because it was a nuisance
Crowdstrike is to blame for the Crowdstrike outage. Why do we always look for other excuses when multi billion dollar companies fail at basic quality controls. It's simply putting scaling and selling before building a quality product/service and in the case of Crowdstrike it couldn't be more obvious. We need to punish this behavior and not act as if they have no accountability because they are on the 'same team'. They did worse damage than most malware, and need to be treated accordingly.
OMFG!!!! Yo, your that guy lol holy shit dude I for sure thought they took you to some black site lol. Yo please make a short video or course on how to RE a botnet so it can be mapped, nothing like learning from the master himself. I'm so stoked on finding your channel, hell yeah, keep at it homie.
@@dyu4634but the correct analogy would be the renter plugging something in which caused your faulty wiring to burn down your house and those of everyone else in the block. yes, they caused it by plugging in something way overpowered, but you enabled it by not fixing your faulty wiring, so you get to share the blame. same here with microsoft.
IMO as the details of outage were stated, both Microsoft and Crowdstrike are to blame. MS for not fixing this whole dangerous aproach long time ago and Crowdstrike for omiting best practices in favour of expediency of updates - injecting update thru uncertified file that is processed by kernel driver, instead of changing the driver and having it tested and re-certified by MS lab procedure. PS: As was hinted in several videos on the subject, Crowdstrike is not the only one who does this. I wonder for example, if Windows gaming anticheat technologies are not used to do this "hack" when updating as well. That is ofc just layman outlook. In the end, it will be down to technicalities of law and legal experts.
leonard french did a legal deep dive on his and to sum it up, both crowdstrike and microsoft have some exposure for gross negligence here, and lawsuits are being prepared against both.
leonard french did a legal deep dive on his and to sum it up, both crowdstrike and microsoft have some exposure for gross negligence here, and lawsuits are being prepared against both.
Games don’t install their own kernel drivers, so it’s not even slightly a similar thing. They generally use installer-level patches for updates, and that would go for anti-cheat updates to control new cheat strategies as well.
@@simontillson482 This used to be true but a lot has changed in the last 5 years. Many games employ kernel level anticheat nowadays (see Valorant as an example). They actually do deploy their anticheat as kernel drivers, and yes, its bad.
@@leonzewe Wow, that is indeed news to me. Seems a rather over the top solution. I bet it hasn’t eliminated cheating either - there’s so many ways to modify online gaming that don’t even need to change the game itself. I’ll do some reading, thanks for the tip.
Personally I think the solution lies in kernal processes, different from user processes. A user (and thus any user app) can kill any user mode process (even if they have to enter a password for root access) but they'd have to drop down to kernel level permissions to kill a kernel process. Anything that involves directly talking to hardware would be put in the kernel itself but for anything else it would involve pipes between kernel processes. The kernel can maintain the security of kernel processes and kernel processes can maintain the security of user processes. This new type of process would require the creation of a new user with greater permission than root which would resolve the problem of security being compromised due to users using admin accounts for their main account. It also means greater kernel stability because most of the kernel would be in kernel processes, with security & hardware code being the only exceptions. Could create something like klibc to share among the kernel processes too.
When security is an afterthought rather than a core requirement of your OS ... you get Crowdstrike type mass outages ... will it be the last or just the first of many ... ???
this really bugs me. it does not matter what language you write in, it can have bugs. if it then runs in kernel mode without going through enough testing, it can crash the kernel. this was a kernel or user space issue, combined with lack of testing. language had nothing to do with it.
good deep dive into the historical problems individuals and businesses have experienced with Microsoft's deficiencies in the design of Windows OS's from the very earliest days of an operating system that was basically designed for individual users/standalone PC"s to its' migration to networks, business and corporate use. Can we blame Microsoft for the disaster of the CrowdStrike global crashing of computer systems? Indirectly, yes - but had CrowdStrike been run by competent and responsible CEO's who make sure their Falcon anti-malware software patches are FULLY TESTED before dumping them on en masse to millions of Windows networked PC's all around the world AND that they would only do such software patches via staged releases to their Falcon software clients, this disaster would not have occurred. If the first point was practiced by CrowdStrike then nothing would have occurred to Windows computers running Falcon software on July 19th and if point 2 had been implemented, the problem would have been quickly reported back to CrowdStrike's HQ and the staged releases to 'next-in-line' recipients of that Falcon software patch would not have gone ahead. So the blame has to be primarily with CrowdStrike's management of software patches distribution and secondly to Microsoft in a much less important sense. CrowdStrike Falcon software runs on Linux and MAC's too but this faulty Falcon software patch only occurred on Windows computers.
@@tomstoob i don't dispute that in any way, but microsoft is also liable due to not fixing the bug in 8 years which caused the machines to stay down, which is why the lawsuit from delta airlines is naming both while trying to recover the 500 million they lost due to the outage.
@@grokitall I would blame MS for allowing the Crowdstrike boot required driver to pass WHQL certification process. Since the driver reads external files, UNSIGNED BY MS, the parser should have been cross examined inside out and bombarded with all kinds of sane and insane things, including NULLs. Still, as many others pointed out, Crowstrike is mostly at fault here, since they didn't take even basic measures to avoid this failure.
@@Hexanitrobenzene i agree that crowdstrike caused this, but microsoft enabled it by not fixing known issues involved in this for many years. that is why they are also being named on the lawsuits.
I think the problem is also that clearly there was code or at least a sophisticated format being read into the kernel by way of these files being dropped in. Why are Microsoft certifying drivers with WHQL that change their behaviour based on the system files? Who says it had to be Crowdstrike who added this file? Could just as well have been a malicious actor who worked this out, and worse still maybe they would work out a way to craft a malicious file of this proprietary format and cause an exploit to reveal itself, allowing arbitrary code execution in the kernel space by escalating privileges through Crowdstrike.
The outage happened for a simple reason…absent of adequate software dev and release process with a bit of negligence. Would the “content update” (which was full of code!) had been tested, the outage would have been prevented.
Hey Marcus, I’ve been getting really interested in learning ethical hacking and was wondering if you could help me out with some tips or guidance on where to start. I’d really appreciate any advice you can share!
If the kernel is ring 0, and user is ring 3, didn't they leave rings 1 and 2 for drivers and high privilege processes like malware protection, but didn't actually use them? But would this slow down Windows a lot? Remember in NT4 they moved at least part of the graphics drivers into ring 0, to save a lot of context switches.
7:46 Right now the EU, and other nations, wants backdoor access to people's private encypted communications for law enforcement puposes. Tho some companies, like Apple, argue this would also make it easier for bad actors to expoit - you can't have that stuff just for the good guys without the bad guys finding them as well. The eternal struggle continues.
Have you addressed the other factor Dave considered comparing Microsoft to Apple? Apple doesn't care about backward compatibility. I suppose that's ok with a smallish cult like user base, but Windows is dedicated to ensuring backward capability and with the relative depth and breadth of the user base compared to Apple that seems to me to be a significant factor.
Not true. OS cannot be the policeman to prevent all security software mistakes from happening. Crowdstrike also affected Linux at April 2024 timeframe. So no OS alone have real fix.
This is the first time I hear someone mentioned EU as a party to blame. I know blaming windows is popular but I think the fault is on Crowdstrike you know. the party responsible for pushing an empty file for the sensor to use?
microsoft tried to throw the EU under the bus to distract from the fact they have not fixed the boot loop bug since 2016. everyone but fan boys immediately called it out as bs.
Microsoft's IT monopoly is the real ongoing problem, and this incident SHOULD have raised questions about so many industries putting all their IT eggs in one corporate basket-case that the world comes crashing down in a single point of failure, but people couldn't see the Operating System behind the forest of Blue Screens seen all over the planet, 'cause it's all CrowdStrike's fault... :)
firewalls can now be implemented in user space, on access virus scanners can as well, and packet filtering. it all depends on what parts have been made visible to user space.
Endpoint protection should be Windows functionality, opening kernels for random 3rd party companies is no different than introducing virus into the system. And then nobody is responsible for anything.
the only alternative to third party drivers is to have a limited number of all in one boards like apple has, then write all the drivers yourself. that solution looks worse than the problem. the better solution is to persistently track which module is starting, and if it crashes the kernel, disable it on reboot.
@@grokitall Another solution is to limit the privileges of drivers. All major operating systems currently have an all or nothing approach, where every part of the kernel and every driver has access to everything. Most architectures, and certainly x86, actually provide far more abilities to fine tune that. It is entirely possible for example to allow a driver access to one specific device without having access to anything else. This is a design choice by MS (and Linux, and MacOS) made a long time ago. Changing that would be a massive overhaul. It would have other downsides as well, but the pros and cons have shifted massively since that choice.
@@jbird4478 I'm not sure that is actually possible. while i don't doubt that processors have multiple rings as a security measure, what you are talking about is something like having a user mode in kernel space, which i can't see any way for the hardware to manage. this is why operating systems only use kernel mode and user mode. this additional mode would need you to tell tell it it could access this small range of memory, but none of the other memory, and while i can see how the operating system provide service functions to do so, it would not give you any advantages over providing those same functions to user space, and would have all the same context switching and message passing costs that are inherent in the microkernel model, which nobody has figured out how to make efficient since the 1970s when the idea was first proposed. for something like a serial port, communicating at 56 kilobits per second this could work as a user space driver, but for things like gigabit networking and external hard drives i am sure that the costs involved would slow it down too much, which puts us right back to having the drivers in kernel mode.
The tile alone gets a thumbs up from me. I can't so many people are letting MS off the the hook when it's their DAMN house that Crowstrike is Fn up!!! Also glad you mentioned Dave's Garage because he was trying to let MS off the hook.
No. It is CrowdStrike's fault. They pushed out the defective software, not Microsoft or the EU.. It is the EU who forced Microsoft to allow third party software operate at the Kernel level. Microsoft did not want to allow this, they were forced to by the EU. That is why it is argued it is the EU's fault. No. It is CrowdStrike's fault. They pushed out the defective software, nobody else.
Microsoft did provide the ELAM driver infrastructure as a supported way of hooking into kernel activity (which is what Crowdstrike uses). It's just that Crowdstrike deployed inherently unsafe code, and ran it in an unsafe privileged context. If they ran their configuration code in Ring 3, there would have been no BSoD. If they ran their code in the safety of a sandbox, windows would not have blue screened If they tested their own code properly, their driver would not have BSoDed. If they implemented their automated deployments correctly, the faulty 291 file would never have reached everyone's machines. If they implemented proper memory probing and error checking in their driver, it would never have bug checked. Their subreddit was filled for three years with end users complaining about their driver causing blue screens. If Crowdstrike not ignored those warnings, they would have found their kernel driver was fundamentally broken, would have hired developers to fix it, and we would not have seen the outages that occurred this month. The only place where Microsoft went wrong is granting Crowdstrike their WHQL stamp of approval. Though Microsoft should have seen in their Windows telemetry that CSAgent.sys is not a driver that deserves to bear the WHQL certificate.
No, ELAM was provided as a way for antivirus drivers to start before other 3rd party drivers. It doesn't solve any of the problems discussed in this video.
Hello! Marcus I wanted to know if, you ever thought about creating a security software for individuals and businesses to help secure their personal online data and financial data against computer hackers, ramsonware and identity thieves. And showing more online how people can block hackers from sending viruses into their computer systems. Offering a free and paid software and videos. Creating simple software and videos that teach beginners how to code in creating technical products like a computer operating system to video games. Take care and thanks.
At a fundamental level though is Microsoft legally responsible for crashes caused by 3rd party software? Probably not. You are claiming the 3rd parties had no choice but provide products as necessary? Maybe it just comes down to who caused the crash, would be cloudstrike?
@@MalwareTechBlogReally the point stands though that it was a known thing, so the one doing the update should have been checking. It's like blaming your car manufacturer because you were injured in a car crash in a way that would only happen if the airbag wasn't there.... After YOU took the airbag out. You can argue about whether or not the way it works is good, but it's not their fault in any sense of the word. It's like when people blame Nintendo for some Switch ports not being very good, even though it was the developer or publisher cheaping out and not doing a good port. There's no such thing as a system that can't run a game (most ports used to be at least somewhat custom made for the system for a reason), only a game that a dev or publisher is not willing to put any effort into, so it's not Nintendo's fault even though they're the ones that chose to have a lesser-powered system. The devs and publisher are the ones who are joining onto a system and then not doing the work necessary to make sure it works well. Just like Crowdstrike did.
@@RunicSigilsactually, microsoft have had recurrences of the boot loop problem since 2016. the fix is also fairly simple, they just have not done it, leaving them potentially liable under gross negligence laws. the eula only provides a get out of jail free card to the extent allowed by law, which does not cover negligence.
No I'm going to side with Microsoft that once it let antivruses see signatures in user mode, they have most of what they need, signature and network motioning, not EDR. the fact that they still needed kernel protection from being removed by malware is a fair argument. Ok to be fair I'm a pleb and not an expert, but even if you're right that Microsoft needed to let antiviruses into the kernel, this explanation needs to be clarified and more specific since it doesn't sound convincing to a typical officer worker compared to Dave.
UA-camrs hey smh, in that case I guess many of us are ridiculously over qualified to speak on this but we are not UA-camrs, we still do the work we do 😂
The WHQL driver requirement predates Windows 7 and Vista. The thing where Windows refused to install drivers that weren't WHQL signed, started with the Windows XP operating system (though the WHQL itself is much older than that).
The Windows 10 process is different. It requires an Extended Validation code signing certificate issued by Microsoft themselves, which requires you to go through Microsoft's own validation process. Previously any code signing certificate was suffice.
in MY view - the fact that CrowdStike is using a MS Certified driver -- that went thru the certification - NOW we learn that it is ALLOWED to run off and execute code that has NOT been validated as safe - (the driver needs to check that everything that it is going to execute is a valid safe set of code) (I get it their patch was bad we know that even the most basic validation of the crowdStrike file this would have not caused the issue - to ME - Microsoft needs to TIGHTEN the certification process why can a CERTIFIED TO BE SAFE kernal driver - load ANY code to execute without checks that the code is valid - think of an emulator that runs emulation - hmmm, that emulated instruction is not legal - so - I am NOT going to allow you to execute (or TRY to execute code that we know is not valid - THROW an error - sorry bub - that will not fly - so - sure the crwowstrike patch would have received ERROR - messages but NOT caused a BSOD ( which of course is doing exactly what it was designed to do - PROTECTING the system ) I,E. there can be and possibly should be recovery from an attempt to execute an INVALID set of code - instead of throwing up BSOD - additional recovery from that situation migh also be a solution. JTB
Your idea on Microsoft wanting to monopolize on anti-virus isn't conspiracy theory!! They been doing this for years, take Netscape for example! Microsoft SLOWED DOWN Netscape browser on Windows back in the days on purpose so they can monopolize on the browser space! I don't like virus scanners myself and would rather rely on Microsoft new Hypervisor + CET but you know they are trying to monopolize in this space.
Microsoft is just so insecure as to be laughable. I don't understand why they still are so popular. People can take free classes to switch to Apple, and Linux is getting easier to use. Windows is an embarassing joke, Azure sucks for stability and Sharepoint is a curse. I just...hate them a lot actually.
I did not understand a single thing but I feel sufficiently educated now to comment on Reddit like an expert on this topic.
join /pcmasterrace you're already overqualified! 😁
This is so funny and true
😊😊😅😅😂😂
Feel the sarcasm...
Microsoft made it somewhat easier for Cloudstrike to make major mistakes, but it was Cloudstrike that neglected to not do the appropriate testing and phased distribution that should be done for software with such low level access.
Are you sure you understand how markets work?
All due respect. But a bad business decision by microsoft did not deploy faulty code without staging around the globe.
Crowdstrike neglected any - and i mean any, even the most basic, care in deploying, creating and testing that update. And they alsow managed to kill debian linux earlyer this year...so one could make a case for them beeing incompetent. And rightfully so.
If a product is bad on microsoft windows because microsoft does not allow you to access the kernel in a smart way, sue them (class action), or limit its functionality.
But they decided to backdoor their stupid written and lazy ass tested software, intentionally circumventing WHQL.
Which js far bejind neglectance....thats intend.
I would not be surprised if they get sued to oblivion over at least one of those points.
Crowdstrike is to blame for not being able to stage their updates. If we let an OS start without a critical component like an EDR, I think we should not operate on such machines
true - but even with or without staged updates, it is blaringly obvious this software 'patch' WAS NOT tested at all and should NEVER have been released without good through testing. Crowdstrike is an untrustworthy company because 1) it does not test EVERY patch it released to its' kernel drivers and 2) it does not practice staged updating in every instance - if at all.
They were not parsing signature file for valid structure.
They trusted the input.
For many years.
So it is not political or anything.
Crowdstrike just pushed shitty code to production, that exploded with shitty Crowdstrike signature update.
They have not fuzzy tested themselves...
Not to mention them bypassing the Microsoft signature requirements for drivers by not updating the driver itself, but a config file the driver read. Which means they can change how the driver operates without updating it and getting another signature, despite being made a critical startup driver. Kinda defeats the spirit of signature required software @@tomstoob
Agreed! Over the years I have been amazed in how more "business applications" were being managed much like Smartphone Applications! Everyone should recognize that MANY Smartphone Applications are updated on a WEEKLY to DAILY basis! This means that only small amount of testing to executed before pushing out to production! Why? Current view is that any "bugs found" would be fixed in the next release! In a perfect world of not mistakes, it is awesome for all machines/systems get the latest important updates! However, as the CrowdStrike situation proved, we still need to protect ourselves for complete shutdown of outages due to bad updates! Staggered updates give to to learn that there are issues and the latest update should be removed from production ASAP! Wow! 🍀🙀🌟 😇
Why should some random software company be allowed to push code into the kernel!
Very good, I really like Dave's channel, I think it would be great to have you two talking about the subject.
Had the same idea, left a comment on Dave’s video about this too. Let’s make this happen folks.
It’s so clear that so many of you are commenting without having listened to, or processed the contents of this extremely complex and informative video. I’m a Mac and Linux guy who works with Windows managers and I’m learning more from this video than I do working full time with certified Windows engineers! Really great work.
Excellent video. Your expertise and presenting skills really show
I DO love to see this kind of video performed from a REAL PROGRAMMER who understand and know about O.S. 👏👏👏
"Fixing blue screens with more blue screens"
Anybody who can't separate data from code is too incompetent to be trusted to run anything at kernel level.
On top of that no code running at kernel level should be able to auto-update itself without the authorization of the IT administrator.
Ooooooh-kay
Thank you for the insights! 👍 (Navigated here thanks to Steve Gibson and Security Now!)
Thanks 🙏
Wow! This is the best video about this topic. When I heard about the Crowdstrike outage I immediately wanted your thoughts. Thank you so much for the historical details that helped explain why this happened. I'm not a programmer but I understood your explanation. Fantastic video. My big takeaway - Deploying without fully testing was Crowdstrike's error but Microsoft must take part of the blame for helping create the environment in which the error became so disruptive. I'm eagerly awaiting future videos from your channel!
Thanks for doing an analysis of the Crowdstrike crash without any yelling.
Amazing historical view - best video on the subject!
Loved the insight! Calming voice!
The only thing that would make the video even better for me is to have links to the sources in the video description. I know it's possible to manually find them all but it would be a nice touch.
I've been inspired to try my hand at cybersecurity thanks to you! I've been developing software for a while but this is a whole different ballgame lol. I've known of you since 2017 as most of us did, but just now found out you have a youtube channel. I have to say, it's an honor to be able to listen to a history maker like yourself.
Great insight. Why do you think Crowdstrike seems to have had no validation, at all, on their channel file updates? Gross negligence?
Yep, no deployment strategy (deploy on small number of machines first to see if it works properly, then the rest), or the deployment strategy was skipped for some reason.
@@x_ph1l Yes, they did not have a tiered deployment strategy, but it's actually worse than that.
They didn't perform a sanity check test on a single Windows box.
The Falcon parser running in the kernel was sent code updates multiple times each day. These aren't just definition files, they contain actual code that is run by the Falcon parser within the kernel. These 'channel files' weren't securely signed. The Falcon parser didn't perform a hash check. The Falcon parser didn't even look for a file header. There appears to have been no checks at all. Crowdstrike's kernel parser just accepted and ran whatever file it was sent.
If even one of those validation steps had been in place (validation on a windows box, tiered deployment, signed channel file, file header in the channel file, hash check of the channel file), this would never have happened.
worse than that, the CEO said it was company policy due to the patches being so urgent that it would take too long to check them.
i think he meant to say that it would take to long to go through the microsoft validation, but if not, he has no business having power over technical issues because he just does not know enough to make those calls.
basically, if you don't have time to do it right and test it, you really don't have time to do it wrong and have to fix it.
@@grokitall wow, that's eye opening))
Read the report which came out yesterday/today.
They definitely did... and it was definitely inadequate.
Like, power-cycling your test machines seems a pretty important part of the process, because it takes way less effort vs the rest of the test suite & will at least catch egregious errors like this one. Then you move on.
Bonus: when code fails a unit test, maybe don't respond by fixing the unit test so the code passes unless you're very sure the test is broken.
“Uniquely qualified”
*locks in and watches the vid intently*
Liked & subcribed at that point!
🤨
Thank you for your perspective 🙏🏻
Why perspective, Dave is just wrong. Its not a perspective its just correct.
A lot of history was covered and Marcus provided a retelling of it from his personal experience and added his opinion.
I found it interesting and enlightening. Based on his very publicly documented expertise in this field, his views carry a lot of credence with me and his insights are valuable. Hence, my comment of gratitude.
I'm afraid I know nothing of you or your expertise in this matter and therefore your unqualified assertion of fact carries very little credence with me whether I agree with you or not.
I stumbled onto this channel and i really enjoyed your content. i look forward to seeing more.
We have quite a similar background 😊 though I chose the more predictable route of a job managing a team of penetration testers.
I like the rational, cool-headed presentation Marcus.
New sub gained.
as a rookie aspiring pentester I didn't understand 80% of this but that fact motivates me to keep learning as much as I can because these clearly aren't simple endeavors good god
You are the best Channel in the tech world. And you are my Hero
If you WERE an EX-malware developer, then you ARE a malware developer NOW! BUSTEEEEEED! Great vid, thanks Marcus xx
He actually fixed a worldwide virus that cost people around 4 billion USD in damages, so when he got caught for his EX malware production he got very minimal punishment a UA-camr named "CrumbRS" made a really good video about it
@@zipz7622 Was just jibbing on it cos he said "was" an "ex", which means not an ex in present tense, ergo, he's a malware developer again today. It's not funny now I have to explain it.
@@marcot8549 appreciate the explanation as I surprisingly missed your wonderfully pedantic joke despite the capitalisation.
@@MadScientist512 I think most people missed my genius. It's hard being me sometimes.
I truly enjoy your insight.
Was looking forward to this!
Hyped to hear u speak about it and even more hyped that you are not afraid of going against dave garage his claim
It is nice to finally see someone from the US willing to look deeper than Microsoft's whining about the EU.
MS tried to stack the deck in their favour and the EU called them out for it. Forcing someone to play on an even playing field is not forcing them to make their system vulnerable.
Good move to call out Daves Garage and others for their skewed perspective. Trying to protect MS while blaming the EU. Apple standing in the background wondering what all the fuss is about is just the killing blow to the MS fanboys. Just insane.
Crumb video got me here.
Crowdstr8ke should be consulting to MS because they are changing the kernel, lack of QA and certification to MS
Really good high level overview of user permissions in windows thanks
Great vid, thank you. And grats on sanctifying out of bad ways, we are all born wicked but not all of us graduate from it!
Wow, this is so interesting. I would love to see more content about such low-level techniques as described here.
Probably more of a course thing than a UA-cam thing
@@MalwareTechBlogI know you are not doing this for the money
You are doing it for the cause I believe that thousands percent
Were you referring to AMSI API for process injection (memory monitoring), or other API group?
can you give your thoughts on Apple's changes to the filesystem and generally how they're moving everything that's kernel related from any access points from userland?
KB5028997 and KB5034441 plus Windows allowing the system to be locked from repair is why I’ve been giving CrowdStrike the benefit of the doubt. Microsoft now saying the EU caused Windows to be this far broken is bonkers. Microsoft has had over a decade to get this right and it’s only been getting worse.
I could listen to you for hours
When Microsoft introduced a warning system that you are about to do something on your system intentionally or unintentionally, everyone disabled it because it was a nuisance
Crowdstrike is to blame for the Crowdstrike outage. Why do we always look for other excuses when multi billion dollar companies fail at basic quality controls. It's simply putting scaling and selling before building a quality product/service and in the case of Crowdstrike it couldn't be more obvious. We need to punish this behavior and not act as if they have no accountability because they are on the 'same team'. They did worse damage than most malware, and need to be treated accordingly.
OMFG!!!! Yo, your that guy lol holy shit dude I for sure thought they took you to some black site lol.
Yo please make a short video or course on how to RE a botnet so it can be mapped, nothing like learning from the master himself.
I'm so stoked on finding your channel, hell yeah, keep at it homie.
Hi Marcus, this is a real long shot, but could i email you about some odd behaviour in macOS and how it handles some image formats?
Please bring back the MalwareTech podcast 😞
Great explainer, thanks.
Nah, this is on croudstrike, not microsoft
I feel the same too. If a renter rents my house and burns it down along with the whole neighbourhood, I'll be pissed if they hold me accountable.
@@dyu4634but the correct analogy would be the renter plugging something in which caused your faulty wiring to burn down your house and those of everyone else in the block.
yes, they caused it by plugging in something way overpowered, but you enabled it by not fixing your faulty wiring, so you get to share the blame. same here with microsoft.
@@grokitallexactly
IMO as the details of outage were stated, both Microsoft and Crowdstrike are to blame.
MS for not fixing this whole dangerous aproach long time ago and Crowdstrike for omiting best practices in favour of expediency of updates - injecting update thru uncertified file that is processed by kernel driver, instead of changing the driver and having it tested and re-certified by MS lab procedure.
PS: As was hinted in several videos on the subject, Crowdstrike is not the only one who does this. I wonder for example, if Windows gaming anticheat technologies are not used to do this "hack" when updating as well.
That is ofc just layman outlook. In the end, it will be down to technicalities of law and legal experts.
leonard french did a legal deep dive on his and to sum it up, both crowdstrike and microsoft have some exposure for gross negligence here, and lawsuits are being prepared against both.
leonard french did a legal deep dive on his and to sum it up, both crowdstrike and microsoft have some exposure for gross negligence here, and lawsuits are being prepared against both.
Games don’t install their own kernel drivers, so it’s not even slightly a similar thing. They generally use installer-level patches for updates, and that would go for anti-cheat updates to control new cheat strategies as well.
@@simontillson482 This used to be true but a lot has changed in the last 5 years. Many games employ kernel level anticheat nowadays (see Valorant as an example).
They actually do deploy their anticheat as kernel drivers, and yes, its bad.
@@leonzewe Wow, that is indeed news to me. Seems a rather over the top solution. I bet it hasn’t eliminated cheating either - there’s so many ways to modify online gaming that don’t even need to change the game itself. I’ll do some reading, thanks for the tip.
Personally I think the solution lies in kernal processes, different from user processes. A user (and thus any user app) can kill any user mode process (even if they have to enter a password for root access) but they'd have to drop down to kernel level permissions to kill a kernel process. Anything that involves directly talking to hardware would be put in the kernel itself but for anything else it would involve pipes between kernel processes. The kernel can maintain the security of kernel processes and kernel processes can maintain the security of user processes. This new type of process would require the creation of a new user with greater permission than root which would resolve the problem of security being compromised due to users using admin accounts for their main account. It also means greater kernel stability because most of the kernel would be in kernel processes, with security & hardware code being the only exceptions. Could create something like klibc to share among the kernel processes too.
When security is an afterthought rather than a core requirement of your OS ... you get Crowdstrike type mass outages ... will it be the last or just the first of many ... ???
Fuzz your interpreters people. Write them in memory safe languages only.
this really bugs me. it does not matter what language you write in, it can have bugs. if it then runs in kernel mode without going through enough testing, it can crash the kernel.
this was a kernel or user space issue, combined with lack of testing. language had nothing to do with it.
good deep dive into the historical problems individuals and businesses have experienced with Microsoft's deficiencies in the design of Windows OS's from the very earliest days of an operating system that was basically designed for individual users/standalone PC"s to its' migration to networks, business and corporate use. Can we blame Microsoft for the disaster of the CrowdStrike global crashing of computer systems? Indirectly, yes - but had CrowdStrike been run by competent and responsible CEO's who make sure their Falcon anti-malware software patches are FULLY TESTED before dumping them on en masse to millions of Windows networked PC's all around the world AND that they would only do such software patches via staged releases to their Falcon software clients, this disaster would not have occurred. If the first point was practiced by CrowdStrike then nothing would have occurred to Windows computers running Falcon software on July 19th and if point 2 had been implemented, the problem would have been quickly reported back to CrowdStrike's HQ and the staged releases to 'next-in-line' recipients of that Falcon software patch would not have gone ahead. So the blame has to be primarily with CrowdStrike's management of software patches distribution and secondly to Microsoft in a much less important sense. CrowdStrike Falcon software runs on Linux and MAC's too but this faulty Falcon software patch only occurred on Windows computers.
microsoft was responsible for not fixing the boot loop problem, which has been recurring since 2016, and is what caused the machines to stay down.
@@grokitall CrowdStrike was the primary player at fault for the 2 reasons I listed
@@tomstoob i don't dispute that in any way, but microsoft is also liable due to not fixing the bug in 8 years which caused the machines to stay down, which is why the lawsuit from delta airlines is naming both while trying to recover the 500 million they lost due to the outage.
@@grokitall
I would blame MS for allowing the Crowdstrike boot required driver to pass WHQL certification process. Since the driver reads external files, UNSIGNED BY MS, the parser should have been cross examined inside out and bombarded with all kinds of sane and insane things, including NULLs.
Still, as many others pointed out, Crowstrike is mostly at fault here, since they didn't take even basic measures to avoid this failure.
@@Hexanitrobenzene i agree that crowdstrike caused this, but microsoft enabled it by not fixing known issues involved in this for many years.
that is why they are also being named on the lawsuits.
I think the problem is also that clearly there was code or at least a sophisticated format being read into the kernel by way of these files being dropped in. Why are Microsoft certifying drivers with WHQL that change their behaviour based on the system files? Who says it had to be Crowdstrike who added this file? Could just as well have been a malicious actor who worked this out, and worse still maybe they would work out a way to craft a malicious file of this proprietary format and cause an exploit to reveal itself, allowing arbitrary code execution in the kernel space by escalating privileges through Crowdstrike.
The outage happened for a simple reason…absent of adequate software dev and release process with a bit of negligence. Would the “content update” (which was full of code!) had been tested, the outage would have been prevented.
Hey Marcus,
I’ve been getting really interested in learning ethical hacking and was wondering if you could help me out with some tips or guidance on where to start. I’d really appreciate any advice you can share!
The blame is actually on the company IT managers for not having proper policies in place.
If the kernel is ring 0, and user is ring 3, didn't they leave rings 1 and 2 for drivers and high privilege processes like malware protection, but didn't actually use them? But would this slow down Windows a lot? Remember in NT4 they moved at least part of the graphics drivers into ring 0, to save a lot of context switches.
7:46 Right now the EU, and other nations, wants backdoor access to people's private encypted communications for law enforcement puposes. Tho some companies, like Apple, argue this would also make it easier for bad actors to expoit - you can't have that stuff just for the good guys without the bad guys finding them as well.
The eternal struggle continues.
I do not want AV on my PC without being able to decide if I want it or not. Especially not at the KERNEL level.
Loading "third party" programs into the kernel sounds like a bad method. Microsoft should find another/better way.
I'm sure Crowdstrike insisting that this was not Microsoft's fault is nothing to do with them depending on Microsoft for their market.
@15:20 are you talking about "Daves Garage" ?
im sure he is.
He is; he refers to that channel at the start of the video
1:20-2:10
*Kernell* have always been a problem. Cause *Rootkits* expanded in those 'Early days'
Have you addressed the other factor Dave considered comparing Microsoft to Apple? Apple doesn't care about backward compatibility. I suppose that's ok with a smallish cult like user base, but Windows is dedicated to ensuring backward capability and with the relative depth and breadth of the user base compared to Apple that seems to me to be a significant factor.
Welp, we finally got our Y2K, just 24 years later.
W Marcus, I just watched the video on his life
Bro you are my hero
Not true. OS cannot be the policeman to prevent all security software mistakes from happening. Crowdstrike also affected Linux at April 2024 timeframe. So no OS alone have real fix.
Totally agree - it’s totally a Micro$oft fault
This is the first time I hear someone mentioned EU as a party to blame. I know blaming windows is popular but I think the fault is on Crowdstrike you know. the party responsible for pushing an empty file for the sensor to use?
microsoft tried to throw the EU under the bus to distract from the fact they have not fixed the boot loop bug since 2016. everyone but fan boys immediately called it out as bs.
Microsoft's IT monopoly is the real ongoing problem, and this incident SHOULD have raised questions about so many industries putting all their IT eggs in one corporate basket-case that the world comes crashing down in a single point of failure, but people couldn't see the Operating System behind the forest of Blue Screens seen all over the planet, 'cause it's all CrowdStrike's fault... :)
Yeah, using Linux would not help, crowdstrike caused Linux kernels to panic in the past also... But keep hatin if you wish so
What's are some examples of a good windows ecurity product that does not require kernel access at all?
Unplugging the computer
@@MalwareTechBlog So there's really no practical solution to this problem yet.
Not until Microsoft makes user mode replacements for all the capabilities security products need
firewalls can now be implemented in user space, on access virus scanners can as well, and packet filtering.
it all depends on what parts have been made visible to user space.
@@MalwareTechBlog What about TDSSKiller or GMER?
if Vin Diesel was into computers
Endpoint protection should be Windows functionality, opening kernels for random 3rd party companies is no different than introducing virus into the system. And then nobody is responsible for anything.
the only alternative to third party drivers is to have a limited number of all in one boards like apple has, then write all the drivers yourself. that solution looks worse than the problem.
the better solution is to persistently track which module is starting, and if it crashes the kernel, disable it on reboot.
@@grokitall Another solution is to limit the privileges of drivers. All major operating systems currently have an all or nothing approach, where every part of the kernel and every driver has access to everything. Most architectures, and certainly x86, actually provide far more abilities to fine tune that. It is entirely possible for example to allow a driver access to one specific device without having access to anything else. This is a design choice by MS (and Linux, and MacOS) made a long time ago. Changing that would be a massive overhaul. It would have other downsides as well, but the pros and cons have shifted massively since that choice.
@@jbird4478 I'm not sure that is actually possible. while i don't doubt that processors have multiple rings as a security measure, what you are talking about is something like having a user mode in kernel space, which i can't see any way for the hardware to manage. this is why operating systems only use kernel mode and user mode.
this additional mode would need you to tell tell it it could access this small range of memory, but none of the other memory, and while i can see how the operating system provide service functions to do so, it would not give you any advantages over providing those same functions to user space, and would have all the same context switching and message passing costs that are inherent in the microkernel model, which nobody has figured out how to make efficient since the 1970s when the idea was first proposed.
for something like a serial port, communicating at 56 kilobits per second this could work as a user space driver, but for things like gigabit networking and external hard drives i am sure that the costs involved would slow it down too much, which puts us right back to having the drivers in kernel mode.
Interesting on the EU judgement
brother this dudes speaking a different alien language
Its clear they need THREE levels, not two.
why anyone used administrator accounts as daily accounts on windows
The tile alone gets a thumbs up from me. I can't so many people are letting MS off the the hook when it's their DAMN house that Crowstrike is Fn up!!!
Also glad you mentioned Dave's Garage because he was trying to let MS off the hook.
The person or company that does a thing is responsible for having done it. The end. No further logic needs to be applied.
No. It is CrowdStrike's fault. They pushed out the defective software, not Microsoft or the EU..
It is the EU who forced Microsoft to allow third party software operate at the Kernel level. Microsoft did not want to allow this, they were forced to by the EU.
That is why it is argued it is the EU's fault.
No. It is CrowdStrike's fault. They pushed out the defective software, nobody else.
Microsoft did provide the ELAM driver infrastructure as a supported way of hooking into kernel activity (which is what Crowdstrike uses). It's just that Crowdstrike deployed inherently unsafe code, and ran it in an unsafe privileged context.
If they ran their configuration code in Ring 3, there would have been no BSoD.
If they ran their code in the safety of a sandbox, windows would not have blue screened
If they tested their own code properly, their driver would not have BSoDed.
If they implemented their automated deployments correctly, the faulty 291 file would never have reached everyone's machines.
If they implemented proper memory probing and error checking in their driver, it would never have bug checked.
Their subreddit was filled for three years with end users complaining about their driver causing blue screens. If Crowdstrike not ignored those warnings, they would have found their kernel driver was fundamentally broken, would have hired developers to fix it, and we would not have seen the outages that occurred this month.
The only place where Microsoft went wrong is granting Crowdstrike their WHQL stamp of approval. Though Microsoft should have seen in their Windows telemetry that CSAgent.sys is not a driver that deserves to bear the WHQL certificate.
No, ELAM was provided as a way for antivirus drivers to start before other 3rd party drivers. It doesn't solve any of the problems discussed in this video.
Microsoft blew it. No excuse.
Hello! Marcus I wanted to know if, you ever thought about creating a security software for individuals and businesses to help secure their personal online data and financial data against computer hackers, ramsonware and identity thieves. And showing more online how people can block hackers from sending viruses into their computer systems. Offering a free and paid software and videos. Creating simple software and videos that teach beginners how to code in creating technical products like a computer operating system to video games. Take care and thanks.
At a fundamental level though is Microsoft legally responsible for crashes caused by 3rd party software? Probably not. You are claiming the 3rd parties had no choice but provide products as necessary? Maybe it just comes down to who caused the crash, would be cloudstrike?
No, they're not legally responsible for the crash, they just made the conditions that forced antimalware products to behave in this way.
@@MalwareTechBlogReally the point stands though that it was a known thing, so the one doing the update should have been checking.
It's like blaming your car manufacturer because you were injured in a car crash in a way that would only happen if the airbag wasn't there.... After YOU took the airbag out.
You can argue about whether or not the way it works is good, but it's not their fault in any sense of the word.
It's like when people blame Nintendo for some Switch ports not being very good, even though it was the developer or publisher cheaping out and not doing a good port.
There's no such thing as a system that can't run a game (most ports used to be at least somewhat custom made for the system for a reason), only a game that a dev or publisher is not willing to put any effort into, so it's not Nintendo's fault even though they're the ones that chose to have a lesser-powered system.
The devs and publisher are the ones who are joining onto a system and then not doing the work necessary to make sure it works well. Just like Crowdstrike did.
@@RunicSigilsactually, microsoft have had recurrences of the boot loop problem since 2016. the fix is also fairly simple, they just have not done it, leaving them potentially liable under gross negligence laws. the eula only provides a get out of jail free card to the extent allowed by law, which does not cover negligence.
Bro... did you not get your gift card? what's it gonna take, $15?! 🤯
No I'm going to side with Microsoft that once it let antivruses see signatures in user mode, they have most of what they need, signature and network motioning, not EDR. the fact that they still needed kernel protection from being removed by malware is a fair argument. Ok to be fair I'm a pleb and not an expert, but even if you're right that Microsoft needed to let antiviruses into the kernel, this explanation needs to be clarified and more specific since it doesn't sound convincing to a typical officer worker compared to Dave.
Signature and network monitoring is not “most of what they need” the entire EDR market exists because of that
I blame society!!!😂
UA-camrs hey smh, in that case I guess many of us are ridiculously over qualified to speak on this but we are not UA-camrs, we still do the work we do 😂
Not posting on UA-cam wasn’t a qualification last I checked
Hey, nice vid. Couldnt think of anyone who could explain this to such a level of quality.
9:34 maybe fix the vouln in your seiling btw.
man, MAC users on twitter/X must be embarrass and delete their tweet by now.
So quick to jump on the wagon shitting on MS
i hate microsoft but this is crowdstrike fault
The WHQL driver requirement predates Windows 7 and Vista. The thing where Windows refused to install drivers that weren't WHQL signed, started with the Windows XP operating system (though the WHQL itself is much older than that).
The Windows 10 process is different. It requires an Extended Validation code signing certificate issued by Microsoft themselves, which requires you to go through Microsoft's own validation process. Previously any code signing certificate was suffice.
What about MUSHROOMS ? 🤔 🤔 🤨
in MY view - the fact that CrowdStike is using a MS Certified driver -- that went thru the certification - NOW we learn that it is ALLOWED to run off and execute code that has NOT been validated as safe - (the driver needs to check that everything that it is going to execute is a valid safe set of code) (I get it their patch was bad we know that even the most basic validation of the crowdStrike file this would have not caused the issue - to ME - Microsoft needs to TIGHTEN the certification process why can a CERTIFIED TO BE SAFE kernal driver - load ANY code to execute without checks that the code is valid - think of an emulator that runs emulation - hmmm, that emulated instruction is not legal - so - I am NOT going to allow you to execute (or TRY to execute code that we know is not valid - THROW an error - sorry bub - that will not fly - so - sure the crwowstrike patch would have received ERROR - messages but NOT caused a BSOD ( which of course is doing exactly what it was designed to do - PROTECTING the system ) I,E. there can be and possibly should be recovery from an attempt to execute an INVALID set of code - instead of throwing up BSOD - additional recovery from that situation migh also be a solution. JTB
Liked and Left 😊
Your idea on Microsoft wanting to monopolize on anti-virus isn't conspiracy theory!! They been doing this for years, take Netscape for example! Microsoft SLOWED DOWN Netscape browser on Windows back in the days on purpose so they can monopolize on the browser space! I don't like virus scanners myself and would rather rely on Microsoft new Hypervisor + CET but you know they are trying to monopolize in this space.
don't forget novell
1st time You learned of *Sarcasm?*
Windows has always been a garbage operating system. Microsoft is absolutely to blame.
the kernel is supposed to block unprivileged access from the beginning. to do otherwise is to write a bad kernel.
Microsoft is just so insecure as to be laughable. I don't understand why they still are so popular. People can take free classes to switch to Apple, and Linux is getting easier to use.
Windows is an embarassing joke, Azure sucks for stability and Sharepoint is a curse. I just...hate them a lot actually.
Microshaft and Cloudstroke both in bed.
Use built-in antivirus, keep admin privileges to ONLY those admins who actually know wtf they are doing. Move on!
this is all helpful info but would you mind citing your sources? possibly adding them to the description. it will help with researching. thanks
Which parts are you looking for sources for?