UAB computer forensic expert discusses CrowdStrike disruption
Вставка
- Опубліковано 5 вер 2024
- Gary Warner Director of Director of Research in Computer Forensics at the University of Alabama at Birmingham (UAB) discusses CrowdStrike disruption -- MORE ➡ shorturl.at/kc0iw
.
.
.
.
.
.
#alabama #alabamnews #abc3340 #uab #crowdstrike #microsoft #outage #forensic #computerforensic #research #it #techsupport
Subscribe now www.youtube.com...
----------
Follow us on other social media:
ABC 33/40 on Facebook / abc3340
ABC 33/40 on Twitter / abc3340
ABC 33/40 on Instagram / abc3340
For more information, visit abc3340.com
Have a newstip? Send it to us! share@abc3340.com
ABC 33/40 is an Alabama based ABC Television affiliate owned and operated by Sinclair Broadcast Group. Sinclair Broadcast Group, Inc, is one of the largest and most diversified television broadcasting companies in the country today. Sinclair owns and operates, programs or provides sales services to 163 television stations in 77 markets, after pending transactions. Sinclair's television group reaches approximately 38.7% of US television households and includes ABC, Fox, MyTV, CW, CBS, NBC, Univision and Azteca affiliates.
#news #sports #weather #alwx #birmingham #anniston #tuscaloosa #alabama
Somebody did not do diligent testing. Pushing untested updates is extremely bad.
That's what probably has happened and I was waiting for them to mention it, that there was a breach in the process
9:04 Nah, as a software dev, I respectfully disagree. Deploying patches without testing them beforehand has a much higher risk than delaying them a few minutes for testing - especially when we're dealing with kernel-space apps.
Actually I'd do several layers of QA / quality gates before deploying anything.
And if speed is the issue, as a rule of thumb, the development phase is expected to take considerably longer than the QA testing.
At least customers should be allowed to configure if they perfer new and untested patches to slightly delayed but tested patches.
This is why you work in software and not security
Also, who doesn't check if a pointer is NULL before accessing it?
The last questions regarding what could be done should be asked of a Software Engineering professor instead. Software developers know it is not time consuming to test a software update - actually, tests should be fully automated. Something went really wrong at Crowdstrike.
He flat-out justified the practice of pushing out untested software that runs at ring 0 at the 5:00 mark in the video.
@@AmericaAndAllies the problem was a content update. the software behaved as designed. the testing/QA group are not the baddies
designing a system that borks on bad input? amateurish. building mission-critical infrastructure on a foundation this poorly considered? this is a cultural issue involving incentives at odds with the culture itself
@@SFDestiny There was no testing prior to pushing out the update. This disaster revealed several issues at Crowdstike. The emperor has no clothes and there is no where to hide.
Still don’t understand why large enterprises don’t test this update, even if it is daily. Because it hardly takes any time at all, and can prevent a lot of problems and damage. Not sure if this software allows for a delayed rollout of updates, if I understand correctly every agent on every single pc and server can be updated without intervention from the IT department.
This is strange considering the fact this software is mainly used in enterprise environments where development, test and production environments are separated. In this case they should have just installed the update on one machine to find out it was broken. A job that would take a few minutes at most. Obviously that doesn’t include testing all applications running in the enterprise.
You are thinking about this the wrong way. Enterprises would not have much control over this. This is a content update (think of it like an incremental update) that provides more info on new attack patterns. This is delivered by Crowdstrike. Think of it like a definiton update on an AV. You do not test every definition update as that would be impossible (no matter how much manpower you have). Also, normally these updates are auto-allowed for security software (like Crowdstrike) so that bad actors do not have time to exploit between them pushing out the update and you approving it after a thorough testing. Having said that, this puts a lot of testing onus on Crowdstrike as they are operating at such a low level and they know that if they push out any wrong update, it has the potential to bring down the system (which is what happened in this case). If you ask me, it looks like someone (or many someone) dropped the ball at Crowdstrike. Some of us have looked at the corrupt file, and it looks like it is a corrupt driver. This should have been picked up in the build process and stopped from deploying to customers. Also, why they didnt roll out updates in a phased manner is beyond me. Looks like they pushed out the update to everyone at once which is why the effect was so dire.
Enterprises don’t have that much control. The vendors don’t allow that anymore
Testing an auto updating anti virus from a very reputable vendor is on the bottom of the list for any IT company. Hindsight is 20/20. There have been a bazillion updates from edr/xdr softwares in the past, mostly successful.
That delay in testing and deploying the update in a controlled or development environment should not be a factor because it is the standard practice for software update. You do not just roll out something this impactful without prior testing.
This begs the question, what was the actual purpose of the corrupted update anyway?
Zero-day threats pose a conundrum because the response really needs to be tested thoroughly ( not only in a test environment), but in each organization, it should be deployed on ONE machine in production first to see how it works. A test environment can never replicate real-world production conditions.
Not only that, the fix might need to be rolled out to ONE machine in production in different geographic locations (or similar) because of different conditions in those areas. This is how to prevent the BSOD fiasco that just happened. However, the clock is ticking with zero-day threats. So the IT community needs to come up with ideas and solutions on how to handle this going forward.
That is why you do not test on a single machine and call it good. You have a pool of test machines where you are supposed to test the code for the very thing that you are trying to detect and prevent. That is called an automated test environment. The idea that things must be rushed out because a bad guy has a new trick is not justifiable and yet people like this 'expert' does exactly that. You have zero chance against zero-day with this mentality.
excellent explanation. thank you.
He is wrong. You must test before deployment, yes it will take some time, but Very basic testing would have prevented an awful lot of economic damage and medical consequences. The software may not be nominally part of a safety critical system, but large scale systems failures by their very nature have safety implications critical for society as a whole.
You are right, he is dangerously wrong headed, a menace mentality if you ask me. People like this guy triggered a global dumpster fire. Sad, pathetic.
UAB is a powerhouse of knowledge across the realm. Excellent explanation!
Excellent explanation. Thank you!
This is a good reason to delay new updates for a week or two and make sure this won't happen to you, IF it's an option.
But they are being updated constantly. Windows 11 is an almost nightmare . It's like every day thing . And now this 3rd party thing .. It's just nuts !
Just get a Linux if you can't keep Windows updated. There are thousands of viruses, malware, etc. etc. that are being written every single day. The only way to protect your windows machine from them all without auto updates is by getting a Linux or mac
Please Don't let Crowdstrike Get away with this Please sue them.
Around the 5 minute mark, the claim is made about the urgency and the tempo of updates as a rationale to risk tripping a BSOD in a billion computers. That is ludicrous. It is dangerous. It is wrong.
Crowdstrike and Microsoft has been compromised.
Unix has a way to roll back any updates to prevent issues like this. Crowd strike or Microsoft needs to implement something similar. Plus testing before deployment.
AFAIK what you describe exists in Windows too but how this software works and that the drive is locked by TPM which is similar in both Linux and Windows likely means that both OS are similar in implementation and vulnerability
Perhaps the only thing saving UNIX machines is a general lack of interest for this type of software today but that could change
The UNIX (and Linux) version of CrowdStrike has the same weakness but got a valid virus definition update file.
Really good explanation
Where I work they try to make me say that we are undergoing updates when I am having system issues, so the old *update* excuse doesn't really math well with me, they make and update and didn't test it, thereby making a release a virus update which nukes the system. Crowdstrike it's self is just a predatory sounding name, I call wm CrowdSTROKE.
When it happened I thought it was my computer so I just started reinstalling windows by the time I was done I found out what had happened lol.
In IT, we used to call this "patch and pray."
Possibly every agent vendor (such as Crowdstrike) needs to include a definition of what a valid, well-formed file looks like at the same time they push out the patch. So that the receiving system can identify if the patch is valid or not. I'm just brainstorming here; I don't know if this is a practical solution.
And this doesn't prevent the definition itself possibly having errors, but we're getting into rare scenarios here. The overall lesson: make sure there are NO errors in your file before you push it.
It's called a file integrity check and in many cases performed by doing a checksum
Although it's often done automatically by the download function, in some operating systems like Linux, the developer has options and the check isn't done. Who knows, Crowdstrike might have been pushing their updates from a Linux server to the Windows machines at customers and no one might have noticed the discrepancy.
Und someone added code to do an integrity check in their own they might be accused of writing bloatware
It is clear they have no integrity checks and data validation checks. It is an amateur move.
What went wrong was that Crowdstrick did not adaquately test in limited mode.
Testing would have prevented this.
No testing?
so your supporting patching without testing in order to protect. right
Hello, what’s difference between Crowdstrike and Palo Alto Network Protection?
yes its takes few min to fix but cost ?
Didn’t crowd strike deploy this update to theirs system?
A better response to corrupted or malformed files (and other anomalies) at the low level is needed. Microsoft needs to work on this (and possibly other OS vendors as well).
I don't see why Microsoft is to blame in this case.
Well, there's possibly an opportunity for MS to review their boot process and see if it could have handled this event more gracefully. That's what I'm saying. An event like this is not 100% one party fault. A lot of things lined up for the perfect storm. Including how CS's customers accepted this low-level update.
@@lak1294 my dude the failure is BEFORE boot
@@SFDestiny I got that. Are you telling me *nothing* can be done to improve the pre-boot process? Then Houston, we have a major problem. And I'm a gal, not a dude.
@@lak1294 I'm off to research the genderization performed by "my dude" and smh that a string of letters and numbers is intended to convey special meaning...
Risk based testing is probably what should have been done to determine which type of tests needed to be run given the short timeframe this critical update has to be released into the field. Perhaps that was done and less weight was given to Windows 10 because it is expected to reach end of support by Oct 2025. However, Microsoft and CrowdStrike may have been unaware of the number of critical applications still running on Windows 10.
Did you just imply that crowdstrike should prioritise urgency over pre-release testing?
I so hope not!😮
This lack of, or failure of the release process did more damage than any single hacker could.
Where was the pre-release test to a small sample group?
How was there no automatic fail detection then return to a previous stable version. This isn't very fail safe software. They may be great at intrusion detection, not to good at high reliability.
And yes software can detect fails and take action ... such as reversion to a previous working copy. Check your risk matrix, system up has a priority over intrusion detection.
What went wrong with CS process that caused this mishap?
the problem isn't technical per se. the problem is cultural. we don't reward competence
Yesterday was a big win for CrowdStrike. Finally a virus protection program that disabled the most prolific spyware program on the internet - Microsoft Windows.
No Linux/Mac products were harmed.
Is it really true that simply removing the faulty patch would restore the machine? It's been said elsewhere that the update didn't create a restore pount.
And if course nowadays since bitlocker has been mandated on most Windows machines, just gaining access to do any repair if missing bitlocker keys may be impossible.
if your organization cannot manage keys... smh
No long term impact, hah! We are painting ourselves into a corner with computers within side a burning house!
yes but the reason they update is to keep up with new threats so if they revert to the previous system they can exploit those vulnerabilities not yet patched? At the end He say's that this type of software needs to be pushed out without delay to avoid giving attackers a window if opportunity but before that say's there's no threat for attacks because by removing the new patch you would just revert back to the previous version.?.....also even when the OS systems are down, can't they can still target the network, map network infrastructure, identify potential entry points, and gather intel for future attacks etc?
ok test like first priority what the problem?
It's not just bad code here it's corrupted file, that opens as null in c/c++who doesn't check if a pointer is NULL before accessing it?
This is pathetic, and by the way we tested CS it's not even that better than defender, what's going on here?
According to several commentators (eg, ua-cam.com/video/ZHrayP-Y71Q/v-deo.htmlsi=I5Ekl6zYoXqS73ZT), the update file just contained all zeroes, possibly triggering the null-pointer dereferences. If that is true, there were either NO testing whatsoever against this update file, or their build process was seriously broken.
It's Green Witch Mean Time... of course.
So many medical personal are doomed 😂😂😂😂
i disagree you can't test this and has to go out immediately. With the timezones assisting on this, they could have rolled out to companies in Australia first (during work hours), and do a rolling rollout. if things don't go well, roll it back immediately. Australia could be the canary.
This did impact Australia during the work day. Shut down our airports supermarket etc. so put that idea where you what doesn’t shine
Test, test and test.... In production! 😂 Smh 🤦🏾♀️ 🤷🏾♀️ 😒 😑
Question 1 is unnecessary
I've had black friday :) as a IT helpdesk.... and know how to solve the problem... and He is totaly right... it works that way
need to test everything what you talking about??
I do not support releasing before testing, creating that sense of urgency can only lead to disaster.
Criminal negligence.
Show the person who wrote the code.
Linux saves lives.
don't blame the snake oil salesperson. this is a social disease
Microsoft Winows is sh#t.
what you talking about ? by pass test environment because you need fast? try on your own computer! try on boss comp first see what he sad!!
this comment section is biased and uninformed. disheartening from various perspectives