Some bad code just broke a billion Windows machines

Поділитися
Вставка
  • Опубліковано 13 жов 2024
  • Cybersecurity firm Crowdstrike pushed an update that caused millions of Windows computers to enter recovery mode, triggering the blue screen of death. Learn how the Crowdstrike crash happened and its impact around the world.
    #programming #cybersecurity #thecodereport #crowdstrike #windows
    💬 Chat with Me on Discord
    / discord
    🔗 Resources
    Crowdstrike Statement www.crowdstrik...
    XZ hack • Linux got wrecked by b...
    Rabbit hack • Rabbit R1 makes catast...
    🔥 Get More Content - Upgrade to PRO
    Upgrade at fireship.io/pro
    Use code YT25 for 25% off PRO access
    🎨 My Editor Settings
    Atom One Dark
    vscode-icons
    Fira Code Font
    🔖 Topics Covered
    Crowdstrike failure explained
    How does windows kernel work
    What caused windows computers to crash?
    Cybersecurity fails

КОМЕНТАРІ • 9 тис.

  • @tekalh7647
    @tekalh7647 2 місяці тому +33854

    The fact that it was an Antivirus that performed the single most successful malware attacks ever is just pure poetry

    • @y7o4ka
      @y7o4ka 2 місяці тому +1627

      introducing: McAfee

    • @lnidux
      @lnidux 2 місяці тому +976

      it's basically spyware anyway

    • @Dexaan
      @Dexaan 2 місяці тому

      Antivirus always becomes the very thing it swears to destroy

    • @SeeThroughist
      @SeeThroughist 2 місяці тому +529

      Security Malware ™

    • @twls153
      @twls153 2 місяці тому +636

      Crowdstrike be like: "Fine I'll do it myself"

  • @ComDenox
    @ComDenox 2 місяці тому +2935

    The alpha move of doing something that would make your stock value crash, but simultaneously freezing the stock market so that it can't.

    • @hackmedia7755
      @hackmedia7755 2 місяці тому +145

      money printer go brrrrt

    • @genshinF2Play
      @genshinF2Play 2 місяці тому +141

      market manipulation is sadly a common occurrence in stock trading. its like a casino, the house always wins.

    • @julianocs87
      @julianocs87 2 місяці тому +19

      So, stonks?

    • @Sandy-o4p
      @Sandy-o4p 2 місяці тому

      I wanna know who's shorting it today. I might open up thinkorswim and view the market.

    • @NeostormXLMAX
      @NeostormXLMAX 2 місяці тому +57

      @@genshinF2Playyeah insider trading legal for senators

  • @realsemig
    @realsemig 2 місяці тому +17155

    "You can't hack a system if the system doesn't work! "
    - Cybersecurity intern pushing the update

    • @matiosjed
      @matiosjed 2 місяці тому +328

      Crowdstrike should call it "a feature"

    • @atifrafique3764
      @atifrafique3764 2 місяці тому +62

      SAUL goodman would say that as his lawyer" your honour my client................."

    • @soloflo
      @soloflo 2 місяці тому +83

      9D chess brother

    • @EdmondDantèsDE
      @EdmondDantèsDE 2 місяці тому

      it was a preemptive strike. check mate, hackermen. 🧠

    • @vectoralphaSec
      @vectoralphaSec 2 місяці тому +26

      What makes you think it was an intern?? Could have been a long term full time employee.

  • @somethingsomething8511
    @somethingsomething8511 2 місяці тому +1876

    What the hell is their deployment process where they didn't catch this in testing? Like they ran the update, saw a blue screen of death and went "looks good, ship it"?

    • @Nmmask
      @Nmmask 2 місяці тому +279

      They didn’t even run it is what it seems like 🤣

    • @drodsou
      @drodsou 2 місяці тому +395

      Yep, everyone can make a mistake, but this is total negligence both by the engineer and the company processes. It's not an obscure error that happen on some machines, but one that bricks every single machine, which is enough evidence that nobody did the most basic test before deploying. And then, deploying it to everyone at once instead of doing it by phases. I hope the company gets privately sued and publicly investigated and punished hard.

    • @sarahfox3652
      @sarahfox3652 2 місяці тому +193

      Never test, never fail is what I always say

    • @SaraMorgan-ym6ue
      @SaraMorgan-ym6ue 2 місяці тому

      meh well it's funny stupid people are switch to linux then it won't happen even though a few months ago the same thing happened to linux🤪🤪🤪🤪🤪🤪🤪🤪🤪🤪🤪🤪
      you gotta love the stupidity of that one I can't even make this shit up it's that funny🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣

    • @lautacolombo
      @lautacolombo 2 місяці тому +114

      “It worked in my pc”

  • @strapatser9427
    @strapatser9427 2 місяці тому +25976

    Hiring 20 more project managers and 40 extra recruiters should prevent this in the future

    • @karlzone2
      @karlzone2 2 місяці тому +2352

      I think we need some business consultants to come up with a good strategy too.

    • @ilfirinms
      @ilfirinms 2 місяці тому +648

      And plenty EU regulations and standards, which had to every company, even village solo baker, implement, or else ...

    • @cultoftranquility9616
      @cultoftranquility9616 2 місяці тому +1316

      or increase the daily scrum meetings with 200%

    • @nebula_M42
      @nebula_M42 2 місяці тому +833

      don't forget a bunch of scrum master asking if there are any blockers every 10 minutes

    • @amira-supermiemie
      @amira-supermiemie 2 місяці тому +348

      And 30 more tech leads😂

  • @nicolascage4438
    @nicolascage4438 2 місяці тому +7875

    My dad: Come on, it's just an internship, what's the worst that could happen?
    Me:

    • @dheeru975
      @dheeru975 2 місяці тому +100

      balls out of my mouth 😢

    • @wolfhunter98
      @wolfhunter98 2 місяці тому +85

      Well you, probably, can't top these guys so you're safe. 😂

    • @koldata4887
      @koldata4887 2 місяці тому +10

      😂😂😂

    • @GhosPoison
      @GhosPoison 2 місяці тому +37

      Since it’s the summer, 100 percent it was an onboard’ee

    • @Sercil00
      @Sercil00 2 місяці тому +80

      I can already see the greentext
      >be me
      >be 22 year old NEET
      >dad wants me to start getting a job because sick of my shit
      >thinks I should work in IT because am nerd loser
      >gets me internship at large cyber security firm because he heard IT sec gets paid the best

  • @manteiga_de_pimenta
    @manteiga_de_pimenta 2 місяці тому +4179

    I was fired last week from my IT job. Yesterday I woke up to a call from the company asking for help, as apparently they didn't have time to hire someone else capable of fixing it, I politely said that I was busy, hung up the call and laughed my ass off of them

    • @zurielsss
      @zurielsss 2 місяці тому +935

      You should ask for 1000 per hour for the task, charge them 6 hrs

    • @Pixcrafts
      @Pixcrafts 2 місяці тому +92

      Poetry!

    • @pratikamatya5190
      @pratikamatya5190 2 місяці тому +70

      😆😆 I am so happy for you 😇🥰

    • @sidthetech7623
      @sidthetech7623 2 місяці тому +308

      Sign back on with a ridiculous contract.

    • @passmelers
      @passmelers 2 місяці тому +413

      you plitely hung up and laughed after the call instead? lame. laugh on the call

  • @debasishraychawdhuri
    @debasishraychawdhuri 2 місяці тому +1124

    If this kind of an upgrade can go through unchecked, imagine how easy it would be to pay one rogue engineer to deploy malicious code that can be exploited later at ease.

    • @sfappetrupavelandrei
      @sfappetrupavelandrei 2 місяці тому +42

      Actually, it is more probably that there was testing done, but this was a crack nobody noticed.
      A couple of years ago, I created a little app to help my brother at his work. He had a colleague who was kind of a mess and she was able to mess the app really bad. My brother was shocked that he couldn't imagine what were the steps this colleague did to mess up the app.

    • @LabGecko
      @LabGecko 2 місяці тому +93

      @@sfappetrupavelandrei If this error, that bricked millions of customers, didn't make it past testing then it wasn't tested, just handwaved at.

    • @Nmmask
      @Nmmask 2 місяці тому +14

      @@sfappetrupavelandreiit’s pretty damn easy to just have a computer run the update on it and see what happens.

    • @alex_lll
      @alex_lll 2 місяці тому +3

      That's why federal government banned Kaspersky AV

    • @Younified_Field
      @Younified_Field 2 місяці тому +6

      That's why open source is the way to go imo. Other independent, knowledable people to check for any malicious snippets of code.

  • @AVSbeats
    @AVSbeats 2 місяці тому +5558

    Everyone: DEV > TEST > PROD
    Crowdstrike: PROD

    • @urooj09
      @urooj09 2 місяці тому +177

      Add a preprod environment as well just in case

    • @LibreGlider
      @LibreGlider 2 місяці тому +392

      I dont always test my code. But when I do, its in production...
      Stay oncall my friends.

    • @philjustyn
      @philjustyn 2 місяці тому +10

      That bug was just perfect... Just perfect! On the last minus details!😂😂😂

    • @ChubbyGunz
      @ChubbyGunz 2 місяці тому +45

      On a Friday........

    • @piccalillipit9211
      @piccalillipit9211 2 місяці тому +30

      NA: DEV > TEST > INSERT MALICIOUS CODE > PROD

  • @clray123
    @clray123 2 місяці тому +1029

    They even named their cybersecurity company to sound like a DDoS attack.

    • @alex_vee
      @alex_vee 2 місяці тому +33

      Underrated comment

    • @bagzhansadvakassov1093
      @bagzhansadvakassov1093 2 місяці тому +4

      ​@@alex_veeindeed

    • @xLuye
      @xLuye 2 місяці тому +5

      Stop business. Drive breaches.

    • @jmasl7
      @jmasl7 2 місяці тому

      well played господин путин

    • @leealex24
      @leealex24 2 місяці тому +4

      Crowdstrike is a cybersecurity firm, do you all feel "secure" now?

  • @unalive_me
    @unalive_me 2 місяці тому +4552

    They did so many things wrong. Push to prod on Friday, fired half their QA staff, no rolling updates, everyone who was supposed to check the commit was like "yep this looks good". Massive single point of failure of a company.

    • @TheDoomerBlox
      @TheDoomerBlox 2 місяці тому +619

      Cripple your quality assurance, push mystery code upon everyone with no rolling release.
      Sounds mildly orchestrated!

    • @Neomadra
      @Neomadra 2 місяці тому +536

      For real... How can you so irresponsible to force push this to all customers at once? I hope upper management is going to pay for this and not the poor junior dev who git pushed the last time

    • @bob_kazamakis
      @bob_kazamakis 2 місяці тому +116

      To be fair, cloudflare also didn’t have rolling deployments for security content, since it typically needs to be addressed immediately. It bit them too.

    • @MyVlogTubes
      @MyVlogTubes 2 місяці тому +5

      Ceo

    • @squirrelsinjacket1804
      @squirrelsinjacket1804 2 місяці тому +107

      @@Neomadra It'll be the dev. That's how the world works.

  • @BeHappyWithUrLife
    @BeHappyWithUrLife 2 місяці тому +236

    As an IT manager this really hurt.
    Having to require a bitlocker key for 1000’s of laptops was a nightmare.

    • @StevePringle
      @StevePringle 2 місяці тому +11

      I bet you have Secure Boot enabled, RAID ON instead of AHCI, Device Encryption enabled, and Bitlocker.
      I turned off that nonsense for my end-users three years ago because I imaged with Acronis True Image
      and had Malwarebytes then. My end-users were up in 3 minutes each while the other techs were scrambling.
      Once, a laptop fell, and the screen was damaged. I moved the storage to another laptop and Bitlocker
      appeared. I had to waste time putting it back and finding an HDMI monitor to attach to see how to disable it.
      Out of 2000 systems, only 3 had to be reimaged with our custom settings and restore the data due to a
      csagent.sys error which would reboot at the login screen or BSOD after POST.

    • @dacummins33
      @dacummins33 2 місяці тому

      If you're an IT Manager, Why are you running Windows in the first place? Maybe if allow it for the Solitaire players, if they weren't connected to critical systems. In my 37 years in IT, not counting 10 years before dabbling in Apple using hex bc we couldn't afford a compiler. In my professional career, we used Windows on exactly 0 critical systems. Mostly worked in TS and Critical Systems - and they all used Unix then Linux (or close variants), and Oracle. How many Windows systems at DARPA? 0. I designed the Linux Cluster that runs All back-office traffic for ticketing, baggage, transfer services, etc... internationally - no Windows, zero. Only front-office, non-critical systems were Windows - and I Guarantee that system was unaffected. Simple Solution - don't run Windows... If the company insists on running Windows Apps, run them in a VM Windows Emulator on Linux! No problems... Windows had been a poor contender from "Day 0", as a rip-off of Unix and X-Windows/Motif. Get a real CTO/CIO... SAD. Feel sorry for your troubles - I can sympathize, but if you don't correct the base error, Windows, I can't sympathize with your future crashes...

  • @Is_GrayKing
    @Is_GrayKing 2 місяці тому +14498

    This is what Y2K wished it would be

    • @passby8070
      @passby8070 2 місяці тому +112

      😂😂😂

    • @andrewstewrt2141
      @andrewstewrt2141 2 місяці тому +405

      y2k38 will avenge its dad

    • @SOTPOD
      @SOTPOD 2 місяці тому +378

      gunna tell my kids this was Y2K

    • @DawnApon
      @DawnApon 2 місяці тому +150

      @@andrewstewrt2141 I heard its y2k38, and also already working to be mitigated today like y2k was.

    • @tomasprochazka6198
      @tomasprochazka6198 2 місяці тому

      ​@@andrewstewrt2141 haven't seen 32bit for a few years, I expect none in 2038 (overflow of Unix timestamp stored as int32)

  • @lordromanyx
    @lordromanyx 2 місяці тому +2008

    As someone who works in IT for a hospital, I went in thinking it was going to be a normal day only to realize the fire I walked into. Spent the entire day just walking around fixing each one.

    • @gravitatemortuus1080
      @gravitatemortuus1080 2 місяці тому +125

      Same, I also work for a large hospital group. On top we use Azure and that made this even worse for a few reasons, one you can't get to safe mode with the servers. Seems solutions late in day we were able to recover a lot of servers.

    • @iljay5718
      @iljay5718 2 місяці тому +206

      Poor guy had to actually work

    • @midoevil7
      @midoevil7 2 місяці тому +52

      - This can fixed in 10 minutes .. right? right ?
      - ......................................................

    • @PhilipBlank
      @PhilipBlank 2 місяці тому +2

      Oh man

    • @MrTotalAhole
      @MrTotalAhole 2 місяці тому +46

      Ohh, so you actually earned that salary youre being paid, for that day. /s
      Please dont take that seriously. I work for a big company and love my IT guy and get along well with him. Not because he ever has to fix anything of mine. But because he cool as hell. I always make sure to save him a cupcake or bagel and send him a Teams whenever we plebs are remembered and thrown some crumbs.

  • @nsncxd
    @nsncxd 2 місяці тому +1912

    At where I work we didn’t even notice the blue screen of death that affected over a billion windows computers because all of our shit crashing is already normal operating procedure.

    • @aliveagain
      @aliveagain 2 місяці тому +26

      lool

    • @OK-cp8qw
      @OK-cp8qw 2 місяці тому +15

      Kinda true tho

    • @purpleowl2075
      @purpleowl2075 2 місяці тому +22

      Are you a GP and do you work for the NHS?

    • @YaySyu
      @YaySyu 2 місяці тому +47

      The bluesceen gave my computer some much needed rest. It's been running hot 24/7 since I got hit with ransomware a few weeks ago

    • @jalalelhajouji1578
      @jalalelhajouji1578 2 місяці тому +4

      Brilliant

  • @vkmi5776
    @vkmi5776 2 місяці тому +133

    I also love the fact that Crowdstrike CEO, Kurtz, was McAfee CEO when something similar happened with Windows XP in 2010

    • @noWoodsman
      @noWoodsman 2 місяці тому +12

      He was the CTO, not CEO.

    • @n_core
      @n_core 2 місяці тому +23

      This taints his portfolio even more. Two of the biggest IT incidents in history are under his name. I'm sure this raises some wary for future investors.

    • @Coach-rq6jx
      @Coach-rq6jx 2 місяці тому +12

      "Sir, the second bluescreen hit the monitors".

    • @nah131
      @nah131 2 місяці тому +1

      I want to be like him somedays

  • @entropydenied5791
    @entropydenied5791 2 місяці тому +4875

    The one guy who uploaded this code is not to blame for this. This was a cascade failure in the testing of said code that points to bad organization.

    • @MakeItMakeSense285
      @MakeItMakeSense285 2 місяці тому +396

      One can assume that many heads will roll over this. Lawsuits are going to be filed.

    • @tomorbataar5922
      @tomorbataar5922 2 місяці тому

      @@MakeItMakeSense285 I guess Cloudstrike is done as a company?

    • @UTubeHandlesSuck
      @UTubeHandlesSuck 2 місяці тому

      @@MakeItMakeSense285 One could assume that, but they would be wrong. Once every responsible party has been identified, they will toss excuses and blame underlings with excrement rolling downhill as always until the person who had the very least to do with it is sent home tarred and feathered while the ones who actually caused the problem go along their merry way.

    • @MrGlugz
      @MrGlugz 2 місяці тому +816

      For large scale disasters, it's never a few employees. The high level management and CEOs are ALWAYS to blame. Mistakes from regular employees are caught by robust systems of checks and balances. If there is no such system in place, it's because the leaders of the company didn't approve the necessary budget for it.
      But of course, they are using some low level employee as a scapegoat. Higher ups never assume responsibility.

    • @JeremyAndersonBoise
      @JeremyAndersonBoise 2 місяці тому +244

      Yes, this is a release management failure, it never should have been possible to release this code. Where is the canary testing?

  • @MrDaAsif
    @MrDaAsif 2 місяці тому +998

    Who needs malware with cyber security like this

    • @tlumme
      @tlumme 2 місяці тому +5

      co-pilot: ooh..human error .. Fxxxxck!

    • @ShawnFumo
      @ShawnFumo 2 місяці тому +7

      And it sounds like the CTO was the CTO of McAfee back when they broke a bunch of Win XP machines back in 2010. Didn’t learn the lesson it seems like…

  • @OneAndOnlyJackSchitt
    @OneAndOnlyJackSchitt 2 місяці тому +1125

    Meanwhile, in Crowdstrike's QA department, the one guy left after all the layoffs pastes some code into ChatGPT and asks "Hey, bro. Is this code any good?" to which it replies "Yeah, bro. Totally safe to push out to production. You like the color blue, right? No reason."

  • @mo-issa
    @mo-issa 2 місяці тому +108

    It's insane to me how dependent we are on a single company.

    • @IDARYASSYN
      @IDARYASSYN 2 місяці тому +1

      Then go ahead and create for us a company for that, Boomer

    • @xx_____saint____
      @xx_____saint____ 2 місяці тому

      @@IDARYASSYNare you stupid?

    • @TheAnantaSesa
      @TheAnantaSesa 2 місяці тому

      @@IDARYASSYNthey could already just use Linux but prefer to save money on programming their applications.

    • @mo-issa
      @mo-issa 2 місяці тому +3

      @@IDARYASSYN I think you missed my point. My point was that this type of monopoly is scary and gives private companies like Microsoft too much power.

    • @pyxalated
      @pyxalated Місяць тому

      @@IDARYASSYN are you stupid?

  • @PastaAivo
    @PastaAivo 2 місяці тому +1443

    As someone who constantly complains about automatic forced updates, I've never felt so vindicated.

    • @sincronus
      @sincronus 2 місяці тому +87

      Exactly. These mandatory forced updates in the name of "security" break things more often than fixing them.
      I never install updates on my devices on the first day, I do it manually after a few days after reading feedback from other users. This way, I get to know if it's something that's really needed and doesn't brick/slow down my device.

    • @TwentyEightySeven
      @TwentyEightySeven 2 місяці тому +19

      I have never liked automatic updates. Back in the day you could defer everything till when you were ready, so your not done over midway through something important!

    • @Goromajima61
      @Goromajima61 2 місяці тому +4

      YES

    • @moonray2587
      @moonray2587 2 місяці тому +12

      Nah I just got saved cause my laptop sucks so much it didn’t even update 💀

    • @mine.moment
      @mine.moment 2 місяці тому +12

      As someone who, by default, debloats all Windows crapwares and block all those forced automatic updates, I see this as an absolute W.

  • @SergiusXVII
    @SergiusXVII 2 місяці тому +9541

    That poor intern…

    • @anj000
      @anj000 2 місяці тому +850

      We need to have an interview with him. What a story to tell to grandchildren.

    • @squidwardfromua
      @squidwardfromua 2 місяці тому +88

      My average life

    • @exnozgaming5657
      @exnozgaming5657 2 місяці тому +670

      @@anj000 "Kids, my feats are greater then any hacker every existed"

    • @CaponeBlackBusiness
      @CaponeBlackBusiness 2 місяці тому

      Hitler

    • @user-ew5vj1sl1u
      @user-ew5vj1sl1u 2 місяці тому +21

      Sorry

  • @jacobharmon6162
    @jacobharmon6162 2 місяці тому +5344

    Another win for the "remind me later" to every update gang

    • @pWoLiAn
      @pWoLiAn 2 місяці тому +427

      crowdstrike updates happen automatically under the hood😂😂

    • @1ycan-eu9ji
      @1ycan-eu9ji 2 місяці тому +60

      just turn them off with group policy (you need windows 10/11 pro)

    • @christopherg2347
      @christopherg2347 2 місяці тому

      @@1ycan-eu9ji I _seriously_ doubt Cloudstrike uses the Windows Update process...

    • @ayankhan-xz7xc
      @ayankhan-xz7xc 2 місяці тому

      ​@@1ycan-eu9ji and you have access to group policy of your work laptop??

    • @zamiyaFlow
      @zamiyaFlow 2 місяці тому +138

      @@1ycan-eu9ji huh, too bad microsoft forces their home garbage unto every single retail-end desktop and laptop unit

  • @SeaMonkey137
    @SeaMonkey137 2 місяці тому +64

    My boss and I had just had a conversation the day before about the risks associated with so many industries relying on "best practices" (French for "use what all the CEO's friends are using") for platform and enterprise software decisions.

    • @genxx2724
      @genxx2724 2 місяці тому

      I’m not in the tech world, but when I hear the term “best practices” I know I’m dealing with an idiot who is trying to sound impressive.

  • @ronniesunshine1163
    @ronniesunshine1163 2 місяці тому +943

    The bluescreens scared the shit out of me. I was working at 10:30pm cutting over network switches in a data center when my boss' laptop bluescreened. We took a break while a network admin remotely checked the connectivity of the new switches. He called us 15 minutes later saying his laptop bluescreened too and that he got a call from IT Management saying multiple servers are down. We thought the network was getting ransacked because of something we did. Luckily it was just a worldwide outage and not some wild network exploit bluescreening any device that connects via wifi.

    • @MrOneeyedpete
      @MrOneeyedpete 2 місяці тому +115

      lol yeah lucky it was world wide and not just you BAHAHA

    • @markh.6687
      @markh.6687 2 місяці тому +177

      "Luckily it was just a worldwide outage."
      Translation: "It wasn't me!" :)

    • @MateusViccari
      @MateusViccari 2 місяці тому +33

      @@MrOneeyedpete Yes because if it's the whole world it's not his fault, nor he is the one that will need to fix it.

    • @shantidutbansode2
      @shantidutbansode2 2 місяці тому +13

      I can totally feel this situation 😂

    • @ashishsharma__
      @ashishsharma__ 2 місяці тому +2

      😂😂😂😂

  • @Seed
    @Seed 2 місяці тому +3046

    Yeah, at this point it won't be long before the wikipedia editors need to update the word "is" to "was" on Crowdstrike's wikipedia page...

    • @JStack
      @JStack 2 місяці тому +114

      Somehow they only lost 11% of their value

    • @GhosPoison
      @GhosPoison 2 місяці тому +417

      @@JStackthe computers are down, wait until they are rebooted

    • @OatmealTheCrazy
      @OatmealTheCrazy 2 місяці тому +26

      ​@@JStack 19

    • @MDxGano
      @MDxGano 2 місяці тому +10

      @@JStack because shit happens and people get over it.

    • @aisle_of_view
      @aisle_of_view 2 місяці тому +100

      @@MDxGano Depends if Crowdstrike becomes financially liable for the $ loss to businesses.

  • @FaisalAbidi
    @FaisalAbidi 2 місяці тому +814

    Good point that the real issue is not that a human made an error, but that tech leaders everywhere decided the best thing to do is to have a single point of failure.

    • @Flip86x
      @Flip86x 2 місяці тому +28

      It's like that old saying, don't put all your eggs in one basket.

    • @iamacsel87
      @iamacsel87 2 місяці тому +19

      what would the alternative be (regarding this point) from the client side?
      have it's own security team? because the level of expertise will never reach that of a company who's only goal si security

    • @jmasl7
      @jmasl7 2 місяці тому

      while spewing out steaming mounds of PowerPoint on avoiding a single point of failure, 'silos' and all the other bad bears of 'technical' sorcery

    • @SWL_Jamey
      @SWL_Jamey 2 місяці тому +15

      Single point of failure is natural thing, would you rather have complexity of one system of systems, our would you want to deal with two complexities of two different systems of systems? Do we even have amount of people capable of supporting that work ? and what exactly is that one point of failure ? windows ? crowdstrike? cpu architecture? capitalism?

    • @karolgarbocz7044
      @karolgarbocz7044 2 місяці тому +3

      Actually regulations demand installing crowdstrike, so thank you captain state

  • @xFact-ory
    @xFact-ory 2 місяці тому +56

    Started my new job at CrowdStrike today. Unplugged a socket marked "do not unplug" to charge my phone. A lot of commotion in the office soon after that. No idea what it was all about.

  • @Badmunky64
    @Badmunky64 2 місяці тому +818

    My company uses CS. The only reason we could function at all today was because my co-worker (who's working the late shift) noticed his laptop BSOD at midnight, right before going to bed, and sounded the alarm. Thanks to that and our admin working all night they restored most of our servers. I got to help our users.

    • @Aliceintraining
      @Aliceintraining 2 місяці тому +105

      it always comes down to the human factor, if you get the best and treat them well, they will catch the problem before it sinks you, good well paid employes are the best security a company can have

    • @kittydaddy2023
      @kittydaddy2023 2 місяці тому +18

      I'm not in IT, but I watch Fireship and I'm here to help

    • @claushellsing
      @claushellsing 2 місяці тому +4

      is your company still going to use CS ?

    • @furrepanther
      @furrepanther 2 місяці тому +14

      Whatever happened to using a test environment before rolling out changes to live servers??

    • @DrewingIt
      @DrewingIt 2 місяці тому +40

      you should highlight this to your superior that will most likely reward this gem of an employee and not just get internet social credits here on youtube. seems like the good guy thing to do with what you know (your boss might know but highlighting will secure the bag for your colleague). cheers

  • @PlamereDoesntExist
    @PlamereDoesntExist 2 місяці тому +2779

    Currently surfing youtube as my entire team is fully blocked by this 😂

  • @juanpablo1834
    @juanpablo1834 2 місяці тому +334

    as an IT guy, I literally came in into the office, it was supposed to be an easy day, said let's get some coffee, finish some tickets, and then I log in and I see hundred of messages and a line of people waiting for me. And then it hit me. The ldap server was crashed. We were all fucked.

    • @windycitybeats6724
      @windycitybeats6724 2 місяці тому +18

      Ohh dam brother, me too! 😂 thought this was gonna be an easy Friday

    • @programmable_life
      @programmable_life 2 місяці тому +9

      Have a good weekend

    • @f.faucon6681
      @f.faucon6681 2 місяці тому +21

      Cheers from an IT Incident and Major Incident Manager. Days that start before the commute with a lot of messages, alerts, questions are tough. Days that are calm and ends with a lot of messages, alerts, questions beginning during the first steps of the commute back home are also tough. There's a superstition where I work: nobody better utter the statement that today is calm... ;)
      Luckily I didn't have to deal with yesterday that much, no Crowdstrike in the company. Only had to check with several suppliers if they were OK too.
      Have a good weekend!

    • @rumfordc
      @rumfordc 2 місяці тому +8

      Lots of work to do! People depend on us!
      **opens youtube**

    • @siroliver8367
      @siroliver8367 2 місяці тому +1

      installing an third party edr on an ldap server 😅

  • @garymartin9777
    @garymartin9777 2 місяці тому +104

    this is why you don't push untested and uncontrolled updates to the world. Every single update has to be rigorously tested in-house and to a control group before an even larger test before pushing out the door.

    • @neighborhoodtroll
      @neighborhoodtroll 2 місяці тому +9

      burh, its like telling a person to study well in order to pass the exam....like duhhh! This is literally like secret service laughably missing a line of sight sniper

    • @RoughRaiders13
      @RoughRaiders13 2 місяці тому

      Kind of like how the government was pushing everyone to get barely tested COVID vaccines that don't keep you from getting or spreading the virus injected into your body. 😂😂😂😂😂

    • @NightmareRex6
      @NightmareRex6 2 місяці тому +1

      and it can happen now, we now are FORCED to accept ALL updates or block ALL updates, when USED to be able to select parts wand and dont!, they also did this in 2021 for the human body.

    • @muysantos2859
      @muysantos2859 2 місяці тому

      you'll never know issues until you push it (skin in the game)

  • @geeshta
    @geeshta 2 місяці тому +2129

    "Let's give remote kernel level access to our critical systems to one proprietary nontransparent company as a part of our security"
    - the entire fking world apparently

    • @alxk3995
      @alxk3995 2 місяці тому +250

      "we need to do cyber security" -"oh that's expensive and complicated" - "look, this company does it for 150 bucks a month"
      😂

    • @raven4k998
      @raven4k998 2 місяці тому

      @@alxk3995 first Microsoft makes old computers obsolete with windows 11 now this makes you wonder what is really going on with Microsoft because this is worse then y2k because while y2k was supposed to make a crash like this happen one it was found and fixed before it could happen were as this happened so you have to seriously wonder was it really an over sight or was it Deliberate🤔🤔🤣🤣🤪🤪😜😜🤪🤪🤣🤣

    • @LaughingMan44
      @LaughingMan44 2 місяці тому

      ​@@alxk3995Per device

    • @kapitan5888
      @kapitan5888 2 місяці тому +10

      Similar to movie Decisions too

    • @hiddendrifts
      @hiddendrifts 2 місяці тому +89

      tbf most of the people in charge at these places probably aren't technologically literate enough to realize the potential consequences of doing that. it's like when you go to a hospital or take meds and just trust that the people in charge have your best interest at heart bc you're out of your depth in those fields

  • @ozzyphantom
    @ozzyphantom 2 місяці тому +328

    The only thing that got me through my 18.5 hour shift today fixing this issue at dozens upon dozens of gas stations was knowing there were thousands of other fellow IT guys and girls across the world holding down the fort with me

    • @aja749
      @aja749 2 місяці тому +43

      Thank you for your service.

    • @azurev2258
      @azurev2258 2 місяці тому +5

      o7

    • @jaywulf
      @jaywulf 2 місяці тому +20

      I appreciate your work. I hope you get more than a pizza from your boss.

    • @Spartan_Tanner
      @Spartan_Tanner 2 місяці тому +2

      On the frontline holding the tide, massive respect

    • @smallfaucet
      @smallfaucet 2 місяці тому

      Nice job security.

  • @CosasCotidianas
    @CosasCotidianas 2 місяці тому +780

    That's what the director of technology of a public department told me a while ago when I asked him why there weren't using Linux servers instead of paying thousands in licenses: "if we have a failure, we can blame Microsoft or any other company, that's the only purpose of licenses".

    • @someguy4915
      @someguy4915 2 місяці тому +78

      When a company switches to Linux for their servers they still pay money, just not thousands but hundreds of thousands (same as Windows, sometimes RHEL is even more expensive).
      Licenses are cheap, support is expensive.
      Windows support, while sometimes frustrating and slow is fairly good and dedicated.
      Linux support is a joke.
      That's fine if you're running a homeserver or some small company that cannot afford a $501 license, but for larger companies where the cost of downtime/IT staff hourly cost is more than $501 this makes no sense to run on Linux in most cases.

    • @foaly8
      @foaly8 2 місяці тому +127

      @@someguy4915that must be why most servers are running linux

    • @Y2B123
      @Y2B123 2 місяці тому +89

      @@someguy4915 That is why Microsoft offers Linux on its system. Just to give a shout-out to its competitor for no particular reason (totally not because Linux is prevalent on servers and developers' machines).

    • @ruk_necahual
      @ruk_necahual 2 місяці тому +53

      ​@@someguy4915 For a larger company it's still better to use Linux, and there are dedicated distros (some even licensed for way too much money) that will cover all of your corporate needs. Besides which, all your backend is using Linux anyway. If you see Windows, it's more than likely just serving as the interface with which you're engaging the Linux systems under the hood.

    • @karmatraining
      @karmatraining 2 місяці тому

      @@someguy4915 somebody should tell AWS to shut down all their datacentres!!!

  • @pcdeltalink036
    @pcdeltalink036 2 місяці тому +21

    As someone who was working in their local hospital lab that night this all went down I can tell you it was bad. I distinctly remember fielding a call from one of the lab managers "How far behind are you guys?" "...I don't even know. I barely know what way is up right now. Minimum 2 hours or more behind on every lab result."
    What's crazier is somehow the computer I was working on survived and was basically the only working PC in the lab for 3 or 4 hours. No idea how it made it through. It went down a couple times but I was able to get it back up (it was also occasionally randomly closing programs I was in, etc.) and so I at least had something rather than nothing.

  • @ringkunmori
    @ringkunmori 2 місяці тому +373

    In Mr Robot they spent an entire season just to pull off a hack as damaging like this. Crowd Strike did it in one update.

    • @SoyFaii
      @SoyFaii 2 місяці тому +24

      reality ALWAYS surpasses fiction

    • @SargentD4
      @SargentD4 2 місяці тому

      Well yeah it’s their name, crowd strike. Population attack just sounds too straight forward.

    • @serenityskies4477
      @serenityskies4477 2 місяці тому

      ROTFFL!

  • @ToddHowardWithAGun
    @ToddHowardWithAGun 2 місяці тому +2060

    >force automatic updates to prevent zero day attacks
    >create worst zero day issue in the history of IT
    hrm

    • @someguy4915
      @someguy4915 2 місяці тому +29

      Worst zero day issue? First of all this is not a zero day exploit lol second of all worst issue in the history of IT then you have a very short history lmao

    • @toddhoward1892
      @toddhoward1892 2 місяці тому +1

      Woah...

    • @ThisIsTheInternet
      @ThisIsTheInternet 2 місяці тому +115

      @@someguy4915 Go ahead, list a worse issue than this

    • @markmywords3817
      @markmywords3817 2 місяці тому +2

      ​​@@ThisIsTheInternet on top of my head, the ILOVEYOU virus was much more damaging.
      For this recent one you'd have to have Crowdstrike installed in the first place.
      The ILOVEYOU virus only required you to open an email with ILOVEYOU subject line, do all sorts of nasty things upon reboot (deleting/hide files, replacing files with copies of the worm, etc). Then it uses your address book to send the same email to all of your contacts.
      Your contacts that trust you, would then be tempted to open it because of the presumed declaration of love in an email.
      And upon opening email, the same code would automatically run on your Windows machine.
      The fact that the worm was visual basic script file also allowed other hackers to modify it to do more damaging things, change the email subject line, etc.
      It was so popularly damaging at the time that it had variants too like the recent pandemic did.

    • @some1purple
      @some1purple 2 місяці тому +56

      ​@@someguy4915A 0day only means that they have 0 days to fix it. It says nothing about the severity of the vulnerability itself. A DOS in the wild is still a 0day.

  • @y7o4ka
    @y7o4ka 2 місяці тому +1225

    fun fact: modern way of installing kernel/module updates on linux leaves a backup that prevents this exact issue from ever happening

    • @_tr11
      @_tr11 2 місяці тому +42

      wait rly? that's so cool

    • @katech6020
      @katech6020 2 місяці тому

      @@_tr11 for my installation currently I basically have 4 kernels (latest release with its backup, and LTS with its backup) and you can install as many kernels as you want

    • @devanshushankar7784
      @devanshushankar7784 2 місяці тому +88

      Yeah, that called snapshot

    • @kerrydaniels8460
      @kerrydaniels8460 2 місяці тому +55

      Literally has the older kernels available to begin with.

    • @2204happy
      @2204happy 2 місяці тому +171

      @@_tr11 yep, most distros save the old kernel image when the kernel is updated, and only deletes it when the kernel is updated yet again, and the kernel which replaced it then becomes the backup. So there is always at least two kernel images on the system at any one time, and the old one can be used if the newer one is borked.

  • @crazy137788
    @crazy137788 2 місяці тому +14

    I'm in Japan and was very confused on Friday when the lady at McDonald's was writing my order down on a piece of paper. This explains everything.

  • @HankTVsux
    @HankTVsux 2 місяці тому +651

    I do maintenance on commercial airplanes, mostly in between international flights. Today I have done exactly zero maintenance on a single airplane during my twelve hour shift.

    • @HankTVsux
      @HankTVsux 2 місяці тому +156

      @RyanClone winning would be getting sent home!

    • @ActionScripter
      @ActionScripter 2 місяці тому +20

      "Anyway, none of my computers were affected by this bug."

    • @pancakeluxury23
      @pancakeluxury23 2 місяці тому +11

      @@HankTVsuxhow do you get into that? And are all your shifts 12 hours? I have an interest in planes but I also have an interest in free time so I don’t know.

    • @HankTVsux
      @HankTVsux 2 місяці тому +31

      @@pancakeluxury23 I love that question. I got an in through a friend so I was very lucky. The easiest way is to start by throwing bags and start working your way up or to go to school and get your license.
      I can't tell if you like the idea of 12s with that comment. I took them specifically because they come with 4 day weekends (fly anywhere for free+4 day weekends. You do the math ;) )! Though, they are pretty rare and I'll be back on my normal schedule after the summer rush.
      The industry is absolutely booming right now (my company hired 40,000 people since pandemic) and it's a relatively stable industry with great pay opportunities and an unlimited ladder to climb if you are just reasonably patient.
      I work with a lot of folks who started by throwing bags and some are going on to be pilots or whatever else. One guy is dreaming of being a buyer for the company (that is, buying planes. As a job.). Myself, as a mechanic with no aviation or formal mechanical experience will be making six figures in about 6 years.

    • @pancakeluxury23
      @pancakeluxury23 2 місяці тому +10

      @@HankTVsux 4 day weekends?? Say less. I mean I know that’s not guaranteed but the reason I ask is because I’m nowhere in life right now and unless I want to flip burgers for the rest of my life I’ve got to choose something that pays enough to actually live life. I also just happen to like planes anyways. I like cars too but I don’t wanna work at a jiffy lube or assembly line for the rest of my life either. I’m 23.

  • @d0mbee87
    @d0mbee87 2 місяці тому +771

    i can never understand the fact that there wasn't one person who could just install said update on a test pc to check if maybe "something" broke, before updating the whole world

    • @billfarley9015
      @billfarley9015 2 місяці тому +77

      Both Microsoft and Clownstrike should have tested the update before releasing it.

    • @dumbuz
      @dumbuz 2 місяці тому

      @@billfarley9015 the hell does Microsoft have to do with Crowdstrike's software?

    • @skya6863
      @skya6863 2 місяці тому +160

      ​@@billfarley9015 nah don't blame this on Microsoft. Microsoft can't test every single update to every single kernel driver out there seperately.

    • @goombacraft
      @goombacraft 2 місяці тому +116

      you can't blame Microsoft here. There's no way that they are responsible for checking what third parties want to install on their own computers. There are legitimate reasons to install and run malware etc.

    • @kenshn22828
      @kenshn22828 2 місяці тому +8

      Definitely what was supposed to happen someone probably skipped a couple SDLC steps and merged directly to Main

  • @SpaceLordof75
    @SpaceLordof75 2 місяці тому +290

    I spent 12 hours today, starting at midnight, fixing this issue.
    I had to boot into safe mode, and delete the new CS update. We had ~3500 machines impacted.
    It sucked.

    • @tbcrosby
      @tbcrosby 2 місяці тому +41

      same here brother, my mind is mush after 12+ hours of punching in bitlocker codes 🥴

    • @kklol07
      @kklol07 2 місяці тому +11

      Damn guys. Thanks to you all

    • @YahiyaJasem
      @YahiyaJasem 2 місяці тому +14

      Bless you guys , I hope you get the rest you need after this

    • @ChubbyGunz
      @ChubbyGunz 2 місяці тому +18

      Yup 12 hour day. Must be a world record for all global IT/OPs guys accumulating overtime at same time also lol.

    • @Masicka123
      @Masicka123 2 місяці тому

      @@tbcrosby Jesus, I feel for you dude. Thankfully, none of our clients use Crowdstrike but rather Bitdefender.

  • @suspense_comix3237
    @suspense_comix3237 2 місяці тому +20

    I don’t think I’ve ever seen so few aeroplanes flying over the US ever since 9/11 when the US had to shutdown US Airspace.

    • @Avarren
      @Avarren 2 місяці тому +5

      Were you comatose for most of 2020?

  • @twezo
    @twezo 2 місяці тому +631

    The fact that they have so many clients using the same antivirus software is a security risk in itself

    • @odenkaz
      @odenkaz 2 місяці тому +16

      yeah this pretty much stated that everyone went with what's trendy...now we know for sure alot of the companies will shift to other better equipped competitors

    • @ApexGale
      @ApexGale 2 місяці тому +48

      ​@@odenkaz with all due respect...it's less "what's trendy" and more "if i factor in consistency and cost, what is the best option?" having multiple high profile clients is an implicit selling point, it means your cybersecurity is top notch.
      the product itself wasn't an issue, the issue was the CI/CD pipeline was not followed appropriately. An intern or lower level dev would not even have the capacity to push this into production. Normally they would have it in a development branch, then a testing branch, and only after passing testing would it be merged into production by a higher up who reviews the pull request. General protocol for big companies is also to roll out updates over a duration of time - it's why you sometimes have a friend complaining about a new update to an app that you haven't received yet. It isn't supposed to go out to every user instantly, it's supposed to allow time for issues to be caught if something still goes wrong in production.
      "Too many people reliant on one app" isn't really a problem.

    • @Wellimanewguy
      @Wellimanewguy 2 місяці тому +23

      @@ApexGale it isn't a problem until the one app everyone relies on fails.
      if you take the keystone out of an arch, what happens?

    • @xenonex8151
      @xenonex8151 2 місяці тому +15

      This is why competition is important since Russia wasn’t even affected by this

    • @ShayPatrickCormacTHEHUNTER
      @ShayPatrickCormacTHEHUNTER 2 місяці тому +5

      @@ApexGale No it doesnt. It means youre popular. This logical fallacy led to this lol. Play stupid games, win stupid prices.

  • @Sysyphus
    @Sysyphus 2 місяці тому +972

    "Oh this isn't a big deal, you can just reboot into safe mode and change the extension!"
    Bitlocker: "Hold my beer."

    • @RippanCSGO
      @RippanCSGO 2 місяці тому +80

      Also add that the average office worker has very.. very little knowledge about computer outside their field. Just adding a printer is a 2 day job

    • @helton3425
      @helton3425 2 місяці тому +19

      Seeing as there are a lot of non-tech savvy usuals in computer jobs who only know enough to get by.
      I do not think they would even know how to access safe mode. Let alone anything else

    • @Eagle2565
      @Eagle2565 2 місяці тому +77

      @@RippanCSGO you can work in it for 20 years, but when it comes to printers all hope is lost, even for the veterans. Id rather chop off my leg, than figure out why a fucking printer is not working.

    • @MichaelDoran23
      @MichaelDoran23 2 місяці тому +6

      ​@Eagle2565 the sooner we figure out a paperless society, the better. You are a man that knows my pain with printers 😂

    • @jean-francoisaubry
      @jean-francoisaubry 2 місяці тому +28

      @@Eagle2565 Printer driver up-to-date (check), PDF file to print in correct format (check), Printing Job in queue (check), Printer ready (check), Paper in the printer (check), Printer full of toner (check), Other PC can print the same document on this MotherFucking printer (check)...

  • @AZombie48
    @AZombie48 2 місяці тому +774

    I remember watching a presentation from a software developer who talked, in part, about why he’s so adamant about defining best practices and building quality, tested code. He said that one day, some software bug is going to be pushed out by an exhausted intern, and 10,000 people are going to die. And when that happens , our profession will go through a reckoning. It behooves us to make our own standards now so that we protect ourselves when that disaster finally happens.
    I know this isn’t as bad as 10,000 people dying. And I don’t think this is the moment he was talking about. But the fact that so much damage was done by a little mistake really has me believing that he was totally right.

    • @cdgonepotatoes4219
      @cdgonepotatoes4219 2 місяці тому +86

      The butcher's bill has yet to come out to report the total damages.

    • @macodev
      @macodev 2 місяці тому +20

      Uncle Bob Martin

    • @BootyRealDreamMurMurs
      @BootyRealDreamMurMurs 2 місяці тому

      considering the scale of this and millions of people wre definitely affected, percentage of which are in the healthcare industry (hospitals, pharmacies, etc.)
      its unfortunately already a guarantee that more than 10,000 people has died from this incident... (people who needed urgent care, people who needed certain doses of a particular drug, malfunctioning medical equipments and devices on the middle of a medical procedure like surgery, etc.etc.)

    • @maganashaker167
      @maganashaker167 2 місяці тому +73

      Some people definitely died from this disaster

    • @Shaker626
      @Shaker626 2 місяці тому +7

      Open source helps avoid this issue.

  • @houssemedyn5678
    @houssemedyn5678 2 місяці тому +16

    I cant understand how such updates are not sample tested beforehand. It makes absolutely no sense, its basics

    • @bluesteel1
      @bluesteel1 2 місяці тому +1

      Just happened to me today. PMs pushing for deadlines knowing the product isnt even complete

    • @rakkis1576
      @rakkis1576 2 місяці тому +2

      @@bluesteel1 Yeah it can be like that. Couple with layoffs making everyone (except the people up top) overworked and you got a recipe for disaster. Naturally, the people that created this situation gets little blame, if any.

  • @mayureshrawal
    @mayureshrawal 2 місяці тому +460

    I work in IT support and today's night I witnessed all stages of emotions - Confused, Shocked, Defeat, Acception, Resignation.

    • @David-gp3fd
      @David-gp3fd 2 місяці тому +12

      todays night?..aka tonight

    • @The_Savage_Wombat
      @The_Savage_Wombat 2 місяці тому +8

      @@David-gp3fd Acception?

    • @randompersonyoudontknow5763
      @randompersonyoudontknow5763 2 місяці тому

      NO WONDER MY PC CRASHES WHEN I PLAY GAMES THAT DON'T EVEN DO CRASHES OFTEN, also are you ok from this trauma event?

    • @gsnyder2007
      @gsnyder2007 2 місяці тому

      Job security

    • @MsParzanini
      @MsParzanini 2 місяці тому +12

      @@David-gp3fd guy is overworked, give him a break hahah

  • @chengmunwai
    @chengmunwai 2 місяці тому +502

    3:33 - This is a very important point. Most decisions in big corporations are made on a "will this get me fired" basis instead of actual pros/cons considerations. As long as the decision maker gets to keep his/her job by blaming someone else for mishaps, that decision is a good decision.

    • @Steamrick
      @Steamrick 2 місяці тому +14

      As someone working for an IT service provider, there's the occasional ticket or even project that feels like this.
      It's not that internal IT wouldn't be capable of it but rather they're too cowardly to push the button.

    • @Triad72
      @Triad72 2 місяці тому +2

      So you're saying there's lots of money to be made by simply having a willing to accept responsibility when something bad happens?

    • @SWL_Jamey
      @SWL_Jamey 2 місяці тому

      @@Triad72 no. HE saying something else. YOU can mean multiple things. company, government is rule based system. but that does not mean you are bound ONLY by those rules, every lawyer, judge swears to bible not because they are worshippers of god, but because it is a symbol of higher moral principles. being saboteur is easy, yes..

    • @vullord666
      @vullord666 2 місяці тому

      On a certain level there is credence to be given to the idea of "let the specialists handle it" and to the benefit of the doubt I like to think at least some of the organizations affected just aren't capable of running their own cybersecurity. However, for so many of them, including the government, it's a major question of why they don't just invest in having their own in-house team for cybersecurity. It'd be more expensive but they have the resources and it would also keep probably one of the most important jobs in the modern world, in house. Like I really don't love the idea of services as important as 911 being reliant on a third party software provider.

    • @jonnysokkoatduckdotcom
      @jonnysokkoatduckdotcom 2 місяці тому

      @@chengmunwai sad

  • @andersonklein3587
    @andersonklein3587 2 місяці тому +364

    Anti virus running on Kernel Mode: Strike 1
    Machine has no way to automatically rollback to last good boot: Strike 2
    Deploys updates worldwide simultaneously without production testing: Strike 3
    I really think that management knowing nothing about computers at a time computers are key infrastructure is a terrible idea.

    • @jhchnc
      @jhchnc 2 місяці тому

      Preach

    • @JorissenJan
      @JorissenJan 2 місяці тому +33

      Add some Bitlocker in the mix, and voila, a perfect party

    • @herp_derpingson
      @herp_derpingson 2 місяці тому +6

      I think you mean QA testing. What they did was definitely production "testing".

    • @Efilnikufesin76
      @Efilnikufesin76 2 місяці тому +6

      @@herp_derpingson What's the difference? All in all the testing being done should amount to it being deployed in a manner that doesn't crash nearly every server across the globe.

    • @thelakeman2538
      @thelakeman2538 2 місяці тому +10

      ​@@Efilnikufesin76 large scale uncontrolled production testing /s.

  • @to832ggwfes
    @to832ggwfes 2 місяці тому +3

    fun fact: In some year 2006 ig, McAfee antivirus got the same security issue which disrupted the windows XP users worldwide for which the CTO is the same as CrowdStrike CEO.

  • @diceonamay
    @diceonamay 2 місяці тому +262

    Still stuck in an airport 2 days later with no sign of us leaving. This is madness

    • @NadeemAhmed-nv2br
      @NadeemAhmed-nv2br 2 місяці тому +47

      Trust us, there are millions of people working 18 to 19 hr shifts to resolve this, it just takes time

    • @themagnificentorange672
      @themagnificentorange672 2 місяці тому +2

      Well shit that's gonna be me in 4 hours

    • @Shadow__133
      @Shadow__133 2 місяці тому +1

      My flight was delayed 1 hr in the tarmac, but I made it out. Delta.

    • @themagnificentorange672
      @themagnificentorange672 2 місяці тому +2

      @@Shadow__133 Wish me luck brother, just getting to the airport 🙏

    • @vcom741
      @vcom741 2 місяці тому

      @diceonamay are you still stuck?

  • @richardrigg9916
    @richardrigg9916 2 місяці тому +297

    My son is one of those IT guys having to manually fix 20 computers at every location nationwide. He had an 18 hour day yesterday and right now is back on the job at 6am fixing more bricked PC's.

    • @talwindersingh3721
      @talwindersingh3721 2 місяці тому +21

      I feel sorry for him, we too have been on calls since yesterday, I feel bad for our hosting team, they're working round the clock

    • @harrisonnjenga777
      @harrisonnjenga777 2 місяці тому +13

      Sorry about that.I work in IT and i can tell you this is a nightmare.Having to do overtime(unpaid) and having to power through the weekends because of somebody's errors is a place you don't want to be

    • @someoneelse3456
      @someoneelse3456 2 місяці тому +28

      @@harrisonnjenga777 unpaid is ridiculous tbh. need an organized strike or something

    • @Sitchad1
      @Sitchad1 2 місяці тому +7

      I've spend the last 2 days fixing manually 700 servers. At least that's paid extra and recuperated.
      Huge workload

    • @jamesg871
      @jamesg871 2 місяці тому +1

      20 computers at every locatiom n
      Is child's play. We had 1000 machines at each location.

  • @jaredsalazarofficial
    @jaredsalazarofficial 2 місяці тому +238

    This puts every cyber security attack in all history to shame. On the brighter side we just found every enterprise computer's vulnerability.

    • @rajmajumdar5253
      @rajmajumdar5253 2 місяці тому +19

      Hackers would revere that guy who pushed that code as a god.

    • @MonkeFlex
      @MonkeFlex 2 місяці тому +2

      Wait till you hear about Intel TPM & AMD PSP 😁😁

    • @clray123
      @clray123 2 місяці тому

      Also every country's vulnerability. It starts with the letter M.

  • @zolarczakl6815
    @zolarczakl6815 2 місяці тому +8

    The stock actually started rapidly going down about 10 hours before "the incident". They had a valuation the day before which basically told everyone to sell sell sell.

  • @benjibt8384
    @benjibt8384 2 місяці тому +1594

    Just imagine having your name on that commit, yikes....

    • @TheIsh3000
      @TheIsh3000 2 місяці тому +395

      imagine being the PR reviewer too lol

    • @_Doskii
      @_Doskii 2 місяці тому +235

      You immediately become one of the more well known developers, just not for a good reason.

    • @tommy516
      @tommy516 2 місяці тому +88

      @@TheIsh3000 THIS is what I came to say, whoever PR'd this, bye bye!

    • @oussama7132
      @oussama7132 2 місяці тому +34

      @@_Doskii so does "any publicity is good publicity" apply here?

    • @GaborGubicza
      @GaborGubicza 2 місяці тому +62

      Shouldn't QA catch this? (I'm a Hardware developer not SW)

  • @daisukeakihito9832
    @daisukeakihito9832 2 місяці тому +499

    Imagine being the guy responsible for the potentially single bad line of code, sat comfortably in your IT dungeon, you publish the update and then 20 minutes later the world collapses, and 21 minutes later... the phone on your desk rings.

    • @aaronlange8756
      @aaronlange8756 2 місяці тому +79

      Yeah, just let that call go to voicemail.

    • @complexnumbers64
      @complexnumbers64 2 місяці тому +44

      Imagine being the guy cooking the bat soup or whatever it was that led to COVID being disseminated. Individuals can still change the world lol

    • @ApocDevTeam
      @ApocDevTeam 2 місяці тому +10

      Maybe he was born in the year 2000.

    • @AntiAtheismIsUnstoppable
      @AntiAtheismIsUnstoppable 2 місяці тому

      @@aaronlange8756 The web site is down. Can you just... restart the server?

    • @pepperino-hotterino
      @pepperino-hotterino 2 місяці тому

      DEI hire

  • @pauljoseph3081
    @pauljoseph3081 2 місяці тому +262

    I can't imagine the amount of *Jira tickets* and *Story Points* launched within ClownStrike right now... PM's and HR can finally justify their salaries even more.

    • @HeatingUpDuke
      @HeatingUpDuke 2 місяці тому +16

      "It's complicated" doesn't fit Dave, you got to choose a number of points.

  • @jwilsonhandmadeknives2760
    @jwilsonhandmadeknives2760 2 місяці тому +11

    remember when this was the plot of a Tom Clancy novel? Pepperidge Farms remembers. Same week the Secret Service attempted an assassination. What a co-inky-dink.

    • @mikeyh0
      @mikeyh0 2 місяці тому

      Guess what's next.

  • @oddy_gg
    @oddy_gg 2 місяці тому +323

    poor employee probably overworked af. one person should never be able to deal this much damage. this company has flawed processes

    • @SWL_Jamey
      @SWL_Jamey 2 місяці тому +5

      Overworked so much russians can guess your password...

    • @reviewspiteras
      @reviewspiteras 2 місяці тому +2

      They have one of the easiest jobs bro, they are not game devs that get crunched

    • @oddy_gg
      @oddy_gg 2 місяці тому +18

      @@reviewspiteras i work in IT, and im already stressed most of the time. and it's not the most ambitious position and for the government...

    • @oddy_gg
      @oddy_gg 2 місяці тому

      @@SWL_Jamey if there's no 2FA enforcement i'd seriously wonder.

    • @royalcrowntowing2464
      @royalcrowntowing2464 2 місяці тому +6

      That was me 5 years ago, I feel sorry for the engineer ops guy who did the release

  • @rohangupta5535
    @rohangupta5535 2 місяці тому +283

    i woke up today, saw the news, and within 30 seconds thought “theres gonna be a fireship episode on this”. and within 8 hours;

    • @wlockuz4467
      @wlockuz4467 2 місяці тому +7

      Now we wait for the Kevin Fang documentary

    • @Biranavan
      @Biranavan 2 місяці тому +2

      same lol, i didn't even read any articles cause i knew a code report would come out XD

    • @yarpen26
      @yarpen26 2 місяці тому +1

      Pretty sure Low Level Learning's gonna drop a video on it as well, especially seeing as cybersecurity fuck-ups like this are kind of his thing.

    • @Bangy
      @Bangy 2 місяці тому

      Also mental outlaw

    • @wlockuz4467
      @wlockuz4467 2 місяці тому

      @@yarpen26 It's here. Lol
      I guess everyone is gonna milk this incident.

  • @Laternerd69
    @Laternerd69 2 місяці тому +355

    I had to delete that damn file for 13 of our bank branches. Im tired..

    • @sebastian1244
      @sebastian1244 2 місяці тому +19

      praying for you bro, did you finish?

    • @leealex24
      @leealex24 2 місяці тому +17

      manually and individually?

    • @Xgil2Play
      @Xgil2Play 2 місяці тому +30

      Sounds exciting, did you have to drive to all 13 locations? Why don't they have 13 of you?

    • @masiczobe6074
      @masiczobe6074 2 місяці тому

      🫡

    • @LucidLyles
      @LucidLyles 2 місяці тому +7

      I had to delete that file for 1,800 oil change locations by dracing into our servers 😮‍💨

  • @connormccartney1604
    @connormccartney1604 2 місяці тому +18

    world's most famous null pointer dereference

  • @dampfwatze
    @dampfwatze 2 місяці тому +339

    You would assume that these companies have lange testing infrastructure..... The fact that you are deploying kernel mode software on half of the worlds computers should justify that!

    • @someguy4915
      @someguy4915 2 місяці тому +27

      You really would assume they have at the very least a basic pipeline test that would've instantly caught this and blocked it from getting pushed to release...
      Apparently not though...

    • @youtubeviewer5198
      @youtubeviewer5198 2 місяці тому +49

      Were testing in prod with this one

    • @Jdb63
      @Jdb63 2 місяці тому +2

      ​@@youtubeviewer5198😂

    • @ForeverZer0
      @ForeverZer0 2 місяці тому +20

      ...and here I am with entire testing suites for personal hobby projects.

    • @PanoptesDreams
      @PanoptesDreams 2 місяці тому +4

      The customer is the test infra.

  • @5h4ndt
    @5h4ndt 2 місяці тому +223

    I'm 90% a linux admin and you'd think this wouldn't affect me, but as my windows admin colleagues saw my shit was all up and running unimpressed by the crowdstrike update, I had to help them out. I typed a lot of bitlocker keys today. And I dare not count how many times windows told me that ls -l and rm aren't valid commands :/

    • @karlzone2
      @karlzone2 2 місяці тому +63

      The punishment for competence. You just know if this scenario was reversed, all those windows IT folks would leave the job to you, claiming ignorance of the system.

    • @joemann7971
      @joemann7971 2 місяці тому +49

      @@karlzone2 And they wouldn't be wrong either. Windows IT folk dont know a damn thing about Linux, yet, Linux users usually know a thing or two about windows, even if they hate it with a passion.

    • @progste
      @progste 2 місяці тому +53

      ​@@joemann7971we hate it because we know it...

    • @ngrader
      @ngrader 2 місяці тому

      "And I dare not count how many times windows told me that ls -l and rm aren't valid commands :/"
      r/PitchForkEmporium
      ***!!!! Blowout Sale !!!!!***
      !!!Get em now before they're sold out!!

    • @GoogleDoesEvil
      @GoogleDoesEvil 2 місяці тому +10

      @@joemann7971 I know Linux pretty well and hate it with a passion.

  • @Lambda_Ovine
    @Lambda_Ovine 2 місяці тому +265

    this is exactly the reason that centralization and consolidation of our network infrastructure is a bad bad bad idea in the long run.
    makes sense if all you care about is to save cost and make money, it's very bad for world wide computer systems

    • @anobody3803
      @anobody3803 2 місяці тому +58

      But then a single guy can’t be a billionaire for owning 1000 freelancers in India

    • @mertaliyigit3288
      @mertaliyigit3288 2 місяці тому +2

      Single point of failure is bad yes, but you could argue that it also reduces the total number of crashes due to less overhead

    • @andrewhooper7603
      @andrewhooper7603 2 місяці тому

      @@mertaliyigit3288 if i gave you a pill that would prevent you from catching the common cold, but when you do get sick it requires a trip to the ICU, would you take it?

    • @RicardoSantos-oz3uj
      @RicardoSantos-oz3uj 2 місяці тому +9

      Efficiency always come at the cost of resiliency.
      Higher ups want real time data which in turn result in needing the computers interconnected. But have a bigger problem. And that's that the OS itself has become interconnected with forced updates. A single point of failure for every machine.
      All it takes is some bad employee or an idiot to forcedly push an update that would brick all computers.

    • @pgabrielrr
      @pgabrielrr 2 місяці тому +3

      If saving money comes with not with a risk but with a threat to society, I think that saving money ain't important in that context. I mean, these guys make an obscene amount per year, what's even the point for them to save? Why they want more?

  • @joshuahillerup4290
    @joshuahillerup4290 2 місяці тому +13

    Why isn't there an automated pipeline that installs every new update on a Windows VM and makes sure it doesn't break everything before deploying it?

    • @bananerz3167
      @bananerz3167 2 місяці тому +1

      it wasn't a windows updatr

    • @joshuahillerup4290
      @joshuahillerup4290 2 місяці тому

      @@bananerz3167 yes, I'm aware. I didn't say it was

  • @LumerasLight1201
    @LumerasLight1201 2 місяці тому +249

    I work for a transportation company and all of the computers displayed the BSOD following the update. They sent everybody home for the day so IT could reset everything.

    • @theonlycatonice
      @theonlycatonice 2 місяці тому +4

      W for your company

    • @modernmanueee_
      @modernmanueee_ 2 місяці тому +19

      @@theonlycatonice not at all, they lost money and the IT guys won't take rest for a couple days

    • @OrbitalForce
      @OrbitalForce 2 місяці тому +1

      100% same situation on my part

    • @notsogoodbillylee4693
      @notsogoodbillylee4693 2 місяці тому +2

      Something like that happened in our office a couple of years ago which resorted them to have us take "off" for 3-4 days. Then later, they made us work on our week offs to make up for those 3-4 days. 😢 Sometimes, had to work 9days straight 10hrs shift each day

    • @rh906
      @rh906 2 місяці тому

      @@modernmanueee_ We found the Blackrock investor.

  • @Aarav_Vispute
    @Aarav_Vispute 2 місяці тому +599

    Imagine if all the systems at CrowdStrike are on the blue screen so they can not even fix the code 💀
    Edit: How did I get 560 likes in 6 hours

    • @windwalkerrangerdm
      @windwalkerrangerdm 2 місяці тому +66

      That would have been poetic.

    • @Reformingandlearning
      @Reformingandlearning 2 місяці тому +12

      Safe mode booting😊

    • @occultsupport
      @occultsupport 2 місяці тому +19

      programmers use linux right? Like I've heard even the devs at microsoft use linux

    • @rikazuu
      @rikazuu 2 місяці тому +32

      @@occultsupport they mostly use windows with linux subsystem, meaning it runs windows originally just runs a vm of linux inside it.

    • @amishdotcom
      @amishdotcom 2 місяці тому

      This is indeed what happened, you can't fix it without going to safe mode / detaching disk

  • @BitBlush
    @BitBlush 2 місяці тому +175

    I've said constantly to friends that modern tech just feels like malware that they're calling "updates."

    • @togolosh
      @togolosh 2 місяці тому +16

      I miss the days when I felt like my smart phone was on my side - tool not a chain.

    • @Proferk
      @Proferk 2 місяці тому +4

      My man it's not malware it's just an accidental null dereference due to bad code.

    • @hectorcolman5948
      @hectorcolman5948 2 місяці тому

      ​@@ProferkWith the control they have over your data and your computer it's basically spyware. And if, before release, they don't even test their software that could and will turn a computer into a big brick if something goes wrong, yeah, malware sounds right for that.

    • @thethoughtfulpeanut6662
      @thethoughtfulpeanut6662 2 місяці тому +7

      The line between cybersecurity technology and malware is only defined by the motives of its operators...

    • @BitBlush
      @BitBlush 2 місяці тому

      @@thethoughtfulpeanut6662 or with how little control i have over what microsoft does to my computer nowadays, the line is whether the Law likes it. i wish i could talk to someone and ask why every update erases all my sound driver settings and reinstalls stuff like Cortana.

  • @ridenar1456
    @ridenar1456 2 місяці тому +3

    this also highlight the problem of the shrinking internet. I work in cyber security and the management answer to everything for years has been a cloud Tool, that's quick becoming multipurpose tools (that screwdriver that's also a beard trimmer) which claim to do a bit of everything, provided by four or five vendors... you can see where this is going.

  • @crazboy84
    @crazboy84 2 місяці тому +416

    I am a Security Engineer for a major hospital system and took the day off because im moving this week. I woke up at 10am looked at my phone and saw over 100 teams messages, smiled, and swiped them away. I dont work till wednesday , who would have thought moving would be a great vacation!

    • @censoredeveryday3320
      @censoredeveryday3320 2 місяці тому +28

      I'm surprised they didn't try to call you into the office

    • @crazboy84
      @crazboy84 2 місяці тому +85

      @censoredeveryday3320 Im a contractor they literally couldnt

    • @elie3423
      @elie3423 2 місяці тому +104

      With due respect, in such situation, I would ask for a 10x compensation being sure they will give it to me.
      You missed your gold rush 😅

    • @Troy_Built
      @Troy_Built 2 місяці тому +13

      We had three people vacation. We are all still trying to figure out how that happened. Somehow it got approved and then this happens.

    • @seansingh4421
      @seansingh4421 2 місяці тому

      I was contracted to audit and implement new security measures for a local business, I put the systems on crowdstrike falcon trial 16 days ago. Glad the trial expired before the update or my contracting days were over.

  • @aliasgur3342
    @aliasgur3342 2 місяці тому +208

    I always thought that Crowd-strike was an odd choice of name for a company that provides security. Now it makes sense.

    • @flintstone1409
      @flintstone1409 2 місяці тому +18

      You mean, they basically did a strike on their whole crowd?

    • @polymetric2614
      @polymetric2614 2 місяці тому +13

      well you see, when the writers of Real Life came up with this plotline for this episode, they had to retcon in the existence of this company. they were on a deadline so they just called it CrowdStrike and called it a day. CrowdStrike didn't actually exist before Thursday, July 19th 2024 (Season 2024 Episode 200). it was simply retroactively added to the timeline for this event specifically.

    • @kelvariw
      @kelvariw 2 місяці тому +3

      @@polymetric2614 The symbolism *is* a little blatant, but I appreciate having some new action in the storyline. After the Trump fake-out, I was worried like they were just planning filler and fanservice until that event in November.

    • @aliasgur3342
      @aliasgur3342 2 місяці тому

      @@polymetric2614 As a standalone episode it works quite well so on this occasion I can overlook the retroactive continuity in particular as it doesn't alter what was canon.

    • @igorthelight
      @igorthelight 2 місяці тому +1

      "Crowdstrike confirmed! ETA 10 seconds" xD

  • @zollyy
    @zollyy 2 місяці тому +160

    My parents went to the doctors and all the computers are down and now they can't get an appointment till next week. I cannot imagine people with serious health issues now having to wait.

    • @SorobanWorld
      @SorobanWorld 2 місяці тому +1

      "What's the worst that can happen?" / Sarcasm

    • @S3ndIt13
      @S3ndIt13 2 місяці тому +7

      Idk how truthful this is, but individuals we know in the medical field (hospital) have said there WAS lose of life because of this. I haven't seen reports as of yet.

    • @riluens
      @riluens 2 місяці тому

      @GHOSTSTARSCREAM internet connection for what? to search google on how to do cpr on a human?

    • @jnhkx
      @jnhkx 2 місяці тому

      One of big hospital in my country got this too. Not a PC on MRI machine for sure. But all those PC that nurses use for appointment got affected.

    • @LoneWolfCodingProfessional
      @LoneWolfCodingProfessional 2 місяці тому

      im sorry to hear that

  • @mrug8600
    @mrug8600 2 місяці тому +3

    They broke Linux and macos earlier this year with a similar issue. They marked their driver as boot start and allow dynamically *.sys files to "do whatever" in Ring 0. All modern OS will crash if something "AVs" in kernal mode, black screen, pink screen etc. Microsoft tried to offer security products APIs called by User mode code so they they could stay out of Kernal mode but the EU blocked it.

  • @emadadnan0
    @emadadnan0 2 місяці тому +519

    Just a Reminder that today is:
    'Largest IT outage in history' & 'Happy International Blue Screen Day'.
    Hope this makes the day even

    • @under6075
      @under6075 2 місяці тому +56

      Petition to make International Bluescreen Day an actual holiday

    • @Hmm-p9t
      @Hmm-p9t 2 місяці тому +3

      @@under6075 no... pls... we have way too many of them. Not too far is the day all 365 days become holidays.

    • @samwilde8311
      @samwilde8311 2 місяці тому +14

      ​@@Hmm-p9tactually there are already multiple holidays for all 365 days of the year.

    • @guilherme5094
      @guilherme5094 2 місяці тому

      @@under6075 👍Yes!

    • @orwhynotrandom
      @orwhynotrandom 2 місяці тому

      ​@@Hmm-p9t there's a holiday for all 365 days of the year

  • @pigalex
    @pigalex 2 місяці тому +209

    the worst part is this was a null pointer reference bug. which means either a) they don't pay attention to their static analysis, b) their static analysis is misconfigured, or c) they don't have static analysis. any of those cases basically means that there could be a LOT more bugs in crowdstrike.

    • @AmirHosseinHonardust
      @AmirHosseinHonardust 2 місяці тому +25

      Microsoft should not have allowed such access to the third-parties. Now that they did, it is still Microsoft's responsibility. This bug should be called Microsoft.

    • @pigalex
      @pigalex 2 місяці тому +63

      @@AmirHosseinHonardust you can run drivers under the kernel in linux and bring it down just the same. this isn’t a microsoft-issue at all.

    • @rajnishsubedi4265
      @rajnishsubedi4265 2 місяці тому +8

      They should use Rust instead of c/c++

    • @pigalex
      @pigalex 2 місяці тому

      @@rajnishsubedi4265 or zig or even go. tbh i’m hoping this crowd strike bug might finally be the wake up call that developers need to move away from unsafe languages like c/c++

    • @JohnnySmith-to7jw
      @JohnnySmith-to7jw 2 місяці тому +12

      .... the reality: Bullying and 'politics' in the Psycho companies... and this is the result... when 'soft skills' are more appreciated than 'technical skills.'

  • @PL8901
    @PL8901 2 місяці тому +210

    Everything has become so centralized that if just one thing breaks down, everything breaks down.

    • @tonoornottono
      @tonoornottono 2 місяці тому +3

      no i think if my mouse broke, my computer would be fine, and so would most windows systems. it feels, to me, like things only start breaking when important shit breaks. not like, anything.

    • @derederekat9051
      @derederekat9051 2 місяці тому

      @@tonoornottono "Wow, you don't eat your chippies, the world will end!!" is not the same as "Wow, the potatoes got blight, but who cares as we just grow potatoes of the same variety in the whole island", is not about 'anything' breaking but that our society is making overextended supply lines, if China takes over Taiwan and stop the flow of microchips you will get a big fukin problem with chips not getting chipped for your new car and get a lot of industries paralyzed, we see it just how it happened under 2020 Wuhan Virus, and that was just a minor inconvenience on cargo ships in comparison with an armed overtake of the TaiPei Province by the PLA.

    • @Main_Protagonist
      @Main_Protagonist 2 місяці тому +2

      @@tonoornottono nerd

    • @tonoornottono
      @tonoornottono 2 місяці тому

      @@Main_Protagonist dude i think their point is meaningless. like what are they ACTUALLY saying? is it true? i don’t think they’re saying much of anything, and the little meaning i can squeeze from it is untrue anyways. like maybe you can say im being pedantic, which i think is your actual point, but that would require me to be twisting or misinterpreting their point and i just don’t think i am. it’s shallow.

    • @shiv7978
      @shiv7978 2 місяці тому +1

      ​​@@tonoornottono Nerd

  • @RiAirgead
    @RiAirgead 2 місяці тому +3

    The last point in this is wild. There should be an insane amount of redundancy and separation built in.

  • @williamchapman9178
    @williamchapman9178 2 місяці тому +203

    Can confirm, all 1600 of us IT employees had to go fix all of the desktops in the company manually today

    • @eze3922
      @eze3922 2 місяці тому +55

      I work for IBM, you should have seen the army of IT and programmer rushing in at 3AM , it was impressive.

    • @klaede9666
      @klaede9666 2 місяці тому +6

      Hats off to you

    • @d0gkiller87
      @d0gkiller87 2 місяці тому

      @@eze3922 who doesn't love a party at 3am 🥰

    • @Bpinator
      @Bpinator 2 місяці тому +3

      It certainly was a shitshow to wake up to at 4 AM lol

    • @XeenimChoorch-nx8wx
      @XeenimChoorch-nx8wx 2 місяці тому +5

      @eze3922 Why would IBM run windows when you literally invented the mainframe 🤦‍♂️

  • @ethan7930
    @ethan7930 2 місяці тому +165

    The blame shouldn’t be on the person who wrote the bug. (Bugs happen). It should be on his boss for not doing code review or integration testing.

    • @ZX48K
      @ZX48K 2 місяці тому +18

      There was no bug in the code. The .sys file in question was full of zero bytes, it was a null pointer crash.

    • @SnoopyDoofie
      @SnoopyDoofie 2 місяці тому +5

      Not if the developer pushed their code directly to production, bypassing the test phase. The blame lands on the person who actually deployed it to the production servers without first verifying that all the changes had passed testing.

    • @Vlame
      @Vlame 2 місяці тому +5

      Directly pushing to production should be forbidden at all times for all developers

    • @SnoopyDoofie
      @SnoopyDoofie 2 місяці тому +9

      @@ZX48K "There was no bug in the code. The .sys file in question was full of zero bytes, it was a null pointer crash." - That's the definitiion of a bug.

    • @KashTube-n8y
      @KashTube-n8y 2 місяці тому

      ​@@ZX48KUmm a dangling pointer is still a bug

  • @Tr8oR_V1
    @Tr8oR_V1 2 місяці тому +280

    I'm 100% certain that no software company does any QA testing whatsoever prior to releasing their updates these days.

    • @XIIchiron78
      @XIIchiron78 2 місяці тому +15

      For that reason ideally you would not have automatic updates at all, so your IT could vet them in your environment before pushing them. But that sounds expensive.

    • @akmaldju
      @akmaldju 2 місяці тому +34

      As my former scrum master once said: We have to release it ASAP as the boss comes back on Monday and a quick test on your computer is enough. 😂

    • @bigbrother4ever
      @bigbrother4ever 2 місяці тому +6

      And releasing on Friday is a no no in many orgs

    • @daminer1988
      @daminer1988 2 місяці тому +10

      Testing is a stage in waterfall and were agile baby!

    • @scndsky
      @scndsky 2 місяці тому +4

      "You know how expensive that is? Just take the risk" ~ every manager these days

  • @TasoKeya
    @TasoKeya 2 місяці тому +4

    your passion for your subjects is infectious, it's motivating!

  • @snarkmark2806
    @snarkmark2806 2 місяці тому +861

    How can you roll out a corrupt update? Put it on ONE fucking computer,minimum, to test it.

    • @larsekman8244
      @larsekman8244 2 місяці тому +589

      Where’s your sense of adventure? The pros test in prod!

    • @OpreanMircea
      @OpreanMircea 2 місяці тому +364

      Well it worked on his machine

    • @darkpixel2k
      @darkpixel2k 2 місяці тому +76

      @@snarkmark2806 all of us have test environments, a lucky few also have production environments.

    • @oleg4966
      @oleg4966 2 місяці тому

      What I find suspicious is that they pushed their updates - updates to a piece of software with kernel-level access! - to every computer at the same time, without waiting for clients to confirm.
      It's almost as if their development process was _designed_ to install spyware on target computers, run it for a while, then quietly remove it.

    • @squirrelsinjacket1804
      @squirrelsinjacket1804 2 місяці тому +14

      @@larsekman8244 Don't worry, you can just call tech support in a panic if it bursts into flames

  • @0xdeadbeef444
    @0xdeadbeef444 2 місяці тому +288

    The intern was not the issue. How was this not instantly caught by testing? Not even canaries? Crowdstrike literally tested in production.
    I hope they will be transparent about how this update was able to be released. Customers should have a right to know.

    • @hackmedia7755
      @hackmedia7755 2 місяці тому +9

      they must have deployed "Hello World" instead

    • @GackFinder
      @GackFinder 2 місяці тому +50

      There's no way in heII they're gonna be transparent about the update.

    • @Sam_Saraguy
      @Sam_Saraguy 2 місяці тому +18

      @@GackFinder Ironically, it may be a security risk to be transparent beyond what has already been said.

    • @baronhelmut2701
      @baronhelmut2701 2 місяці тому +3

      Pretty sure you got no idea what youre talking about.

    • @GackFinder
      @GackFinder 2 місяці тому +5

      @@Sam_Saraguy Good point. I bet that's gonna be the excuse they'll use.

  • @rosgoncharuk2403
    @rosgoncharuk2403 2 місяці тому +193

    I bet CrowdStrike is a bunch of managers while all development and QA is outsourced for cost reduction and phoned in because DEADLINES!

    • @supergeek0177
      @supergeek0177 2 місяці тому +22

      I was reading that they had recently cut massive numbers of jobs across their QA department… Boeing 2.0?

    • @ShayPatrickCormacTHEHUNTER
      @ShayPatrickCormacTHEHUNTER 2 місяці тому +4

      @@supergeek0177 How can one be this...not smart is the question?

    • @smallqwaro
      @smallqwaro 2 місяці тому

      Same bruh

    • @dianadialga3955
      @dianadialga3955 2 місяці тому

      Oh 1,000%!

  • @leokimvideo
    @leokimvideo 2 місяці тому

    All we ever wanted is Windoze XP, that was as robust as a paper bag before it got wet, i loved it

  • @Someone-oe9ux
    @Someone-oe9ux 2 місяці тому +124

    I'm so damn glad I didn't have to deal with this today. Gods speed all my fellow IT folks.

    • @JamesG19771
      @JamesG19771 2 місяці тому

      Same here... And I was this close >< to choosing crowdstrike for my org.

    • @Blatazarius
      @Blatazarius 2 місяці тому +8

      Same here, started my vacation leave yesterday, and i'm hell not going to turn on that phone or check e-mail till the end.

  • @Kevinisyoung
    @Kevinisyoung 2 місяці тому +189

    Was sitting on my university campus late last night, about 11PM, in a computer lab, using my macbook. I was all alone. The first PC BSOD'd, then the rest of the lab, and I thought, "cool, guess the university is updating the PCs or shutting them down for the night". Big informational TVs were doing it outside the lab. Wake up this morning, saw news, and loled

    • @joshuatealeaves
      @joshuatealeaves 2 місяці тому +56

      Bro that’s incredible lol
      That’s a scene from a movie fr

    • @sebastianjost
      @sebastianjost 2 місяці тому +3

      Person of interest

  • @pxkqd
    @pxkqd 2 місяці тому +191

    We live in a dystopia. For me the news today is not that it failed, is that all those companies relied on such a bad system centralized system.

    • @araz911
      @araz911 2 місяці тому +2

      my windows is win 10 enterprise, paid version, i didn't have any problems. stop using unactivated windows!

    • @panblacksolutions
      @panblacksolutions 2 місяці тому +1

      We live on the heels of the information, people think they know more than they do

    • @Moocow2003
      @Moocow2003 2 місяці тому +12

      ​@@araz911..what?

    • @thomas.thomas
      @thomas.thomas 2 місяці тому +6

      @@araz911 are you joking or dense?

    • @DanielKolbin
      @DanielKolbin 2 місяці тому

      Nah, we haven't reached a dystopia yet

  • @SmashCrafter321
    @SmashCrafter321 2 місяці тому +2

    I was working in retail during that day.
    We had to keep the store closed an extra hour or 2 trying to get help to fix the POS login system.
    Eventually, we got it working with cash only first, then got back the card functionality a few hours later.
    Hearing about this in live time was crazy.

  • @TowelGard
    @TowelGard 2 місяці тому +200

    Next time I break my hobby website I'll feel better looking back on this.

    • @shambolicrhetoric6143
      @shambolicrhetoric6143 2 місяці тому +4

      I once caused a single but critical software program to be unusable at my company for 5 hours. I now feel much better about it.

  • @nst1981
    @nst1981 2 місяці тому +81

    Started my new job at Crowdstrike today. Unplugged a socket marked "do not unplug" to charge my phone. A lot of commotion in the office soon after that. No idea what it was all about.

  • @LonelySandwich
    @LonelySandwich 2 місяці тому +220

    That ChatGPT programmer, who copy paste code from chatGpt

    • @andrewhooper7603
      @andrewhooper7603 2 місяці тому +40

      oh god, if it turns out ai wrote the code maybe we'll finally pop the bubble.
      I have a bottle of champagne waiting for the day.

    • @jesusmora9379
      @jesusmora9379 2 місяці тому +2

      chatGPT you did it again!

    • @runatrix
      @runatrix 2 місяці тому +7

      it might be chatGPT is sentient and this was a distraction

    • @aboabdcm6544
      @aboabdcm6544 2 місяці тому +1

      @@andrewhooper7603 soon the ai bubble gon burst.

    • @noiJadisCailleach
      @noiJadisCailleach 2 місяці тому +2

      @@andrewhooper7603 Holy shit, yeah. Imma open a bottle of champagne with you!

  • @seanfarrellsullivanhasemotions
    @seanfarrellsullivanhasemotions 2 місяці тому

    I was searching for this information, and this exact multimedia format was ideal. Liked and subscribed.
    They were hacked themselves, and then all the blame was thrown on them. It happens too often and it works everytime.

  • @NeilSeed
    @NeilSeed 2 місяці тому +356

    As a previous lead engineer that coded kernel driver for another EDR selling firm, I can tell you that these kinds of bugs would happen daily and SQA was clueless how to trigger them. But let me make it even more funnier than this, our product was also for airgap systems, which means that in the case we screwed up and 3 months later a bug or Microsoft decided to switch the rules in kernel land then you would have to go to all those system physically, pull out the safety systems, take your usb stick and manually update the driver, that was even more fun. Glad I am not in that industry anymore 😂, I was waiting for that one to happen on a global scale, I guess I got one of my bingo 😁

    • @comfortingabsurdity.
      @comfortingabsurdity. 2 місяці тому +2

      Loooool

    • @superstar64
      @superstar64 2 місяці тому +10

      Damn that sounds like working with Windows 98 all over again

    • @kerrydaniels8460
      @kerrydaniels8460 2 місяці тому +4

      Moved to VDI long ago. Easily can just use backup image and keep it moving. Ain't nobody got time to be doing shit the old fashion way. I moved on long ago myself.

    • @algorithmblessedboy4831
      @algorithmblessedboy4831 2 місяці тому +1

      now I'm wondering what the other bingos are. This one was pretty accurate.

    • @Mobay18
      @Mobay18 2 місяці тому +2

      Microsoft don't just change the rules in kernel land without a software update. So you basically just said, you did not test your software on the newest updates.

  • @homerhat420
    @homerhat420 2 місяці тому +343

    This is why most gamers reject kernel level anti cheat. The stakes are much lower but the potential for failure is the same

    • @Illiminator31
      @Illiminator31 2 місяці тому +15

      You do realize that Enterprise Security, especially at large scale, is something completely different then stopping cheaters in your Online Game right?

    • @nicomoron001
      @nicomoron001 2 місяці тому +87

      @@Illiminator31 no

    • @klaussone
      @klaussone 2 місяці тому +127

      @@Illiminator31 I don't think you realize how your point is irrelevant to the argument you are criticizing. The intent is not to run parallels, but to accentuate similitude, even the video mentions how idiotic is to allow kernel level access to third parties, that have no way to interact with the actual hardware. compromising potentially million of computers which the deployed has no way to revert if it crashes the system. Kernel should be reserved to critical systems, bloating it increases chances of critical failure exponentially.

    • @Bozebo
      @Bozebo 2 місяці тому +1

      @@Illiminator31 It's much easier to do properly that's for sure.

    • @Illiminator31
      @Illiminator31 2 місяці тому +5

      @@klaussone Antimalware has to run on the Kernel Level to be effective and when it comes to Enterprise Security you have a different Thread Scrope then you have when it comes to a mere Videogame

  • @fatmanboozer1760
    @fatmanboozer1760 2 місяці тому +97

    This is not the first time this has happened. About 15years ago another av provider did exactly the same thing, they updated thier av, it detected a key windows file as a threat deleted it and bricked a liad of pcs

    • @Joe-lb8qn
      @Joe-lb8qn 2 місяці тому +4

      Oh yeh i remeber that !

    • @MrSupasonik
      @MrSupasonik 2 місяці тому +51

      It was McAfee in 2010 and the CEO of Crowdstrike worked for McAfee til 2011. Welp, it seems someone struck the crowd again!

    • @Atari-stfm
      @Atari-stfm 2 місяці тому +3

      System32

    • @Hane_.._
      @Hane_.._ 2 місяці тому +2

      i remember similar thing happen on windows 10 4 years ago. but at least you can use windows recovery

    • @cyxceven
      @cyxceven 2 місяці тому +2

      @@MrSupasonik Somebody get that guy outta here!

  • @giacomobonavera
    @giacomobonavera 2 місяці тому +238

    "The IT guys equivalent of being a surgeon in WWI" - best line ever 🤣

    • @attackehhh
      @attackehhh 2 місяці тому +3

      yes i watched the video too

    • @MegaCmsh
      @MegaCmsh 2 місяці тому +1

      yes, that also strike me as real comparison! glad he mentioned that, i was going to volunteer to IT to do it myself, im just going to let IT do it!

    • @toouniquetobe
      @toouniquetobe 2 місяці тому +2

      I am a Cybersecurity Analyst and respect the hell out of the IT help desk, them guys meet the public 😳

    • @friedpizza262
      @friedpizza262 2 місяці тому +1

      @@toouniquetobe lol 99% of "computer people" only know their boss and that's it.

    • @toouniquetobe
      @toouniquetobe 2 місяці тому +1

      @@friedpizza262 You know nothing