How I survived a DDoS attack

Поділитися
Вставка
  • Опубліковано 9 чер 2024
  • After posting a video about running my website off a remote Pi cluster, I was hit with three DDoS attacks. This video explains what I did to make sure my website stayed online, and how it relates to Russia and Ukraine.
    Support me on Patreon: / geerlingguy
    Sponsor me on GitHub: github.com/sponsors/geerlingguy
    Merch: redshirtjeff.com
    2nd Channel: / geerlingengineering
    #DDoS #DarkWeb #Ukraine
    Contents:
    00:00 - DDoS - February 9th, 10:41 AM
    01:20 - How it happened
    03:11 - First mitigations
    05:35 - Stable but slightly broken
    07:56 - 2nd and 3rd DDoS attacks
    08:55 - Preventing DDoS attacks
    11:05 - Russia's invasion and UA Cyber SHIELD
    12:20 - DDoS attacks are here to stay
  • Наука та технологія

КОМЕНТАРІ • 1 тис.

  • @JeffGeerling
    @JeffGeerling  2 роки тому +680

    6:49 - It was DNS.
    RedShirtJeff.com

    • @baylinkdashyt
      @baylinkdashyt 2 роки тому +34

      It's always DNS.
      Or Lupus.

    • @declanmcardle
      @declanmcardle 2 роки тому +7

      It was vegans...

    • @TheJacklwilliams
      @TheJacklwilliams 2 роки тому +2

      Thanks for the post Jeff. Full of great information and I'm glad you were able to battle back against it. Per your point re centralization vs de-centralization, agreed wholeheartedly. Now, a challenge to you because well, I'm an insanely huge fan of what you do and how you do it. If you, or I, or another fan(s) were to NOT cloudfare in a case like this, what could be done to stop the attacks? The biggest issue I see that you called out, is the average home user of bandwidth is going to be doing Spectrum or AT&T, and well, pay for bandwidth...
      Thanks again, as always, great post, highly informative. Glad you buttoned it up.

    • @GrandPlatClips
      @GrandPlatClips 2 роки тому +9

      You Leaked your IP Address 4:48

    • @daveamies5031
      @daveamies5031 2 роки тому +6

      @@GrandPlatClips That was his previous IP before dhcp renewed 🤣🤣🤣
      Pretty sure he mentioned he had a static ip in a previous episode.

  • @ghangj
    @ghangj 2 роки тому +1023

    Currently on the Cyber Security road and it is amazing how much I have learned from this video, "DOCUMENT EVERYTHING".

    • @RoelBaardman
      @RoelBaardman 2 роки тому +60

      Not from security experience, but general network-admin experience:
      Don't just document what went wrong... also document what went right! This reveals positive patterns, shows improvements and (perhaps most important in a company) documents why the expensive tools are worth it.

    • @ghangj
      @ghangj 2 роки тому +25

      @@RoelBaardman Thanks for the tip * *scribbles something* *

    • @vaisakhkm783
      @vaisakhkm783 2 роки тому +5

      I also learned the same lesson a few days ago..
      I never going to miss anything

    • @HoloScope
      @HoloScope 2 роки тому +3

      @@RoelBaardman this!

    • @danielstellmon5330
      @danielstellmon5330 2 роки тому +14

      Document what went wrong for you. Document what went right for the boss.

  • @marcogenovesi8570
    @marcogenovesi8570 2 роки тому +734

    the big question is why would anybody waste all those resources to ddos a content creator personal website. Tell us the truth, you have been working with secret stuff and these are the other secret agents coming at you

    • @JeffGeerling
      @JeffGeerling  2 роки тому +310

      I'm going to have to ask Red Shirt Jeff what shenanigans he's been up to...

    • @kalam564
      @kalam564 2 роки тому +138

      @@JeffGeerling We've traced the DDoS and it's coming from inside the house. (insert suspenseful music)

    • @JTSabre
      @JTSabre 2 роки тому +35

      Probably an attack against the VPS hosts range of IPs, rather than a direct attack on the single site.

    • @PBRichfield
      @PBRichfield 2 роки тому +5

      @@JasonWade plausible false flag attack ttp

    • @JWSmythe
      @JWSmythe 2 роки тому +12

      It's most likely not a government. It's some script kiddie with his botnet, trying to impress his friends. That happens a lot, most people just don't see many of them. For botnets, it isn't a lot of "resources". The resources are the infected or exploited machines scattered all over the world.
      If he had done more logging and analysis, he could have formed a good idea of how many separate attackers there were.
      If he were less public and less accountable, he could have looked at the attacking machines. They may have all had the same kind of remote exploit. There are a lot that even use things like Word Press to relay their attacks. I have been called to help fix exactly that. Someone used some unpatched exploit, to deploy a bunch of back doors and relay code, and attacks were being run from their site. I really hate Word Press just because there are so many holes. Every WP site is just waiting for a script kiddies crawler to discover, and to add to their botnet.

  • @izzieb
    @izzieb 2 роки тому +542

    Some people are sad and have too much time. Why would anyone DDoS Jeff's site?!

    • @jordanlake9151
      @jordanlake9151 2 роки тому +241

      Well at least it led to content

    • @jeremygmail
      @jeremygmail 2 роки тому +51

      Why? It is the Internet. Things happen for the dumbest reasons like because I can or because I am doing it for the lolz. The only thing you can do is protect yourself as much as you can and watch/alert on suspicious anomalies. That is more than just high traffic these days so your alerting has to be on point.

    • @ApolloSevan
      @ApolloSevan 2 роки тому +19

      Red shirt Jeff egged them on I’m sure! 🤣

    • @luis449bp
      @luis449bp 2 роки тому +6

      Just for fun

    • @JeffGeerling
      @JeffGeerling  2 роки тому +233

      @@jeremygmail most likely someone did it for the lulz. They're probably chuckling to themselves watching this video :P

  • @TechnoTim
    @TechnoTim 2 роки тому +188

    I feel your pain Jeff! Each time I released a video talking about self-hosting security I get DDoS'd

    • @Disatiere
      @Disatiere 2 роки тому +12

      I can see people seeing it as a challenge

    • @dieSpinnt
      @dieSpinnt Рік тому +2

      @@Disatiere I can see people going to jail ...

    • @Disatiere
      @Disatiere Рік тому +6

      @@dieSpinnt I mean usually they drive there

    • @dieSpinnt
      @dieSpinnt Рік тому

      @@Disatiere Yeah, you are right. Just couldn't resist to make a pun, based on your comment:)
      because in reality: some of the attackers get caught ... for doing childish BS.

    • @dragnar12
      @dragnar12 Рік тому +5

      U: look i have my own private server.
      The poeple: Lemme test how good it is

  • @jeremygmail
    @jeremygmail 2 роки тому +47

    Thank you for sharing @Jeff. As someone who was on an IT team which dealt with a Top 3 (at the time) attack where they sustained 300Gb/s and wreaked havoc on the network. I can relate. Good luck keeping ahead of the botnets!

    • @halo4life166
      @halo4life166 2 роки тому +3

      Out of curiosity, was this an OVH one? I seem to remember a Blackhat talk about something similar in the past

    • @jeremygmail
      @jeremygmail 2 роки тому +5

      @@halo4life166 no it was a different one. Just saw they got to 1Tbps. That is pretty egregious for 2016!

    • @jesseclutterbuck6617
      @jesseclutterbuck6617 Рік тому

      these days its done with time servers. you can request alot of data from a time server and we all know how fast there connection speeds are. this technique is called the amplified ddos attack. in the past some amplified ddos attacks have reached as high as 2.5 tbps.

  • @rickharold7884
    @rickharold7884 2 роки тому +94

    Wow. That’s nuts. Great learning experience and love that u share it. Much appreciated

  • @hubertnnn
    @hubertnnn 2 роки тому +113

    One suggestion to make this kind of DDoS less problematic: Use two servers.
    One that will be handling heavily cachable data for most people and another one for handling POSTs etc.
    This way if you get DDoSed, only the vulnerable POST server will get hit, and the GET server will survive pretty much unharmed.
    You will loose the ability to send comments, but the website will still be up and fine.

    • @logangraham2956
      @logangraham2956 2 роки тому +5

      you could still send comments , just accept posts only from your post server and ignore everything else on your get server.
      than when i comment is posted
      client -> post server -> get server ->client

    • @noviccen388
      @noviccen388 8 місяців тому +2

      Do most ddos use post requests?
      do the Get requests only have minimum impact on the server?

    • @hubertnnn
      @hubertnnn 8 місяців тому +5

      @@noviccen388 No, but GET requests can be cached and POST cannot. Introducing a caching layer can increase the number of requests per second 100-1000 times.
      So the same server will be able to handle just 100 POST requests per second or 10.000 GET requests per second.
      It will be much harder to DDOS the second one.
      Also most traffic in websites is GET traffic, so sacrificing a POST server to DDOS will only limit functionality instead of killing the website completely.

    • @Quint2105
      @Quint2105 8 місяців тому

      @@hubertnnnJust curious. I’m running a VPs which got hit by small scale ddos attacks recently. Hereby they targeted the vps ip itself. The problem we faced is that our network bandwidth ran out, our system resources such as CPU and RAM where almost unaffected. Our normal legitimate traffic could not get trough the network bombardment of the attack anymore. What could I do to prevent this from happening apart from switching to a higher bandwidth network?

    • @hubertnnn
      @hubertnnn 8 місяців тому

      @@Quint2105 It really depends on the specifics of your system. Not sure what do you mean by VP.
      But as generic rules I would start with a CDN network with DDOS protection like cloudflare (it has free tiers).
      Next thing would be to reduce the size of responses, if they filled your network then a lot of data had to be transferred.
      After that I would add some kind of IP based throttling (typical configuration is 60 reqests per minute per IP), it wont help against huge botnet, but will at least limit the effects and help against smaller botnets (limiting each bot to just 1 req/s).
      And yes, increasing the available bandwith could also help.
      You could also try auth protecting some of the larger data behind short lived tokens that require authentication and captcha, though capthas recently can be easily solved by machines while being hard to solve by humans..

  • @paulmichals
    @paulmichals 2 роки тому +43

    Thank you Jeff for this very topical video. I've been doing IT since the early 90's (yes I am older than dirt) and DOCUMENTATION (often paper notebooks) is the best bit of information to take away for those who watched this video.

    • @turbopro10
      @turbopro10 2 роки тому

      I've been doing IT since the 70s before it was called IT, so there ...

    • @paulmichals
      @paulmichals 2 роки тому +2

      @@turbopro10 in the 70's I was underway under water on watch as a Reactor Operator on the US Nuclear Powered fast attack submarine USS Queenfish - SSN 651. But in about '73 I do remember messing around with punch card readers at a local community college's computer lab.

  • @skug978
    @skug978 2 роки тому +6

    Thanks for sharing Jeff. Good that you dealt with the problem and gained the experience from it.

  • @almostmatt1tas
    @almostmatt1tas 2 роки тому +11

    This is one of those videos that makes me realise I don't know as much about computing as I thought I did. Time to spend the day googling acronyms! Thanks for sharing your experience Jeff.

  • @scottwilliams895
    @scottwilliams895 2 роки тому

    Jeff, it's very cool of you to share what happened, how you responded, and what you learned.
    Content like this is why you earned my Sub years ago, and why I keep coming back for more.

  • @pendragonscode
    @pendragonscode 2 роки тому +9

    I had this happen to me once before! Instead of doing what you did, I decided to make my site give a link to youtube... A rickroll link. After that, I added captcha and then my site got up. Meanwhile for some bizzare reason, the rickroll thing i added actually attracted more attention as my friends who knew about my site started sharing it around lol. I didn't know my site was down until the 2nd day, added the rickroll on the 2nd day, left it like that for almost a week. (Was also at the same time getting a new machine to host the thing.). So yes rickroll helped me.

  • @DevOdyssey
    @DevOdyssey 2 роки тому +22

    Awesome breakdown Jeff! This really affirms everything I've learned in my job so far!
    I've certainly learned about DDoS, but never seen such an eloquently "documented" video, describing your real life, personal experience in a timeline manner. Happy Cloudflare came in and saved the day.
    And of course, the obligatory, "Thanks DNS, thanks" 😒

  • @LifeIsRecusive
    @LifeIsRecusive 2 роки тому +157

    "I got hacked, because i revealed my infrastructure"
    *Makes additional video exposing more of the infrastructure information*
    Red shirt jeff should have done this video lol
    Thanks for the explainer, always appreciated

    • @trbry.
      @trbry. 2 роки тому +15

      I always thought **hacked** was more of 'you locked that door not this' instead of "I'm gonna put all these billions on pebbles on the road so you can't drive here".

    • @Max24871
      @Max24871 2 роки тому +8

      At this point he just using us to test his hardening efforts

    • @levelup1279
      @levelup1279 2 роки тому +2

      I have a broad definition of hacking, & that's just manipulating computer systems in a way which the designer never intended, or just doing general hacker things. It's hard to define hacking because of how broad it is. The 90's were much more liberal with who qualified as a "hacker".
      Now there are all these keyboard warriors who get angry if you don't use the correct terminology. "That's not hacking you idiot, its exploiting".
      Nope, it's all hacking.

    • @Mmmm_tea
      @Mmmm_tea 2 роки тому

      @@trbry. some people line their drives with pebbles,they don't stop you driving just stop you driving fast... if you like your windows.

    • @bjw8qsrmhgxn4wwk30
      @bjw8qsrmhgxn4wwk30 2 роки тому

      Security by obscurity is a farce.
      With some sleuthing you’d be able to determine almost any information about Jeff’s site.

  • @RicardoVargas03
    @RicardoVargas03 2 роки тому

    Man! You are AMAZING! This is the first time I have seen your videos; there is a LOT of value here! Thank you!

  • @roguethinker6284
    @roguethinker6284 2 роки тому +3

    Smokin Video Geoff. In 11 minutes you've covered just about everything I know about mitigating DDOS attacks. Took me years. My brain is getting old

  • @CraigEngbrecht
    @CraigEngbrecht 2 роки тому +35

    Thanks! I appreciate all your arguments here, and the wonderful breakdown of the information.
    I have always argued for monitoring, however, rarely implimented it correctly. :P

  • @filovirus1
    @filovirus1 2 роки тому +54

    whoever comes up with a way to pinpoint DDoS attackers so we can reach out and slap their physical faces should win a Nobel prize

    • @johndododoe1411
      @johndododoe1411 Рік тому +1

      More likely an award from a military team, such as Nobel's original corporations.

    • @chrisakaschulbus4903
      @chrisakaschulbus4903 Рік тому

      YES! Then darknet users can finally find out who is ddosing their markets. That'd be great.

    • @angryjoshi
      @angryjoshi 8 місяців тому +3

      Child assault is illegal 😂

    • @glynnetolar4423
      @glynnetolar4423 6 місяців тому

      A little "wet work" might curb that kind of activity.

  • @Jason-mk3nn
    @Jason-mk3nn 2 роки тому

    Great video, on all levels! Great work and thank you for sharing!

  • @maartentoors
    @maartentoors 2 роки тому

    I love the transparency of your content/tutorials. As for monitoring I use NEMS myself (awesome package).As for mitigating the 3rd attackon your site, 30 mins response/mitigation... KUDOS!
    Cheers from a fan!

  • @xuldevelopers
    @xuldevelopers 2 роки тому +88

    I see you can use pretty nifty awk and friends. The usual thing I do in these cases is to use it to select, say, all IPs that requested more than 10 pages without ever downloading any other resources (CSS, JS, images whatever is required for your pages). That always works because these DDoS do not simulate browsers completely so it is easy to differentiate robots. One must deal with legit SE later. Then I feed the firewall. Once it runs again there is a plenty of time to do other things. The biggest list of IPs I selected that way was 35.000 individual IPs during one attack.
    Also there are those tell signs, that you can target with grep|cut|awk|sort|uniq|... most of DDoS attacks rotate UA strings that get logged in your logs. So selecting and grouping all requests from given IP and seeing how many different UA strings compared to requests is there turned out to be very often a reliable way. With other signs it is close to 100% accuracy. Not to mention that if all it does is to hit POST page then it is easy to identify all "weird" IPs. Worst cases are those DDoS attacks that simulate a normal browsing. I've seen behavior where there were sequences of dozen or so pages that each robot followed pretending to be a normal user. That was a tough one because server was under pressure and everywhere I looked all appeared to be a legit user browsing until you figure out that all you see is the same browsing sequence over and over.
    I added logging of select cookies and some HTTP headers like supported languages and such so I have more info to use for selection. Those robots very rarely support cookies. Especially tracking JS-set cookies is something attackers don't support. But lately I noticed that few attackers were stuffing standard cookies like tracking cookies or session cookies with random numbers but it is really rare.

    • @JeffGeerling
      @JeffGeerling  2 роки тому +26

      This is true, but at one point today DigitalOcean shut down my main IP after it was getting hit with 2.3 million PPS, at which point I basically hid the entire server behind Cloudflare.
      Some providers may be willing to work with DDoS mitigation, but usually once it's persistent and high volume, they want that traffic out of their DC.
      For the initial attacks, I could've handled it by setting up fail2ban with Nginx logs, but once it got going, I fear I would've needed to invoke CF regardless :(

    • @Supremax67
      @Supremax67 2 роки тому +2

      @@JeffGeerling -- Also a reason why decentralized public ledger are trending.

    • @francois1e4
      @francois1e4 2 роки тому

      @@Supremax67 What do you mean?

    • @Supremax67
      @Supremax67 2 роки тому +2

      @@francois1e4 -- You call them blockchain, but that is over simplifying it. Not every public DLT is a ledger and not every blockchain is actually decentralized.
      In a sea of noise, only a few of them shows promise. The next decade should be interesting.

    • @francois1e4
      @francois1e4 2 роки тому

      @@Supremax67 True that!

  • @TheNillquest
    @TheNillquest 2 роки тому +18

    Please, more videos about prevention of ddos and ransomware, btw brazilian here, sorry for our country been a part of the attack IT security here is minimal.

    • @JeffGeerling
      @JeffGeerling  2 роки тому +27

      I don't blame individuals ;)
      Some people like to block entire countries-and that can help to an extent-but I would rather leave things as open as possible because even in the countries where it seems the worst of these attacks originate (especially Russia and a few south Asian countries), there are still plenty of legitimate users who just want to learn something, and who am I to shut them off?

    • @Private-GtngxNMBKvYzXyPq
      @Private-GtngxNMBKvYzXyPq 2 роки тому

      Glasnost -> Peace
      Cooperation -> Mutual Benefit
      I second the request for more videos on security. Thank you.

  • @martinc.7424
    @martinc.7424 2 роки тому

    Thank you for sharing your experience Jeff.

  • @meddlin
    @meddlin 2 роки тому

    Good work, man! This is awesome, and inspires me to beef up my website more.

  • @wartlme
    @wartlme 2 роки тому +7

    Hope no one hits my site with a DDos attack. Glad you made it. Thanks for sharing.

    • @volkhen0
      @volkhen0 2 роки тому +1

      What’s your website? ;)

  • @thrillscience
    @thrillscience 2 роки тому +7

    Who would have anything against a RaspberryPi guy? Big Arduino?

    • @JeffGeerling
      @JeffGeerling  2 роки тому +5

      Heh, but Arduino's making a board with the Pico on it now. Not sure who would care that much!

  • @falazarte
    @falazarte 2 роки тому

    Amazing video! Keep up the good work.

  • @younisamedi
    @younisamedi 2 роки тому +1

    God bless you Jeff! We're all with you brother.

  • @hse5.0
    @hse5.0 2 роки тому +5

    Nice documenting the attack. Also looking forward to the GPU project for the pi. Looks like someone deserves a well rest this weekend 😜.

  • @microm4n
    @microm4n 2 роки тому +13

    This was great. I was debating putting my website behind Cloudflare in preparation for an attack that I can't cope with myself, along with some of their other offerings (like the new anti phishing email stuff). I too am not a fan of the centralisation of traffic but for now it's about the only option we have, and CF are still "good guys", at least for now.

  • @FelipeFonsecaRocha
    @FelipeFonsecaRocha 2 роки тому

    Nice from you to not stop sharing man... Really learn a lot from you...

  • @RobertFabiano
    @RobertFabiano 2 роки тому

    This was a great video! Real meaty subject with good level of detail

  • @AndrewDanne
    @AndrewDanne 2 роки тому +7

    Good to hear you are back on line and in 1 piece after this. Can you suggest how I would test/monitor my IoT, Raspberry PI's, Network, to see & monitor if I am contributing to a BOT net? Cheers

  • @MarcoGPUtuber
    @MarcoGPUtuber 2 роки тому +6

    0:54 It's a good thing you use CRTs. The lack of smart features make them UNHACKABLE!

  • @vagellan_8842
    @vagellan_8842 2 роки тому +1

    Bro! Awesome video! Love the shirt. I still plan on getting into IT professionally instead of just studying, and tinkering, and grumbling about every commercial setup I see or have problems with. Love the shirt and just bought one!

  • @Rosco785
    @Rosco785 2 роки тому

    Loved this deep dive into this, bell and subbed for sure.

  • @qingdom
    @qingdom 2 роки тому +5

    "Anton died so that we could live!" - Gilfoyle, Pied Piper

  • @john_hawley
    @john_hawley 2 роки тому +45

    So from what I'm gathering from your analysis: For the home guy with limited bandwidth and hardware your options are:
    1. Buy a PaaS (i.e. Cloudflare)
    2. Shut 'er down

    • @JeffGeerling
      @JeffGeerling  2 роки тому +22

      Pretty much. Though if you are close friends with a local ISP, you might be able to work with them on a solution. But good luck with that if you're 99% of people. Spectrum won't give me the time of day :(

    • @abhimaanmayadam5713
      @abhimaanmayadam5713 2 роки тому +5

      Cloudflare does have a free tier

    • @jeremygmail
      @jeremygmail 2 роки тому +7

      @@JeffGeerling Ha! when we got ddos'ed our provider took us offline because the botnet was killing their network too :)

    • @Vangard21
      @Vangard21 2 роки тому +2

      I'm no crypto-advocate (and ATM it's like 90% scams), but distributed Web 3 is an alternative to Cloudflare/AWS/Google control of the internet.
      IPFS for statics, Ethereum dapp backend. And a ~2 minute page load for end users :/
      But it might well turn out to be the best alternative to web centralization.

    • @Xamy-
      @Xamy- 2 роки тому +5

      @@Vangard21 no mate. That shit is all just a ploy to promote crypto scams, don’t talk about it. Watch “Line goes up - the problem with NFTs” (and crypto)

  • @Bianchi77
    @Bianchi77 7 місяців тому

    Cool info, thanks for sharing, well done :)

  • @yalopov
    @yalopov 2 роки тому +1

    This video is amazing, some of us like to self-host things but that comes with risks we need to be aware of!

  • @eyesofnova
    @eyesofnova 2 роки тому +24

    I don't know much about it, but I've run across the github repo for Gatekeeper. Its open source DDOS protection. I'd be curious how well it functions in practice, or how hard it is to get it configured correctly.

    • @lorenzo42p
      @lorenzo42p 2 роки тому +6

      probably not a fix for a ddos. the best you can do is drop the packets, but the flood of packets still needs to reach the firewall before they can be dropped. bottleneck is your internet connection, which gets swamped and overloaded. there are some possible options to drop the packets before they're sent to your internet connection, but those technologies are usually reserved for the big companies.

    • @johndododoe1411
      @johndododoe1411 Рік тому +1

      @@lorenzo42p Yeah, I wish there was a common ICMP extension for a swamped server to request upstream dropping of high volumes of packets. Something that could be quietly running on the Cisco backed routers and prioritize blocking requests that reject the most attack traffic in any given moment, letting through low bandwidth traffic that happens to hit site firewall rules that send too many block requests. Ideally the router priority software would also detect if multiple recipients are requesting protection against the same outside source, ultimately resulting in zombified machines getting blocked closer to their own connections, following by an angry letter from their ISP.

  • @user-zb3zh7ch9s
    @user-zb3zh7ch9s 2 роки тому

    Another great video thanks Jeff

  • @fonte935
    @fonte935 2 роки тому

    Fascinating! Thanks Jeff.

  • @ernstoud
    @ernstoud 2 роки тому +4

    Years ago already the adagio was that the only way to stop DDOS is making sure your pipe is bigger than theirs. There is no way around companies like Cloudflare who have the budget for those big pipes.

    • @MatthewDeveloper
      @MatthewDeveloper 2 роки тому

      This is true, I've tried blocking IP's on iptables, after a while iptables are actually using all the CPU usage on my small server.
      I turned the server down, waiting for the attack to be done.

  • @davidbubble6863
    @davidbubble6863 2 роки тому +58

    Curious how those attackers choose their targets. Jeff's web site of all things? Makes no sense at all.

    • @JeffGeerling
      @JeffGeerling  2 роки тому +97

      I once learned from a wise old man... "Some people just want to watch the world burn."

    • @davidbubble6863
      @davidbubble6863 2 роки тому +5

      Well that's one reason 😂

    • @guiorgy
      @guiorgy 2 роки тому +3

      @@davidbubble6863 Just for the LOLs because YOLO? Or they happen to be a viewer who wanted to challenge Jeff, or give him a reason to make this video ¯\_(ツ)_/¯

    • @RetroGameStream
      @RetroGameStream 2 роки тому +7

      Yeah I wonder that same thing. I host over 200 websites and the few times I've had to deal with this they were always the smaller sites that didn't make any sense, like a ma pa grocery store or small church. Not sure what they got out of that unless they just chose their sites randomly.

    • @AudreyRobinel
      @AudreyRobinel 2 роки тому +8

      @@RetroGameStream perhaps they are just trying their tools, see what works or not? maybe they are akin to "interns" in their fields, and this is their assignment before leveling up?

  • @syntheticperson
    @syntheticperson 2 роки тому

    Very insightful. Thanks

  • @bagorolin
    @bagorolin 2 роки тому

    Thanks for sharing!!❤️

  • @DanielLopez-up6os
    @DanielLopez-up6os 2 роки тому +4

    40 Mb/s Attack seems HUUUGE, Then I remembered the SpamHouse attack cloudfare protected, and it was somewhere along a 1 TB/s attack. Cloudfare is amazing AF!

    • @sergsergesrgergseg
      @sergsergesrgergseg 2 роки тому

      40 mb/s is quite low.. you can buy stressers that hit a lot more than that for less than 10 dollars

    • @DanielLopez-up6os
      @DanielLopez-up6os 2 роки тому

      @@sergsergesrgergseg those stressers usually are incomplete http request based tho, so quite easy to mitigate.

    • @sergsergesrgergseg
      @sergsergesrgergseg 2 роки тому

      @@DanielLopez-up6os you would be surprised on the level of sophistication some of these cheaper underground services can offer

  • @janhumpolicek8373
    @janhumpolicek8373 2 роки тому +4

    Holy cow you saved me! I amexperiencing this rn!!!! Thank so so much.

  • @unserfa
    @unserfa 2 роки тому

    Thank you for sharing!

  • @michaeldesilets7528
    @michaeldesilets7528 2 роки тому

    I enjoy your videos for entertainment. When I let my head get out of entertainment mode and back into semi work mode I learn a bit and enjoy your videos more. Thank you.

  • @muhammadazmi3323
    @muhammadazmi3323 2 роки тому +3

    yep I can confirm this, most people in my country don't care about cybersecurity even on a government level, no wonder how many botnets have already been installed on individual devices

  • @ur1friend437
    @ur1friend437 2 роки тому +4

    OMG Jeff I hope the attacks didn't took too much time from your family time. I admire your work and honesty, so for that reason is heartbreaking to see you being punish, for your good work and honesty.

  • @constantiusdamar1925
    @constantiusdamar1925 2 роки тому

    Great Video Jeff,

  • @chuxxsss
    @chuxxsss 2 роки тому

    Thanks for the tip about cloud flare, Jeff.

  • @AndrewBeeman007
    @AndrewBeeman007 2 роки тому +29

    When I saw your video about the cluster on a farm I was curious as to why you didn't have it behind Cloudflare. I agree with the idea of not contributing to centralization, but there are too many bad apples out there to not have a layer of protection like Cloudflare IMO.

    • @monsterhunter445
      @monsterhunter445 2 роки тому +2

      In theory cloudflare could snoop traffic if unencrypted?

    • @AndrewBeeman007
      @AndrewBeeman007 2 роки тому +4

      @@monsterhunter445 If it is unencrypted, you have more significant problems. But in theory, yes.

    • @webfreezy
      @webfreezy 2 роки тому

      Just to note - you could also use AWS Cloudfront - but I don't think they have a free tier.

    • @AndrewBeeman007
      @AndrewBeeman007 2 роки тому +1

      @@webfreezy In my opinion, Cloudflare is far less evil than Amazon

    • @soundspark
      @soundspark 2 роки тому +2

      @@AndrewBeeman007 Even though Cloudflare looks the other way at abuse?

  • @SutherlandBoswell
    @SutherlandBoswell 2 роки тому +3

    Documenting everything is the type of advice that seems obvious but is easy to skip over. I wish I had documented it, but in the past I dealt with what appeared to be a pretty small DDoS attack that turned out to actually just be a clever way a virus was trying to phone home. The domain was a simple two word name, and what I seem to remember is that both of those words happened to be in an array the virus would use to build a long list of domains to try phoning home. The malicious party could easily come back after buying a different domain from the list if they were ever shut down, and I assume it made it harder to trace back to the creator since many of the randomly generated names were already owned by legitimate sites like mine.
    Since my memory of it isn't great I really wish I had followed that advice, because it was an interesting learning adventure.

    • @JeffGeerling
      @JeffGeerling  2 роки тому +1

      At this point it's just my instinct-if something weird happens, immediate screenshot.
      If it turns out it wasn't something interesting, I can always delete the screenshot later! I've almost never had a moment where I regretted saving off some extra data during one of these moments.

  • @RuiFungYip
    @RuiFungYip 2 роки тому +1

    The nice thing about cloudflare tunnels, is that it turns an incoming connection into an outgoing connection.
    Which is pretty handy when you want to host a site and you're behind a CGNAT.

  • @driver34579
    @driver34579 Рік тому

    I once saw a server rack that had a glass window. There was a sign inside that read: In case of DDoS attack, break the glass and cut the cables.

  • @adversHandle
    @adversHandle 2 роки тому +4

    I accidentally dosed thr online learning portal for my college once .
    The webpage wasn't loading I left the tab open and did other work.
    2 Hours later the admin knocked on the door of the study room asking if I was in there 🥺😱 what.
    He was cool about it I had no idea I took down the website 😅

    • @JeffGeerling
      @JeffGeerling  2 роки тому +1

      Haha, though that shouldn't be on you, probably an application bug that caused your browser to keep reloading something in an infinite redirect loop or something!

  • @Alok_raj
    @Alok_raj 2 роки тому +3

    Mine also got ddos
    Thks it might help me.

  • @phlizneinbleedblop2318
    @phlizneinbleedblop2318 2 роки тому

    Woo Jeff! thanks for the info hopefully we all can prepare for inevitable attack

  • @JonDoe-gi5zf
    @JonDoe-gi5zf Рік тому

    This video is very informative.

  • @sebastiannielsen
    @sebastiannielsen 2 роки тому +3

    Note that using a firewall (instead of Cloudflare which he uses in this video) doesn't work if you have a limited line to your ISP. If the strength of the DDoS attack is bigger than your incoming internet line, only person in charge to stop the DDoS is your ISP or upstream hosting provider. This because even if you have a imaginary, perfect, firewall that is able to absorb 100% of the DDoS attack and let 100% of legitimate traffic in (which doesn't exist in reality), your internet line would still be swamped with the DDoS attack, which means the filtering must happen before the bandwidth is reduced.
    Another reason mitigations must be upstream, is if you have a so called metered connection. Even if your firewall blocks the traffic, it will usually still count against the metering, why you need to talk to hosting provider regardless.
    As saw in the video, he is using Cloudflare, which acts as a big firewall before it even reaches your hosting provider, thus your smaller internet line isn't affected. This is equvalient with mitigating at your hosting providers' backbone.
    Smaller DDoS attacks however, can be mitigated with a good anti-DDoS protection to not load down the server.

    • @ewookiis
      @ewookiis 2 роки тому +1

      All lines / connections are limited ;). Cloudflare and services as such does have firewalls, but the descision is not always made at the lowest level at first on these kind of services. The saving grace is the blocking (fw's) of known bad, loadbalancing, caching and the much higher ceiling of bandwidth since they have a multitude of ingress points - also the known flows of sender and destination across cloudflare setup accumulate quite a nice dataflow, in conjunction with known addresses from botnets etc etc. in short - one always needs an backup ;).

    • @sebastiannielsen
      @sebastiannielsen 7 місяців тому

      @@appxprt4648 Yes 50% of total capacity, since the system wont be able to respond. But usually, broadband is metered in like 100mbit/100mbit, so a DDoS attack has to fill either of these to 100%, which is equal to 50% total.
      Backplane capacite is usually number of ports / 2, so a 16 port gbit switch usually have 8gbit backplane, so you would just not be able to flood it unless you have access to multiple ports on that switch. Or have access to a unfiltered uplink port.
      But these types of DDoS attacks can be mitigated by a firewall, ergo, make sure there is a filter before uplink port. Its when the DDoS are bigger than your ISP connection that you are in trouble.

  • @rbunpat
    @rbunpat 2 роки тому +5

    A question, could Cloudflare prevent this?
    Edit:Nevermind, I got the answer.

  • @driodeiros
    @driodeiros 2 роки тому

    Thank you for your videos and content Jeff, I find them helpful and entertaining.
    This video inspired me to improve my monitoring stack (prometheus + grafana also) but it also made me think about the backup/restoring strategies, particularly for your prometheus time series data and the grafana alerts and dashboards. What strategy do you follow to backup and restore those in your environment?
    Thank you so much again and keep up the good work!

  • @Tim-Kaa
    @Tim-Kaa 2 роки тому

    Thanks, very useful

  • @alexlandherr
    @alexlandherr 2 роки тому +10

    Sad to hear that, I run my own little Dark Web site hosting satellite images on a Pi4B 8GB using Nginx. I hope I never experience this.
    EDIT 2022-03-16 19:46 UTC: It’s a static site so not that much going on, it’s meant to be lightweight. No JS, only CSS for styling using Atomic.

    • @them2545
      @them2545 2 роки тому

      Oh cool mind dropping the onion link

    • @skorpion1298
      @skorpion1298 2 роки тому

      @@them2545 I like onions

  • @MarcoGPUtuber
    @MarcoGPUtuber 2 роки тому +5

    Was it DNS?

  • @soultracer
    @soultracer 2 роки тому

    Thanks for sharing.

  • @povilasstaniulis9484
    @povilasstaniulis9484 2 роки тому

    Thank you for sharing.
    Documenting as much information as possible is an incident response 101 for pretty much everyone who is hosting their own servers. Of course, not everyone runs high-profile websites/projects to require very extensive monitoring, but some level of monitoring is pretty much a must. And monitoring is useful not just for cyber incidents but for monitoring overall server health too.
    A good example of a tool many admins don't bother to set up is root emails. They aren't that hard to set up and a simple email from smartctl that you hard disk is going south can prevent data loss and downtime. Or an email telling that your backup script didn't run properly.

    • @JeffGeerling
      @JeffGeerling  2 роки тому

      For me it's usually the once or twice per year certbot starts complaining about certs... I then fix it before the cert expires :D

  • @xephael3485
    @xephael3485 2 роки тому +4

    This is basically an advertisement for cloudflare... you didn't handle anything. Also 3000 requests per second? That's pretty weak bro.... 2kpps is minimum alarming level for most DDoS mitigation products.

    • @techbriefing
      @techbriefing 7 місяців тому

      yeah 3k RPS is very low and if your site collapses at that level of traffic it's a bit embarrassing
      most modern DDoS attacks on medium to large services are now 2-3M RPS+

    • @techbriefing
      @techbriefing 7 місяців тому

      the largest ever DDoS attack was performed by someone I know, who owns the Meris botnet. that achieved 400 million RPS by exploring a vulnerability in HTTP/2 (now known as the Rapid Reset vulnerability). he has previously taken the entire Cloudflare network offline, taken Google offline, taken Amazon offline, among other huge services. he's been thwarted by Cloudflare and Google teaming up but he's already found a new vulnerability although I don't know the details.

  • @pranaypallavtripathi2460
    @pranaypallavtripathi2460 2 роки тому +23

    When you are as smart as Jeff, you can make a whole video on why your viewers are not able to view your website and gain even more views. Take that DDoS attacker. 😂

  • @henkdevries5042
    @henkdevries5042 2 роки тому

    Holy!! You too?? Wondering how much sleep you have missed, it did not show. Good work Jeff, keep up the great work!

  • @carstenr.1682
    @carstenr.1682 2 роки тому

    Thank you for sharing your insights on this.
    I'm using the Cloudflare Argo Tunnel. It's super easy and the big benefit is, that you do not need to open port 80 and 443 for inbound traffic.
    I can also recommend to use an AWS Lightsail instances - they are cheap, fast ...

  • @linuxastro
    @linuxastro 2 роки тому +1

    Yep, it has gotten bad of late. I had a "what's that noise" event when an alarm went off for the first time last week (10 years after install).

  • @nhalliday89
    @nhalliday89 2 роки тому

    Wow isn't that a trip... .It appears that you have been super busy with this madness honestly i envy your patients with how the situation has presented itself...

  • @luminescentlion
    @luminescentlion 7 місяців тому +1

    10:53 I know the pain, it was 5Mbps up before I we switched from Comcast to Fidium now its 1Gbps up symmetrical with my down for half of what you pay.... which is nice.

  • @bepowerification
    @bepowerification 2 роки тому

    Man. i am glad you survived!!

  • @albertcanon9185
    @albertcanon9185 2 роки тому

    nice educational video,, thankyou sir

  • @superbrain3848
    @superbrain3848 2 роки тому +1

    reminds me on the Mirai Botnet that managed to shut down a quite large part of the internet back in 2016.
    Some kids managed to create a massive botned in an accident, and then launched an Attack to the wrong IP, causing the DNS provider Dyn to run into issues.
    Dyn provides a DNS service for websites like Spotify, soundcloud and Twitter.

  • @yacahumax1431
    @yacahumax1431 Рік тому

    thank you for the info.

  • @karter61
    @karter61 2 роки тому +2

    I run quite a popular website that gets multiple daily massive DDoS attack attempts. Cloudflare is a godsend as without it there is no way the site would be able to stay up. I have got quite a bit of complicated rules running on CF to help prevent these attacks. The best thing is that CF has really great API's so I have been able to automate everything to keep the site online

  • @MarksGoneWicked
    @MarksGoneWicked 2 роки тому +2

    Several years ago, I had noticed my bandwidth taking a hit. I went into the network monitoring on my router and watched my router being slammed by requests. They were hitting the dynamic IP assigned to the router by my provider. Thankfully, only a slow connection was the only result.

  • @kjyhh
    @kjyhh 2 роки тому

    very imperative. good vid

  • @patsypryor9850
    @patsypryor9850 2 роки тому

    I just stumbled in to your site, and at my green level of understanding,I am just terrified to use my devices at all. Just unplugged my internet and went to bed. May go old school off grid,yikes!!! what an education. thnx

  • @uhthin
    @uhthin Рік тому

    Well done

  • @luvxinh
    @luvxinh 2 роки тому +1

    Interesting. I was also thinking of what you've been sharing with the world in the spirit of open source. It's basically the documentation for your infrastructure made available to the public

  • @AlanDampog
    @AlanDampog 2 роки тому

    this is a great video!!!

  • @optimalchoice270
    @optimalchoice270 2 роки тому

    Thanks!

  • @Star-xf8rd
    @Star-xf8rd 2 роки тому

    For Network monitoring I can recommend PRRG it’s quite powerful with lots of sensors prebuilt, you can even create own sensors for numerous systems

  • @tech_craft
    @tech_craft 2 роки тому

    Love the thumbnail!

  • @cheetobambito9724
    @cheetobambito9724 2 роки тому

    you know what all these ddos attacks after years of never having a single one? YOURE MOVING UP IN THE TECH LIFE AND SHOULD BE PROUD!(:
    Good stuff Jeff cant wait for your next video

  • @hamburger2430
    @hamburger2430 2 роки тому

    I like your funny words, magic man!

  • @johncnorris
    @johncnorris 2 роки тому

    Sounds like a tough day at the office but at least you've learned a lot about defensive measures.

  • @Wordsnwood
    @Wordsnwood 2 роки тому +2

    Yup, that thumbnail is 🔥
    (And I've tried to get my team to adopt your "it was DNS" shirt for our team uniform, but so far no go.... 😉)

  • @Space_Reptile
    @Space_Reptile 2 роки тому +2

    *adds Jeff's website to the list of websites unreachable when cloudfare has an issue again*
    Would love to see a project of where you make your "own cloud flare" so it won't be affected by outages like half of the Internet at this point but still be protected

    • @thewhitefalcon8539
      @thewhitefalcon8539 2 роки тому +1

      Cloudflare can do what Cloudflare does because it has hundreds of terabits of bandwidth, and that's the only way to do it.
      How much do you suppose that costs?

  • @FlygisTheFlygis
    @FlygisTheFlygis 2 роки тому

    “How I survived a sneeze attack” up next on this channel. So glad you’re alive bro

  • @RyanHenrie999
    @RyanHenrie999 2 роки тому

    Maybe you could do a cyber security basics video in the future. I found this video very insightful.