Why was Facebook down for five hours?
Вставка
- Опубліковано 3 чер 2024
- Facebook was down for five hours last week. What happened and what do DNS and BGP have to do with it?
Facebook's explanation: engineering. 2021/10/05...
Support these videos on Patreon: / beneater or eater.net/support for other ways to support.
0:00 DNS
7:13 Caching DNS
10:34 Hop-by-hop routing
14:07 Default-free routing
18:28 Peering
19:50 BGP
26:08 The outage
------------------
Social media:
Website: www.eater.net
Twitter: / ben_eater
Patreon: / beneater
Reddit: / beneater
Special thanks to these supporters for making this video possible:
Aleksey Smolenchuk, Anders Carlsson, Andrew C. Young, Anson VanDoren, Anthanasius, anula, Armin Brauns, Ben, Ben Cochran, Ben Kamens, Ben Williams, Benny Olsson, Bill Cooksey, Binh Tran, Bouke Groenescheij, Bradley Pirtle, Bradley Stach, Bryan Brickman, Burt Humburg, Christopher Blackmon, Dale Andrew Darling, Daniel Jeppsson, Daniel Pink, Daniel Tang, Dave Burley, Dave Walter, David Clark, David Cox, David Dawkins, David House, David Sastre Medina, David Turner, David Worsham, Dean Bevan, Dean Winger, Dilip Gowda, Dissy, dko, Dmitry Guyvoronsky, Dušan Dželebdžić, Dzevad Trumic, Emilio Mendoza, Eric Dynowski, Erik Broeders, Eugene Bulkin, Evan Thayer, Eveli László, fisksvett, fxshlein, George Miroshnykov, Harry McDow, hotwire33, Ingo Eble, Ivan Sorokin, James Capuder, james schaefer, Jared Dziedzic, Jason DeStefano, Jason Dew, JavaXP, Jaxon Ketterman, Jay Binks, Jayne Gabriele, Jeremy, Jeremy Cole, Jesse Miller, Jim Kelly, Jim Knowler, Jim Van Meggelen, Joe Beda, Joe Pregracke, Joel Miller, John Fenwick, John Hamberger jn., John Meade, Jon Dugan, Jonn Miller, Joseph Portaro, Joshua King, Jurģis Brigmanis, Kai Wells, Kefen, Kenneth Christensen, Kitick, Koreo, Lambda GPU Workstations, Larry, László Bácsi, Lucky Resistor, Lukasz Pacholik, Marcos Fujisawa, Marcus Classon, Mark Day, Marko Clemente, Martin Noble, Martin Roth, Mats Fredriksson, Matt Krueger, Matthäus Pawelczyk, Matthew Duphily, Max Gawletta, Maxim Hansen, melvin2001, Michael Tedder, Michael Timbrook, Michael Weitman, Miguel Ríos, mikebad, Mikel Lindsaar, Miles Macchiaroli, Muqeet Mujahid, My Yiddishe Mama, Nicholas Counts, Nicholas Moresco, Nick Chapman, Not Yet Wise, Örn Arnarson, Paul Pluzhnikov, Paul Randal, Pete Dietl, Phil Dennis, Philip Hofstetter, PixelSergey, Porus, ProgrammerDor, Randal Masutani, Randy True, raoulvp, Renaldas Zioma, Ric King, Richard Butler-Malone, Richard Ertel, Rick Hennigan, Robert Comyn, Robert Diaz, Robey Pointer, §çīŤøşHï ŇåĶąýŐbŕÔ, Scott Holmes, Sean Patrick O’Brien, Sergey Kruk, solderspot, SonOfSofaman, Stefan Nesinger, Stefanus Du Toit, Stephen Kovalcik, Stephen Riley, Stephen Smithstone, Steve Jones, Tayler Porter, TheWebMachine, Thomas Bruggink, Thomas Eriksen, Tim Walkowski, Tom, Tom Yedwab, Tommaso Palmieri, Tyler Latham, Vincent Bernat, Walter Montalvo, Warren Miller, William, Wim Coekaerts, Wraithan McCarroll, xisente, Yee Lam Wan
Facebook going down was a small price to pay for getting this incredibly informative video from you, thanks Ben!!
Let's bring down some other large network in some weird way to get next video! ...just kidding.
I would pay for Facebook to be down.
You forgot to add "forever". :)
It’s a win/win
Not for Mark ahaha
I'm pretty sure Ben Eater is actually a physical manifestation of knowledge.
He not only knows everything, he also have time to live, breathe and share his knowledge
All you need is a default route and the ability to read, and you can acquire knowledge too. Exciting times.
That sounds like he is a named version of eaters from Digimon cuber sleuth.
Eaters are only interested in collecting and sharing information across it's hive mind.
Ben Eater is an A.I. this physical form that we see is a computer manifestation of his real self, Ben Eater is like the M.C.P. on Tron Part 1 LOL
@Penguin Hey I remember you! You commented on my video once :)
"I'm connected to a router on AT&Ts network that I happen to have access to"
As a former NOC tech for an ISP this was absolutely hilarious to hear you say this so casually. They don't just give anyone console access to random routers!
I think he still works as a nw engineer. If im not wrong he even worked for juniper
Im a network engineer for a medium size ISP (primarily enteprise fiber). This has become my goto video to explain to people what I do. I work with BGP and the like daily.
This is better explanation of DNS than the one I had at university, thank you.
That's because he is self thought, not sure if Ben has high school diploma.
@@dm3on "self thought"
@The Poison Donut not only that but for even a person that isn't very knowledgeable in this field, he makes it very interesting to watch and all his videos are very easy to follow and i almost never find myself lost trying to figure out what is going on, ben is the best.
@@Ripsaw460 oH nO, a slight spelling mistake that totally obfuscates the meaning of his message, what will I do?
@@Nick-lx4fo you dont have to do anything, just thought it was funny.
To quote a random Tweet: "it's hard to even explain to a layperson how badly you have to fuck up to get BGP and DNS both as part of the same problem. Like, you need multiple diagrams."
LOL -- that was my exact reaction when I heard about this. Like, what happened to break DNS and BGP, that took 6 hours to recover from? That is one COLOSSAL screw-up. With such huge stakes, too. It did make me suspicious what _really_ happened.
It took them 5 hours and 50 minutes to find someone with a physical key to let them into the building.
@@johnm2012 That's false. It DID take them that long to get someone with the necessary protocol expertise to fly in and fix the mess. It's silly to assume that no one on site would have a key to the building. It's reasonable to assume that a corporation like Facebook understaffs their infrastructure and doesn't retain experts to maintain that infrastructure.
On site, they use badge readers to access the building. Those badge readers ran on Facebook's network.
@@zacker150 Again, this is false. Why would a badge lock system run on Facebook's global network, as opposed to an internal, segmented network? It makes no sense.
Watching one person setup global router configurations is seriously invoking wizardry levels of mastery. I am truly humbled to watch a work of fine art like this and gives me goosebumps thinking how much power one can wield with the right knowledge and access to modern society.
Now multiply this times 70 000 or so, because that is the number of AS-numbers in use in the world, thus close to the number of independent Internet networks who connect with everyone else.
Nobody can explain like Ben does. Everything is always at exactly the right pace and detail.
I kinda feel smarter learning from this man.
"A router on AT&T's network that I happen to have access to" WHAT A FLEX :D
He works there...
Not really a flex. That server is public and available for anyone to use to validate their BGP routes
There's a few of them spread around, usually referred to as a "looking glass".
Lol
I like the "my address has changed by now" disclaimer. Don't poke me please. You are just poking my mouth breathing neighbor.
@@tiporari It's bad enough with bots scanning ports and trying to dump payloads into the open ports they find yet alone the comment section of youtube seeing what holes or services are avalible.
Man, networking is so convoluted to me. Its a miracle that the global internet works as well as it does on a daily basis.
The number 1 reason it works so well really is the redundancy. Notice how multiple time through this video he keeps talking about dozens if not hundreds of this and that. And that is what makes things work so well, everything is connected in hundreds of ways, so if a route or 2 goes down for whatever reason, who cares, there are still hundreds of routes left to take instead.
There are further tools that make automatization much easier. You set up a few things in the beginning, and then sit back and relax.
I tell people that the entire internet only works with gentleman's agreements and handshakes. And that's when I'm being nice about phrasing it.
@@GummieI except its rarely the case with preferential routing. ISP's sign contracts with third parties to reduce overhead and operating costs by having them operate long-distance runs. - This gives these single-points of failure a Scapegoat if it fails.
US and UK are the worst offenders of this, literally blocking or banning competitors networks at hops that would route through their network rather than around it. This means that regardless of speed or outage state, a singular ISP can ensure the bulk of their traffic goes through and is on their and partner networks.
This gets worse when we start talking undersea cables. Exit Nodes (backbone) connections to externally connected networks are entirely priority based. When it comes to routing overseas, these contractors get priority. Why? rather than operating their backbone at a flat-cost for a nation, and telling the Goverment to cover the cost of operation so all may benefit...
They instead choose to charge rate-based, with tiered flat based contracts available to ISP's. - This means that, your provider has to both pick a rate limit(and quality[jitter] limit) for its clients and limits routing to the connecting country to ONLY this backbone. | Sometimes they have failover's to competitors backbone lines, but since its rate-based there is caps client and SEVERE data-rate penalties.
Amazon is the worst offender of this, offering basement priced backbone lines that are PAINFULLY slow(except to /some/ of their services) and this results in much of the worlds internet being measurably slower as a result...
Australia has the worlds slowest internet exclusively due to no one hosting services inside australia (draconian encryption laws), and relying exclusively on Amazon and Telstra's dedicated Undersea cables to route outside of Australia(to the UK, the longest possible run).
Since nearly everything is hosted in the US or germany. This means reaching these places is usually several hundred hops. - Turning the Fibre network that makes up Australia's rather robust internet infrastructure into a cesspool of garbage slow performance.
@@edenrose2374 this is actually super interesting. thanks for sharing.
That had to be the best explanation of this I've seen. Not dumbed down, no hand-waving explanations, but carefully and effectively explained with a live demonstration to boot. Thank you.
I just enrolled in a CCNA/CCNP course being offered at my University and yes I am as vanilla as they come as long as Networking is concerned but this video was so crystal clear that I may have just revised effectively about 16 credit hours in a single 30minute video!
At the end should have been a cutaway to a team of breadboard computers all networked together, that made all this possible. 😊
Cutaway scene like from the movie “The Prestige” with an eerie music that shows a lot of those breadboards, ICs, wires and then Ben using the monitor and keyboard connected to it.
Maybe that´ll be the topic of an upcoming video?
@@TNTsundar You mean the breadboards that generate the AI known as Ben Eater? 🙃
The actual servers aren’t that different, just a lot of modularized sheet metal chassises in big housing units with so, so many hard drives
Next video: "I built 30 breadboard computers so I could make my own internet... The Eaternet."
I must've watched a dozen videos about the Facebook outage, but somehow only Ben is able to explain it not only in a way that I can understand, but it actually seems simple, even though it absolutely isn't.
This is the best tech channel on UA-cam, no contest.
Except he didn't explain it, he talked about what the servers were doing and how the rest of the internet would know they were down and then said "oh and the reason was they typed something wrong"
I mean complexity is built on understand and learning the simple stuff. This still isn’t like rocket science or anything lol
You know a person is a good teacher if they can communicate clearly about DNS
The sound of Ben's keyboard is totally ASMR for a guy like me. Brings back lovely memories of my first PC back in the 90's.
yeah the clackity clack clack just sounds super comfy
Goes back further for me, at the keyboards for an IMSAI and a Polymorphics in the late 1970's, and the college SOROC terminals attached to a PR1ME mini in the mid 1980's.
what keyboard/switch he use?
@@izzurzuhri probably built it himself from breadboards and electrical switches. Each key is the size of a foot and he jumps on them
@@izzurzuhri sounds like clicky Alps to me, might be wrong though, a few other switches “ring” the same way
Ben, just found your channel and I have to say this is refreshing to see a DIY learning channel starting from the beginning... you keep learning strong... thank you for your time...
One of the best explanations of dns, routing, and BGP that I have seen.
I've been an internet professional since 1994, and i knew all this already, but your explanations are SO GOOD that i watched it just for fun. Great job!
Yea I spent years banging away on dn42 but this is BY FAR the best and most succint and insight-dense "class" on DNS and BGP I've ever seen.
He does give excellent explanations of things, he lays it out better than I've ever been able to a non-IT person.
The subnet mask description was so succinct that I'm using it internally when explaining networking to both our techs and clients.
not much changed since 90s, i mean the protocols. right?
Same. I took a grad networking systems course, Ben’s explanation of BGP is *so* much better than the course, I wish I had this then
Ben this is an amazing video, as a network engineer in the industry I always like to listen along to these kind of videos to refresh my knowledge and remind myself of ways in which to explain various content and this was perfect!
I was really thinking of making a video on the topic but this does the topic justice and I don't think I could add anything else to the topic!
Keep up the great work!
you know the video will be as informative as it possibly can when the tutor is building a GPU on a breadboard on his free time
Wow! That seems like a lot of work to set up the demo you did, but then again I don't happen to have access to a router connected to AT&T's live network 😏
How does he even have access to that?
@@hexagonist23 my guess is that his full time job is a network engineer at AT&T?
Maybe he rents a server there or installed his own? He has his own website that likely gets a lot of traffic.
@@Jeff-ss6qt but that wouldn't give him access to a top level route table for an AT&T router
To even be able to login to something like that is a huge deal
AT&T and several other ISPs host public route servers and/or a BGP looking glass where you can view their BGP routes. Their around to assist with troubleshooting multi-carrier internet routing issues. Maybe that’s what he’s connected to? But yeah, the demo he built is incredible.
I just had a class about this today and I understood it so much better in this video. Thank you so much for your work Ben
It says a lot about me when I still don't understand it
That's kind of what I was going to say - He boiled down probably 2 or 3 days of a CCNP class into 20 minutes or so.. Of course the "real" class would have confused you with all the other options you could put on those bgp routing statements, but it does make me think about maybe this video should be a prerequisite for diving into those sections of a network course. It's really good to see the basics working first!
It's a shame the education system is failing people, I didn't start truly learning until I dropped out of college. It's bizarre how much they charge for schools considering the word is Greek for "free time".
That's called applying knowledge. Next time, explain it to someone else. Try the Feynman method.
Isnt The Current Educational System Still For Factory Jobs?
Subscribing to your channel is one of the best decisions I've ever made in my life. I didn't even know anything about these and you explained it so smoothly. I guess I'm blessed to have a teacher like you. Keep up the good work!
You are one of the most truly educating people on UA-cam. Unlike so many others that are just so superficial with their details, mainly because they don't really understand the subject matter, you still down and give the real information. Now I understand DNS and how it works and it was just nice to see it properly explained. Would have been a bit nicer to have had an explanation about the DNS protocol, but this video is still an amazing explanation. Great work.
Hi Ben. I am a network engineer for an ISP and this was a VERY well done video that talked about the "house of cards" effect of the outage. Great work!
That outage just happened seconds after the update of my local Pi-Hole DNS instances, that was quite a head-scratching experience. Thanks a lot for those explanations.
I'm studying for the Network+ right now, and this video is the best explanation on these topics I have found anywhere. I could go on and on about how great your videos are, but mainly I just wanted you to know that I appreciate you sharing your knowledge with us and putting in the effort to make it easy to understand and fun to watch.
I must say, you're one of the best teachers out there, not only is your explanation simple and easy to understand, but it also lends itself to allowing people to learn on their own, rather than shoving information into our faces (thanks school). Whatever you do keep teaching, you're amazing.
everybody gangsta till Ben pulls out the heckin AT&T router
@Maxime St-Louis , Ben has ssh access to a router on the AT&T network, and he blurred out its IP address in the video. That is not public.
@Maxime St-Louis do u have access to an AT&T router? Not every Tom, Dick and Harry has it
@@MxSherwood Wasn't that just his home router?
@@HaraldSangvik No, he specifically said he's connected to an AT&T router happens to have access to and blurred out the hostname.
@@HaraldSangvik No, it's part of the actual for real AT&T network, the one that may suddenly stop working and cost millions of dollars if you type a wrong command
You are undeniably one of the greatest content creators on this platform.
I watched this when it was a fresher video, but I'm taking an introductory networking course this semester. While we haven't covered BGP yet, we _have_ talked about DNS, and so this actually made sense this time. Fascinating stuff, even for someone who doesn't really use Facebook
Great job explaining this by the way. It's extremely challenging to explain this in a simple way for everyone to grasp, but you manage to do it perfectly. Great work with the lab stuff too.
Now, THIS is how you explain things!! Foreword, introduction, background, examples and real-world demonstration... The best explanation for how the DNS architecture works (and how the rest of the Internet works lol)
Fantastic. I came for some more details on the outage than given in Facebook's statement and I got the best BGP tutorial I've ever seen. I'm more of a system guy myself, so local routing and DNS don't have much mystery left for me. But internet routing and especially BGP has always been kind of black magic to me. Thanks to you, I've got a grab of all this and the whole picture of that outage makes sense now.😁
Great video. Your simulated networks (and routers, I guess, apart from the AT&T one?) are impressive.
I'm just gonna keep it as Black Magic!
I have to say that the way you explain things in all of your videos by showing all the steps and distilling things into simple to understand chunks is really fantastic. If you were my college professor, you'd be my favorite.
Awesome visuals. As a fellow route/switch guy - I can't pass up a video with BGP in the title. About 7 years ago when I was starting my journey in networking, I watched nearly all of your network videos. The fundamental concepts that series covered set me on the right path.
Ben, I just wanted to say how much I enjoy watching your videos. I've learned breadboarding techniques, logic design, microcode and processor design, and hardware bit error detection and correction (among others). Then, when you create a video that is my wheelhouse, you do so in such a concise and informative fashion that I'd be hard pressed to do the same. You cover the fundamentals of Internet name resolution and route exchange in 30 minutes in what would normally be an intense week to month long seminar. I'm impressed with the level of detail you provide, along with the time it took to create a lab environment to demonstrate the operation of BGP routing.
I had not noticed the Facebook outage myself, but at the start of your video, I suspected it might have been a result of BGP route dampening due to network interface flapping, but that would have been for the entire AS and not just specific subnets. Your insight and use of BGP looking glass servers to demonstrate how specific routes were missing is remarkable. I don't have any particular insight as to how this occurred either, but perhaps Facebook is using an internal route reflector or confederation to handle routing Anycast traffic for DNS and ICMP. I don't have any experience with Facebook's network infrastructure, but this outage should hopefully help Facebook and other networks design and maintain more resilient networks.
Thanks again for all your time and effort in creating these videos. You have always been a favorite content creator of mine, and this latest video just adds to my adoration. I could tell you were running JunOS, but I'm curious if you were using GNS3 or some other simulator. In any case, kudos for your time and devotion to create such an informative video!
isnt the commit command a thing only on junos?
@@hariranormal5584 It's a thing for JunOS and Palo Alto firewalls (PanOS). Juniper devices also let you do a confirmed commit where if you lock yourself out by doing something dumb, it will automatically rollback to the previous configuration. Cisco devices typically commit changes immediately and so can cause issues if commands are entered incorrectly or in the wrong order. The only recourse then is to reboot the device and let the "startup-config" replace the running config. That's why you typically see commands like "copy running-config startup-config" or old school "write mem" to finalize a configuration.
I am in awe at how many views a video, that is in large part a Juniper router configuration demo, has. This has got to be the most viewed video that details Juniper router BGP configuration. Amazingly well done.
I have been studying for my Network+ Exam and am taking it on Monday. This video was so awesome to be able to relate some of what I am learning and actually see it in a real life situation to really get a better understanding of how it works rather than just reading about it. Awesome video! Thank you!
Ben, I'm no scholar at this, but again (as I say 'bout most everything you do) you'vedone a superb job of detailing intricate(unfamiliar) connections in a clear and lucid manner. Thanks!
Agreed, thank you!
Nice to see someone practically explaining dns lookup.Even I have never tried to do it
Thank you so much Eng. Ben for this pro explination, I think you have to do another one for what happen yesterday.
Ben you absolute legend, I had a good portion of this already figured out, but the way you explain it and glue it all together is just next level!
I feel more ignorant every time I watch one of his videos.
Same here.
Dig your feet in and don't give up.
What is Facebook? The only one I remember is one from the movie ARMY of DARKNESS or Ash v.s. The Evil Dead. 👻🙃
Haha, and this one is one of the most beginner-level videos he has around.
@@ZeroUm_ I'm big into his breadboard PC series but not into networking. They're very different, this may be beginner but still wack for someone not familiar
Thank you for this deep dive! I really appreciate it. I'm a computer professional, but my knowledge of networking is limited to the ip command, modifying /etc/network/interfaces, and cursing at systemd. I hadn't even heard of BGP before this kerfuffle. I watched a few other videos on the topic since the outage, but this one is far and away the best I've see. I feel like I actually understand what's going on. Thank you again.
Same. Best I can do is IWD and copy paste commands
@@12-343 same LOL. Networking is really hard :(
I've never managed to get my head around BGP until now (despite working with network engineers whose explanations just become white noise). Massive thanks!
Beautiful video. That's a whole lot of effort that went into demonstrating routing and BGP. Very clearly explained and easy to understand. Thank you for you effort.
Can you please do a video incorporating NAT. You honestly have explained subnets, dns, and ip better than anyone Ive ever seen.
that would fit into the explanation of this video quite well. I'll try, probably not as clear as Ben could do it:
His router has two addresses, one public IP address that is routable in the internet (i.e. his ISP advertises routes to that address via BGP) and one on his local network, that is not. Only his router knows what to do with addresses on that local network.
Now when he sends a packet from a host on his local network to the internet, it gets send to the router (the default gateway). What a NAT router does is now in addition to just passing the packet on to his ISP's router, is to slightly modify it. It replaces the source address (i.e. the 'return' address that the host on the receiving end should send its answer back to) with his own public IP address. It also stores the info of where it actually came from in a table in memory.
That way it looks to the host on the other end like the request was sent from the router's public IP address, not the local IP address of the host in Ben's network. Any reply will be sent back to that router's public address, that everyone knows how to route to.
Once the router receives a reply, it'll look up in his table where the request actually came from and rewrites the destination address for that packet back to the original local IP. Now it can route that reply packet via the local network, and sends it to that local address.
This allows one to use a big number of local addresses but still access the internet, even though the ISP gave you only one single publicly routable IP address. As Ben mentioned, IPv4 address space is limited (to about 4 Billion addresses), so NAT is an essential crutch to keep IPv4 working today, since that address space would otherwise be to small to address every device on the planet.
@@Hugh_I not bad, a little bit simple, since the NAT typically ends up rewriting the port number as well as the IP address, but that's a minor point compared to the rest of your explanation.
@@Hugh_I my favourite is when there’s two or three layers of NAT, so port-forwarding breaks because it’s only aware of its own internal NAT table, and not the layers beyond itself.
I really like your style of explaining stuff completely. That really helps me with understanding, most other people would gloss over "details" like the root DNS servers.
I love how you just casually answered my questions about the internet that have been absolutely boggling me my whole life and that I didn't know how to ask for
Damn this was really cool, I'm glad you took the time to explain this in such detail.
You're the only person I think who understands all the systems from the ground up.
Again, a great video. Never disappoints man!
This is a really good video.
I just finished watching your Networking playlist and would love more videos like this one focusing on setting up routers.
Amazing graphics! Made it real easy to follow as a fellow network engineer. Great explainations as well, I learned a lot about bgp
plot twist: it was not simulated network, Ben just was recording while repairing facebook xddd he just needed few days to do montage : D also without recording he would repair it much faster! thanks for vid Ben!
Ben took down Facebook just to create an explanation video.
That was a seriously intersting video, would love to see more stuff like this!
I like the nice illustrations
And your way of teaching, I always learn so much from your videos :)
Your work is great and you have an amazing way of showing and explaining things that is almost hypnotizing
My god, all this knowledge for free! Thank you Ben!
"Why was Facebook down for five hours?"
A better question is why was ot ever up in the first place.
Agreed, the time Facebook was down probably got families to actually talk with each other, and hopefully get a better relationship. I hope all these unsocial networks would cease to exist.
Technical question != interweb-social question
But agreed in general :)
@@TheLamefeed Actually, it made people crazy.
@@TheLamefeed back in my days we used sms instead of facetime
@@TheLamefeed pornhub usage increased massively 😂
Hats off @Ben Eater, you are a gifted man, the way you flow explaining all this is so smooth. I used to work as a Network Engineer but reached up to OSPF from the client-side, not ISP. I just learned a ton of useful aspects about BGP and DNS from the ISP Internet perspective. Cheers!
Hey Ben, Your videos are always a treasure trove of knowledge. You've inspired me a a lot. Keep it coming. Best wishes to you man.
Another amazing video! Thanks for sharing.
Wonderful explanation, thank you.
Could you in some future video show how did you prepare mock networks?
thanks so much Ben. Thanks for putting in all the work to make those simulated setups and walk through those commands, plus the work to show access to an actual router setup. We appreciate it!
The production value of this is just insane! Amazing job.
I learn a lot from your videos. May have to watch videos multiple times to get a firmer grasp on what you're saying, but my god have I learned a lot.
Also, I don't think Facebook is telling the truth about why it was down for 5 hours. Then again, I never fully trusted facebook from the beginning.
Your way of explaining things is incredible! I've worked in web hosting for about 6 years and now I really understand DNS :)
You're working for 6 in years in web hosting and you didnt know what dns was?
@@crit7_ for web hosting dns is a table like /etc/hosts :)
@@ilyadaemon plus you don’t need to know anything after the point where your web server network connects to your ISP!
Another very clear and well done video. Looks like I'll be working with BGP at work very soon, so this came at an opportune time!
Brilliant explanation! Tks for taking the time!
Amazing explanation.
One funny thing, I've always been saying IPv6 will not become mainstream, because no network engineer can make sense of the numbers. Its funny it was actually Ben's only reason not to use it.
I mean, for trying to explain something, it's impractical. But Copy-paste can be your friend. And it works in a fairly similar way to v4, far as I can tell.
This is one of the best videos that I had watched ! Subscribed. As to the reason, when software engineers stopped the access to data, backup procedure will be fast.
I've been waiting for the technical analysis, ty
If I were to guess, they have something roughly inspired by OpenFlow SDN internally to propogate config changes. Configs store things like DOS protection thresholds, routes to key services (likely algorithmically generated and pushed based on an understanding of machines and distances between them). I suspect a config change they are talking about damaged the route generation.
nohh. they just disconneced Mark Zuckerberg 10 years old laptop :)
@@myname377 fb hire this man right now
LOL -- yup. The SDN controller was running on Phil's workstation, because that's the IP address it was configured to use when they were first configuring the dev POC. And unbeknownst to anyone, including Phil, it never got changed. So, when Phil shut down his workstation to do an OS update, just two minutes before a scheduled schema replication, the entire controller cluster went offline and started dropping routes.
You know... or something like that. :-) If I've seen it once, I've seen it a thousand times.
@@nickwallette6201 damn you’ve genuinely seen that before? That’s depressing. I’ve had stuff like that happen with my NAS and various desktops, but to imagine professionals in a (public) company doing it… actually no, that makes perfect sense. The feet tend toward the well-trodden path and all that.
Thanks!
great video, very clear explanation and illustration. good work👍
@Ben Eater, I came here to see why it happened but I definitely leave this video with a good baggage of information that would probably took me waaay longer to get. Thanks! You've done a great job explaining not only what happened but also the theory behind it.
The fact that I missed the entire event says a lot about how much I value Facebook.
And WhatsApp and Instagram as well.
Only reason I knew about it is because my mom was flustered she couldn't reach me on Whatsapp... ugh
Only reason I know it is because I don't have the apps on my phone but I was checking if my professor answered my question in the private Facebook group we have for a lesson
My mom was crazy about accessing her cadle and resetted the internet modem because even if she understands its the first diagnostic step, she does not understand why and when to do it.
You still had internet, mom. Just not those websites.
i didn't even notice it. My friends were talking about it and thats how i learnt about it
"So I'm in the console on one of AT&T's routers that I just happen to have access to"
I love that he just casually glosses over that and none of us even bat an eye. "Yeah ofc he has access to it, why wouldn't he, nothing to see here"
Well now. Mr. Eater, thank you, thank you, thank you! Also, thank you! Clear, concise, full overview I will be reviewing and attempting to understand for years! I have been setting up networks for BAS for years, not often, but as a need would pop up. Struggling through it. First time I have had a complete view of the picture . I just had a look at your other videos…. More to learn. Thanks for that too. I do wish i could type even a quarter the speed you do.
Awesome! Thanks Ben. Your content is always making stuff easy to learn.
when you said AS number 7018 I immediately said AT&T. Ive peered with AT&T so many times now that their AS number is permanently engrained in my head. haha.
I do know 174 is cogent and 6939 is HE, and 1299 is telia, too much interest on the hosting community so that's just stuck on my head xD
@@hariranormal5584 Years of working in carrier networks and looking at BGP tables has me reading these like maps...It's funny how numbers stick inside the human brain.
I decided to look up who AS 1 is, and it's Level 3 these days. Ironically, they are not AS 3; that's MIT. They should broker a trade ;) (I used to work at 111 8th Ave, and Level 3 proudly had space on the 3rd floor. Always enjoyed that.)
@@JonathanRockway then they move to England and everything breaks x)
@@minirop is it because of break-xit ?
14:20 but do they know you have access? 🤔
imagine getting rce on an at&t router just make a youtube explainer
He could have a read-only account for auditing security, nothing he did on the live box needed rce.
Woww. Mind blown. Awesome explanation. Wish you could do these deep dives into current network related events. Thanks a billion for this video.
Loved how you went through all of this so fluidly. Well done. When I try to explain these things -- nope, now sending people to this video heh
Hey Ben what software do you use to simulate the routers and peering? Great explanation btw, pitty we don't get more insights from Facebook
I would like to know as well.
I assume since he has access to an at&t router that he works in the industry, which would mean he has access to disk images of the router operating system. Then it's just a matter of putting it in several VMs and connect them together, using the VM provider of your choice.
My guess would be either GNS3 or some vSRXs running in VMware or similar.
Logical systems are a possibility, but the interface names suggest one of the first two.
He is using a Juniper vMX virtual router.
Cisco Packet Tracer and GNS3 are the most commonly used ones
This guy went from looking at a routing table in his LAN to a BGP table on Tier 1 ISP's network within 10 seconds. Damn.
Nope...
That was awesome! Please don't stop making these super informative videos.
Great deep explanation, as always.
I did know it before, but now I know a little bit more.. 👍
Thanks Ben!
You are Inspiring, You are my Knowledge Angle, Bless me. Praneeth from India.
Angle?
@@khealer I think they meant "angel".
@@khealer Don't be surprised if tomorrow there is a temple dedicated to Ben in India.
"knowledge angle" ahahaha
Bridging Gap Protocol :) Basically the reason for the outage was a human error that was propagated to >1000s of Facebook's backbone BGP routers through the massive use of network automation and management from a single pane of glass! It goes to show that automation doesn't get rid of human error but lets it propagate at a large scale...Don't forget the basics and think thrice before you hit that push button :) Excellent BGP and DNS overview Ben!
I had to come back to comment. It's unbelievable how well you explained BGP. We touched the subject at school and I couldn't get it. Now I did.
Not that I'll ever need it, for reasons, but I did get it. Thanks!
Thank you Ben Eater for this valuable explanation. I also struggled in understanding BGP and this video explained it clearly as a highlevel explanation.
Hey Ben, Here in Australia today there was a nation wide outage for a major ISP and the outage lasted 9 hours. It seems that bgp was the culprit. It would be great if you could explain to us how, because in this case no dns was involved
For context, Ben Eater's expertise is actual computer networking. He speaks about his amazing journey on his podcast. He is absolutely brilliant and his story is inspiring.
Yeah, this is the subject he started out learning and his pro experience. The rest is just for fun.
Didn't understand a huge amount of it inherently but you did explain it very thoroughly and calmly! Excellent video!
I've been meaning to get around to trying to understand the basics of BGP for ages and now I do, thanks.
Thousands of outlets and channels covering the Facebook outage: zero knowledge and a clusterfuck of buzzwords.
Ben Eater covers the outage and ends up creating a video that (informally) explains DNS & BGP better than almost everyone else out there. This is in fact a perfect video now to show to someone who asks "how does the internet work".