We all know Linus' real endgame is having enough storage to finally upload his consciousness into a machine, then he can run LTT in perpetuity as a sentient AI that constantly drops database tables
It is always so great to see big name content creators/tech wizards like Jake Tech Tips and Anthony Media Group collaborate with young and upcoming screw driver enthusiasts like Linus.
As a server engineer, every time I watch Linus and Jake work on servers I cry a little inside. Please, for the love of your HR department and your own health, please purchase a server lifter. You spend big money on lots of things, this will save your backs and potentially Linus' body if he keeps choosing to lie under servers... I also love the change management Normal Company: I need to raise a change with 5 business days lead time with install and back out plans, passed by Change Advisory Board and approved by change manager\service control manager. Notice sent out to all impacted users. LTT: YO GUYS IM CHANGING STUFF, GET OFF THE DISKS!!!
To be fair they did give advance notice, they just went the extra step of walking around to say its happening now instead of the usual "your fault for not listening, your fault its now gone"
If Linus keeps going around like this, there might be the event of him dropping a server on any part of himself. In the event of the part of him that is crushed being his head, the outcome maybe really bad and the endgame part a bit too real. Technically the way they are doing things, the event of any of Jake's extremities being in peril is not to be excluded from the equation. That would be as bad, and a bit of a liability for the company, too.
I shudder to think of what would happen to me if I was doing this on PRODUCTION servers just on a whim like how its depicted here. Now I get it, a smaller operation like this isn't going to have Lab/Dev/QA/Prod environments, but to go basically from the lab to production and just "hope and pray" it works, I cry along side you. I sent one of these videos to our OH&S guy a few years back, I honestly think he had a stroke because he just sat there staring at the screen mouth agape and twitching the whole time. All humour and joking aside, they REALLY need to have better safety practices, every one of these videos is just a worker's comp agent's worst nightmare.
Hope Wendell can do a video on how he restored some of Linus's bit-rotted files. I've never made this mistake but it would be interesting to see the repair.
@@LiLBitsDK bit-rot is when the bits that make up your files are changed / corrupted over time. If data is just stored on a hard drive, it will degrade and you can lose the data. ZFS has something called "scrubbing" which checks this and fixes issues before they become unrecoverable. I believe LTT did not have scrubbing setup or it was setup incorrectly so they were not protected against bit-rot
You might be interested in this video ua-cam.com/video/tU0Wc6Kj6IQ/v-deo.html Wendell goes full computer janitor on recovering data from an old Enterprise SSD.
17:45 Linus be like: "Yeah let me just lay here under a sever with 60 drives in it that isn't mounted correctly and play with the mounting hardware. Not that it could just fall down and will splash my head."
Take some advice from a man that has had two back operations in that last year DO NOT LIFT LIKE THAT get help, wear a brace of use a telescoping cart or you will be sorry like I am. When I was your age and younger I did the same stuff and I am paying for it now BIG TIME?
That's what happens when you constantly choose a technology which is fundamentally not designed to be expanded (even if it with a lot of work can be but not designed for it). And discounting technology because "no one here knows how to set up and maintain it", and instead choose a different technology that ALSO clearly no one there knows how to set up and maintain and are learning bit by bit on the fly...
We only live physically once so we shouldn’t be afraid to do anything bro, i smoke weed on my UA-cam channel and i ate Mcdonald’s inside Walmart 😈😈, screw anyones opinion 🎯
This went smoothly compared to some jobs I've done. One rack was in a room so small that the 35kg servers had to be angled at more than 45 degrees just for the font of them to clear the wall and there was only space for one person to do so...from the side of the rack.
Might I recommend the acquisition of hydraulic table carts. Makes transferring heavy things like that from different heights much easier and safer for both the equipment and your staff.
2 роки тому+435
you know, when I worked in Inventec, we had lifting thingies for carrying servers, so you could just adjust height and slide even full 150gk 4u server in and out of test rack by yourself without help, no matter if it was almost on ground level or 2m high (it wasn't in regular racks, it was more like store shelves with rails, so putting in and pulling out servers was easy) I bet logistics guys would love this thing
I think Linus needs help, this whole storage thing becomes like an addiction. I don't think there's enough storage in the world to satisfy Linus' hunger for more.
Nah, if that were true he'd be investing in knowledge of Ceph. With a single head node, he knows what the Storage upper limit is. With a proper cluster, he could scale to dozens of PB
It sounds like a lot, but there's plenty of people who have something approaching (or exceeding) a petabyte at home as a hobby (so, not making money specifically with it)! He runs a business, so even if he wouldn't need quite _that_ amount of storage, a couple of petabytes is pretty reasonable, no?
honestly i love the server content more than the typical videos. because its legit just linus and jake spending a day nerding out and having a laugh while they do it. and this server stuff is far more interesting than another overpriced gaming pc
LMG guys: I am curious... this will allow you to have access to all those projects.. how do you know which project file to grab what desired clip from? What is a typical list of keyords/tags you might use for a given project and how does your search for relevant video get achieved? Is there any kind of AI invloved in creating metadata for future searches or is it all manual? That might make for an interesting video.. at least I'd watch it. Not a lot of hardware to it.. but the whole workflow thing is interesting and worthy of a video.
@@joshjlmgproductions3313 sure but often a simple video title doesn't tell you all the things in a video, especially when you have a specific product or clip in mind that wasn't the highlight of the video...
Having followed the new server videos for a while, I’m glad to see things get finalized. Although part of me wonders if Linus was partially inspired by the SSD-based server used for the new house.
Jake's job is so cool. He gets to design and build these amazing projects for Linus and spare no expense you normally would. With the house and everything he does at the office hes become my favorite face on LTT.
@@danieljonsson8095 as an enterprise IT Server Engineer, I cringe at everything Linus chooses for his server and network core solutions. It is obvious that Linus, nor anybody who works for him has ever worked in corporate/enterprise IT.
I think the perfect backup for your archives would be an automated tape storage room Edit : yeah tape writing and accessing is slow but it is just a backup for a very dark day
@@5dashes They intentionally (totally intentional, wasn't a mistake at all, we swear!) delete/corrupt all your data every 6 months to keep you on your toes and remind you that backups are important!
You need a hydraulic lifting table (jack). It's normally used to lift stuff onto the bed of a pickup truck, but seems like a perfect fit here too. They even have wheels so you can cart heavy stuff around.
They seriously better get around to getting some of that before anyone of them seriously injures themselves. Being fun and goofy is cool until people get hurt.
You really need a larger server room - something on the order of 5x larger. That one rack should be 3. You really need 5 feet in front and back of clear space for access, and then at least a half a rack for network appliances and patch panels. This would make working on the physical plant a whole lot easier.
By single point of failure, are you referring to the 'head' server? If that goes down, they will temporarily lose access to their data, but the data itself will still be safe, they'll just have to import the zpool on another server, same as they did on the new server in this video. So while there may be a single point of failure, I don't think it is inconsistent with the goal of avoiding losing data in the future.
@@Cyberguy42 A clean hardware failure is far from the worst case scenario. I've seen processors with incorrect AES s-boxes, silently mis-encrypting all data. I've seen memory bit flips lead to incorrect indirect jumps in the processor. Or power spikes (lightning) taking out multiple drives simultaneously. Or just simple software bugs. The list goes on.
@@jmickeyd53 I mean, you still need backups, yeah. I don't think they contested that? Doesn't Linus always try to throw at least one "RAID is not a backup!" into these videos?
@@jmickeyd53 For pure curiosity: How would your preferred setup look like? Because I don't see how two head servers will solve your mentioned issues from my basic infrastructure knowledge.
How does Linus not have a "proper" server room/rack by now? I've seen Cogen plants that have better server rooms. I get that he has absolutely insane SERVERS, but they'd probably last a lot longer with proper cooling, ventilation, spacing, etc.
@@shapshooter7769 he really doesn't though. That server room is way too small for the contents of that rack. If AC goes out or is underperforming, the smaller the room, the quick things burn out/up; hence the comment in the video about their UPS burning up in the past. Also, they should be insulated (both thermally and audibly) which they've kind of done once (based on past videos) but clearly not well since both sound and heat can leak out the vented door. I love Linus and especially all his server videos, I just figure by now he'd have a state of the art server room to match his BETTER than state of the art servers 🙂
It does look like where the janitor keeps his supplies after hours. LTT has enough square footage, I'd like to see something more 'bright tech clean-room' and less 'broom closet vibe' :) With how often they end up filming segments in there you'd think they'd want more space.
I agree, being able to walk in, have a few shelves for bits and bobs and being able to walk the whole way around the rack is night and day, probably not feasible due to the effort of terminating everything somewhere else though
Ok I'm gonna have an aneurysm from watching Linus laying under the server. Gah! Also, WHERE are your steel toes? Those sneakers, yea no. Maybe some basic safety precautions you guys? I mean at least they wore ear protection, sort of.
Yeah not a fan of those. I get the thumbnails, YT makes the rules, and however annoying they are you get shanked if you don't play by them, but it gets annoying when it spans the first minute of the video.
I love LTT, but I'm genuinely considering unsubbing with the childish tiktoky style choices they're making these days :( I just really grates on my nerves for some reason
Hey great work on these, I love seeing you guys work on datacenter hardware/software solutions ! Next logical upgrade : geo replication on those storinator (now that you have multiple buildings)? I think trueNAS has some implementation of gluster Georeplication in the SCALE version, so that could be an interesting thing to try out !
@@animefreak5757 Well, it depends. From what I remember, they're a few hundred meters apart (maybe 90-150m apart or smthg). It's definitely enough for fire to not spread across the two buildings, and it puts both storage cluster on different power circuit, which is probbly enough for a company the size of LMG.
13:30 always love when Linus & Jake "playing" around with the server, comes the time to shut the server down and had to hesitantly tell the editors. But this episode comes with a twist lol
17:37 when that happens, get a hex wrench of the right size and L it up in there. If you don't have enough leverage, just get a hex driver with the right tube wrench and make yourself an extra lever. CAUTION, if the rack is 100% full, you risk bending the leg. I usually grab the offending corner and then push the rail in. ALSO, i can't believe you put the rack directly on the floor. WTFrak. No shock mounts?
@@marc0523 also depends on where you are locating i suppose, if its a building with any possibility of having substantial vibrations shock mounts would make sense.
Linus: this Server, that needs four people to lift is not mounted correctly and only held in place by a piece not designed for heavy loads Also Linus: says that with his head directly beneath it
Reminds me of every experience I've had doing projects in IT, especially new SAN / storage migrations. There's a million little things that are slightly off, with many tasks that have to be finished to deal with the last lingering issues before everything is "right". About what you said on the zones, most people don't use these kinds of setups, and there are always issues that come up. Only vendor appliances are truly headache-free (Nimble / Pure arrays). They are the reason most enterprises don't use these kinds of setups, appliances are vastly more easy to standup and their support is worth its weight in gold--they are the definitive experts so we don't have to be.
@@Braindead154 I'm not so sure about that. I'm a bit of a data hoarder myself and every time people suggest me to use cloud to store my data, I redo the math and it always ends up being cheaper to just store it locally, even at my scale (under 100TB). For example, using an unofficial glacier calculator, with 3.6PB of data, just to store (not retrieve) it for a year, it's giving me a quote of 175K USD.. And if you are actively retrieving data, you soon end up adding tens of thousands of dollars - as an example, the retrieval cost for 20TB costs 25K USD and requires a minimum of 4 hours. Now, compare to their new server - it's probably close to 200K in hardware (too lazy to do the parts list math myself), but they get so much free stuff from sponsors that I'd be surprised if it ended up costing more than 50K for the build (they got most of the drives for free, got the RAM for free, etc). And, maintained properly, the yearly cost of operation will be much lower than any cloud storage (it's just power plus the occasional hard drive replacement) and you get unlimited retrieval for no additional cost (and it's much faster than 4 hours for 20TB).
They aren’t storing 3.6PB of data, they are storing a fraction of that and using raid for redundancy. You don’t have to do any of that with S3, durability is just there and to a much higher level. Even if they are getting paid to build with the equipment they use, I still hope that they have some offsite data storage.
@@ddognine ... with dedicated support teams who watch the health of the arrays over time, regular hardware maintenance and operations documentation in a trio of purpose-built datacenters, with proper fire suppression and spread across a large enough geographic area that any one minor disaster doesn't lose your data, instead of someones spare time between script reviews and a closet.
"Leftover" 30 TB NVME Drives from Dream's build. Man the stuff that "falls off a truck" at LMG in any given week is prolly worth more than my house. . .
I love when Linus and Jake are doing server stuff :) Also I love how they have an order of magnitude more TB of SSD storage than the capacity of my HDD based NAS :D
Damn dude, as someone who's lost poetry and artwork that only existed in digital form, I feel your pain on whatever wasn't recovered. People say, you can create it again... And it's like no.. you can't.. Glad y'all got the majority recovered. You guys are living my dream, lol not counting UA-cam stuff. Stay awesome
One benefit in their case is that the majority of the content is actually still on YT and can be downloaded, just in significantly reduced quality and organization.
I love how unassuming high end hardware gets. Those SAS cards look so much like some trash tier GPU. Green PCBs are like the beige boxes of circuit boards now
I think we forget sometimes that it's really only "enthusiasts" who care about the looks of anything inside their case (and not even all of them, either). Regular users never open their computers, and even if they do they don't expect anything "pretty". And businesses don't care either, do they? Both have no emotional connection to their hardware, and they don't "present" it to others or see it as decorative part of the room, either.
I work in IT with systems and other stuff that will crush any "enthusiast" nonsense. It's a completely different game where cost is no object and nobody gives two shits how things look. You're talking budgets where millions doesn't describe it, that's a budget error nobody cares about. A single rack costs millions. And you're talking multi acre data centers. The raw scale of it all would blow most peoples minds. Unless you work for a for real institution like Microsoft, Amazon, the military, facebook, you have no godly idea how stupid things get. It's all laughable. And while I don't directly do data center stuff now talking to people on headsets wandering around multi acre facilities while they have to use hear protection and wear winter gear is still a thing. And there are multiple facilities on that site. "We have the linux complex, the ibm AIX complex, the windows complex" and complex is multiple warehouse sized monstrosities. Walking into one is like walking into a jet engine that pumps cold air. Tons of the stuff is classified and even though you are cleared there are armed guards and you check your stuff at the enterance. Many ban smart devices completely as they can be used to trace movements, so leave in car or lose job.
Love the show and all you do! I gotta speak up here though. As a content creator and IT professional, managing this large of a redundant data set is going to grow in complexity as your data grows. Eventually, your internal staff will no longer be able to manage it on their own and you will need to hire a full time Tech just to monitor and maintain it. A tape archive system isn't as sexy and you're not going to have instant access to the raw content of your oldest videos [How often do you really need that?] but it's a hell of a lot cheaper and functional for decades. As you've seen in the past and as you will continue to see, having that instant access on a ginormous NAS costs a lot more than the hardware, eventually, it's going to cost you the salary of a dedicated Tech that does nothing else except manage that storage. Then again, you make bank at LTT so this might be something you're prepared to do but I'm already seeing the video title from LTT, 'Guys, I was wrong about the NAS, again, we lost everything!!!!...Maybe' video title in 2023 or 2024. 3.6 Petabytes isn't going to be nearly enough when you consider the steep ramp of camera tech and thus storage requirements.
Linus has got to be one of the only people to make a video building a new data server sound like it's going to be an epic tale that will one day be sung about. Also, he makes it all seem so easy but you can tell when they're talking about adding insulation and temps in the room, his hair at his forehead is soaking wet, meaning he's been sweating his ass off.
I would love a tutorial on how to turn multiple machines into a single server. I probably won't build something like this on a large scale, but it would be nice to have this knowledge in my back pocket for when I eventually expand my NAS to a server rack in the near future
I just set up a 4-machine high-availability Kubernetes/K3S cluster thanks to Techno Tim's videos on here. It's pretty cool to see an application running on one machine and then be able to move it over to another within a few seconds. YOu can even do it with Raspberry Pi's or old servers that companies are getting rid of.
This was a great video, but I am very hopeful that when LTT labs becomes a functioning channel, you can literally do videos about all the nitty-gritty tech stuff and like how to use the operating system and how to set up everything in detail so we can learn how to do it.
Based on my understanding, the point of labs is to generate new information, to answer questions that haven't been reliably answered before. There are already plenty of tutorials on setting up and using a basic Truenas system, so IMO there's no need for videos about that. However, if there are questions without clear answers (for example, which nitty gritty settings are best for certain use cases), then videos from labs that answer those questions would be helpful.
Just a thought, maybe try using MLV (mass loaded vinyl) to try and cut down on the noise escaping from the server room. Companies like Audimute sell it in rolls. Using something like mineral wool or acoustic foam isn't going to do much for stopping sound transmission.
Because of LTT I started to build my own NAS with Truenas Core and tried to Backup anything that I can't recover from the Cloud. It's not easy because you can buy only this much as a Teenager but soon I can store the Backups on another place, so that I much safer from getting a data loss than the average citizen.
@Ian Visser I only backup important document and photos but I don´t want to lose these, but the rest is in the cloud or not important enough so..... And I have not nearly this amount of raw data, must of the things on my PC are games that are already in the cloud.
Every time you post a video about your storage servers (which is a lot of videos now) it's just midly infuriating knowing that you could use tape drives which would be much cheaper, much more sustainable and probably more reliable. What's more, I literally learned about those 12TB tapes from you guys. I get it, you can't really milk them as much, since there probably aren't many tape sponsors, and buying a bunch of tapes isn't much of a video, but it's still annoying.
5:20 Most modern operating systems will automatically use read caching for most file systems. This is not a special feature of ZFS, I'm 100% sure Linux does this by default for most filesystems. Basically any unused RAM is used for read caching. Empty RAM is wasted RAM, so it usually won't actually be empty.
Haven't they been using 8k cameras for many projects for quite a while? Nothing like recording /storing stuff in 8K so that folks can watch it in 1080P (1/16th it's original res)
Should not be using a single card for a JBOD. They should be balanced across all three cards, so if one card dies, you still have connectivity to your storage arrays despite degraded
We all know Linus' real endgame is having enough storage to finally upload his consciousness into a machine, then he can run LTT in perpetuity as a sentient AI that constantly drops segways to our sponsor.
I don't know if I'm right, but it feels like your editors all have slightly different styles that show up in the videos they edit. I really enjoy the variation.
We all know Linus' real endgame is having enough storage to finally upload his consciousness into a machine, then he can run LTT in perpetuity as a sentient AI that constantly drops database tables
inb4 his DROP TABLE permission is revoked :P
Next LTT video idea?
Portal style?
WHAT THE FUCK LOL THIS IS FUNNY
Ah I see what you did there😅
With the rate that Linus is going, he'll make a seperate channel talking about all his servers
Great, that's what I care about the most
I would subscribe to that instantly.
I would subscribe to that past century.
i’d binge watch it
“Linus Linux Tips”
It is always so great to see big name content creators/tech wizards like Jake Tech Tips and Anthony Media Group collaborate with young and upcoming screw driver enthusiasts like Linus.
Definitely one of the, if not THE, most underrated comment in this thread!
Well, you could say Linus has a talent to... Screw things up....
As a server engineer, every time I watch Linus and Jake work on servers I cry a little inside.
Please, for the love of your HR department and your own health, please purchase a server lifter. You spend big money on lots of things, this will save your backs and potentially Linus' body if he keeps choosing to lie under servers...
I also love the change management
Normal Company: I need to raise a change with 5 business days lead time with install and back out plans, passed by Change Advisory Board and approved by change manager\service control manager. Notice sent out to all impacted users.
LTT: YO GUYS IM CHANGING STUFF, GET OFF THE DISKS!!!
Absolutely true and absolutely NECESSARY to start good business practices as his business is growing.
To be fair they did give advance notice, they just went the extra step of walking around to say its happening now instead of the usual "your fault for not listening, your fault its now gone"
If Linus keeps going around like this, there might be the event of him dropping a server on any part of himself. In the event of the part of him that is crushed being his head, the outcome maybe really bad and the endgame part a bit too real. Technically the way they are doing things, the event of any of Jake's extremities being in peril is not to be excluded from the equation. That would be as bad, and a bit of a liability for the company, too.
The same thought, i got, when i watched this video... why dont they buy some kind of forklift for handling the Servers in their rack 😓
I shudder to think of what would happen to me if I was doing this on PRODUCTION servers just on a whim like how its depicted here. Now I get it, a smaller operation like this isn't going to have Lab/Dev/QA/Prod environments, but to go basically from the lab to production and just "hope and pray" it works, I cry along side you.
I sent one of these videos to our OH&S guy a few years back, I honestly think he had a stroke because he just sat there staring at the screen mouth agape and twitching the whole time.
All humour and joking aside, they REALLY need to have better safety practices, every one of these videos is just a worker's comp agent's worst nightmare.
Hope Wendell can do a video on how he restored some of Linus's bit-rotted files. I've never made this mistake but it would be interesting to see the repair.
bit rotted?
@@LiLBitsDK bit-rot is when the bits that make up your files are changed / corrupted over time.
If data is just stored on a hard drive, it will degrade and you can lose the data.
ZFS has something called "scrubbing" which checks this and fixes issues before they become unrecoverable.
I believe LTT did not have scrubbing setup or it was setup incorrectly so they were not protected against bit-rot
@@Steven-d4c it was not a bit rot issue...it was a totaly missconfigured setup...
You might be interested in this video ua-cam.com/video/tU0Wc6Kj6IQ/v-deo.html Wendell goes full computer janitor on recovering data from an old Enterprise SSD.
he did mention it a few times . the server build series might be helpful as well. Wendell covers a lot of topics including software and backups.
17:45 Linus be like:
"Yeah let me just lay here under a sever with 60 drives in it that isn't mounted correctly and play with the mounting hardware. Not that it could just fall down and will splash my head."
Poor linus, He has to lay in a sewer just to make ends meet my condolonces 🙏
@@TheFalseShepphard Ikr poor linus and his very overpriced Sewer system
he would be just fine, yes it is heavy but he would still be fine
@@LiLBitsDK 60 drives would make a jbod weigh 210 lbs or so idk
@@LiLBitsDK actually in the video it says 290 lbs and that may be when empty
Take some advice from a man that has had two back operations in that last year DO NOT LIFT LIKE THAT get help, wear a brace of use a telescoping cart or you will be sorry like I am. When I was your age and younger I did the same stuff and I am paying for it now BIG TIME?
Can't wait for the next "endgame" server next year.
We all know how this story goes 😂
That's what happens when you constantly choose a technology which is fundamentally not designed to be expanded (even if it with a lot of work can be but not designed for it). And discounting technology because "no one here knows how to set up and maintain it", and instead choose a different technology that ALSO clearly no one there knows how to set up and maintain and are learning bit by bit on the fly...
We only live physically once so we shouldn’t be afraid to do anything bro, i smoke weed on my UA-cam channel and i ate Mcdonald’s inside Walmart 😈😈, screw anyones opinion 🎯
I give them 9 months
He just needs a separate server for server videos
I love how professional these videos are… right up until they start installing things
Belive me, it's sometimes more near to reality how they do things then one would think. :D
Can confirm. It’s all good until someone opens the debug console, or grabs a screwdriver. Then it’s all asses and elbows
This went smoothly compared to some jobs I've done. One rack was in a room so small that the 35kg servers had to be angled at more than 45 degrees just for the font of them to clear the wall and there was only space for one person to do so...from the side of the rack.
Might I recommend the acquisition of hydraulic table carts. Makes transferring heavy things like that from different heights much easier and safer for both the equipment and your staff.
you know, when I worked in Inventec, we had lifting thingies for carrying servers, so you could just adjust height and slide even full 150gk 4u server in and out of test rack by yourself without help, no matter if it was almost on ground level or 2m high (it wasn't in regular racks, it was more like store shelves with rails, so putting in and pulling out servers was easy)
I bet logistics guys would love this thing
Yeah, I don't know why they don't have something like that - that would make it a ton easier and safer.
@@invisi1407 . Probably don’t mover servers that much.
They just need a lifting cart.
He likes Jake's help.
Yeah, any kind of rolling scissor jack.
I've used them for automotive stuff, like lifting transmissions into vehicles.
Love you too
@Jonathan Z they will. Wendell tease it on twitter.
I think Linus needs help, this whole storage thing becomes like an addiction. I don't think there's enough storage in the world to satisfy Linus' hunger for more.
haha. You should talk to a Google employee. Linus' data hoarding is childs play.
Haha my starting home nas is 40TB, i don't even know what I'm gonna use it for yet. But data aggregation is almost exponential.
Linus is the IT equivalent of Galactus confirmed
Nah, if that were true he'd be investing in knowledge of Ceph.
With a single head node, he knows what the Storage upper limit is.
With a proper cluster, he could scale to dozens of PB
It sounds like a lot, but there's plenty of people who have something approaching (or exceeding) a petabyte at home as a hobby (so, not making money specifically with it)! He runs a business, so even if he wouldn't need quite _that_ amount of storage, a couple of petabytes is pretty reasonable, no?
ZFS for the win! Great pick guys! This time we know your data is safe (unless someone drops a server)!
unless? - until.
@@Nalianna fr
I can't wait for Linus to create a new channel just to show off his backup servers. He could call it Linus Backup Tips?
More like "Linus Backup and Restore Tips"
As long as it's not "Linus backs his tip up", 'cause that's not exactly UA-cam material.
Linus Back Tips
Linus' Back That Thing Up, Just the Tip, and Only for a Second!
Linus server tips
Linus: The weight of this heavy server isn’t being supported
Also Linus: Yeah I’m just gonna rest my head under it.
My thoughts exactly, and he was poking at the support.
OSHA is a privilege
He had on protective headwear at least?
I love how linus just casually lies directly under extremely heavy servers which are poorly held in place.
honestly i love the server content more than the typical videos. because its legit just linus and jake spending a day nerding out and having a laugh while they do it. and this server stuff is far more interesting than another overpriced gaming pc
LMG guys: I am curious... this will allow you to have access to all those projects.. how do you know which project file to grab what desired clip from? What is a typical list of keyords/tags you might use for a given project and how does your search for relevant video get achieved? Is there any kind of AI invloved in creating metadata for future searches or is it all manual? That might make for an interesting video.. at least I'd watch it. Not a lot of hardware to it.. but the whole workflow thing is interesting and worthy of a video.
They probably date their videos and name them the same name as the UA-cam upload.
@@joshjlmgproductions3313 or the internal name for the project, rather than the clickable title on UA-cam
they could also search for a video on youtube and then have more info about raw material, if there is some naming convection consistency
@@joshjlmgproductions3313 sure but often a simple video title doesn't tell you all the things in a video, especially when you have a specific product or clip in mind that wasn't the highlight of the video...
pretty sure this is only archival, not active projects, so they probably don't go back to it very often
Having followed the new server videos for a while, I’m glad to see things get finalized. Although part of me wonders if Linus was partially inspired by the SSD-based server used for the new house.
Or the million dollar all SSD server that they made a video of and then completely forgot about.
@@jackhemsworth7515 its not optimal to run this many ssds, a million for a server is not cheap
@@jackhemsworth7515 they had to return the parts to the sponsor, it wasn't meant to be a stable thing
Jake's job is so cool. He gets to design and build these amazing projects for Linus and spare no expense you normally would. With the house and everything he does at the office hes become my favorite face on LTT.
I wonder if this is just the way Linus deals with his own mortality, and the need for permanence in time
Even so, we all will eventually be forgotten, no matter what we do.
It’s similar to how i do it.
@bruh I can only guess that's the final point, to make a copy someday and for it to be tax deductible hehehehehe
As someone that manages enterprise IT equipment at an MSP, I love these videos about your backend infrastructure.
Don't use LTT as a source for good enterprise IT management... Just don't... You're going to be in for a rough time if you do.
16:45 For the others who enjoy Linus' backend
@@danieljonsson8095 It wouldn't be half as entertaining otherwise lol
@@HilbertXVI Certainly. I'm just saying, be well aware that LTT is an ENTERTAINMENT channel, not "good practices" channel.
@@danieljonsson8095 as an enterprise IT Server Engineer, I cringe at everything Linus chooses for his server and network core solutions.
It is obvious that Linus, nor anybody who works for him has ever worked in corporate/enterprise IT.
I would finally be able to install my entire steam library with a setup like this.
3 games max
I think the perfect backup for your archives would be an automated tape storage room
Edit : yeah tape writing and accessing is slow but it is just a backup for a very dark day
VHS or Beta?
@@donc-m4900 fish tape
As a backup yes, for reading 100% NO
900,000 feet of tape!
he would probably need LTO10. Even then... could you imagine trying to backup all that data he has?
If there is ever a shortage of hard drives, we know Linus has bought them all.
Can I just say how much I LOVE videos with Jake and Linus, you can tell these 2 are bros and genuinely enjoy their time together
“It’s LeviosAAAAH”
That reference did not go to waste. 11/10
3 years later...
Linus: We bought another unit, and we're turning it into the LMG Data Center!
AWS competitor -- Linus Web Services
@@5dashes They intentionally (totally intentional, wasn't a mistake at all, we swear!) delete/corrupt all your data every 6 months to keep you on your toes and remind you that backups are important!
270? Naw…. Go absolutely insane…1,200 hard drives at the maximum amount of storage per disk possible. THAT’S game over!
It’s your endgame… for now, you’re gonna have a new server by 2023 lol
If they do it right, this should be extendable by updating the "head" servers or adding more arrays.
I love that Jake knows he's in the untouchable zone, ordering food not only infront of the bossman but on video
Jake is Linus' adopted son, he is untouchable
And insists the editor leaves it in! :D
ive noticed an inversion of the early Jake/Linus dynamic, nowdays it feels like Jake is the adult and Linus is the child.
This is why Linus hires employees - so he is free to act like a child in camera.
Which is of course, content.
I absolutely love their dynamic, and I’m constantly impressed at Jake’s knowledge and skills.
"Endgame"
we all know a "bigger, better, and faster" one is coming in a few months for Lab 2.
You need a hydraulic lifting table (jack). It's normally used to lift stuff onto the bed of a pickup truck, but seems like a perfect fit here too. They even have wheels so you can cart heavy stuff around.
They make ones specifically for loading racks.
Yep, can get them in a veritable sizes
Or use Jake and another guy instead...
See my comment earlier: battery powered server lift. I installed an entire datacenter by myself with it!
They seriously better get around to getting some of that before anyone of them seriously injures themselves. Being fun and goofy is cool until people get hurt.
You really need a larger server room - something on the order of 5x larger. That one rack should be 3. You really need 5 feet in front and back of clear space for access, and then at least a half a rack for network appliances and patch panels. This would make working on the physical plant a whole lot easier.
I don't think they actually have enough need for that. It'd make sense if they were doing more than basically meming with the server room.
Linus: We almost lost all of our data. Also Linus: We're going to build a new storage system with a single point of failure.
By single point of failure, are you referring to the 'head' server? If that goes down, they will temporarily lose access to their data, but the data itself will still be safe, they'll just have to import the zpool on another server, same as they did on the new server in this video. So while there may be a single point of failure, I don't think it is inconsistent with the goal of avoiding losing data in the future.
@@Cyberguy42 A clean hardware failure is far from the worst case scenario. I've seen processors with incorrect AES s-boxes, silently mis-encrypting all data. I've seen memory bit flips lead to incorrect indirect jumps in the processor. Or power spikes (lightning) taking out multiple drives simultaneously. Or just simple software bugs. The list goes on.
@@jmickeyd53 I mean, you still need backups, yeah. I don't think they contested that? Doesn't Linus always try to throw at least one "RAID is not a backup!" into these videos?
@@jmickeyd53 For pure curiosity: How would your preferred setup look like? Because I don't see how two head servers will solve your mentioned issues from my basic infrastructure knowledge.
@@ray89520 Ceph cluster would have been better immho.
How does Linus not have a "proper" server room/rack by now? I've seen Cogen plants that have better server rooms. I get that he has absolutely insane SERVERS, but they'd probably last a lot longer with proper cooling, ventilation, spacing, etc.
But he does though.
it IS a proper server room, albeit small AND it is a professional rack... so what are you talking about?
@@shapshooter7769 he really doesn't though. That server room is way too small for the contents of that rack. If AC goes out or is underperforming, the smaller the room, the quick things burn out/up; hence the comment in the video about their UPS burning up in the past. Also, they should be insulated (both thermally and audibly) which they've kind of done once (based on past videos) but clearly not well since both sound and heat can leak out the vented door.
I love Linus and especially all his server videos, I just figure by now he'd have a state of the art server room to match his BETTER than state of the art servers 🙂
It does look like where the janitor keeps his supplies after hours. LTT has enough square footage, I'd like to see something more 'bright tech clean-room' and less 'broom closet vibe' :)
With how often they end up filming segments in there you'd think they'd want more space.
I agree, being able to walk in, have a few shelves for bits and bobs and being able to walk the whole way around the rack is night and day, probably not feasible due to the effort of terminating everything somewhere else though
I love Jake. LTT just has such an amazing crew.
Ditto. He is clearly so knowledgable, continues to amaze me what he knows and the broad spectrum of things he knows and is capable of!
Yeah. He's like a super chatty LAN admin. You don't see those very often. It's fun.
Can you run DOOM in it tho? And Crysis? And Minecraft?
18:14 I love how enthusiastic Linus promotes their screwdriver like "EASY!" while Jakes knee is almost getting crushed by the weight of the JBOD 😂
Then the sponsor is for a competitors screwdriver 🤣
I'm a scientist at a multi-million dollar tech company and Linus has a better IT infrastructure than we do.
I'm a gamer!
@@djhokage1 I'm a data scientist at a cybersecurity company so its all IT but okay sure.
To be quite fair, they're shooting 8K/ 8K HDR content which are LARGE files + they're also a multi million dollar company hahah
@@valismind Yeah 1000p. Their data needs are beyond massive. This kind of content is half the reason I subscribe.
Which company
Ok I'm gonna have an aneurysm from watching Linus laying under the server. Gah!
Also, WHERE are your steel toes? Those sneakers, yea no. Maybe some basic safety precautions you guys? I mean at least they wore ear protection, sort of.
LTT video editor: "How many flashy text fields do you want? How big shall they be?"
LTT marketing guy: "Yes."
Yeah it's gotten worse over the past few months, more clickbaity and seems to treat their audience as children or preteens
Yeah not a fan of those. I get the thumbnails, YT makes the rules, and however annoying they are you get shanked if you don't play by them, but it gets annoying when it spans the first minute of the video.
I love LTT, but I'm genuinely considering unsubbing with the childish tiktoky style choices they're making these days :(
I just really grates on my nerves for some reason
@@Shanghaimartin same
Go pay for Floatplane then.
Hey great work on these, I love seeing you guys work on datacenter hardware/software solutions ! Next logical upgrade : geo replication on those storinator (now that you have multiple buildings)? I think trueNAS has some implementation of gluster Georeplication in the SCALE version, so that could be an interesting thing to try out !
the two buildings he has are VERY close together. I don't think that's what your usually going for with geo replication.
@@animefreak5757 Well, it depends. From what I remember, they're a few hundred meters apart (maybe 90-150m apart or smthg). It's definitely enough for fire to not spread across the two buildings, and it puts both storage cluster on different power circuit, which is probbly enough for a company the size of LMG.
13:30 always love when Linus & Jake "playing" around with the server, comes the time to shut the server down and had to hesitantly tell the editors.
But this episode comes with a twist lol
17:37 when that happens, get a hex wrench of the right size and L it up in there. If you don't have enough leverage, just get a hex driver with the right tube wrench and make yourself an extra lever. CAUTION, if the rack is 100% full, you risk bending the leg. I usually grab the offending corner and then push the rail in.
ALSO, i can't believe you put the rack directly on the floor. WTFrak. No shock mounts?
I've never seen shock mounts, is it possible you live in an earthquake zone and that's why you have them?
@@marc0523 also depends on where you are locating i suppose, if its a building with any possibility of having substantial vibrations shock mounts would make sense.
Only place I have seen racks with shock mounts is on ship installs.
It's not like Vancouver never gets earthquakes, either.
Linus: this Server, that needs four people to lift is not mounted correctly and only held in place by a piece not designed for heavy loads
Also Linus: says that with his head directly beneath it
At this rate, maybe Linus will make his own cloud storage services available to the public.
Reminds me of every experience I've had doing projects in IT, especially new SAN / storage migrations. There's a million little things that are slightly off, with many tasks that have to be finished to deal with the last lingering issues before everything is "right". About what you said on the zones, most people don't use these kinds of setups, and there are always issues that come up. Only vendor appliances are truly headache-free (Nimble / Pure arrays). They are the reason most enterprises don't use these kinds of setups, appliances are vastly more easy to standup and their support is worth its weight in gold--they are the definitive experts so we don't have to be.
For their use case, they should just toss it in a Amazon S3 Glacier bucket and call it a day… would save him so much money and time.
@@Braindead154 I'm not so sure about that. I'm a bit of a data hoarder myself and every time people suggest me to use cloud to store my data, I redo the math and it always ends up being cheaper to just store it locally, even at my scale (under 100TB).
For example, using an unofficial glacier calculator, with 3.6PB of data, just to store (not retrieve) it for a year, it's giving me a quote of 175K USD.. And if you are actively retrieving data, you soon end up adding tens of thousands of dollars - as an example, the retrieval cost for 20TB costs 25K USD and requires a minimum of 4 hours.
Now, compare to their new server - it's probably close to 200K in hardware (too lazy to do the parts list math myself), but they get so much free stuff from sponsors that I'd be surprised if it ended up costing more than 50K for the build (they got most of the drives for free, got the RAM for free, etc). And, maintained properly, the yearly cost of operation will be much lower than any cloud storage (it's just power plus the occasional hard drive replacement) and you get unlimited retrieval for no additional cost (and it's much faster than 4 hours for 20TB).
@@Braindead154 There is no such thing as the cloud, just someone else's computer.
They aren’t storing 3.6PB of data, they are storing a fraction of that and using raid for redundancy. You don’t have to do any of that with S3, durability is just there and to a much higher level. Even if they are getting paid to build with the equipment they use, I still hope that they have some offsite data storage.
@@ddognine ... with dedicated support teams who watch the health of the arrays over time, regular hardware maintenance and operations documentation in a trio of purpose-built datacenters, with proper fire suppression and spread across a large enough geographic area that any one minor disaster doesn't lose your data, instead of someones spare time between script reviews and a closet.
"Leftover" 30 TB NVME Drives from Dream's build. Man the stuff that "falls off a truck" at LMG in any given week is prolly worth more than my house. . .
most of it ends up at linuses house as well i reckon
5:49 Linus is getting "high" performances
I love when Linus and Jake are doing server stuff :) Also I love how they have an order of magnitude more TB of SSD storage than the capacity of my HDD based NAS :D
Linus: "Hard drives are dead"
Also Linus: "I bought 270 hard drives"
I just hear Linus saying the Kylo Ren *MORE* meme 🤣
Time to get a bigger server room
I love seeing the behind the scenes of LMG, it was hilarious seeing the logistics banter
Server and Moving Vlogs are still the best. Watching since 'whole room water cooling' 😁
Now you just need all those drives to be those Exobyte SSD drives for the largest SSD storage you could ever get
I love the Server projects especially with AMD. They are epyc.
That is the art of Linus (and Jake to be fair) topic was ok but they made it SO engaging and fun to watch!!
Amo gus
Real and True
Very
same
Amo Gus 😔
Based and sus
Damn dude, as someone who's lost poetry and artwork that only existed in digital form, I feel your pain on whatever wasn't recovered. People say, you can create it again... And it's like no.. you can't..
Glad y'all got the majority recovered. You guys are living my dream, lol not counting UA-cam stuff. Stay awesome
One benefit in their case is that the majority of the content is actually still on YT and can be downloaded, just in significantly reduced quality and organization.
I love how unassuming high end hardware gets. Those SAS cards look so much like some trash tier GPU. Green PCBs are like the beige boxes of circuit boards now
I think we forget sometimes that it's really only "enthusiasts" who care about the looks of anything inside their case (and not even all of them, either). Regular users never open their computers, and even if they do they don't expect anything "pretty". And businesses don't care either, do they? Both have no emotional connection to their hardware, and they don't "present" it to others or see it as decorative part of the room, either.
I work in IT with systems and other stuff that will crush any "enthusiast" nonsense. It's a completely different game where cost is no object and nobody gives two shits how things look. You're talking budgets where millions doesn't describe it, that's a budget error nobody cares about. A single rack costs millions. And you're talking multi acre data centers. The raw scale of it all would blow most peoples minds.
Unless you work for a for real institution like Microsoft, Amazon, the military, facebook, you have no godly idea how stupid things get. It's all laughable. And while I don't directly do data center stuff now talking to people on headsets wandering around multi acre facilities while they have to use hear protection and wear winter gear is still a thing. And there are multiple facilities on that site. "We have the linux complex, the ibm AIX complex, the windows complex" and complex is multiple warehouse sized monstrosities. Walking into one is like walking into a jet engine that pumps cold air. Tons of the stuff is classified and even though you are cleared there are armed guards and you check your stuff at the enterance. Many ban smart devices completely as they can be used to trace movements, so leave in car or lose job.
Love the show and all you do! I gotta speak up here though. As a content creator and IT professional, managing this large of a redundant data set is going to grow in complexity as your data grows. Eventually, your internal staff will no longer be able to manage it on their own and you will need to hire a full time Tech just to monitor and maintain it. A tape archive system isn't as sexy and you're not going to have instant access to the raw content of your oldest videos [How often do you really need that?] but it's a hell of a lot cheaper and functional for decades. As you've seen in the past and as you will continue to see, having that instant access on a ginormous NAS costs a lot more than the hardware, eventually, it's going to cost you the salary of a dedicated Tech that does nothing else except manage that storage. Then again, you make bank at LTT so this might be something you're prepared to do but I'm already seeing the video title from LTT, 'Guys, I was wrong about the NAS, again, we lost everything!!!!...Maybe' video title in 2023 or 2024. 3.6 Petabytes isn't going to be nearly enough when you consider the steep ramp of camera tech and thus storage requirements.
Linus has got to be one of the only people to make a video building a new data server sound like it's going to be an epic tale that will one day be sung about. Also, he makes it all seem so easy but you can tell when they're talking about adding insulation and temps in the room, his hair at his forehead is soaking wet, meaning he's been sweating his ass off.
When "Leftovers" from one Person is better than everything you ever had, combined.
Clever way of showing “logistics” which has been mentioned many times but hasn’t been seen (by me) until now.
Random thought: I'm trying to picture Linus 40 years from now, will he still be making videos or will he pass the role onto his kids?
Undoubtedly there will be something on a New, New, (x25) Whonnock....*whatever the hell whonnock is supposed to mean anyway*...
15:25 Linus meant a Firewall
Imagine waking up in the morning and his new video ITS HERE
Me: wow
Can’t forget the fire in the old old server rooms and how many you guys had it was crazy but did a very nice recovery on that one
I would love a tutorial on how to turn multiple machines into a single server. I probably won't build something like this on a large scale, but it would be nice to have this knowledge in my back pocket for when I eventually expand my NAS to a server rack in the near future
I just set up a 4-machine high-availability Kubernetes/K3S cluster thanks to Techno Tim's videos on here. It's pretty cool to see an application running on one machine and then be able to move it over to another within a few seconds. YOu can even do it with Raspberry Pi's or old servers that companies are getting rid of.
That transition a few seconds after 9:37 💯
I mean it's a pretty basic slide transition lol, nothing crazy tbh
18:50 "Breaking: CEO crushed under too much server"
Linus really has is own full scale data center
Sometimes I think Linus' company is worryingly inefficient, but then I go to work
Then you look back at Linus and his team and go "You know what, those guys are doing a great job! I wish my company was more like theirs."
This was a great video, but I am very hopeful that when LTT labs becomes a functioning channel, you can literally do videos about all the nitty-gritty tech stuff and like how to use the operating system and how to set up everything in detail so we can learn how to do it.
Based on my understanding, the point of labs is to generate new information, to answer questions that haven't been reliably answered before. There are already plenty of tutorials on setting up and using a basic Truenas system, so IMO there's no need for videos about that. However, if there are questions without clear answers (for example, which nitty gritty settings are best for certain use cases), then videos from labs that answer those questions would be helpful.
Just for those wondering, they make a cart that also sissor lifts the server to move it into the rack...
“It’s LeviosAAAAH”
That reference did not go to waste. 11/10
STOP STOP you're doing it WRONG. It's levOOOOOsah not LeviosAHHH.
@@TheOrangeMelons I was actually thinking of Oney’s video… oops…
@@DarkSideKyp it's Levios-AUUHHHH
I actually heard that in Hermione's voice... :)
Oh stop it Linus!
Just a thought, maybe try using MLV (mass loaded vinyl) to try and cut down on the noise escaping from the server room. Companies like Audimute sell it in rolls. Using something like mineral wool or acoustic foam isn't going to do much for stopping sound transmission.
Linus used cavity wall insulation (recycled jeans) instead of actual sound dampening materials when he set up the room.
The issue with vinyl is that if it gets heated beyond a certain point it combusts
Thank you for always putting your sponsors in the actual video so I can skip the shit out of them.
Because of LTT I started to build my own NAS with Truenas Core and tried to Backup anything that I can't recover from the Cloud. It's not easy because you can buy only this much as a Teenager but soon I can store the Backups on another place, so that I much safer from getting a data loss than the average citizen.
@Ian Visser I only backup important document and photos but I don´t want to lose these, but the rest is in the cloud or not important enough so..... And I have not nearly this amount of raw data, must of the things on my PC are games that are already in the cloud.
Every time you post a video about your storage servers (which is a lot of videos now) it's just midly infuriating knowing that you could use tape drives which would be much cheaper, much more sustainable and probably more reliable. What's more, I literally learned about those 12TB tapes from you guys. I get it, you can't really milk them as much, since there probably aren't many tape sponsors, and buying a bunch of tapes isn't much of a video, but it's still annoying.
Ahh the 2022 Q3 mass storage server video.
How many storage endgames will this man have, it feel like yesterday he made his last massive server
I just love how Linus is being all serious and Jake is more interested in when his burrito will arrive 🤣
There are nearly 6000 videos on the LTT channel alone, add the others and you're probably in the 10's of thousands of video projects!
Nice! Server content is my favorite part of the channel lately.
Does Wendell have a video on the data recovery process? I think that would be absolutely fascinating to watch.
Yes please!
'How often do you want to wipe the strand of hair from your face?' - Linus: 'Yes'
WOAH THAT'S HUGE
That’s what she said 😏
Bruh
I almost had a heart attack when linus ran in there with the smaller server
5:20 Most modern operating systems will automatically use read caching for most file systems. This is not a special feature of ZFS, I'm 100% sure Linux does this by default for most filesystems. Basically any unused RAM is used for read caching. Empty RAM is wasted RAM, so it usually won't actually be empty.
Seeing Linus lie under like really heavy JBODs made my anxiety go through the roof. One (admittedly unlikely) slip and his chest would be crushed
Noti gang
Yessir!
Congrats on being actually 1st
THX :)
I have no idea why I watch all of these videos
JBODs? but I do… I do.
literally cannot the amount of space needed for these videos we realize now is absolutely insane
Haven't they been using 8k cameras for many projects for quite a while? Nothing like recording /storing stuff in 8K so that folks can watch it in 1080P (1/16th it's original res)
1:05 *manta*
Manta
@@zetsu2816manta
Should not be using a single card for a JBOD. They should be balanced across all three cards, so if one card dies, you still have connectivity to your storage arrays despite degraded
We all know Linus' real endgame is having enough storage to finally upload his consciousness into a machine, then he can run LTT in perpetuity as a sentient AI that constantly drops segways to our sponsor.
I don't know if I'm right, but it feels like your editors all have slightly different styles that show up in the videos they edit. I really enjoy the variation.