They Told Me NOT to Do This... - Building a Node of the $1,000,000 PC
Вставка
- Опубліковано 14 тра 2024
- Thanks to MANSCAPED for sponsoring the video! Get 20% OFF + Free Shipping at manscaped.com/TECH
Use code LINUS and get 25% off GlassWire at lmg.gg/glasswire
We've built a LOT of crazy servers, but when Supermicro asked us to not rebuild one of the storage servers for the Petabyte of flash project, well, we had to do just the opposite!
Discuss on the forum: linustechtips.com/topic/14222...
Check out the Kioxia CD6 Drives: lmg.gg/rkDFz
Check out AMD Epyc 32 Core Processors: lmg.gg/5vQAv
Check out AMD Epyc 64 Processors: lmg.gg/pN6lh
Check out the Supermicro Storage 1124US-TNRP: lmg.gg/6Qqcu
Check out the Supermicro GPU 4124GO-NART: lmg.gg/bgJiM
Check out Samsung DDR4 ECC Memory: lmg.gg/GAN0z
Check out the NVIDIA MSN3700-VS2F Switch: lmg.gg/0SKZE
Check out the NVIDIA ConnectX-6 200GbE Cards: lmg.gg/rfSEo
Buy some Infinite Cables Fibre Cables: lmg.gg/HGrgD
Buy some Infinite Cables Custom Cables: lmg.gg/K4JK8
Buy some FS.com 100GbE Transceivers: bit.ly/3IX9qpr
Purchases made through some store links may provide some compensation to Linus Media Group.
► GET MERCH: lttstore.com
► AFFILIATES, SPONSORS & REFERRALS: lmg.gg/sponsors
► PODCAST GEAR: lmg.gg/podcastgear
► SUPPORT US ON FLOATPLANE: www.floatplane.com/
FOLLOW US ELSEWHERE
---------------------------------------------------
Twitter: / linustech
Facebook: / linustech
Instagram: / linustech
TikTok: / linustech
Twitch: / linustech
MUSIC CREDIT
---------------------------------------------------
Intro: Laszlo - Supernova
Video Link: • [Electro] - Laszlo - S...
iTunes Download Link: itunes.apple.com/us/album/sup...
Artist Link: / laszlomusic
Outro: Approaching Nirvana - Sugar High
Video Link: • Sugar High - Approachi...
Listen on Spotify: spoti.fi/UxWkUw
Artist Link: / approachingnirvana
Intro animation by MBarek Abdelwassaa / mbarek_abdel
Monitor And Keyboard by vadimmihalkevich / CC BY 4.0 geni.us/PgGWp
Mechanical RGB Keyboard by BigBrotherECE / CC BY 4.0 geni.us/mj6pHk4
Mouse Gamer free Model By Oscar Creativo / CC BY 4.0 geni.us/Ps3XfE
CHAPTERS
---------------------------------------------------
0:00 - Intro
2:03 - The Supermicro A+ Server 1124US
3:47 - Opening it up
9:38 - Installing the AMD EPYC 7543
11:06 - Installing the RAM
12:04 - Installing the Boot Drive and Storage
16:13 - Powering it up and Configuration
19:43 - Testing
21:45 - Conclusion - Наука та технологія
After Linus mistaking NVMe over fabrics (NVME-oF) with optic so many times, we deserve a techquickie explaining the basics and underlining the differences.
@@tamron6014 Cmon man, everyone makes mistakes. You can’t expect them to be subject matter experts on something they’ve only experienced a fraction of before.
They’re still informative regardless.
@@tamron6014 Why..? This content is for entertainment. No one should be watching this to make decisions about their data center lol!
@@rc6431 Most people on this channel have no clue about data center and they will start to think that this is professional stuff. you handle things like that and everyone is running their business like that.
Linus could do videos on fashion too, it would be the same outcome.
NVME is a protocol, and NVME-of is using the NVME protocol over different transports, like Fibre channel, Ethernet, Infiniband, PCI express, etc. “Over fabrics” is all encompassing, it can be run on top of any transport, really.
@@5urg3x NVME over Pigeon?
Jake: "I installed thousands of CPU's"
David: "yeah i know, but that guy dropped half of those"
And most of them still worked anyway.
@@alfredkugler3043 except for that one that cost 10k
@@snarkylive its the only we know about not working...
@@satakrionkryptomortis except that he actually fixed it later.
Hahahaha
It's crazy that I've watched Linus do tech shit for 10 years but I still trust Jake to install anything infinately more
Where is the crazy part? Linus is dropping sth in nearly every single episode.
Sometimes I think Jake is a bigger douche than Riley.
I've always thought Linus is the dumbest smart person I have ever seen.
Linus drop tips exists for a reason
You know, Linus actively generates HUGE amounts of anxiety in me whenever he removes components. Jeezus.
Why? It's not like this comes out of your pocket.
yeah even tho i never build a pc of any sort i feel every nerve in my body send alerts from the way he handles it but its still somehow enjoyable lol
The way Linus aggressively yanks things that cost thousands of dollars apart will never not make me wince
@Nine9 as strong as deez
With enterprise servers, you would be surprised at how "in there" they are.
It's just people that think that electronics will break just by looking at them. People that have experience with hardware know that you won't hurt it by doing want of that.
@@waltch5711 NUTS! Ha got em!
I’ve used similar supermicro servers before and those risers are not easy to snap on properly. They’re kinda jank. You really do have to pull on it super hard and then push it down really hard to put it back. They’re not likely to be used or changed all that much outside of testing and design / PoC, but you would think that they’d put a little bit more effort into the metal work and mechanisms. It’s definitely not finished like an Apple product that’s for damned sure lol.
You could easily fit more sponsors in this video
they already got a couple servers worth a million
I smoke weed in my UA-cam videos as my job lol
@@SevenHunnid im doin it wrong apparently.
@@SevenHunnid nobody cares
@@TheMetamist I think the OP is referring to the April Fools video
Linus has been particularly energetic lately. He's clearly in a happy place.
I believe it's artificial. He seems to be cranking the energy to appeal to the casual viewer, especially in his "shorts" videos.
Or breaking down
@@Squitdoogenz Doubt. Linus has always loved data center related stuff, and theyve been doing that a LOT lately.
I would argue he is much less stressed with the reduced hours, hiring sucsess and having moved, even if cash flow was awkward with physical products.
Yea getting that kind of hardware would make anyone happy.
Ah Jake, the man of culture, informing Linus of the fabled “spit roast” technique.
Hello verified, I'm first
Had a good laugh at linus's face.
and here I thought it was a simple Eiffel Tower maneuver
Because of that, Jake is now the homie
ok bot account
Linus is like a 6-year-old tearing into Christmas gifts at full speed. Yanking wires and knocking drives off the table, while Jake is the patient parent who keeps telling his toddler to slow down.
A 6-year-old is not a toddler anymore.
LoL! It’s great chemistry.
Yeah it's called acting
I think this is the dynamic Linus requires to operate
@Dream you got em good
Linus: "They Told Me NOT to Do This..."
Me: Obviously they've seen your videos.
"Hey Linus, don't do the thing...." *wink wink* - Nvidia wants to sell their shit to anyone, even if it is miners. $ money talks $ baby
Seeing Linus pull on everything the way he did both made me laugh and made my stomach fall trough my body from pain...
These things are meant to be installed/maintained en masse. Hundreds or even thousands of units in a warehouse, all managed by a team made up of 200lbs of pure muscle and brain. They’re meant to take abuse like this. Redundant power supplies.
@@alecwhatshisname5170 Just because our body can repair damage doesn't mean we should break our arms on purpose 😉
@@nickyhr I'm wondering if anyone tried to set a record for most fractures in a single arm (without any bone frailty).
@@Cymes *Someone desperate to get a Guinness record*:
"WRITE THAT DOWN, WRITE THAT DOWN"
I'm deploying a HA cluster at work in a couple weeks. So this series has totally got me hooked and I'm not jealous at all.
"could they [nVidia] hate us any more at this p.."
Jake: "yeah?!"
Guys - it reallllllly looks like you weren't actually benchmarking your ZFS array in FIO tests at the end... If you look at your terminal, you are running the fio tests from within the root directory (/root) which is on your intel NVMe device........... That would explain the low 4k random WRITE IOPS... So, while you may have set metadata only for the ARC for your NVMe array, I doubt you did that for your boot device - That means your read performance numbers of 20GiB/s is most likely just you benchmarking your ARC (memory) performance.
Yeah, let‘s hope they have realized this off-screen!
Well, good thing they said this was a series, so hopefully they address this in the upcoming episodes.
@@ReturnJJ I hope they do, but honestly when the entire point of the video is about the performance of this server you’d think they’d verify that their tests are actually working as intended lol
Yea what he said
oh
the production value is a big part of what makes these videos.
you wouldn't really think much of it, but it's super helpful when they highlight relevant key information on text heavy screens, like when highlighting the IOPS speed in that terminal screen.
these videos are more than just informative; they're extremely entertaining and digestible.
I really love the server type videos. Keep bringing them on.
I've been watching Linus for about 13 years (good NCIX times). I am now 27, working in datacenter for 3 years. Love how this content is keeping up with me
Yes. It's a good strategy. My bet is that by the time we're far enough into our careers to be in charge of purchase decisions, LTT will be deep into enterprise vendor sponsored content. That's where the real money is.
@@davidspagnolo4870 doubt it, enterprise segment has people who chose Hardware and they are already well informed (Thats the reason they are hired, to avoid being scammed by advertisement). This project Was also meant as a ad for Toshiba drives, it just so happened that they managed to do something more complex
@@cyjanek7818 Eventually we will be the people in those positions. Also, if you think that enterprise infra purchasers are immune to scams, I have a bridge to sell you.
@@davidspagnolo4870 well, I wasnt clear enough. Obviously there is Still marketing involved but Different Kind, more Personal one. People from company are contacting Potential buyers directly, show Samples etc But they never let random comparisons to be made on Internet. They would never allow reviewer to compare their enterprise products (just like nVidia didnt care to give Linus A100 for testing, even though he is big person in tech space) because they have too little to gain there and to much to Lose if some test wont go as planned. Doubt it will ever change, because Linus, as big he will be, wont be better for sales than some salesman contacting Person from company directly.
@@cyjanek7818 the fact that this video exists and has as many sponsors as it does shows that you're at least, in part, wrong.
Man, Jake is great. Banters with Linus, knows his stuff, good voice and presenter. Moar Jake!
Gingerfans
How things have changed from younger Jake lol
@@Chaosdude341 i never hated jake but i remember when jake was newer and you would see a lot of comments hating on him
@@Pro_DRIFTZ That's because his banter was childish and annoying.
@@JETWTF the reality is it fits the primary viewer demographic so it was perfectly fine. spend enough time on the discord and you'll see what i mean, lol.
This is such an awesome series! Thanks to all the companies that put their very expensive eggs in this particular basket
I love that ending. Linus was so happy with the seamless Jake transition!
I was a data center engineer. Specifically storage (WD). I’ve configured and deployed 100s of these supermicro servers. If you have any questions I’d be happy to answer them.
What's the best item in the datacenter cafeteria?
I'm a storage architect at 45Drives myself. Has anyone else realized yet that they weren't actually even benchmarking the ZFS array they built? LOL.... They were running the fio tests from the root directory (/root).. I know this because they're logged in as root user and there is a tilda in the shell meaning they are in their home directory ~ and the fio tests do not specify disks or mount points - which means fio was running from whichever directory you ran it in......
They mention they disable the ARC for everything but metadata on their NVMe array, but they definitely didn't do that on the boot pool - so when they ran their randomread tests, what they were actually benchmarking was the ARC performance (RAM) lol and of course its going to be amazing (20GiB/s) ----- but when you see when they finally benchmark the write perfromance it is actually only what you'd get out of a single NVMe device (hint hint lol because it was a single NVMe) and they kinda hand waved it away.... but that is exactly what was happening. They were hitting their intel boot NVMe drive the whole time.
@@Fay7666 Probably the Chips.
@@mitcHELLOworld Poor little boot drive shouldn’t have to endure that torture.
@@kamikamieu haha indeed :P
You're a million dollars in my heart, Linus & Friends
wholesome moment
New age golden girls. They should have "Golden Nerds". You're a friend and you break stuff
I love the art style on these videos. I love this series already!
Great project you're starting, great to learn more of how much can tech be pushed. Also, please, keep the timer thing, it's funny to know that an imminent Linus-thing is coming XD.
I used to build in these super micro chassis all the time when i worked in the data center. The way they’re thought out and designed is pretty crazy. I moved on before AMD was even back in the cpu game but man I would have loved to use some of these CPU’s
Just thinking about all the physics simulations that can be run on this server makes me excited.
You can probably break the world record of computing pi digits on these systems
imagine if it were analog :)
@@spicybaguette7706 nah those people have literal supercomputers. 62.8 trillion digits in 108 days is pretty impressive
@@spicybaguette7706 this is application specific server
@@dickmeatbootysack2165 ua-cam.com/video/s-Xma3SHHos/v-deo.html
They don't. The system used to break the current world record has 2 32 cores AMD CPUs. The problem is not compute, when you're working with 62.8 trillion digits, so about 25TB of space (you actually need much more), the bottleneck is mostly storage and bandwidth. With the entire million dollar system they definitely have enough of both.
13:02 "I think they call that a spitroast" is the greatest thing he's ever said on the channel
Shoutout to the editors, y'all had a lot of fun on this one and it made the video so much more hilarious
This is going to be one of the best series on this channel. I cannot wait to see what else they figure out
Jake is very quickly becoming my second favourite host for tech explanations! I used to think he was so childish and annoying but his camera presence has gone way up and really highlights his strengths!
he's come into his own as the networking + server guy. not sure i've seen much of him + anthony but i'd love to see both of them do some wild stuff w/' linux servers
@@ashlyy1341 Exactly this. The "preparing for 10gbit internet" video was just great for this specific reason
@Deldarel The truth!
Wasn't he barely an adult back then? I was under the impression he was around 18 when he joined the company.
Most of the servers I've installed usually just run ESXi on those little internal USB headers. They usually come with a little RAID microSD to USB adapter with 2 microSD cards in them and since ESXi takes up so little space itself, it runs off it easy and still has that redundancy that you want in a production server.
Would love to see a complete walk through how to build a server that is a Home Theater, cloud storage (back ups), internet and access point and running Virtual machines for computers around the house.
From what I've researched, you usually don't want your internet access (presuming you mean your router) to be in the same box as something else, because if it goes down or needs update/reboot your whole internet is down during that. And access points are usually purpose built devices. The test though should be theoretically done in one box, though it'd potentially require some very beefy consumer hardware, depending on how many virtual machine you plan to run.
I work with Supermicro servers every day and some of the things you can hear in the video made me flinch XD But highly entertaining as always :D
Same, watching him try to pry that pcie riser card out of the chassis without undoing the latch on the back made me recoil in pain lol.
A water bottle near a critical server, that's CRAZY ! 😁
And it's shaking 19:20 !
They must trust the seal
I would love a breakdown or elaboration on what you talked about at the end of the video in terms of a server being designed for its use case. How is your new neumonic server different in function than petabyte flash?
I'm so looking forward to watching this series
Theres something about those compartments just sliding out that just seems so satisfying
I'll bet you watch a lot of "oDdLy sAtiSfYiNg" videos, huh?
@@littlejackalo5326 honestly, no. I never thought of that, I see where you thought that though!
19:20 The increasing fan sound with Linus's speaking is nice
Can you please do a studio tour again? Just have someone walk around with s camera and show us the place. I really enjoy these videos, and seeing how the office and different building looks like time to time makes me feel like I have s more personal relationship to your channel. 😊.
Mabye even som behind the seems shoots.
Your offices is like a playground for most tech nerds. I live in Norway and will probably never have the opportunity to visit it, so give us the next best thing. And give it to us atleest once a year so we can keep up!🤩
man i really like jake in these series. Always a good laugh and a good vibe. Also has a good chemitry with linus!
13:04 - For the uninitiated, he says "They call that a Spit-Roast"
I am afraid of googling that
I don't think that's true. His Lips move to much you can hear "spi" but there are like 3 more words after
@@CallMePandora You probably should be
@@MrTree421 That is definitely what he said since that is exactly "I'll go at it from one side you go at it from the other" would be
I some times expect Jake to tell Linus to go sit in “timeout” and think about what he just did.
I really dig the Linus Jake team ups. Really wish there was a lot more.
~ 8:00 32x PCi Express! Wow this and Compute Express Link are mind blowing
The scenario sounds good, but I'm not sure about the support required when something malfunctions or it's just a driver error!
These kinds of deployments usually are part of a larger data center that has full time support people on staff. So when the company buys (or leases) new hardware, the support team receives an extensive training…
This wouldn’t be for a home user
If a company can afford this, they can also afford to hire someone to do the maintenance & operation
@@flyfalfly You would be amazed how often they just don't care enough to do this. I've been sent to bail out companies where the IT administrator left the company six month ago and they not only haven't hired anybody but they've thrown away all the documentation that he or she had collected on their IT installation. Others never sent anyone to be trained on their new hardware, and some never bothered configuring the mail address where any error or warning messages were sent so the first indication they had that anything was wrong was when enough drives failed that the arrays became unavailable. In one case the system was configured to spread the data over several arrays, and if one became unavailable it would keep working with those that were still available. Error messages were of course generated but with no mail address set no one got them. They didn't react before the last array finally failed after several years.
Do not trust that a customer actually have any idea of the health of their IT system. Some do, but those are almost never a problem. They will order new drives when needed and keep on top of the service schedule and call before everything crashes and they get corrupted data.
I treat the sponsor transitions as a game these days. Hit fast forward as soon into it as possible. Today I got one word. Pretty good.
That's 2 words
I love the energy between these guys😁
Its so funny to watch these guys be so amazed by the thick PCB's. I worked at a place that made really thick ones like that but the whole area of the chassis.
Love the "Linus drops something"-timer! Please add it to all videos from now on :D
i have an old hdd that i use for bulk large data storage (thing is still in perfect health too) and using a massive block and directory size has helped with io performance on it TREMENDOUSLY. legit a thing to know about if your one to thrift parts and fine tune setups
I love how the production of the videos generate more videos about how is the IT life behind the productions!
If I remember correctly from office update videos they don't have a dedicated IT team at all. It's just random office members who deal with it. Kinda makes sense, because so many people at the office know so much about it.
@@austinblackburn8095 sure! And that's what is awesome about Linus tech tips!
you can see when he gets serious, and is keeping track of what linus is ripping off so he can put it back. Nervous laughter and all, lol.
Love the dubbing over Linus' errors at the 12 minute mark 😅
Same! I was curious to see if someone commented on those and... yep.
@@coriscotupi five overdubbed by six
At 12 minutes that little over dub was perfect and fantastic XD
It's funny how Jake is keeping Linus in line, keeping him from going crazy.
Imagine the carnage if Alex was a co-host of this series.
The voice technicians make a great job in your team. I mean, every A/V medium should do like they do
They’re usually so good, I tend to compare my web exports with ltt videos to check levels haha
I say it everytime but man, I LOVE Linus and Jack's chemistry.
I love how Jake is the ''adult'' of the two and Linus is just a big kid.
If I ever get a chance to meet youtubers, I hope it them!
Yo, Love the content bro keep it up!
You have such a professional approach to your work. Well done Linus Team .
11:28 bruh the anxiety i got when he got water near an open motherboard...
So, I know you have probably shot it, but I hope you do Folding at Home with the head unit. Would be so cool to see how it performs with those types of data sets.
I like how epyc has a server mounting that makes sense. Intel you have to use a tool or do it manually. It has no retention clamps so only thing that holds the CPU in sockets is cooling package.
It's awesome to see ZFS pushed to these awesome limits on new hardware... all I have are old 2nd-hand components. The fact that it works great on both is amazing too.
I had an SQL query in work I ran the other week that took 5 minutes. The thing that dropped my jaw was the total CPU time for the query was 4 days. Total cores was a 4 figure number and total memory had Tb at the end. 64 cores. Meh. Average. Not great, not terrible. Would make a fairly ok dev server.
Maybe it’s time to optimise this shit ;)
The board thickness is nothing about rigidity, and everything about trace density. You have how many dims of memory? And they all need to get to a CPU. You need many pCB layers to do this. And every high speed signal layer, you need a ground layer for noise isolation an signal delivery (every signal needs an equivalent ground reference and return)
Nonse that was the pcb for drive attachment. No ram was connected to it'
I dont understand how, I only game on a PS4, do not and have never owned a PC, and yet I watch every single one of your videos religiously, even though most of the time I have no idea what you're talking about, lol. I think that goes to show the quality of your content and editing is still interesti g and captivating even without the tech breakdown. Keep up the good work team
As for Graid relying only on drive telling if there is a read error, Linux's built-in mdadm also does the same. If you want the extra guarantee, you can use something like integritysetup (which uses dm-integrity). Don't bother with journal or bitmap for it (just use -D mode) as RAID parity can be used to repair instead.
atime is extremely useful when dealing with large datasets . If your data that is residing on very expensive/fast media isn’t being accessed often, migrate it to cold storage.
I’m surprised that we’re getting another episode of the server build so soon, not that I’m complaining. A shame that you won’t be able to keep it, though.
Make your own episodes then.
?
@@armandoaranda639 Do you know how expensive this stuff is? Plus I’m not shitting on them at all.
@@alexfinns6162 It was mentioned in the previous video that they have to send this back.
@@Neoxon619 why should they sent it back?
Can you do a video about the differences between Ryzen, Threadripper, and Epyc CPUs, and which tasks are best for each (for example compiling C++, rendering, or gaming)?
I loved the skit (but maybe not actually a skit) of Linus being over-the-top excited and Jake being the straight man.
"I had a GPU that made that noise, it made her dog throw up". Bruh xD
Holy Linus, you're reverting back to the "I'm going to break a 10k CPU" era 😅
Another classic video of maximum anxiety when Linus does virtually anything at all, but also maximum intrigue and comedy
Never get tired of the fellas doing server stuff
These two really have the “nerd bully” vibe down pat.
Idk why but I'm loving the server content
Ide love to see all ur servers and what they do
12:04 - the boot drive chapter... Linus appears to lose his voice at certain points 🤣🙊. Joking aside, I'm curious to see how the entire cluster performs when put together.
why do i have this feeling that the people who built that are screaming as they watch this video
I know I was when he just yeeted that PCI-E cradle out. 😂
I'm always so invested in these videos and I never have any idea what's happening
It's always kind of impressive how much Jake feels like he did his homework for every video while also looking like someone who doesn't do the homework for subjects he doesn't care about, just a true computer nerd respect men
18:12 atime is file access time. Used quite often when you want to see when your employees used certain files last. But do go on.
BREAKING THE RULES. That’s what makes stuff cool
17:02 - 12 pieces of 6.9 terabyte drives. That's a lot of nice.
Having Jake and Linus doing video's together is great
This was last time SUPER MICRO sents LTT any hardware 5:14
When are these companies going to learn that whenever you ask Linus not to do something, he's gonna do it anyway.
or he just puts tape over the screen in the important places...
I love Jake. He's awesome and smart. Never let him go Linus.
My server has a SDcard slot on the motherboard to use as the boot drive - I currently use a 64GB card as my system drive and it works great as long as I remember to make my VM drives on my actual storage drives instead of the SDcard lol. It also has at least 2 internal USB type A headers for if you wanted to use a USB key as a boot drive.
I'd love to see you guys get to show off some real enterprise storage one day. No offense at all to Supermicro (I love Supermicro and others), but those of us working for the top tier enterprise vendors in the world consider this stuff to be "commodity" equipment. See if you can get into Hitachi Vantara, HPe, EMC or IBM (of course there are several other options as well) kit sometime. Should be fun. But, fair warning, they might REALLY not let you open it up and pull it apart. :) Anyway, great video, as always! Thanks guys!
The way Linus drops things around, he should add this to his merch: Linus KNOCK-OFFs
Linus confirmed to be part cat (when deliberate) part dog (when accidental)
I know you said before that enterprise storage is not really for your audience, but you are getting closer to it. Still would be fun to see you have a good look at a few. Not so much to see what they can do but what they internals are.
So many gems in this video, love this.
finally i can store all my home work
And access that massive library folder in its entirety in nano seconds
Linus got lucky it wasn't falling this time 😂
Poor linus
Big fan of server content.
ive never seen linus actually react to dropping something or knocking something over. this is truly what it takes
How come PCIe 4.0 risers and mainboard implementation was "such an enormous hassle" in the desktop space while servers can get by with unshielded angle adapters, tiny connectors and super long trace lengths and extension cables? Wtf
Ya know, that's a really good question...
cost. The BOM was to costly on OEMs in the consumer market.
The joys of working for the retailer. Bought a "under-specced" server and upgraded all I needed in assembly later