Shiny New 128 Core DevOps Build Server For Greg Kroah-Hartman! AMD Epyc ftw
Вставка
- Опубліковано 9 лют 2025
- We've put together a beast of a system for the number 2 Linux kernel guru, Greg Kroah-Hartman himself! Check it out!
Linux DevOps testing: • Solving Linux DevOps I...
**********************************
Check us out online at the following places:
Website: level1techs.com/
Forums: forum.level1tec...
Store: store.level1tec...
Patreon: / level1
L1 Twitter: / level1techs
L1/PGP Streaming: / teampgp
Business Inquiries/Brand Integrations: Queries@level1techs.com
IMPORTANT Any email lacking “level1techs.com” should be ignored and immediately reported to Queries@level1techs.com.
-------------------------------------------------------------------------------------------------------------
Intro and Outro Music By: Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 3.0 License
creativecommons...
Wendell, you have come such a long way. You seem very comfortable on camera these days, and I really enjoy watching you divulge your knowledge to us mere mortals.
Yeah, he used to do tech content and now he just shills whatever product sponsors him.
what he wasn't like that before ?
@Dominic Cerisano I can understand people need to get funds from somewhere but honestly I am sick of just seeing builds with sponsored hardware. You do realise they used to make much more informative content. They've covered a lot of brands content saying only positive things about them with none of the negatives e.g. asrock.
@@edplat2367just leave then
@@edplat2367 asrock rack's hw is pretty damned good, but I dont really use any of their consumer stuff. I feel like the world wendell comes from is similar - while many of the consumer side products have various issues, I find it pretty easy to gloss over them as those issues can often be attributed to using the motherboard for some server-like workload.
Gigabyte is the one I have problems with, at least personally. Shit support behind any product, no matter how solid it is, makes a shit company. I'll never pick up another server or consumer component from them. I think L1 actually noted some concerns with them as well, but that's neither here nor there at this point.
Wendell, just your friendly computer janitor, with a LOT of keys on his set and knowledge.
I pay attention to the companies that said "Yes" to this stuff. That'll be the tie breaker if I have to choose between any of them.
imagine what a 256 core cpu would be like cause oh yeah it's coming I can feel it
Same (and well, EVGA forever) :)
I'd never heard of this brand of PSU though... i love the idea of it, except... not available in my country. my NAS needs one :)
I like the charity aspects. Keep in mind that Linux literately runs the world, and every hour GKH isn't sitting in front of a computer doing is freeing up some of the most valuable developer time on this earth.
So in whole, this probably donated way more than the hardware cost to the Linux project.
Great vid, but you forgot to shoutout to the SSD folks at the end. Hope they don't get mad, thanks Kloxia.
lol loved the ending,
yet as a gentoo enthousiastic since 2005, that box just seems awesome all the way, am sure GKH gonna put it too some good use ^^
yeah, makes me wonder how long it would take from chroot to desktop login if you stuffed it with 1tb ram and used half of that for a portage ramdisk for building.
Wendell, you tha man! When I first watched your early videos I KNEW that we would see great things from you! Just imagine all of the kernels compiled on this beast. Amazing build!!!! Awesome video!!!!!!
i love that atx sized double PSU, i use it on all my 4U servers
Going to interview the commander in chief again? First one on the topic of computers barely working was awesome, and would love to see you guys geek out over Linux and hardware again.
Totally. Not to mention, Level 1 providing ram and coolers to help GKH is basically directly assisting the entire Linux community so no harm no foul there.
Another fantastic video, always exceeding expectations (which are getting pretty high). I'm glad it's all going to such a good cause again! GKH FTW
Man what a badass setup. Im also really glad to see a good air cooler for that socket that points the fans correctly for server cases. that was one of my concerns with going with an epyc setup in my server.
Wouldn't be a level1 video without the boiler snake😊
I have a Torrent case and I love it. With an AMD 7950X, a PowerColor 7900XTX, 64GB of DDR5-6000 CL30 RAM, and a Noctua D15 cooler. With the included fans the thing just stays so cool and quiet, it's great. Also, a solid metal side and no RGB, how great is that!
Still loving my 3990X workhorse! It just chugs away, day in and day out, without complaint.
I don't like putting those huge tower heat sinks on their sides, though. That's gotta torque the socket... not a 10-year plan IMHO. I always build them so gravity is assisting in the right direction, even if it means using a 3U or 4U case for the larger systems.
For the VRMs, I always offset the intake fan on the tower cooler down towards the motherboard as much as possible, so a small part of the fan is actually pushing air under the cooler instead of through the cooler. That seems to solve the problem for me.
-Matt
Thank you for sharing that fan placement tip.
I do the same thing with my fans on my Noctua NH-D15
The board and backplate are taking most of the load, then it's torqued to spec. Buildzoid had some thoughts on pinout loads and apparently even Intel 1700 could take tens, almost hundreds of KG of pressure over all the pins it uses. TR4/SP3 have even more pins over a larger area. I also wouldn't worry much about stress on the chip itself if they state in the spec it's supposed to be pressed to such extents on the socket.
@@cldpt agreed. Ive had an NHD-15 on one of my systems for 8 years now, its fine and it hangs big time. same on my current threadripper, high end boards arnt made of thin paper, they got back plates etc, plus the included toque wrench on threadripper insures its all correct. nothing to worry about.
This feels cool to see while currently owning a machine that excels at everything I do on it. Nothing like this, but tools keeping out of the way feels amazing.
So cool and infectious to see Wendell geeking out. Awesome video, thanks for sharing :)
Danke!
Wendell sharing the experience of building a PC/Server/Workstation that will impact the Linux community is wholesome to watch.
RE: The special standoff - in the days of nylon standoffs that slotted into grooves in the case to hold the motherboard you could just snip off the part of the standoff that would go into the groove to support the motherboard.
Everything new is old again :)
Nice build! That dual power supply is sweet - I can see uses for that. I will never build a system like this but still fun to live through others. If we had UA-cam in the early 90's would it have been as much fun? I wonder...
The power supply form factor is pretty slick
I'm in love with that chasis, and there's a version of it that doesn't have a glass panel!!!
I love the Torrent.
I was about to order it when it was recalled because of the fan hub issue.
It is sooo well thought out and built.
My previous Corsair is also huge and not bad but when I built up the Torrent case everything just made sense.
The only "gripe" I have with it that it could have one centimeter more space on the side behind the motherboard because of the amount of cables I have running through there and the SSD mounts are a bit iffy as I have old SSD's with the screw holes on the "wrong" side.
No glass side for me either.
I love it!
@@FutureChaosTV it's a shame it's not available in my country, it would be very expensive as well.
Love this advice. Define your needs then tailor the solution to those needs.... If only sales would understand that!
Wow what a build. I can only dream of such hardware.
Wendell, Thank you very much for putting on such a fantastic build video, its definitely the best build video I have seen in a long time. The way you present the information is great to digest.
that has to be some kick ass compile times!
That motherboard is ridiculous. I feel like I just watched you tame a beast.
Sick, how entertaining and relaxing to see you at work!😅👍
Hey Wendell, glad you went with a heatsink with appropriate orientation for the server socket.
This powersupply is really cool!
I would love to see a video for a build for compiling rust code. Needs threads and clock speed
Awesome vid! Great support for the linux community. Just one thing that i haven't understood, where the storage went?, Sure, Wendel mentioned Kyoxia and his drives but not shure where they went in the build
That's actually a very neat build for an on-prem build server that doesn't pollute the environment with noise.
I would really like to see some performance testing for pfsense. I know that for 99% of situations using cheap off the shelf hardware can even handle a 10gb connection, but it would be really interesting to see when push comes to shove, where the bottlenecks actually are.
Hot damn
Wow! This is absolutely amazing 😍
I really like those coolers. And great video!
I looked into the FSP Twins for my ATX server build but saw they chose to use a USB interface /w proprietary Windows software for monitoring instead of PMBus like every server board has.
Love your presentation style and jokes.
Weery vell done 👍, very Wendell. 7:40 The HMAA4GR7AJR8N-XN dimms are ECC. Was wondering, so I checked. Much appreciation for your generosity and neighborly welcome.
That case looks awesome
Hope to see more collabs between L1Techs & GKH!
I pressed that like button hard when i saw that "no tempered glass".
Would love to know more about the server config. What tunables did you change? Did you set up a RAM disk for the kernel source code? Are you running the latest kernel with all the new EPYC patches?
then the other videos mentioned are the videos for you :)
@@Level1Techs OK! Thank you!
While I can understand how liquid cooling can be high maintenance by running two liquid cooled computers myself, I have found there are ways to keep the maintenance down. The biggest thing is to use clear coolant. I have been using the clear EKWB coolant for years now, which is a glycol mix with biocide and my flow meter has been measuring the same max flow rate for years now with just a simple yearly flush with the help of an air compressor, proper use of the regulator valve on the air compressor, and a Pyrex bowl with a lid to handle splashing. (The idea being in addition to a well engineered loop that drains well, I use an air compressor to more completely clear out the loop of the old coolant.) When I used coolant that was colored, that food dye or whatever they used just trashed the loop and provided food for the algae to grow. Then I had to get out the toothbrush, long needle brushes, etc and dismantle everything to clean that mess out. I used to hear people say that is what you had to do every year or so when I got into custom liquid cooling loops and with the colored coolant I would agree, but not with the clear coolant I am using now. If you want color in your loop, use LED lights and some colored tubing; don't use colored coolant. But yeah, I would argue a well cooled computer with quality liquid cooling keeping all of the hot running stuff cool (and I am using a monoblock on my workstation board, which is a rarity to find and I had to bug a German guy to make it), it will likely outlast an air cooled setup just because it runs cooler. I suppose that is the other thing, I invested in good sensory and control hardware to keep the cooling system running optimally and cut power in case of a serious cooling fault or if it just gets way too hot in the room. With dual 280W CPUs, that is a whole lot of heat to dump into the case and back to back 120mm coolers is definitely not ideal for that kind of heat output. A high flow rate coolant loop will whisk away that 500+W of heat and barely warm at all between the two CPU water blocks. It would be interesting if you could find a block to stick on those VRMs. I am finding my liquid cooled VRMs with high end pads on them stay rather cool. I have lost hot running VRMs in the past with things like the system just turning off while under sustained full load to a loud bang as a power cap blows out and then the system shuts off with a burnt electronics smell filling the room. So keeping those VRMs cool is important even if they are rated for a high temperature as they don't always last at those temps.
Impressive build ! I would have suggested disassembling the heat sinks before shipping but I suppose it's too late now.
Jeff over at kraft computing tried to put those freezer coolers in a 4u chassis and it was 5mm to tall.
Yes, that's what shied me away from them. It's possible they might fit in some 4u chassis, but I measured my Supermicro 743 and it was too close to call.
I guess with the money already spent it wouldn't be a big deal to check (Epyc 7452, 256GB DDR4 3200 SK Hynix, 4x U.2 drives for VMs, soon to be video card of some description for cloud gaming).
"You should know exactly what problem it is you're trying to solve, before you ever spend the first dollar."
The problem I'm trying to solve is I JUST WANT TO SIT IN CLOSE PROXIMITY TO GOD-LEVEL COMPUTER POWER while I do my mundane office spreadsheet job.
Wendel! Are you secretly building my birthday gift? It's a bit early, August is still months away!
Please don't forget to put an R6900 on it, I also want to try some funky GPU acceleration for things it was not really originally designed for :P
Hey, I just put 8 of those exact DIMMs in my 7452 Epyc homelab server. I got them for a very reasonable price.
modprobed-db and localmodconfig helps allot speeding up kernel compile time since you rarely need all the driver files for a specific system. Just gotta keep in mind its not perfect and sometimes misses out modules so you gotta keep a eye on that but I use it myself and it works quite well even though it forgot to include some core filesystem modules.
Actually, I wonder how a pair of the shiny new 7773-X would do in compiles. it's massive L3 cache should help quite noticably. But maybe these chips are unobtainium right now, and BIOS support is at beta currently.
I'm sure AMD would send these if Wendell asks them. But maybe at a price, even for gkh.
Love this
Lol, I love watching Level1Techs skits
Your channel always fills me with GREAT ENVY..... But ur sparkle effect on the thumbnail looks like snow...🤣
Amazing video! I'm considering one for my company (our first build/proxmox server!) and have some questions:
- we're gonna go with single Epyc 7002 Rome and 64GB over 4x16GB, for cost and initial testing, upgradeability in the future
- I'm choosing between Supermicro H11DSi (for 7002/3) and this Gigabyte (which has Rev1 and 3),: Supermicro boards have 2 benefits, such as upgradeability to Milan (7002+7003) while Gigabyte needs to stick to Rev1 which is 7001/2 only and Rev3 does MIlan. It also has offset sockets which may help cooling. I'm leaning towards Supermicro but the Gigabyte looks fancier and has full x16 slots. Anything wrong here? It is also slightly cheaper. I think they both will fit ATX fine.
- wondering if the Gigabyte Rev1 has firmware upgrades for Milan?
What are your thoughts on such a machine for a proxmox doing both build servers and on-demand Windows/Linux remote workstations and LXC containers?
I like the eco-friendly comment with the nuke the whales background 👌
I just noticed that 🤣
4:48 You've got some kind of "really" deep bass tone that hit right about here for the second time. I think I'm only picking it up because I've got headphones on, but it's kinda weird, and kinda pleasant all at the same time. I think you must have had your camera mounted to or sitting on the table when you plopped that box down onto it and then patted your hand down.
You are making great videos.
Wendell on an iPhone? This truly is the darkest timeline. In all seriousness, very cool to be commissioned by Greg.
_engagement_
Nice; Howitzer of a station!
OK, driving me nuts....what is that handheld light you're using in this vid? I could really use one!
What is this dual U.2 PCIe card at 9:40 ? I would like to use it in my build...
I love Arctic's products and didn't know about this cooler. I guess on a normal motherboard it would not blow air horizontally, but vertically.
big thumbs up
What a build, the snake just wanted to say goodbye to it.
Boiler snake wants to run a code compile on this ASAP.
Afaik FSP also made (or still makes) the be quiet! power supplies
sick HPC build!
For the people wondering about cooling (as I did)
After a quick google search:
apparently water loops can handle about 100W per 120mm fan, and about 130W per 140mm fan for “reasonable temperatures”. This appears to be the case for your general AIO 27mm radiator
EK lists the cooling capacity for one of its own 60mm radiators at up to 480W at 2800 rpm and a ambient/ water temperature delta of 10C
With the important note that radiators get more efficient with a greater temperature delta between ambient and the liquid
Sooo…. That’s nuts
Did not think that would be that much
I came here for the Snake Jazz… didn’t disappoint
No mention of the vrm temps? It's the vrm that gets toasty in dual epyc desktop builds. The CPUs are surprisingly easy to keep cold with the massive dies. But those tiny vrm 6/12 phase heatsinks struggle without massive airflow.
Can I download the Windows system and use it to edit the video, and do you install a screen card 3090 Nvidia on it?
Love it. I wanna one.
"Gotta Nuke somethin'"
Boiler snake...daphuq?
LoL good work buddy.
FINALLY someone builds an actually amazing system on youtube. not just the same typical shit. zen3 epyc amazing.
$20k of CPU's shipped by FedEx?? You're brave! :)
Woot!
I'm surprised you didn't mention the lack of sound damping on the panels. The case isn't cheap (you only save $10 over the windowed version). I expected more from Fractal (used the original XL, with some mods in my main PC for over a decade).
Also, I had to replace the bottom fans as they were resonating with the case. Making 950-1100rpm unusable. While quieter above that zone, they're not quiet.
I can't find the dedicated video on the fsp psu
Why not add a shroud to the dual heatsink/fan sandwich to direct the hot air out of th ecase??
I see that VOIP phone on the desk when are we going to get a VOIP video
Are the CPU cooler fans airflow all one direction? Wouldn't you get the hot air from one feeding the second one making it run hotter? Pulling air from the center and out opposite ends could work but may be tougher to manage air flow or be turbulent.
custom acrylic air flow channels like Puget uses should solve the issue. but it might need some thermal testing.
What distribution did you use for this outstanding build? Did GKH went with Arch for his build server?
for those of us who don't have a lot of moneys is the cheap epyc first and second gens worth it?
I'm in love
Should really put a cap on that valve in the back of that custom copper build.
I think even Dell stopped doing inline cpus on their dual socket workstations to keep the back socket from overheating.
What game was he playing in the beginning of the video?
Damn I like that power supply.
would that cooler be worth it for a 24 core 7401??
Always wondered what kind of cell phone you use, Wendel. I wonder no more! :)
Questions - Why did it turn out that dual socket 32 cores turned out to be optimal, as opposed to quad socket 64 core? Does it have to due with parallelization, Thermal Limitations, Fabric between CPUS, Scheduler issues? Or something else?
You mention it being no maintenance for 10 years, but i imagine after 5-6 years it would be replaced as server hardware will have improved vastly by then, especially in terms of PCIe and CXL developments.
it might still be adequate as a child's Minecraft system then! :)
@@mdd1963 nah, Minecraft is super frequency-dependent so dual 75f3 would've been the way to go 😁
Nice power box
Well Played
Wow, this thing can keep Stellaris from ticking once-every-five-seconds till the next patch!
Now I'm just thinking of this build with the Delta THD2048HT fans that linus showed off.
Wendell, is the build time bottlenecked by io? If so, could they be lowered further by having multiple optane p5800 drives in raid 0?
No
I/o bottleneck
@@Level1Techs thanks for your feedback. Good as always
love it