??? I paid for all the screwdrivers used in my LTT screwdriver review videos - and never recommended them over cheaper alternatives@@bent540 Or are you implying I tried to sell Linus some of my screwdrivers? That... is definitely not true. I like my screwdrivers, don't want to sell them to Linus.
@@jacquesb5248 Honestly from a practicality standpoint, 8 or 10 TB for a SATA drive is probably the max I'd go with (accounting for price + rebuild time for a full disk). 20TB+ is really only best for maximizing storage per lb/volume!
@@JeffGeerling at these sizes the risk of another drive to also break while rebuilding is just a bit too great and while you have backups for that, it's really annoying if you need to actually go and start from there again...
Hey Jeff, It's Mark (met you at the creator summit) So short story on that power switch.. the batch of switches we had in production were not consistently terminated by our supplier. I ended up making a batch of power switch testers for our production guys to use once I found out about this. I'm pretty sure that your unit was one of the first off the line, so it looks like it got through before we knew about the supplier issue. Wouldn't be something we'd see on a full build, as they would have used the power switch mounted to the rear panel to test the system, but since your unit didn't include a mobo they would have used a drop in test bench motherboard. (which is a motherboard mounted to a sliding plate with its own power switch that we install in the back of the unit with enough IO options to accomodate all cabling possibilities offered). They have to leave the rear panel off when testing like this, the same rear panel that houses your untested power switch. 😅
Small problems like these are non-issues, as long as one is able to easily find a description of a working solution and can readily apply it oneself 🙂❤ Much worse is problems (in other computers or software) that takes days of tearing one's hair out, not being able to find anything conclusive to solve the matter.
Jeff violated the first rule of system building, never close the case for your first boot, it's bad juju. ;) Actually another great video, love seeing unusual builds BTW Rocky is my favorite distro, I have several machines running it. Highly recommend!
As a homelabber, I would love to see more 3D printed items in a homelab environment. Lets face it, Homelabbing is about cramming enterprise tech into your house. Seems like 3D printing fits right in there.
Having purchased several 3D printed little adapters over the years from eBay for various enterprise tech that I've used in creative ways, I concur. I just don't need another hobby...
And in an actual datacenter environment, something like a Serverlift SL-500. We went from needing 2 blokes holding the kit up to the rack and a 3rd getting the silders aligned to a single guy being able to so it without any grunting swearing or crushed fingers Yes they're expensive ($8-14,000), but in a work environment they're a lot cheaper than workman's comp if someone gets injured putting a 4U server into a rack For homelab users: You can kludge it with a motorcycle lift or auto workshop scissor table but these only lift about 1 metre at most (these are $250-350 or can be hired) That lift limitation is OK if you can put light stuff into racks or only half populate them but I was needing to fill 42U x 12 racks and had no real control about where new stuff went as old items were retired. You can aim to keep the heavy stuff low but even 2U servers are difficult to lift 7 feet in the air and clip into rack rails
AsrockRack also makes similar motherboards but with an AM4 socket, which I'm using in my own setup. Most or all of the desktop-class, non-APU AM4 CPUs support unregistered ECC. Turn on Eco mode in the BIOS and you've got a powerful, quiet, and efficient homelab/virtualization/storage server using a CPU recycled from your last-gen gaming rig.
Never expected to get a refresher on what can go wrong with a pwrswtch today, but glad to see I am not the only person cursed with learning this the cursed way.
The worst power switch issue I've seen is the computer just randomly shutting down or rebooting. The switch was faulty and sometimes stayed half open and would randomly click based on wind currents or movement of the shelf
Hey Jeff, that 4 pin connector is for, standby, power on, power good and ground. It tells the power supply to turn on and makes sure everything is okay.
Great experiment Jeff! So much fun geeking out with you. And here's a huge bonus - your video dropped 2 mins after I finished watching todays' ExplainingComputers video. Happy Sunday 😁
I'd be interested to see what you can accomplish once everything is all set up and you start playing with C-states, etc. to get the power usage down. Being a mere pleb I have been using Chinese x99 motherboards and cheap Xeons for my recent builds and have managed to do pretty well all things considered as I have only, so far, stood up a "do everything" server that serves as my NAS and home services. A dedicated low power secondary backup target as well as off-site cold storage is in the plans. Just not sure which direction i'm going to go. The new Arm stuff looks like fun but just too spendy for my use case. So either going to go with another Xeon or maybe an N5100 series cpu.
Thanks! And it is, I have been able to get a lot more done, and finally seeing the effects of it in being able to do more than one project at a time now. Plus my wife is very happy having a sewing room at home now haha
Hiya Jeff! Feel free to expand on 3D printing any time you like! I've always appreciated how the 3D print community has rallied around the SBC industry. When I started printing, my first REAL project after cutting my teeth on test models was to print a RazPi3 case that would mount to my printer and run OctoPi. Of course I could have bought something, but being able to conjure a practical and functional piece of computing hardware out of thin air was one of the primary things I enjoyed about 3D printing. I'm really interested to see what you use to print. Personally, I have a a Monoprice, a couple of Creality's Ender3-family printers, a Prusa MK3s+, and a Creality K1 Max. And also, basically an entire wall rack of every interesting type and color of filament I could get my hands on (I got addicted to buying cool plastics for a bit). I am just getting back into adult-level RC cars, so it'll be really fun to start designing some of my own parts. Eventually I'd like to 3D print buck-molds of popular cars as well as my own designs and vacuform shells that will fit the chassis that I'll be racing around! I can't wait!!!
Nice video Jeff, thanks for sharing this bleeding-edge stuff. I just did a NAS build using Epyc Milan using an AMD Epyc 7313P + ASRock Rack RomeD8-2T + 8x32GB DDR4 3200 RDIMMs. Drive config is 16 x Micron 1100 2TB SATA SSDs arranged in two RAIDZ1 VDEVs. I'm seeing nearly identical idle power draw as what you are which is surprising given ARM VS x64, with it going up to just under 200W with multiple VMs running in addition to the TrueNAS Scale. Love that 45HomeLab chassis, but too rich for my blood.
I was wondering how it would compare with a similarly specced Epyc setup. So about the same? I have an ASRock Rack EPYC3451D4I2-2T running TrueNAS Scale, will have to get some power measurements on it.
@@justinclacherty9181To be fair my comparison isn't 100% apples to apples to Jeff's here because I'm using only SATA SSDs and he has spinning rust. But yeah I don't think the ARM power efficiency claims are enough here to justify the boutique nature of an ARM Server for a HomeLab. I'll be able to source at least functional (if not exact) replacement parts for my build for years - I'm highly suspect that will be true for the ARM Server.
OMG I was right when I said Ampere ARM on your image when you asked what we think you'd be doing with this chassis. Wow. Great video, I really want to build and use such a server, but when I saw 158W in idle, I decided I don't want to (can't afford it at the moment anyway), because electricity here is 40¢ per kWh I believe.
Something to note with Asrock IPMI, at least on the board I have, is that you have to manually configure which temperature sensor(s) correspond to which fan(s), as well as setting the fan curve. You also have to make a mapping from temperature to fan speed, or I think you can set it in a mode where it's like a feedback control that steps the fan speed up or down.
Definitely would love to see some 3D printing shenanigans on the channel! It's always awesome to see the unique ways that people utilize additive manufacturing in their professions and hobbies.
Good job on the Video, Your edits are getting so very good! I can't wait to see how Arm Cpu's are going to change as time goes on. It'll be cool to see laptops using them as fully as they do with Intel and AMD cpu's.
Although I am not an Apple fan, their M series shows that RISC / ARM with emulation can perform as well as x86-64 for modern laptops & desktops. I wish there were more powerful & efficient options in the PC hardware ecosystem.
awesome reviews! Love your ARM64 content - especially now when Adlink and Asrock delivers products that are useful! Took the courage to actually invest in a 128core adlink after your latest video! Can inform you that my 128Gig ECC LR DIMMs didnt work, sent them back and populated it with 64gig ECC R DIMMs instead and they work like a charm. Looking forward for your next Ampere video!
9:20 The screw tightening order should be in a "star pattern", not just "round the clock" (or in this case "against the clock"). It stabilizes the pressure load and position of the cooler, making it a downward movement instead of a skewed movement. I learned that from a mechanic, for mounting engine tops. Bonus points if: * Make three "rounds": 1. make the screws just get a grip 2. screw them so they touch the bottom position 3. tighten
You’re not made of money, but these parts aren’t cheap! Jeff always gets to play with the cool toys. Thanks for sharing the setup! Hope you have a happy and healthy 2024
Very interesting project Jeff. And important too, efficiency is both mathematically beautiful and a necessity for the progress and survival of humanity.
Heh. In this case it was an empty little prop. I had it sitting around because I planned on recording with my Dad on set for a Geerling Engineering video-figured it was a fun little thing to have sitting around in this video :)
Likewise I killed the family TV after sitting a glass of milk on top and spilling it down the back. This was the 70s and my family had saved up for a long time just to buy this TV, and I killed it. We didn't have a working TV again for years. Years. TVs cost a fortune in those days.
I recently discovered that I have been doing daily snapshots of the volume containing my virtual machine's hard drives, for 3 years, and forgot about it. I didn't notice it until I went to migrate the volume. It didn't impact performance.
Interesting. A friend of mine is running 6x 3.5" HDD but on a ThreadRipper 3960x with a GTX1650 for some LLM dev. Idle power is 138W with load sitting around 190W when running inference tests. He's been bitching about the power draw for ages but these Ampere numbers make the TR build look pretty good. Might have to copy his build lol
Honestly, having used both the 45Drives chassis, as well as the Rosewill. The 45Drive chassis is definitely worth the price tag. I have a bunch of the Rosewill cases in my lab. They are extremely flexible, and well thought out pieces of hardware for the price point. So if you just don't have the money for a 45Drives system, they're a great second choice. Rosewill has a set of components that allows you to easily change out the front of your chassis.
I recently built out my new storage pool in a HL15... Crammed 12 15Tb Mach.2 drives in there along with some nvme drives. Luckily I didn't have the power switch issue you did but for some reason the power light on the switch doesn't want to work... Given that it's on the back I just stopped caring though... hehe. Your case seems louder than mine was. Not sure if that's the mic picking up a bigger frequency range or something else. Overall it looks like you're putting together a solid build... I'll be interested to see the benchmarking for storage and network throughput when you're fully setup....
I really like my mini-ITX nas I mostly put together from stuff I have lying around and some drives, but this kind of thing is what I actually want to play around with.
Nice build. I would love to have one of those 45 drives cases. My major problem is having short rack space as I am using a network rack in the top of my closet, so I am currently cutting a few inches off the front of my case. 😀
For switches, NC = normally closed, NO = normally open, C = common. It has to do with the default state of the circuit when wired: NO means an open circuit which the switch closes, NC means a closed circuit which the switch opens. C is the common which completes the circuit either way. Since power switches in computers only need to briefly toggle, it should always be wired NO so when the switch is pressed it closes the loop. It knows the circuit change due to a sensing voltage, usually around 3-5 volts of direct current (VDC) but quite low amperage.
On the power switch: NO is Normally Open. NC is Normally Closed. And "C" is Common. There is continuity between NC and C when the switch is not pressed, between NO and C when the switch is pressed.
The adaptor cable is for Power Good (Gray), 5V Standby (Purple), PS-On (Green) and Ground (Black). It needs this to provide standby power to the board and to signal the PSU to turn on. I had to mod the cables that came with my Supermicro M11-SDV and figure these out.
Love your videos! Hehe, that's totally me - always using screwdrivers as knives. Huuu, 150+ Watts in idle... for Test, or if you have paying customers, yes... but probably not for fun!
158 watts is pretty high idle power draw, I have a second hand Dell T620 with dual E5-2650 v2, 192Gb Ram and 11 hard drives and 3 NVME that runs between 185w and 212w when serving the containers and filesystem (10-17% CPU). I was considering whether newer hardware would be more power efficient, but this makes me want to keep this running for a while longer until the hardware is completely obsolete!
I like that you are using ARM for your NAS. I just recently got acquainted with Rocky Linux for a test I am doing at work. I am glad it is available so that non-enterprise users can use it as a direct replacement for RHEL and CentOS. I look forward to see how you complete the setup. Just an FYI, I run TrueNAS Core on my Dell R720 with 4x Seagate Exos 18TB and 4x Seagate SAS 3TB drive with a dual 10G SFP+ NIC and the built-in quad NIC and it only draws 140w on average. I an running dual xeon 8c/16t procs and 128GB RAM. I thought newer hardware and ARM would be less power draw?
I love this board and chip. Keeping an eye out for when they start to hit the secondary market. Also I was hoping that stupid orange overprice bit driver would have gone away by now.
I searched around for a used MB+CPU+RAM combo and settled on a kit that was 5 years old and cost US$1,000. It chews about 60w idle, 70w when file sharing, and can handle a GPU and chew about 110w if I render out BRAW delivery, (TrueNAS Scale-VM Windows-Davinci Resolve). Case and PS were expensive but quiet for home use. Definitely hunt around for bargains. I initially set my sights too high, but after my long journey, I ended up with just the right gear.
Looks nice. Finally. Manual and features look really good. It also looks better than the WS system we saw, that had gimped memory channels, and price is right (the 10G version is little overpriced, but still good). Also it has enough power and VRM to support all CPU models I think. The boards are coming in. There is one review on a newegg. For the file server, virtualization host, and even a router, this is fantastic. I hope they also do work on E-ATX version, with maybe a bit more space between PCIe slots, for a GPU or two. Then I would totally use it as my desktop. But even then, I think it is just about right, I need 1 GPU, and 1 NIC (25/100Gbps). Which with big GPU, will basically use all slots. But then storage can be done using OCulink so that should be fine too.
I would recommend to remove noctua fans for the CPU cooler. As you have the 45drive case fan perfectly aligned to cpu. Basically, just use the noctua heat sink only. You can check the cpu temp regularly if removing the 2 fans is Ok. This will additionally reduce power intake
@@jonathanbuzzard1376 I would rather keep this system a little quieter though-which means a little less airflow/lower rpm/lower CFM from the case fans. In that case, having a fan directly on the CPU cooler is handy-Noctua makes similar coolers for other enterprise sockets too!
I was just working on a PC without an antistatic strap and simultaneously watching Jeff working on a PC without an antistatic strap. I thought "that's going to catch up to us!" Looks like it might have for Jeff but I got lucky! 😂
As much as I would like the 12 volt standard for the desktop, I also see a disadvantage in it. For example, if you want to misuse your desktop platform for other things, you are very dependent on the motherboard manufacturer not wanting to sell the high-priced things again, as is so often the case.
Great video. I just picked up a used, massive, old server with 14 drive bays, 2 power supplies, and lots of space for tinkering. Unfortunately it is too deep for my rack, and I am scared of the power draw. I'll have to hook up my kilowatt and see how it compares to your new setup, Jeff. :)
Be prepared for anything! I've seen a few old systems power and sucking down like 300-600W :O A lot of them were also never tuned for power saving at all-and some platforms don't have much you can tweak either :/
Honestly for a typical *home* NAS, this machine would be waaaay overkill :D Also, one thing I've found with Ampere's chips is they are most efficient for medium to mostly-utilized workloads. If you run it idle most of the time, you can probably get a lower power budget with other CPUs.
Correct me if I missed something, but the idle was approx 150 W (let's say 100 without the drives). That's still more than a 200$ difference in electricity costs where I live. But as Jeff said, this is a server for entirely different use case than a small home NAS. And sorry if the comment came out wrong. It was more about me being silly about a few watts than about comparing server grade hw to small home nas with a plex server.
So its not a cake?
The cake was a lie.
@@JeffGeerling about to say the same thing 🤣
@@JeffGeerlingThe cake was always a lie.
@@JeffGeerlingOkay, so the pivot to baking isn't happening yet. Got it.
no thats extra....
I like how you don’t pretend the video isn’t sponsored because no cash changed hands - integrity move!
It's best to be blatant about the vendor relationship; then you can decide how much you'd like to trust my conclusions!
i dont do things based on how much money involved
this guy tried to sell linus screwdrivers! maybe its because you dont know what integrity means!?
??? I paid for all the screwdrivers used in my LTT screwdriver review videos - and never recommended them over cheaper alternatives@@bent540
Or are you implying I tried to sell Linus some of my screwdrivers? That... is definitely not true. I like my screwdrivers, don't want to sell them to Linus.
This is my dream NAS. When this stuff starts showing up on eBay in a few years I'm going to be so happy.
yeah those 20TB hdd
@@jacquesb5248 Honestly from a practicality standpoint, 8 or 10 TB for a SATA drive is probably the max I'd go with (accounting for price + rebuild time for a full disk).
20TB+ is really only best for maximizing storage per lb/volume!
@@JeffGeerling at these sizes the risk of another drive to also break while rebuilding is just a bit too great
and while you have backups for that, it's really annoying if you need to actually go and start from there again...
Why is it your dream NAS?
@@JeffGeerling true, i have 4 6TB for 15,6TB and have 2TB left so bigger is going to work for me. data hoarder
Hey Jeff, It's Mark (met you at the creator summit) So short story on that power switch.. the batch of switches we had in production were not consistently terminated by our supplier. I ended up making a batch of power switch testers for our production guys to use once I found out about this. I'm pretty sure that your unit was one of the first off the line, so it looks like it got through before we knew about the supplier issue. Wouldn't be something we'd see on a full build, as they would have used the power switch mounted to the rear panel to test the system, but since your unit didn't include a mobo they would have used a drop in test bench motherboard. (which is a motherboard mounted to a sliding plate with its own power switch that we install in the back of the unit with enough IO options to accomodate all cabling possibilities offered).
They have to leave the rear panel off when testing like this, the same rear panel that houses your untested power switch. 😅
Good to know, and glad to meet you virtually again :)
Small problems like these are non-issues, as long as one is able to easily find a description of a working solution and can readily apply it oneself 🙂❤
Much worse is problems (in other computers or software) that takes days of tearing one's hair out, not being able to find anything conclusive to solve the matter.
Jeff violated the first rule of system building, never close the case for your first boot, it's bad juju. ;)
Actually another great video, love seeing unusual builds
BTW Rocky is my favorite distro, I have several machines running it. Highly recommend!
And then he locks it up again after swapping the RAM! No, no, no. Never seal until boot succeeds.
@@BrBill That's just how he rolls 😎
As a homelabber, I would love to see more 3D printed items in a homelab environment. Lets face it, Homelabbing is about cramming enterprise tech into your house. Seems like 3D printing fits right in there.
For sure!
Having purchased several 3D printed little adapters over the years from eBay for various enterprise tech that I've used in creative ways, I concur. I just don't need another hobby...
The older I get, the more a appreciate the value of a workshop trolley when it comes to moving awkward and / or heavy kit around.
Pulling the mag drives before you rack it also helps
And in an actual datacenter environment, something like a Serverlift SL-500.
We went from needing 2 blokes holding the kit up to the rack and a 3rd getting the silders aligned to a single guy being able to so it without any grunting swearing or crushed fingers
Yes they're expensive ($8-14,000), but in a work environment they're a lot cheaper than workman's comp if someone gets injured putting a 4U server into a rack
For homelab users: You can kludge it with a motorcycle lift or auto workshop scissor table but these only lift about 1 metre at most (these are $250-350 or can be hired)
That lift limitation is OK if you can put light stuff into racks or only half populate them but I was needing to fill 42U x 12 racks and had no real control about where new stuff went as old items were retired. You can aim to keep the heavy stuff low but even 2U servers are difficult to lift 7 feet in the air and clip into rack rails
AsrockRack also makes similar motherboards but with an AM4 socket, which I'm using in my own setup. Most or all of the desktop-class, non-APU AM4 CPUs support unregistered ECC. Turn on Eco mode in the BIOS and you've got a powerful, quiet, and efficient homelab/virtualization/storage server using a CPU recycled from your last-gen gaming rig.
Running one of their Intel Core boards with a 12600K and ECC. It's a great system.
they have also AM5 boards
@grizant
Eco mode does fantastic work when it comes to load power consumption, but what about idle? Is it acceptable on ASRock Rack setup?
Ryzens idle quite high, they don't properly support C6.@@antagonista8122
On the switch, you usually have NO - Normally Open, NC - Normally Closed, and C is actually Common, ie a changeover switch.
That makes more sense since Normally Closed would be Closed, normally, heh
There are many tech people out there, but I love your videos and your style. Honest, calm and professional. Thank you for the video!
Never expected to get a refresher on what can go wrong with a pwrswtch today, but glad to see I am not the only person cursed with learning this the cursed way.
The worst power switch issue I've seen is the computer just randomly shutting down or rebooting. The switch was faulty and sometimes stayed half open and would randomly click based on wind currents or movement of the shelf
Hey Jeff, that 4 pin connector is for, standby, power on, power good and ground. It tells the power supply to turn on and makes sure everything is okay.
27:24 I felt this moment in my entire being, and the "sometimes that happens" was the perfect response.
takes 30 minutes to build. takes 45 to remove from packing lol
I used to have an ARM NAS! It was called an Acorn Archimedes 😀 ran RISC OS in ROM and the network card had drivers in it’s ROM!
Blast from the past!
How many hard disk did it hosted and for which capacity ?
Definitely curious about your 3D printing adventures!
Great experiment Jeff! So much fun geeking out with you. And here's a huge bonus - your video dropped 2 mins after I finished watching todays' ExplainingComputers video. Happy Sunday 😁
It's a great pairing!
@@JeffGeerling Like Wine & Cheese!!! 😜😆
love the full setup video, its very informational and shows the whole process of building and racking a server.
I have several Asrock Rack boards. All have been very solid and super stable. They are not cheap but if 24/7 is what you need.
I'd be interested to see what you can accomplish once everything is all set up and you start playing with C-states, etc. to get the power usage down. Being a mere pleb I have been using Chinese x99 motherboards and cheap Xeons for my recent builds and have managed to do pretty well all things considered as I have only, so far, stood up a "do everything" server that serves as my NAS and home services. A dedicated low power secondary backup target as well as off-site cold storage is in the plans. Just not sure which direction i'm going to go. The new Arm stuff looks like fun but just too spendy for my use case. So either going to go with another Xeon or maybe an N5100 series cpu.
Very cool project. Awesome to see this system up and running.
one of the cooler HL15 builds I have seen, very nice keep it up!
Nice to see you have all the same issues any of could have! Well, except for dropping screws into the gubbins - maybe I'm clumsy
Your channel is amazing, thank you for your hard work and sharing the fun with us.
This is the first video of yours I have watch with you in the new space. Studio looks good and functional! Looks like the hard work is paying off.
Thanks! And it is, I have been able to get a lot more done, and finally seeing the effects of it in being able to do more than one project at a time now. Plus my wife is very happy having a sewing room at home now haha
I'm so excited with this setup! The presentation is top-class as always :)
Hiya Jeff! Feel free to expand on 3D printing any time you like! I've always appreciated how the 3D print community has rallied around the SBC industry. When I started printing, my first REAL project after cutting my teeth on test models was to print a RazPi3 case that would mount to my printer and run OctoPi. Of course I could have bought something, but being able to conjure a practical and functional piece of computing hardware out of thin air was one of the primary things I enjoyed about 3D printing. I'm really interested to see what you use to print. Personally, I have a a Monoprice, a couple of Creality's Ender3-family printers, a Prusa MK3s+, and a Creality K1 Max. And also, basically an entire wall rack of every interesting type and color of filament I could get my hands on (I got addicted to buying cool plastics for a bit). I am just getting back into adult-level RC cars, so it'll be really fun to start designing some of my own parts. Eventually I'd like to 3D print buck-molds of popular cars as well as my own designs and vacuform shells that will fit the chassis that I'll be racing around! I can't wait!!!
Nice video Jeff, thanks for sharing this bleeding-edge stuff. I just did a NAS build using Epyc Milan using an AMD Epyc 7313P + ASRock Rack RomeD8-2T + 8x32GB DDR4 3200 RDIMMs. Drive config is 16 x Micron 1100 2TB SATA SSDs arranged in two RAIDZ1 VDEVs. I'm seeing nearly identical idle power draw as what you are which is surprising given ARM VS x64, with it going up to just under 200W with multiple VMs running in addition to the TrueNAS Scale. Love that 45HomeLab chassis, but too rich for my blood.
I was wondering how it would compare with a similarly specced Epyc setup. So about the same? I have an ASRock Rack EPYC3451D4I2-2T running TrueNAS Scale, will have to get some power measurements on it.
@@justinclacherty9181To be fair my comparison isn't 100% apples to apples to Jeff's here because I'm using only SATA SSDs and he has spinning rust. But yeah I don't think the ARM power efficiency claims are enough here to justify the boutique nature of an ARM Server for a HomeLab. I'll be able to source at least functional (if not exact) replacement parts for my build for years - I'm highly suspect that will be true for the ARM Server.
OMG I was right when I said Ampere ARM on your image when you asked what we think you'd be doing with this chassis. Wow.
Great video, I really want to build and use such a server, but when I saw 158W in idle, I decided I don't want to (can't afford it at the moment anyway), because electricity here is 40¢ per kWh I believe.
It should be lower when the hard drives spin down.
CPU should be 6 to 12 W at idle.
Looks fun! Although Im watching every video without having any experience, so you are learning me thanks!
Something to note with Asrock IPMI, at least on the board I have, is that you have to manually configure which temperature sensor(s) correspond to which fan(s), as well as setting the fan curve.
You also have to make a mapping from temperature to fan speed, or I think you can set it in a mode where it's like a feedback control that steps the fan speed up or down.
Definitely would love to see some 3D printing shenanigans on the channel! It's always awesome to see the unique ways that people utilize additive manufacturing in their professions and hobbies.
Good job on the Video, Your edits are getting so very good! I can't wait to see how Arm Cpu's are going to change as time goes on. It'll be cool to see laptops using them as fully as they do with Intel and AMD cpu's.
Although I am not an Apple fan, their M series shows that RISC / ARM with emulation can perform as well as x86-64 for modern laptops & desktops. I wish there were more powerful & efficient options in the PC hardware ecosystem.
Awesome as always! Looking forward to the performance and testing!
awesome reviews! Love your ARM64 content - especially now when Adlink and Asrock delivers products that are useful!
Took the courage to actually invest in a 128core adlink after your latest video!
Can inform you that my 128Gig ECC LR DIMMs didnt work, sent them back and populated it with 64gig ECC R DIMMs instead and they work like a charm.
Looking forward for your next Ampere video!
Really cool! I can't wait to see more ARM servers, especially ones that could be used in the homelab!
9:20 The screw tightening order should be in a "star pattern", not just "round the clock" (or in this case "against the clock"). It stabilizes the pressure load and position of the cooler, making it a downward movement instead of a skewed movement. I learned that from a mechanic, for mounting engine tops.
Bonus points if:
* Make three "rounds":
1. make the screws just get a grip
2. screw them so they touch the bottom position
3. tighten
FREAKING AWESOME video Jeff !! Nice work !!
You’re not made of money, but these parts aren’t cheap! Jeff always gets to play with the cool toys. Thanks for sharing the setup! Hope you have a happy and healthy 2024
Very interesting project Jeff. And important too, efficiency is both mathematically beautiful and a necessity for the progress and survival of humanity.
The coffee mug on the case is triggering me!😅 My Dad had a Heath kit TV die on him when a vase of flowers on it spilled.
Heh. In this case it was an empty little prop. I had it sitting around because I planned on recording with my Dad on set for a Geerling Engineering video-figured it was a fun little thing to have sitting around in this video :)
Likewise I killed the family TV after sitting a glass of milk on top and spilling it down the back. This was the 70s and my family had saved up for a long time just to buy this TV, and I killed it. We didn't have a working TV again for years. Years. TVs cost a fortune in those days.
That is a big honkin home lab NAS, wow.
Thanks for sharing this build with us.
I have to present a project at the end of my apprenticeship. An ARM based NAS is exactly what I was looking for! Great idea! Thanks!👍
RAID is not a backup, but those ZFS snapshots are super handy for day to day restores.
True!
Also superhandy for the day to day revert you latest system software upgrade if it all fails.
I recently discovered that I have been doing daily snapshots of the volume containing my virtual machine's hard drives, for 3 years, and forgot about it.
I didn't notice it until I went to migrate the volume. It didn't impact performance.
@@edwardallenthree When you forget you took a snapshot of a VM years ago in VMWare and have been running on it since. >.
This is amazing content! Arm is the future.
Interesting. A friend of mine is running 6x 3.5" HDD but on a ThreadRipper 3960x with a GTX1650 for some LLM dev. Idle power is 138W with load sitting around 190W when running inference tests. He's been bitching about the power draw for ages but these Ampere numbers make the TR build look pretty good. Might have to copy his build lol
150w with 32 cores and so many pcie lanes is awesome for virtualization, excited to see where ARM goes in the next few years.
You can already do that now with EPYC.
Nice... Clear and useful as always! Tks for sharing this.
Perfect for me! Hater of arm since the early beginning into larger things
Love your rocky shirt as well Jeff :)
IIRC the 2 in 3-2-1 backup strategy means two different types of media. So I'm hoping for a video on a tape robot in the near future. :D
Does Amazon's robotic tape library count? My 3rd backup is in Amazon Glacier Deep Archive :)
"Hello there!" 😂 Always looking forward your Linux/Servers videos. Keep em up!
This was a epic episode! Always enjoye someone building a server
Honestly, having used both the 45Drives chassis, as well as the Rosewill. The 45Drive chassis is definitely worth the price tag. I have a bunch of the Rosewill cases in my lab. They are extremely flexible, and well thought out pieces of hardware for the price point. So if you just don't have the money for a 45Drives system, they're a great second choice. Rosewill has a set of components that allows you to easily change out the front of your chassis.
I didn’t know you had a mug. Gotta get that. Good product placement ❤
I recently built out my new storage pool in a HL15... Crammed 12 15Tb Mach.2 drives in there along with some nvme drives. Luckily I didn't have the power switch issue you did but for some reason the power light on the switch doesn't want to work... Given that it's on the back I just stopped caring though... hehe.
Your case seems louder than mine was. Not sure if that's the mic picking up a bigger frequency range or something else. Overall it looks like you're putting together a solid build... I'll be interested to see the benchmarking for storage and network throughput when you're fully setup....
Awesome build Jeff. Not gonna lie, very jealous of that rig. 🤤
So happy you didn't go with an off the shelf nas software. Truenas is cool but I love the idea of just using Linux
I can't wait for part 2!
I really like my mini-ITX nas I mostly put together from stuff I have lying around and some drives, but this kind of thing is what I actually want to play around with.
Nice build. I would love to have one of those 45 drives cases. My major problem is having short rack space as I am using a network rack in the top of my closet, so I am currently cutting a few inches off the front of my case. 😀
For switches, NC = normally closed, NO = normally open, C = common. It has to do with the default state of the circuit when wired: NO means an open circuit which the switch closes, NC means a closed circuit which the switch opens. C is the common which completes the circuit either way. Since power switches in computers only need to briefly toggle, it should always be wired NO so when the switch is pressed it closes the loop. It knows the circuit change due to a sensing voltage, usually around 3-5 volts of direct current (VDC) but quite low amperage.
I got a few of those Logitech K400 keyboards too. Truly a great keyboard for bench testing! Definitely a favourite on my benches.
Awesome video sound was top notch as well
With all those Seagate drives, you might as well leave the top off, since their MTBF is shorter than the time it takes to put the lid back on :^)
On the power switch:
NO is Normally Open.
NC is Normally Closed.
And "C" is Common.
There is continuity between NC and C when the switch is not pressed, between NO and C when the switch is pressed.
It's pretty common that these switches get wired incorrectly.
So cool, looking forward for part 2 :)
The adaptor cable is for Power Good (Gray), 5V Standby (Purple), PS-On (Green) and Ground (Black). It needs this to provide standby power to the board and to signal the PSU to turn on. I had to mod the cables that came with my Supermicro M11-SDV and figure these out.
Great information, thanks 👍
Love your videos! Hehe, that's totally me - always using screwdrivers as knives.
Huuu, 150+ Watts in idle... for Test, or if you have paying customers, yes... but probably not for fun!
158 watts is pretty high idle power draw, I have a second hand Dell T620 with dual E5-2650 v2, 192Gb Ram and 11 hard drives and 3 NVME that runs between 185w and 212w when serving the containers and filesystem (10-17% CPU). I was considering whether newer hardware would be more power efficient, but this makes me want to keep this running for a while longer until the hardware is completely obsolete!
Cant believe we're in an era where arm cpus are socket able
I like that you are using ARM for your NAS. I just recently got acquainted with Rocky Linux for a test I am doing at work. I am glad it is available so that non-enterprise users can use it as a direct replacement for RHEL and CentOS. I look forward to see how you complete the setup. Just an FYI, I run TrueNAS Core on my Dell R720 with 4x Seagate Exos 18TB and 4x Seagate SAS 3TB drive with a dual 10G SFP+ NIC and the built-in quad NIC and it only draws 140w on average. I an running dual xeon 8c/16t procs and 128GB RAM. I thought newer hardware and ARM would be less power draw?
Oh sweet they used the Super micro rails, nice choice to not have to design a whole new thing. Looking forward to seeing more ARM server stuff coming.
22:41: "Hello there!"... sold!
I'm crediting this video for my massive ARM stock gains yesterday.
Nice video. Will have a look and see what these run for in uk
For those that dislike this particular case the manufacturer Protocase will make any design you like, even as a one-off
I love this board and chip. Keeping an eye out for when they start to hit the secondary market. Also I was hoping that stupid orange overprice bit driver would have gone away by now.
Would love more of your take on 3d printing.
Finally......!!!! My dream project...!!!! A DIY ARM NAS....!!!!🤩🥳
Great video Jeff. Love sth. I want to build a nas w/ solidigm P5336 61T drives when I win the lotto.
That would be... yeah I think I'd have to sell my house haha
Nice video. Can’t wait for the next one!
I just love your sense of humor Jeff :)
I searched around for a used MB+CPU+RAM combo and settled on a kit that was 5 years old and cost US$1,000. It chews about 60w idle, 70w when file sharing, and can handle a GPU and chew about 110w if I render out BRAW delivery, (TrueNAS Scale-VM Windows-Davinci Resolve). Case and PS were expensive but quiet for home use. Definitely hunt around for bargains. I initially set my sights too high, but after my long journey, I ended up with just the right gear.
Looks nice. Finally. Manual and features look really good. It also looks better than the WS system we saw, that had gimped memory channels, and price is right (the 10G version is little overpriced, but still good). Also it has enough power and VRM to support all CPU models I think. The boards are coming in. There is one review on a newegg.
For the file server, virtualization host, and even a router, this is fantastic.
I hope they also do work on E-ATX version, with maybe a bit more space between PCIe slots, for a GPU or two. Then I would totally use it as my desktop. But even then, I think it is just about right, I need 1 GPU, and 1 NIC (25/100Gbps). Which with big GPU, will basically use all slots. But then storage can be done using OCulink so that should be fine too.
Hey Jeff, good work as always. You look healthy, hope you are well!
Very cool use of ARM
Your simple NAS is more powerful than my gaming rig. Game over for me.
I would recommend to remove noctua fans for the CPU cooler. As you have the 45drive case fan perfectly aligned to cpu. Basically, just use the noctua heat sink only. You can check the cpu temp regularly if removing the 2 fans is Ok. This will additionally reduce power intake
True; I might test just one cpu fan instead of two. With this lower wattage cpu that should be sufficient and still quiet
@@JeffGeerlingreal enterprise servers *never* have fans on the CPU's, a passive heatsink with possibly some ducting is what you want.
@@jonathanbuzzard1376 I would rather keep this system a little quieter though-which means a little less airflow/lower rpm/lower CFM from the case fans. In that case, having a fan directly on the CPU cooler is handy-Noctua makes similar coolers for other enterprise sockets too!
thanks very good video looking forward to chapter 2
It's running OpenBMC but an AMI UEFI BIOS? Usually the ASRock Rack boards run Aptio UEFI and MegaRAC remote management.
I was just working on a PC without an antistatic strap and simultaneously watching Jeff working on a PC without an antistatic strap. I thought "that's going to catch up to us!" Looks like it might have for Jeff but I got lucky! 😂
Cool build. I look forward to hearing how it performed over time. If you used encryption, CPU loads and such
As much as I would like the 12 volt standard for the desktop, I also see a disadvantage in it.
For example, if you want to misuse your desktop platform for other things, you are very dependent on the motherboard manufacturer not wanting to sell the high-priced things again,
as is so often the case.
18:06 this made setting up a 60 drive ssd ceph cluster very easy. Can't imagine having to put them all in caddies
Nice build. I hope one day a standard mini-itx or m-atx MOBO Arm64 based CPU
Great video. I just picked up a used, massive, old server with 14 drive bays, 2 power supplies, and lots of space for tinkering. Unfortunately it is too deep for my rack, and I am scared of the power draw. I'll have to hook up my kilowatt and see how it compares to your new setup, Jeff. :)
Be prepared for anything! I've seen a few old systems power and sucking down like 300-600W :O
A lot of them were also never tuned for power saving at all-and some platforms don't have much you can tweak either :/
And here I was trying to figure out how to get my intel setup under 17 W idle.
Honestly for a typical *home* NAS, this machine would be waaaay overkill :D
Also, one thing I've found with Ampere's chips is they are most efficient for medium to mostly-utilized workloads. If you run it idle most of the time, you can probably get a lower power budget with other CPUs.
whats 17 vs 35 watts? honestly you waste and throw away more food than what you'll ever save in power costs.
Correct me if I missed something, but the idle was approx 150 W (let's say 100 without the drives). That's still more than a 200$ difference in electricity costs where I live.
But as Jeff said, this is a server for entirely different use case than a small home NAS.
And sorry if the comment came out wrong. It was more about me being silly about a few watts than about comparing server grade hw to small home nas with a plex server.
Yes i meant 17 watts vs. 35 watts of another system, not jeffs. Of course going from 350 to 17w idle is an entirely different story.
@@JeffGeerlingthat's something very few understand. just because it's arm, doesn't mean it will draw less power than x86 in all cases.
always use an ESD band for latests highe or even more precious vintage and retro components ,-)
Those look like Supermicro rails! FWIW, my 15 drive (rust) old E5 32t Proxmox system runs about 155-170w.
They are!
Given the price of the 15bays case, such loud fans is not excusable