I also have a Proxmox Server with 2 HDD's and multiple SSD's in it. And the i7-6700 is really efficient with just 23W idle, running over 50 LXC's. I love Proxmox 😅
try changing the cpu governor to powersave instead of performance on proxmox this will reduce your turbo frequency and therefore your consumption. on my server it made me go from 300w to 186w in bench
I actually did that yesterday and it maybe made half a watt of difference. After I dug into the issue, it seems the newer AMD and Intel CPUs are already super efficient scaling their frequency to the workload. Even their performance setting scales down a lot at idle.
@@TechDregs it's especially in bench that the difference is seen after it's sure that on recent cpu it feels less. on my old xeon from 2011 the difference is just impressive.
zfs read/write will be done in RAM first, so if your test data fits in RAM, it wont even touch the the spinners until it flushes the transaction groups to disk. additionally, mirrors generally give the best performance. reads will be basically the speed of all drives added together, as ZFS will split the requests. writes will still be limited by your number of vdevs, which is one, so that'll be the speed of 1 drive.
When you were writing to the Toshiba drives, did you copy more than 512 megabytes? The technical spec sheet explicitly lists a 512 megabyte buffer that will most likely operate at or close to SATA 3 speeds, which is what you seem to be getting. This also means that anytime you're copying stuff that doesn't overload the 512 megabyte buffer, you'll continue to get those kinds of speeds - and the buffer will probably clear out at or close to HDD speeds, so you'd probably have to copy more than a gigabyte before it slows down.
@@TechDregs after finishing the video I'm less certain - it would explain reads but not writes. I was only half watching the video too - no offence. ;)
ARC is “built-in” with ZFS. You just need RAM. LOTS of RAM, preferably! ;) There is even L2ARC (layer 2 ARC), which is usually done in flash storage, but it isn’t helpful in most workloads: useful when you have lots of clients working on the same large working data, very unhelpful in homelab environments. 🙏 ZFS ❤
Nice build and thanks for sharing. Any thoughts on what would you improve or do differently after all this? You learnt about ASRock not going higher with the C states. Any thoughts on what would be a better choice when it comes to MB? Do you think you will use that power? Storing stuff from cameras, NAS functionality, home automation can handle old i3 mini pc, or even raspberry. I can see that you are after higher transfer speeds and lots of storage possibilities. But this can be achieved in minipcs as well with a lower power consumption. Although your initial one was not bad I wonder how it increase when you move all the stuff you plan. I run my homelab on old hp elitedesk g3 and when it comes to power it is still a beast for all the mentioned tasks. I run proxmox with 4 VM and more than 10 docker stuff. Considering change because of the case that is just too small to hold all the disks in one box. And also for fun. However not decided yet because currently my idle is under 10W and power consumption is important to me. I think processor are quite power efficient at idle these days so are easier to pick for Homelab than MB that would meet your expectations and not much more, run at high C states, not to mention ECC topic. :)
Other than perhaps finding a different motherboard, IDK. The thing with the motherboards is, I'd probably need to reduce the SATA ports and just add an HBA later if I really ended up need it, because there are very few boards with 8 SATA ports. I haven't researched specific boards though. As far as mini PCs and RPis, none of them can really do everything I'm doing. They lack I/O bandwidth, ports, and bays. Something like the Minisforum MS-01 looks neat, but costs almost as much, doesn't have drive bays, and has pretty much the same idle power draw. An RPi couldn't even run my NIC.
Thanks for the video - I'm planning to upgrade my server next year, and will be going down a power efficiency rabbit hole... was already leaning towards Intel because of the (apparently) lower idle consumption and iGPU transcoding support, but it looks like I'll still need to do a bunch of research 😅 I considered moving my Home Assistant to a VM on my server, but ultimately decided to get a Home Assistant Yellow instead. Whether that makes sense obviously depends on your usecase :) - but for me it means the automations keep running if I need to do maintenance on the server, and since the heaviest thing I run on HA is ESPHome builds, a CM4 has enough computer power, and eMMC storage is a lot more stable than doing a janky setup on a regular Raspberry Pi with SD-cards or (horror!) USB attached storage. The Yellow also has built-in Zigbee and NVMe support, so you can get even faster / more reliable storage than the compute module eMMC. Oh, and the Yellow should also support the new version 5 compute modules, for even more compute juice.
Yeah honestly the main thing I'm eyeballing with moving HA over is how it improves voice assist. However, that's more of a curiosity than anything. I don't actually use voice assist much (kind of like Jeff Geerling's last video on HA voice hardware). But the possibility of throwing a real GPU at it and getting lightning fast high quality voice responses is very tempting.
One advantages on AMD platform ： 4/4/4/4 pcie bifurcation Intel ： we try everything to lock up features variety from pcie bifurcation, voltage and multiplier setting, overclock etc 😂 But if you need transcode encode decode go Intel for sure Possible good option ： AMD 4004 eypc which was on am5 and officially supported ECC memory with correct board
Honestly, I think one of the AMD mobile processors with their IGPU would be pretty awesome for a home server; they just aren't cheap. I've went back and forth on ECC before deciding that I just probably didn't need it.
@@TechDregs their G series is too expensive compared to what they selling on mini PC, don't go for AMD am5 G series EPYC 4004 was similar to desktop am5 ryzen 7000/9000 with a weak 610m in the IO die part, provide very basic decode encode function , which wayyyyy too far back behind compared to Intel quick sync even the one on n100/n305 But on a consumer / basic server board without IPMI a IGP is best you can have to give a little help 4004 start from 4 cores to 16 cores so it just like a budget server version of ryzen (like Intel Xeon on consumer platform before Intel chop them down)
Don't put all your eggs in one basket..... if your CCTV system is an important piece of your security system, leave it on it's own dedicated piece of hardware (with a UPS) And then sync it to your NAS so you have two copies of the video...although you probably want to dedicate a drive or two just for this, and have it write over itself.
My NVR uses dedicated drives that are already mirrored. I'm just going to move those over, so everything will be kept separate other than the processing. UPS back up requires powering the cameras as well, so it takes a pretty decent system to keep them up in a power outage. I'm looking into that for a "maybe next year" thing.
Buy a high quality Seasonic PSU and return whatever cheapo you bought. Add a little extra power budget for future upgrades. You won't regret it...but you might regret a cheap PSU when it takes out your motherboard and drives. I usually run high quality power supplies for about 15 years (multiple builds) before I move them to less critical platforms. Mostly due to fear that the capacitors will dry/leak. The upfront investment is worth it for the lifespan you will get. I can't say the same for cheap PSUs.
I look at them. They are quite expensive. The Leadex isn't cheaply constructed. Superflower has a pretty decent reputation for reliability. But yeah, Seasonic is the gold standard.
That's a premium titanium class psu. You don't get titanium efficiency without using quality components. I would put that toe-to-toe with an equivalent Seasonic.
@@TechDregs I've been through so many disasters with "other" brands that I don't even think I've considered them. I generally don't mess with success. Lol. I used to build computers for people in an area where brownouts and lightning was common, and the only PSUs that didn't fail were Seasonic. After some preliminary research, maybe I'll give them a try for something. That said, if the price difference between a Seasonic and the next best PSU is $100 or less, I'm probably gonna go Seasonic. If you consider the long-term investment, that $100 is peanuts IMO.
@@RK-kn1ud Superflower is an OEM like Seasonic. They are known to build really high quality power supplies, they're just usually branded by other companies.
@@iwh7 _"which spinning disc nas has 10W consumption?"_ My ARM based RAID1 NAS with 2x1TB HDDs takes 15W *maximum* from the grid, and this data is not based just on an assumption made from reading datasheets or somebody else's anecdotes in the Internet but actual measurement done with an Amp meter at the mains. And it doesn't need to fall into a "idle mode" to achieve that. The $27 of annual electricity bill is what adds to & defines the *actual cost* of my home server.
I also have a Proxmox Server with 2 HDD's and multiple SSD's in it. And the i7-6700 is really efficient with just 23W idle, running over 50 LXC's. I love Proxmox 😅
try changing the cpu governor to powersave instead of performance on proxmox
this will reduce your turbo frequency and therefore your consumption.
on my server it made me go from 300w to 186w in bench
I actually did that yesterday and it maybe made half a watt of difference. After I dug into the issue, it seems the newer AMD and Intel CPUs are already super efficient scaling their frequency to the workload. Even their performance setting scales down a lot at idle.
@@TechDregs it's especially in bench that the difference is seen
after it's sure that on recent cpu it feels less.
on my old xeon from 2011 the difference is just impressive.
zfs read/write will be done in RAM first, so if your test data fits in RAM, it wont even touch the the spinners until it flushes the transaction groups to disk.
additionally, mirrors generally give the best performance. reads will be basically the speed of all drives added together, as ZFS will split the requests.
writes will still be limited by your number of vdevs, which is one, so that'll be the speed of 1 drive.
15:22 what file manager and config you use? cheers
When you were writing to the Toshiba drives, did you copy more than 512 megabytes? The technical spec sheet explicitly lists a 512 megabyte buffer that will most likely operate at or close to SATA 3 speeds, which is what you seem to be getting. This also means that anytime you're copying stuff that doesn't overload the 512 megabyte buffer, you'll continue to get those kinds of speeds - and the buffer will probably clear out at or close to HDD speeds, so you'd probably have to copy more than a gigabyte before it slows down.
It was a 37.6GB video file, which is larger even than the amount of RAM I have on the server (32GB). That's what's so odd about it.
There is also the ZFS cache which will highly skew benchmarks. Good for the user though!
Your disbelief of the mirror results - consider ZFS ARC.
Yeah, I need to read up on that. Thanks.
@@TechDregs after finishing the video I'm less certain - it would explain reads but not writes. I was only half watching the video too - no offence. ;)
Yeah, as I mentioned to someone else, the file I was transferring is larger than the RAM on the server. So, it really is strange.
ARC is “built-in” with ZFS. You just need RAM. LOTS of RAM, preferably! ;)
There is even L2ARC (layer 2 ARC), which is usually done in flash storage, but it isn’t helpful in most workloads: useful when you have lots of clients working on the same large working data, very unhelpful in homelab environments. 🙏
ZFS ❤
Nice build and thanks for sharing.
Any thoughts on what would you improve or do differently after all this? You learnt about ASRock not going higher with the C states. Any thoughts on what would be a better choice when it comes to MB?
Do you think you will use that power? Storing stuff from cameras, NAS functionality, home automation can handle old i3 mini pc, or even raspberry. I can see that you are after higher transfer speeds and lots of storage possibilities. But this can be achieved in minipcs as well with a lower power consumption. Although your initial one was not bad I wonder how it increase when you move all the stuff you plan.
I run my homelab on old hp elitedesk g3 and when it comes to power it is still a beast for all the mentioned tasks. I run proxmox with 4 VM and more than 10 docker stuff. Considering change because of the case that is just too small to hold all the disks in one box. And also for fun. However not decided yet because currently my idle is under 10W and power consumption is important to me.
I think processor are quite power efficient at idle these days so are easier to pick for Homelab than MB that would meet your expectations and not much more, run at high C states, not to mention ECC topic. :)
Other than perhaps finding a different motherboard, IDK. The thing with the motherboards is, I'd probably need to reduce the SATA ports and just add an HBA later if I really ended up need it, because there are very few boards with 8 SATA ports. I haven't researched specific boards though.
As far as mini PCs and RPis, none of them can really do everything I'm doing. They lack I/O bandwidth, ports, and bays. Something like the Minisforum MS-01 looks neat, but costs almost as much, doesn't have drive bays, and has pretty much the same idle power draw. An RPi couldn't even run my NIC.
Thanks for the video - I'm planning to upgrade my server next year, and will be going down a power efficiency rabbit hole... was already leaning towards Intel because of the (apparently) lower idle consumption and iGPU transcoding support, but it looks like I'll still need to do a bunch of research 😅
I considered moving my Home Assistant to a VM on my server, but ultimately decided to get a Home Assistant Yellow instead.
Whether that makes sense obviously depends on your usecase :) - but for me it means the automations keep running if I need to do maintenance on the server, and since the heaviest thing I run on HA is ESPHome builds, a CM4 has enough computer power, and eMMC storage is a lot more stable than doing a janky setup on a regular Raspberry Pi with SD-cards or (horror!) USB attached storage.
The Yellow also has built-in Zigbee and NVMe support, so you can get even faster / more reliable storage than the compute module eMMC. Oh, and the Yellow should also support the new version 5 compute modules, for even more compute juice.
Yeah honestly the main thing I'm eyeballing with moving HA over is how it improves voice assist. However, that's more of a curiosity than anything. I don't actually use voice assist much (kind of like Jeff Geerling's last video on HA voice hardware). But the possibility of throwing a real GPU at it and getting lightning fast high quality voice responses is very tempting.
One advantages on AMD platform ： 4/4/4/4 pcie bifurcation
Intel ： we try everything to lock up features variety from pcie bifurcation, voltage and multiplier setting, overclock etc 😂
But if you need transcode encode decode go Intel for sure
Possible good option ： AMD 4004 eypc which was on am5 and officially supported ECC memory with correct board
Honestly, I think one of the AMD mobile processors with their IGPU would be pretty awesome for a home server; they just aren't cheap. I've went back and forth on ECC before deciding that I just probably didn't need it.
@@TechDregs their G series is too expensive compared to what they selling on mini PC, don't go for AMD am5 G series
EPYC 4004 was similar to desktop am5 ryzen 7000/9000 with a weak 610m in the IO die part, provide very basic decode encode function , which wayyyyy too far back behind compared to Intel quick sync even the one on n100/n305
But on a consumer / basic server board without IPMI a IGP is best you can have to give a little help
4004 start from 4 cores to 16 cores so it just like a budget server version of ryzen (like Intel Xeon on consumer platform before Intel chop them down)
Don't put all your eggs in one basket..... if your CCTV system is an important piece of your security system, leave it on it's own dedicated piece of hardware (with a UPS)
And then sync it to your NAS so you have two copies of the video...although you probably want to dedicate a drive or two just for this, and have it write over itself.
My NVR uses dedicated drives that are already mirrored. I'm just going to move those over, so everything will be kept separate other than the processing. UPS back up requires powering the cameras as well, so it takes a pretty decent system to keep them up in a power outage. I'm looking into that for a "maybe next year" thing.
why not intel xeon
Cost/performance. You have to go used Xeons or you have to drop a lot of money. Plus I get an igpu, which could come in useful.
Buy a high quality Seasonic PSU and return whatever cheapo you bought. Add a little extra power budget for future upgrades. You won't regret it...but you might regret a cheap PSU when it takes out your motherboard and drives.
I usually run high quality power supplies for about 15 years (multiple builds) before I move them to less critical platforms. Mostly due to fear that the capacitors will dry/leak. The upfront investment is worth it for the lifespan you will get. I can't say the same for cheap PSUs.
I look at them. They are quite expensive. The Leadex isn't cheaply constructed. Superflower has a pretty decent reputation for reliability. But yeah, Seasonic is the gold standard.
That's a premium titanium class psu. You don't get titanium efficiency without using quality components. I would put that toe-to-toe with an equivalent Seasonic.
@@TechDregs I've been through so many disasters with "other" brands that I don't even think I've considered them. I generally don't mess with success. Lol. I used to build computers for people in an area where brownouts and lightning was common, and the only PSUs that didn't fail were Seasonic.
After some preliminary research, maybe I'll give them a try for something. That said, if the price difference between a Seasonic and the next best PSU is $100 or less, I'm probably gonna go Seasonic. If you consider the long-term investment, that $100 is peanuts IMO.
@@RK-kn1ud Superflower is an OEM like Seasonic. They are known to build really high quality power supplies, they're just usually branded by other companies.
The efficient home NAS & server is when it consumes 10-15W max. Whatever above is not efficient.
What do you expect from a 13th gen Intel CPU? And 10-15W vs 25W idle isn't much of a difference.
which spinning disc nas has 10W consumption?
Yeah, these two HDDs are close to 9W at idle by themselves.
@@iwh7 _"which spinning disc nas has 10W consumption?"_
My ARM based RAID1 NAS with 2x1TB HDDs takes 15W *maximum* from the grid, and this data is not based just on an assumption made from reading datasheets or somebody else's anecdotes in the Internet but actual measurement done with an Amp meter at the mains. And it doesn't need to fall into a "idle mode" to achieve that. The $27 of annual electricity bill is what adds to & defines the *actual cost* of my home server.