Cutting my VM server power draw in half... and dropping 108 cores...
Вставка
- Опубліковано 9 січ 2024
- Patreon: / peterbrockie
Twitter: / peterbrockie
IG (rarely tech related): / peterbrockie
Disclosure: Some of the links below are affiliate links. This means that, at zero cost to you, I will earn an affiliate commission if you click through the link and finalize a purchase.
Stuff used in this video:
Proxmox Intel iGPU SR-IOV guide: www.derekseaman.com/2023/11/p...
PCIe fan mount: www.thingiverse.com/thing:352...
-Intel Version-
ASUS Prime Z790-A WiFi motherboard: amzn.to/3T8lKfj
Intel 14600k (the one I used): amzn.to/3NcrYai
Intel 12600k (bit cheaper): amzn.to/3RqK9LO
Intel 14900k (top-o-the-line): amzn.to/3T9727R
CPU contact frame: amzn.to/3uJFonR
Corsair 2x32GB 5200 kit DDR5 RAM: amzn.to/3uDx3Sp
-AMD Version-
ASUS Prime X670E-PRO WiFi motherboard: amzn.to/47H5k1Q
AMD 7900X (high cores, not cheap though): amzn.to/47Za2b7
AMD 7600 (cheaper, fewer cores): amzn.to/3RpVWtK
AMD 7950X (top-o-the-line most cores, very expensive): amzn.to/46JRomx
CPU contact frame: amzn.to/4159fTD
G.SKILL 2x32GB 6000 RAM "AMD version", probably fine with the cheaper stuff listed for the Intel build as it won't be going anywhere near 6000MT): amzn.to/3GL4plr
-Common Hardware-
Dual 25GbE card: ebay.us/glYEIE
Corsair Commander Core XT fan controller (I use a Commander Pro, but it seems to be discontinued): amzn.to/3uHYtqu
Noctua NF-A12x25 120mm fan upgrade: amzn.to/46F5cig
Noctua NF-A8 80mm fan upgrade: amzn.to/47WoMHK
Noctua NH-U12A (barely fits in the RSV-4500U): amzn.to/418nBmw
Noctua NH-C14S (not quite as good, but will fit most cases): amzn.to/413NSC6
Rosewill RSV-4500U case (15 bays, not hot swap, excellent cooling): amzn.to/3NbJ9IT
Rails for case (not great, but they work): amzn.to/3R7wkjX
PCIe 4.0 compatible M.2 adapter: amzn.to/3HczzCl
Seasonic 750W PSU (high quality PSU - recommended): amzn.to/46HbyO5
Corsair 1000W PSU (overkill unless you have a ton of drives): amzn.to/46LU7vI
LSI 9207-8i 8-port SAS controller: amzn.to/3N5NEof or ebay.us/tTUR2e
WD 4TB SATA SSDs: amzn.to/3N8VeyF
Misc hard drives. Personally I have found cheaper recertified drives to be just as good as new for way less.
12TB recertified SATA drives: amzn.to/4a6qpVk
14TB recertified SATA drives: amzn.to/47Uv7TS
18TB recertified SATA drives: amzn.to/3RrOZsq
Current Filming Equipment:
Sony FX30: amzn.to/3uTmAzb
Sony A7IV: amzn.to/3uqGsdT
Sony FE PZ 16-35mm F4 G lens: amzn.to/3JjOdI5
Sony FE 35mm f/1.4 GM lens: amzn.to/3ootA4e
Sony FE 90mm f/2.8 G Macro lens: amzn.to/3bFBVYD
SmallRig RC 220D video light: amzn.to/3E2ZomO
SmallRig RC 120D video light: amzn.to/3YIzoXg
RGB Video Light: amzn.to/3eqVGGs (not my exact light, but similar)
Desview R6 monitor: amzn.to/34CfKUI
AmScope ME300TZA-2L Trinocular Metallurgical Microscope: amzn.to/355bIkd
DaVinci Resolve: amzn.to/3gm2fLR
Buy me a cup of earl grey:
Bitcoin: bc1qf5yzmguzxmx7wmal36ancfvqf34ge08juvtnfk
Ethereum: 0x1B2df497F1c3bDa4b952Bd3cE266Ccf9D3d6cc30
Chia: xch1qtmtxej0tz2c82wj45mu5ncsu4s6z7nageec44qmc0p4dqexdv7s4a4rlg
Ravencoin: RKtLSABkqdFHBSL2ntb1pdC3PVEuc8Dt2Q
Raptoreum: RSECAdkYjcrGh7BogBYjXdB3oaThYdyFxA - Наука та технологія
I'm using a 4U Rosewill L4500U case with a Thermalright cooler because I'm cheap. The Phantom Spirit 120. It's
I can say with first hand experiance while working on flight simulators, our buildings had garbage power feeds coming in and we saw a lot of outages. When the power went out we scramed to get all the severs down before the ups dropped out.... if it was a long outage we could almost guarentee a dead drive in the building when we powered back up. If the ups didnt hold until you got the stack shut down you knew what rack was getting a new drive and battery tray! I replace my drives critical storage drives ever 5 years and toss the old good ones in my plex box since it always needs more storage.
SR-IOV is such a cool tech that's so underrated. I recently got it working on an 4-port Intel Gbe card in a Lenovo 1-litre PC that's pulling router/firewall duty for me (firewall running in a VM under proxmox). I immediately saw a huge speed improvement over my previous Ubiquiti ER-X, even when it had HW-assisted NAT enabled.
To be fair, that Edgerouter is kinda half baked, best leave that thing on 100Mbit
@@FrankyDigital2000 it's definitely not up to the speeds that are becoming more widely available today. When I got it, I had 200 Mbit/s service. I got an upgrade to 400, and I never saw it do more than 350. Could have been my config fiddling over the years, but with my new virtual FW, I'm over 500 regularly.
@@the_beefy1986 You probably had software switching enabled, I had the same issue on the ER 3 Lite, but disabled that and managed 900+mbps, and thats the lite.
Thanks for the video an appreciate the honesty on the build, so many people shy away because of a fan boy stuff, this video shown your like me i build a computer and select parts based on use case I have AMD and intel through out my network, I think AMD needs a better IO especially when boards come with so much stuff already, I did not know about the Virtualization issues with pass through though as i not come across this issue on my setups but its good to know, thanks bro.yeah the larger drives are crazy prices right now and it makes it harder to determine if larger cap with less drives is going to save on power and make its money back, i would wait a little then go for it as it would take X amount of time before you see the saving, but generally speaking you would save on power but depending how much you pay for the electricity will determine if its a saving or not. In Australia we pay 5 times the amount for electricity so it could save money at that point.
plus our government is on the save the world by increasing peoples usage on power driving up the price even more.
Windows Task Manager tends to report the highest clock of all the cores, which often changes and likely will never match HWINFO perfectly. If anyone knows more or has a correction, let me know.
Also I'm glad I just found this channel. Very cool ^^
ElectronicWizardly did a video about Proxmox on Intel. According to him, Proxmox cannot properly balance workload across P- & E-cores. Hopefully, Proxmox will make it happen eventually. Until then, hybrid CPUs are not the best choice for VM server.
I've seen it. He did his testing with an older kernel just before Proxmox 8 came out, so it possible it is all fixed by now, either way in a normal homelab it's really not something I would ever be concerned with.
Thank you.
I'd be interested in cinibench results for the old server hardware
Curious what your asset management situation is like for tracking things like equipment, sata cables, and squirrels.
I pretty much keep track of where everything is in my head. Except hard drives. I really need to properly record all the serials of the NAS drives since they aren't hot swappable I need to know what slot everything is in.
Squirrels manage themselves.
My linux workstation has 6000mt/s at 98gb (7950x, Trident Z), I'm not sure if 128 is totally different, but I'm sort of surprised that I just hit expo and after I verified it, never had to look at it again. I feel pretty lucky tbh, and I wish it would work better than it does.
The issue with spinning down drives is the strain on the motor at startup. Now that won't kill a drive quickly unless it's being spun up really frequently, but it is an added opportunity for it to fail. And with a decently sized array, you also have to consider the power surge of all that rust spinning up, something that won't of itself kill the power supply, but again, it's another opportunity for things to fail. I feel safer, especially as I use refurb drives, to set power saving to max, without spindown.
pretty sure motor itself can take it no problem, motor driver electronics might be affected though.
But mostly it is probably due to excessive heads parking and difference in vibration and balance stability of spin-up vs nominal RPM state.
Honestly I 'd like to see a real test from a channel which can afford to have a lot of drives going. I don't really buy the motor wear issue because although they are fast, they do take a couple second to spin up, so it's not like it's a massive amount of wear to get it going - especially in a helium drive where it should have way less air resistance.
It's just one of those computer myths that has always bothered me as I've never seen actual testing on it.
@@PeterBrockie A lot of drives in this case would be in 5-6 digit range, not hundreds.
Myself, I do not consider it a myth because historically there were at least issues with WD green drives that parked their heads as was programmed on the lowest level in firmware even though HDD was not set to any powersaving mode externally. And IIRC it did cause noticeable amounts of premature failures when used 24/7 in NAS. Folks used to disable this in firmware eeprom via program called wdidle.
WD Green series failed if you looked at them the wrong way even with the idle timer changed. :D
I've run my NAS with it on and off at different periods (15 drives) and I don't have any more failures than the usual "Is it DoA? Then it'll last a while" type failures which I think are the most common these days. But like your comment, mine is purely anecdotal so it'd be nice to have a proper controlled test. For science!
@@PeterBrockie Nobody is going to pay for test this size where you have to 'only use your tooling part of the time', or 'increase costs instead if decreasing them' if you'd have to use more space for the same amounts of utilization. Only the biggest of datacenters have real-world statistics and they sure won't let drives simply lay there without work or consume energy without producing revenue.
P.S. I ran only 2 greens, both of them for like 70-80k hours, one got couple of rogue remaps in a middle of lifetime, other one still good.. I had better luck with greens, blues and blacks than with RE, seagates and toshibas for example. But my choice is hitachis for sure, so WD HC now (used 1-5k hours, not new / 0h)
So I’m curious. Are all the device IOMMU groups separate on this board? I’m seriously looking at going this route, but would like to know.
For the AMD parts in the description, you could've mentioned the non-X version of the 7900 since it doesn't suck too much power, yet performs really well.
You can also turn down the power on a 7900X to make it perform like a 7900, on my asus MB there's an "AMD eco" setting under the PBO menu where you can choose the TDP you'd like. The only real difference between the SKUs is the 7900 comes with a cooler
I originally meant to mention Eco Mode in the video. I gotta start writing these things down before filming. Haha
I'm sure you can get a similar effect limiting PL1/PL2 on Intel but it isn't an easy to use switch.
@@maxhammick948 plus the non-x is a little cheaper "depending on where you are" so..
Can you share the 3d model of the fan attachment you used?
In the description already. :D
i cant even afford newer hardware for my personal home server currently but i wish i could afford to bht hardware sometimes has hiccups but it still works
One option could have been to use only 1 cpu off your dual cpu server. That can nicely shave off some watts.
Is the base file system zfs, ecc would have been nice?
The problem is you drop half the RAM slots and half the I/O when you run a single CPU. At that point I might as well just use a consumer board. :D
Would have loved to see you build a new server and transfer VM’s over with no / minimal loss of uptime.
If you’re limited to hardware you have, fair enough, but what I don’t see is a lot of homelabbers doing it the enterprise way when they swap hardware.
The thing is it simply isn't needed in a homelab. 99.999% uptime only matters if it actually affects people. When I do my changovers it's on Fridays when I'm off and the wife's at work - no one is affected other than me. There are a lot of videos on Proxmox clustering, etc. I'm sure something is out there that covers it.
It's actually the reason I don't bother having a cluster at home in the first place. It doesn't matter if my Plex server goes down to some weird failure - I'll just fix it when I get a chance to. :D The only thing which would matter is if my OPNsense system went down and my wife was working from home - even then it's easier to tell her "plug in your iPhone and teather to your work laptop" than having an LTE fallover or a cluster of servers all drawing power for the odd chance the fiber goes out (it has been offline for about an hour in ~3 years).
4:48 hold up. Are you sure you mean SR-IOV? this is available? why did no one tell me?
With modern Intel iGPUs. Arc doesn't seem to have it enabled - which is lame as hell. Link to how to set it up is in the description. :D
@@PeterBrockie is there a specific support list (UHD vs Iris Xe)? kind of regret buying the 12100f now damn. Will you do an update video at some point about the reliability?
@@tollertup I haven't personally tested with anything other than 12th and 14th gen. But from what I am reading any iGPU on 11th gen or newer is supported.
Although 11th gen seems to be somewhat buggy, so maybe stick with 12 or newer.
I'm running a 7900x with 128gb at 6000mhz. Passed in all mem tests.
There are no hard and fast rules. It's completely dependant on the RAM used, motherboard, etc. 128/192 is all over the place in speeds depending on what parts you have. It'll get better over time as BIOS updates fix DDR5 oddities.
Did you cut your "R" button in half too?
Just savin' power on typing time. Or something. :D