Proxmox on Erying Motherboards! - Testing Hybrid CPUs in Virtual Machines
Вставка
- Опубліковано 28 січ 2024
- UPDATE - Check out Part 2 - IT WORKS NOW!
• PART 2! - Proxmox on I...
Grab yourself a Pint Glass or Coffee Tumbler at craftcomputing.store
In a recent video, I criticized Intel for their decision to remove Efficiency Cores from their Xeon E-2400 Lineup. BUT, you all pointed out the reason was likely the lack of support for Hybrid CPU architectures in VMware. I don't know if you're aware, but other Hypervisors exist, especially in the SMB market. Today, I'm testing out Proxmox with Intel's big/LITTLE CPUs, thanks to a pair of Erying motherboards purchased from AliExpress.
But first... What am I drinking???
Silver Falls Brewery, Like Yesterday 90s IPA (7.4%)
Links to items below may be affiliate links for which I may be compensated
Erying i9-12900H (ES) Motherboard: s.click.aliexpress.com/e/_DnD...
Erying ITX i7-13620H: s.click.aliexpress.com/e/_DCa...
Follow me on Mastodon @Craftcomputing@hostux.social
Support me on Patreon and get access to my exclusive Discord server. Chat with myself and the other hosts on Talking Heads all week long.
/ craftcomputing - Наука та технологія
![[柴犬ASMR]曼玉Manyu&小白Bai 毛发护理Spa asmr](http://i.ytimg.com/vi/0TsXQ7z2Dh4/mqdefault.jpg)
Proxmox dev here. We have some workstations running with 12700k/13700k and I can't remember any ongoing stability issues that were CPU related. The only stability problem we encountered (AFAIR) was too high clocked memory speeds (early DDR5 modules though). If you can replicate the hangs/crashes maybe you could open a bug report with logs etc. on our bug tracker?
Edit: to clarify, our development workstations run with Proxmox VE of course 😅
I was planning on getting a 12400 to run proxmox for my homelab, so I'm now very curious. I'l keep an eye on this. Good to see you guys are on top of it!
Thanks to you & the team for all your work!
The 12400 only has 6 p-cores and 0 e-cores
A Proxmox dev out in the wild? Thank you for the work you do! 😊 Proxmox makes my job so much easier than the other hypervisers I sometimes have to interact with.
Not using proxmox yet, but my friend says you are the best, he uses it a lot, thank you for your work
I wrote the code to implement the CPU Affinity feature in proxmox. (Dot on the Proxmox Forums)
I haven't tried using P vs E cores for gaming VMs, but I do leverage affinity for E cores to force lower power profiles on the CPU.
By pinning some of my constant-load VMs to E cores, they are never responsible for putting the CPU into a high power draw state.
This saves ~20W of power on average in my usage (security system designed to run off battery backup).
It looks like my comments on this video are being immediately deleted after posting. Hopefully this comment won't disappear *crosses fingers*
Yay, the comment didn't get deleted this time. It looks like adding url links to these comments will get your post deleted.
URLs are disabled on all my videos to prevent malware or malicious links. Do you have a keyword to search for or a github user/project name you can share?
Yeah, there's 2 relevant links to the affinity discussion.
1. , Title: 'CPU Pinning?', forums url endpoint: '/threads/cpu-pinning.67805/'
2. , Bug Number: 3593
A user on the forum asked how one could achieve CPU pinning. One of the proxmox employees, t.lamprecht, suggested using the `hookscript` which calls arbitrary bash before, during, and after the VM lifecylce. During the startup hookscript, t.lamprecht suggested users put the 'taskset --cpu-list --all-tasks --pid 0-11 "$(< /run/qemu-server/104.pid)"' command, which will take a pid (and all of it's children) and restrict the pid to the specific cores. It works pretty well, but it was painful to setup this hookscript for every single vm.
On this forum thread, the proxmox bugzilla bug was created 3593.
I (Daniel Bowder) picked up that bug and made the relevant changes to the proxmox codebase to inject the `taskset` command before launcing the kvm/qemu process for the VM at startup. It was then added in PVE 7.3. It's my only contribution to PVE that I have made, but I am very proud of it. I found it super usefull. (I also got a little scared about the proxmox code base lol, Did you know they are mixing tabs and spaces for formatting!!! tears I can't get VScode to properly format the code files.).
Anyway, the logic behind the affinity is really dumb, but this is now something that can potentially help you investigate further. The taskset command above can be called WHILE A VM IS RUNNING, and reallocate it's cores on the fly.
CORES=0=11
QEMU_KVM_PARENT_PID=$(cat /run/qemu-server/104.pid)
taskset --cpu-list --all-tasks --pid ${CORES} ${QEMU_KVM_PARENT_PID}
You could use this to quickly flip back and forth between cinebench runs. (Running htop on the PVE host is really fun when you do this).
@@CraftComputing My posts even without links are getting deleted. Not sure what's going on. :shrug:
Thanks for doing that rundown Jeff, pretty interesting stuff. Its cool to understand what the limitations might be with these different core layouts
Great video! Was very entertained with all the chaotic permutations of results. I think I read somewhere that virtualising a Proxmox VM, and then virtualising Windows within that leads to better(or rather more stable) results.
Some Winception right there lol
Cheers Jeff! Enjoying a Coronado Brewing Orange Ave wit. Excellent video, your Erying series really interests me for my homelab cluster i am planning on building, thank you for the upload!
I am running MINISFORUM MS-01 and don't have any of these issues. Super interesting to see.
I'll be testing an MS-01 shortly, so thanks for chiming in with this.
I have been running Proxmox on my 12th Gen i7 (8p/4e + HT) for 2 months now, passing trough the GPU to Win11Pro & Fedora Linux VM. I did not have a single issue, and it is rock solid. I only assign 10 out of 20 threads to any of the VMs. I an not treading a hyper-tread core, as something you can assign to the VM, but for Proxmox to use when needed/possible. - I will add some more workloads to proxmox, and investigate the stability. Keep up the great work!
Did you end up noticing any performance issues? Genuinely curious as I've been seeing a lot of mini-PC deals with p/e core CPUs lately
you can set it to be NUMA aware, then set CPU 0 to use E cores and CPU 1 to use P cores.
@@System0Error0Message would this be in bios settings or in proxmox itself?
@@manofwar9307 proxmox. when making a new VM i always used advanced for more settings. under CPU always set NUMA aware and you can also set it to host CPU so the guest OS and software can prefer its own core priority as needed as well. In programming languages they've long supported big little but thats down to the software to implement it.
You can make it look like a 2 socket CPU but im not sure you can properly set the affinity in proxmox.
@@System0Error0Message awesome, thanks!
This was a very interesting video Jeff. Always very interested in the virtualisation topics and learned a lot here !
I'm actually curious about one thing, I am not a drinker, so the parts about beer etc at the end, I usually skip and go to the next video, I'm wondering if that hurts your analytics with people doing that or if I"m just an outlier.. great video as always, thanks man
The reason they're at the end is so people who aren't interested can skip them. If you click off the video at 16:00 or 18:45, it doesn't matter to me. Like and comment while you're there though ;-)
Thanks for covering the big/little proxmox setup. I've been wary of dropping the cash to try it. Hope to see some future updates around this where you get it to work.
just one moment - he's trying it on a weird chinese mobos, kinda a stretch to test it like that for stability
This video was perfectly timed as I was just checking out these Erying boards. I'm pretty happy with my current homelab (P360 Ultra) but i was eyeing the ITX boards for a low power gaming system. They are definitely priced better than the minisforum stuff.
I ran proxmox on a 13900k. It hosted 2 plex servers both with their own gpus and a truenas vm with hba passthrough. I didn't notice any stability issues and all 3 vms ran smooth for the entire time. None of which were Windows. Maybe theres something with KVM and Windows on Big/Little arch.
It seems like this may be more a windows isue.
@@mikehathaway2842Generally if it's even remotely possible for a problem to be caused by Windows... the problem will be found to be with Windows. 😂
if efficiency cores can be statically assigned in proxmox, i would definitely assign them to LXC containers and try to run them :(
For VM CPU Affinity, we run the `taskset` program which allows us to specify which cores a given `pid` (and it's children) will be allowed to run on. We can get the QEMU parent `pid` for any VM by catting the /run/qemu-server/.pid file. (eg: `taskset --cpu-list --all-tasks --pid 0-11 "$(< /run/qemu-server/104.pid)"`)
I don't know how we would do it for LXC. I haven't looked at that part of the pve codebase.
I can confirm this works. Just run lscpu -e and find the core number you want to assign
you can set affinity for separate processes or cgroups, but afaik you can't do that for the hypervisor itself, as it's not 1 process/cgroup
Great video. Been wondering about this for a while now.
Thank you Jeff! You always answer all my proxmox questions & ideas! 😊
last year i intentionally opted for the 10th gen i9 in my home server build just to avoid the weird core sizes.
Hi Jeff! So I can say that based on the places I have worked, it is common practice to overcommit resources and allow the running VM's to hang out waiting to be serviced (like being at the DMV). So as volatile as your test environment was, how much worse would it be if you overcommitted by say 33% or more? It would be curious to see if it was that much worse, much-much worse or surprisingly better (stability/crash wise of course). Just a thought...
Yes, I've seen (and done) quite a bit of overcommitting in my day as well. My comments in this video about scheduling are *pure speculation*, as even when undercommitting CPUs, I still had a ton of instability.
@@CraftComputing, understood sir. Thanks for the reply!
I seem to remember seeing a different video talking about how weak the VRMs are on this board, and to get it working without issue it needed a fan directly on them. I wonder if you were experiencing that issue?
The VRM's get so hot you can't touch them. I added a nvme heat sync to each VRM and problem is solved. Now they barley even get warm. Worth the extra $20 in my book.
No fan needed and they look cool in my book :)
Thermalright HR-09 2280 PRO Black 2280 SSD heatsink
New opening for the video. Wasn't expecting that.
The channel Hardware Haven used a 13th Gen CPU with Proxmox and didn't report any problems. Maybe because he didn't run at full power. You should ask him if his Mini PC crashes at full load.
actually, this gives me an idea. What if instead of running laptop cpus you just aggressively undervolted/underclocked a similarly specced desktop cpu?
Surely you could get pretty similar performance levels at similar power levels, with the added bonus of not being ebay specials. Although that is part of the fun i suppose.
Interesting! My 13600K system (on Gigabyte Z690 Aero D) has been rock solid running Proxmox (multiple weeks uptime without issue). After watching this I decided to give it a little test. With one VM and some containers running in the background (though not doing much), I decided to spin up two VMs each with 10 CPU cores (maxing out the 20 threads of the 13600K with 6 P cores with HT and 8 E cores without HT) and I ran Cinebench r15 multicore on both at the same time (multiple runs) and the runs completed without any issues. CPU usage goes right up close to 100% on the system, but didn't seem to freeze or crash. May be something with the Erying boards (or the mobile CPUs they're using) that's causing an issue.
Awesome video, thanx for testing 👌😊
Most people, even those in the IT space don't realize that Intel has a large enough product stack to hypersegment. They may sell several thousand units to a telecom for a single purpose and that company only needs a certain level of performance. Then another tier could be predicted to sell hundreds of thousands of units. That main tier may have slightly defective units that create yet another segment, sold at a discount to another market.
Thus it's difficult to say that a given product is bad or good relative to another form Intel, since sales (generally) represent a balance of price and performance from the buyer's perspective. Hence why we need reviewers and testers who can judge how well something will work for its intended use case.
Engineering sample CPU could be what's crashing the host. I had one 9700f QS a few years back and even though QS are supposed to be "final" as ES are not, mine crashed the pve host when the guests ran handbrake video conversion, very much like the way you described. I can't be 100% certain because I didn't have an alternative cpu for testing.
I suspect it's the boards that are unstable, MCE usually indicates a hardware error.
this is most likely due to a bad motherboard or BIOS, such Frankensteins have never been stable, especially from Chinese companies
+ there will be zero support, enthusiasts will make working versions of the BIOS
@@rpnXN it was because of the microcode, he made a new video about it
I'm curious if you'd do this experiment again on the ZimaCube Pro (i5 1235U). You know... when it arrives.
I would love to see you run the same tests via Hyper-V, since the Windows OS (as of Win11) is supposed to be BIG.little aware, and should be able to schedule work across the VM threads.
Sadly so far no - HyperV does not utilise ecores
Curious if others have had weird issues with their erying boards. I have a 13420h matx board and first only one ram stick was detected, turned out this was because of using a 2-port (8 sata) hba, a 4-port (16x sata) solved the problem. But then the Realtek LAN stopped working, couldn't even detect it in hardware, in BIOS it's enabled. Lastly wifi randomly doesn't start in Windows (code 10 or 43). It was valued $25 on the package, i kinda get why. 🤷
So I could use a 10G network card on the ES board?
I'm interested to use one of those for a Plex server (NAS has the media storage). Do the ES still have the same intergrated GPU as the retail chips?
And what memmory speed can I go up to on those chips?
How about them LIONS!!!!
Not many YT creators into computers and sports. Good to see one on here.
New intro is cool. Constructive feedback; maybe half or so second shorter. Sound level is a bit too low can't hear it without turning up the sound to max. I thought it had no sound at first.
IMO if your intro is longer than 5 seconds and doesnt have any relevant information in it, it's too long and you need to shorten it.
I did not look at the timer. I just thought to myself, "this is outlasting my attention span".
it's like 30 seconds long the videos usually start at 1 minute though.@@_vilepenguin
I hope this is sorted out. SME and homelab hypervisor is the most promising use case of big little. I'd love a system with 2P cores and 8 or 16 E cores for containers and light VMs
Can you try to iGPU performance on the 13620H? it is a UHD with 64EU (vs. 32EUs in desktop version)
Would be good to test out width the erying boards altogether as they are so weird and hacked together it might just be them causing the issues.
My Tiger Lake boards have been running 24/7 for 8 months with pretty strenuous loads, and have been 100% stable. This was a software issue, not hardware.
Try using a high quality board to rule it out? I run a w680 board with a 13500 it works great
agreed, this might be a board issue.
Also, I use Hyper-V and your GPU-P guide to share my gaming PC with my girlfriend. I use a 12900 KF. Never had a problem, but I have also never oversubscribed system resources.
Hmmm, I'd be interested in whether much lower CPU utilisation would be stable long term for home slabbing. I went with M500+s recently, though I'm liking the M600s now as I can use the 2.5 disk for proxmox and the dual NVMe drives for ZFS mirrored storage.
When running a single VM, I can confirm that Proxmox has effectively identical performance as bare metal on heterogeneous cores.
I have a 13700K system with 64GB RAM. When I gave a Win11 VM all cores + 60GB RAM (4GB left alone for Proxmox itself), it gave pretty much the same benchmark results across the board as an identical bare metal Win11 install. I also ran single-threaded performance tests, not just full blast benchmarks. From that, it seems that P and E core scheduling was comparable for that scenario.
That said, I don't know the intricacies of Proxmox CPU scheduling so it might behave differently with multiple VMs splitting the resources vs. one VM that's given everything.
I can confirm with proxmox running Windows VM, isolcpus and static CPU pinning, it runs at the same speed or faster than bare metal. Have a 13900k with all cores and threads giving me the same cinebench R23 score. If I give it 8 less E cores, use isolcpus to make the hypervisors to run exclusively on those, and statically pin all the threads. I emulated a 13700k and got higher R23 score than an average 13700k.
Ultimately, I did isolcpus to keep the hypervisor and smaller VMs in the last 8 E cores. Then split the rest evenly across 4 Windows VM, each with 6 threads (2P+2E) with GPU passthrough, and even if I run R23 on them at the same time they outperforms a i7 8700.
If you are wondering why high end motherboards have so many PCI-E slots, it is because of crazies like me and the cost is still lower than server-grade components
I purcahsed a Erying 12500H board 13 hours ago (video age 8hours at time of comment) I only plan to use it as an UNRAID NAS / Docker box - HBA card should work in the PCIE slot without too much issue I hope. I have seen some comments of the Engineering Sample ones and or these boards specifically can not do IOMMU passthrough (not an issue for my use case, but something to think about for others?)
The value proposition of these boards even with the ES chips is great for home labbers so glad they are getting some coverage here!
I found the exact same problems with an Erying board with ES i5-12450H. It usually crashed when trying to reboot a VM and PCIe passthrough was downright broken, with the passed through device remaining in an active state despite the VM being powered off, rendering it useless unless the system is rebooted.
Let's say I was thoroughly disappointed about that. In XCP-NG passthrough was equally broken, but the VMs never crashed as XCP-NG is running Linux kernel 4.x still.
Wonderful test! have you ever thought to test it on a router or server with Intel Ultra9?
I've never had an issue with careful over-subscription in proxmox. I've been told that best practice is to not do so... but, consider that it's likely that your VMs won't see full load at the same time, or anywhere close to it.
I should probably spin up a test node and try and see if I can break it though, would be curious to see. I know I've definitely mapped cores 1:1 with the host on VM's that are used for render workloads, but, the real test would probably be running actual stress tests, like linpack or stress to see if actual 100% utilization upsets proxmox.
I would guess that KVM will eventually support all of this eventually, and likely well before everyone else does.
I've overprovisioned CPUs plenty without issue. The comments about underprovisioning to allow cycles for scheduling was completely speculative on my part.
@@CraftComputing ah, I was wondering if you knew something I didn't. It seems like many people say this, but, I can't find anything even manpage adjacent that says that'd be the case tbh
I've had huge instability on a 12th gen (i5-1240P) NUC using promox until I switched the memory to Intel validated ones. I haven't had another crash since.
Good point, bad memory can cause significant problems in servers
I was running the DDR4 at 2666, but didn't check the validation list. Might be worth investigating. Thanks!
I have running Proxmox on NUC12 with i5-1240P and Kingston FURY DDR4-3200 for a month with no issue.
I too am having issues with Proxmox, on the MS-01. Lockups, web interface crashing, VNC constantly failing to work.
Isn't this caused by the differing instruction sets supported by the E and P cores?
That was always the thing holding me back als i like to pass AVX512 to the vms
I passed through each CPU as an x86-64-KVMv2+AES, meaning I was already limiting instructions to sets both are capable of.
@@CraftComputing I wonder if some KVM/scheduling code requires an instruction that's available on the P-Cores but not on the E-Cores, so when the scheduler schedules itself on an E-Core it just hangs.
Tho I guess that would show different symptoms...
Either way interesting results.
@@insu_naThe E and P cores use the same instruction set to eliminate that problem. None of the 13th Gen Intel chips support AVX-512 because of the E-Cores don’t have it.
@@Knirin makes sense
I can't wait to hear your review of the Erying ITX i7-13620H motherborad in terms of gaming I'd also like to know the Xe performance as well. I got a tiny case that would be perfect for it, but I would like to know if the integrated GPU will be good enough to out perform a RX6400, or at least match a RX6500xt I'm wanting to make a low powered SteamOS machine with lots of storage for right now(since it dose have 3 M.2 ports). Once they get Nvidia support working I can later on add a LP 4060 (unless AMD brings out a newer LP 7000/8000 by that time).
hope youre dealing with the loss okay. pretty bad feeling going to bed last night, feeling better about it now. go lions!
Won the division, and came 3 points shy of the NFC Title. Hard to complain about those results after 65 years of futility.
@@CraftComputing It is hard to complain but man we were SOOO close, if only we had kicked that field goal to force overtime. Also, I didn't know you were a Lions fan! #OnePride
We missed you!
@CraftComputing Do those CPU/board combos come in an ITX form factor?
For power saving purposes (much as possible), should I choose an Intel Xeon combo (Aliexpress used) or an Erying card. This is for a virtualization server for docker containers.?
I recently got an Erying with a i9-13980HX for the sole purpose of running VMs. Not everything needs to run on P cores. I run all my demanding stuff on my main server with a six-core i5-8400. But a lot of stuff I run in VMs happily runs on my secondary server with an ancient 15 year old (!) two core AMD CPU. With 12 GB of RAM, it runs 8 (!) VMs, for various low-demanding tasks while also sharing the storage space with the main server. Since that machine is really ancient (apart from the PSU), I plan to replace it with the Erying. That's where the E cores will shine, they will happily run the existing low-demanding VMs, while leaving a lot of room for adding new ones for more demanding tasks.
Love this channel. I mean who doesn't like craft beer and pc hardware? :D
I'm curious to see a couple of scenarios:
1. 4 VM each with 1P + 1E core
2. 3 VM with 4P, the 4E cores reserved to proxmox which should be great for containers
Love the Erying Motherboards, have an M-ATX Erying i7 12700H, using it as a gameserver, had mine since Jun 11, 2023, running 24/7, no issues :) Thinking of getting the New 13th gen Erying with over 5Ghz and DDR5 support.
One of my Tiger Lake 11900H boards is currently running Minecraft and Palworld servers :-)
@@CraftComputing i recently got the ES 13900h from "17029". its only a ddr4 model but unlike most of the other engineering samples my sample has pcie gen 4.0 8x.
i was only able to find the 12700 / 12900 ES sample boards that have pcie gen 4.0 8x but there are also a large majority of them are pcie gen 2 (strangely they state m.2 is still 4.0 x4 so it might actually be feasable to put a pcie 4x card like a rx 6400 and use the card to its fullest.
my supplier has also told me hes willing to give me the modified bios to make the 13900h ES overclockable! :D
I wonder if the reliability would improve using desktop cpus, such as the i5-13500/i5-13500t. I've been eyeing one of those for home lab use for a while.
My Tiger Lake boards have been 100% stable since I installed them. Testing these Alder/Raptor boards in Windows they had no issues whatsoever. I think the stability was purely software.
@@CraftComputing Yeah, in that context that sounds like a valid conclusion. I think I'll have to keep watching this topic to see if big little on proxmox becomes viable, but at least I'm in no hurry.
What CPU type did you use? kvm64, host or something else?
x86-64-v2-AES
@@CraftComputing I'd try other types first and Linux server VMs with GeekBench second.
Proper mobo, desktop CPU third guess.
Did you ever think that your instability problems might not be Proxmox, but board and cpu problems?? The second cpu seem to be more stable....could a better board that isn't so "weird" be an idea to work with??
You could also be working with a flaw in the mobile CPU architecture that may not be a problem with desktop chips.
If you reran these tests with a more mainstream motherboard and desktop chips, you make it different results....
Thank you for the video;
Monte
Could it be that the SSD you have is faulty? Also I assume you are running everything stock. Xmp also not good to have on in this case or at least not during initial testing
I moved mostly away from virtualization in favor of containers. Debian + Docker on 12650h runs flawlessly and makes great use of P and E cores.
How about running all containers vs VM ? and TrueNas as the front end ?
Where do you source your statistics from? Before Broadcom's acquisition of VMware, the size of the companies didn't seem to matter, whether they were small or large. Companies that valued stability and usability often chose VMware. Hyper-V and Proxmox were rarely used in comparison. Some companies might opt for solutions like VirtualBox or VMware Workstation Pro for their virtualization needs :)
Can you just keep one E core available for the hypervisor and push the vms to 100%?
Thanks for the video❤
I have been using Hyper-V with i9-13900H (Minisforum MS-01) and I encounter no stability issue at all. Yes, the performance load balancing is odd and sometimes heavy workload stuck at E cores, but at least I have nothing to worry about if I just want to put more VM running. Normal operation in Win10 inside Hyper-V guest is also very snappy, much snappier than my previous E-2244 / i7-8700.
I like the new intro
Best year for the Lions since i was born! I miss Barry Sanders but I thought we really had a chance 😞
I'd be tempted on doing this with a 14700. Leave a full E core cluster dedicated to proxmox and then have 8 big and 8 little available for the VMs. I would not have more than one VM on the same cluster due to the shared L2 cache though. I have a feeling that could potentially be the source of the trouble.
i never allocate all threads to vm's on any hypervisor. i alwas leave two unallocated for the hypervisor's use.
I9 12th gen had ecc enabled.
Id like to know if it works in these es samples
I have been running Proxmox on Intel NUC12 with i5-1240P for a month with no issues. At this moment I running 9 VMs without CPU Affinity, but only assign 1-4 vCores to each VM
I’d be tempted to try a retail desktop cpu in a retail board if you have one incase it’s something weird with those mobile parts/board combos
Couldn't the issue arise from cinebench using some instruction set which the E cores don't support? A thread get scheduled to a P core, cinebench gets the capabilities of the CPU, uses some P core only instruction set, proxmox schedules the VCPU to an E core -> crash. But it should be caught by an illegal instruction exception which can be gracefully handled, so maybe my theory is incorrect.
Can't wait for my new order of 2 glasses and coffee tumbler.
I have been running 2 servers using hybrid CPUs for nearly a year now without a single issue (i5-12600K (6p/4e, 16 threads) and i3-12100F).
Both running ProxMox with TrueNAS as a VM. The i5-12600K server also running a Gaming VM and some other services (NextCloud, VPN etc...).
I can send more info on my setup if anyone is interested, but those are my two cents.
Whats the idle on the 12600k system?
I'm building a PC for containers but am confused what spec should I do!
Photoprism for photos,
Openmediavault
Adguard DNS
Jellyfin for tv shows and movies
Home assistant for controlling lighting and for 2x home cams
Nextcloud (Want to poweruse it!)
Zfs raid shared 2x 2TB HDD
Also came across Mycroft, like an Alexa/ Google assistant alternative.
PS suggest some quality of life apps;)
@@user-ek5uq6ce2k I assume you mean power consumption?
About 80W without the GPU (measured before I got it). Under somewhat standard load (TrueNAS, 4 HDDs - file access, snapshots, replication etc.) I measured 45kWh over 17 days (roughly 2.5kWh/day).
@@svejdik313 oof thats really high. I am running 8700k, 5 Drives, 2 NVMe drives, 10g SFP+ NIC, 4 constantly on fans, on Windows with Hyper V + 4 VMs for Automation etc and my power draw is 50W at idle. It does around 1.4 kWh per day. I was thinking of upgrading to 12th gen but that high power consumption is too much.
I am running a Seasonic Prime Fanless PX 450W Platinum PSU though...which might be helping a bit
@@user-ek5uq6ce2k Yea... I don't even want to know how much it uses with the GPU and Gaming VM 😅
Honestly the CPU is way overkill for my uses. It averages 5% load and maybe gets up to 30% if I run some extreme compression.
I'd definitely use a less powerful one if I was building this again.
Got a great deal on an HP mini PC with i7-12700T.
Works great for ProxMox, no issues whatsoever.
VM' actively using 21 cores on a Ryzen 8 core 16 thread CPU. I think I'm fine. And loads are pretty low, so does not need to pump high performance. It's my happiness!
Did I miss something on the Erying Boards? I have an older engineering sample i5 board from them, which I also got for around 160 US$, but lately they cost around 600-800 US$. Jeff said, he got them for 160 US$. What are the 12th and 13th gen Erying boards costs for you?
Did you try a similar test but with Linux guests? I've seen some pretty strange behaviour from Proxmox with Windows guests from time to time, but have had no issues with any Linux VMs.
You were going for Mac CPU Commit (and completely avoiding CPU Over Commit which is what VMWare is greatly abused for in corp-land) but what about.... not? Leaving 1 or 2 CPU cores short from Max Commited to VMs and seeing if that pans out for the big.little system? Odds are it will wind up being E cores that are leftover but that should be find for the scheduler and minor system management on average...
I don't do anything like this but I like the 13700k cpu I have. I dedicate the P-cores to cpu mining and the browser to the E-cores. No stutter, youtube plays fine, browser works fine. etc
I have the same erying Engineering Sample board .before using this board I was curious on how bad that pci 2.0 x8 would actually be for gaming. So I plugged in a Rx 5700xt and somehow my gpu reported pci 4.0x8 even though the sticker says 2.0 . I was able to get smiliar framerates to my Ryzen 5 3600 in gaming
Also I also encounter weird crashes with that engineering sample under Debian Linux using docket. even though it ran fine on windows . Since proxmox is based on Debian I expect the issue to be the same .
I’d love to hear if you work out some of these issues. I’m tinkering on it Aswell. Have you ever looked into the hidden bios options? (Toggle by ctrl+f8) maybe there could be a fix but I haven’t found something yet .
I will rather buy a new board before I’ll ever touch windows server so I’m really desperate to find a fix
Another possibility is that the dies being used for the xeon, having the GPU and/or E cores disabled, are dies that failed GPU and/or E core testing. and have little use in the consumer market..
You should use Microsoft's Diag tools to see exactly which cores/threads each vm is using. Been wondering whether I could turn my windows 13900KS Desktop into a Proxmox VE Desktop running my main Windows Daily system and other VM's instead of Windows 11 and VMware, but the whole Big/little core issue of not having enough information out on the internet with Virtualization on Mixed Core Cpus, Ive been too afraid to try. Needing a powerful virtualization & storage server but cant make my mind up. Have everything but a motherboard, drives, & hotswap external drive cages to build a 12900K server with it but not sure if its worth using it for that due to big/little. Over, picking a 16C AMD, a couple of used Gen 1 or 2 32C Threadrippers, or a Dell R740 2x Gold Cpus.
Have you tried blacklisting the p cores so linux can’t use them?
just heard you say vmware kinda does not work on e cores, thats why when i tried vmware on my 12700h laptop, it felt super slow and unusable. is it possible to use it? what do I have to do ?
Basically, VMWare treats a core like a core - it has no way of differentiating between P and E cores, so things will randomly run super slow.
You pretty much have to disable the e-cores entirely to get it to work. VMWare has not made any mention of supporting any BIG.little architecture, and given all of the people Broadcom is making angry with their VMWare licensing changes, it might be a while before they do.
Might I recommend something other than VMWare?
Thank you for your knowledge and sharing it! this is so valuable ! Please I am currently using windows hypervisor on my laptop but on my ryzen desktop I use Vmware , I would appreciate your recommendation on a software ! Thanks again!!@@aetherspoon
He is wrong about that Pci-e 2.0 x8 slot being unusable. You'd probably get around 75% of the fps with a 3080, it's less but definitely usable.
hey Jeff, im running a Unraid server. and im using a i5-13400 and I have no issues what so ever with vms or with docker and vms assign to either p cores or e core. if none where set it will do I for me.
@CraftComputing
Jeff, I'm with you on your opinion that Intel's move was on the bone-headed side. What they needed to do is offer differing models with and without ECores to fit the needed market sectors properly. You are correct into your original assessment.
I have heard that the amount of data that a video card passes through to and from the motherboard is not as much as people think. So, having a a degraded slot for the video card (like the one that has the sticker next to it) may only have a minor effect on gaming performance. It's worth a try to compare a cpu on that board with a given card, then switch the gpu and cpu to a gaming motherboard and see if there is much improvement.
Bandwidth to the video card really depends on your use case and even game to game. AI processing such as LLMs you need as much bandwidth as you can get, AAA gaming at 4K a similar situation, older games not as much, and crypto mining as we've seen with the mining motherboards with many x1 slots and breakout boards hardly needs and bandwidth at all.
I'm running ESXi 8 on a Minisforum NAD9 (i9-12900h). As Minisforum BIOS doesn't support disabling E-cores I had some issues setting it up so I had to add some kernel parameters I didn't get PSODs each time the system started. I have a nasty feeling Vmware won't be my next choice for home lab as Broadcom's acquisition of Vmware is doing it's own IBM for Redhat/CentOS, Oracle for Sun... and it's gonna end in tears for a lot of homelabers :(
I'm working on my first Proxmox server. X79 Dual 2630v2
Have gone with X99 E52680v4. 2 Huananzhi m'board. Will swap one with ASRock X99. Huananzhi support via Aliexpress message board is next to none - Lousy to say the least.
10:36 running 3 "Virtuer" machines lol
har har har, he flubbed a word in a 20 minute long monologue.
@@CraftComputing sometimes “stubbed toes” are worth a laugh instead of tears.
At least with xen and kvm based solutions we can always constrain the pool of CPUs available so it really doesn't matter much.
Cool to see people involved with Proxmox's development in the comments. Hopefully this leads to a followup if/when things get fixed. Considering the amount of people saying they're running 12+ gen chips with no issues, i'm wondering if Proxmox is detecting some oddities in the BIOS or chipset of these Erying boards rather than struggling with the CPUs.
I desperately want these to run well; kind of like what you said, they would make *awesome* homelab boxes with the right pricing.
I saw that that one of the 12th gen M-ATX boards from Erying was getting some pretty bad reviews about CPU clockspeed stability; I'm wondering if those specific ES CPUs have some issues?
i've been running proxmox on 11th gen erying with 8 cores. the only problem for me is finding a good 1U low profile cooler. i run some number crunching VMs from video encoding to AI so using the 12th or 13th gen is a terrible idea. however on a long run it clocks around 3.7Ghz on all cores. stability wise i've never had a hang or crash, and the CPU cores keep trying to clock to 4.8Ghz when possible. power use is also very low with a 20W idle from the wall with 2 HDDs, 2 ram and only the CPU fan. It runs as part of a proxmox cluster with other lesser known power efficient hardware that have no issues running proxmox such as AMD's embedded and does entirely fine combining NICs with usb adapters too.
I'm sure that the crashes and issues you faces are either due to the eirying boards/bios/firmware being buggy or a flaw in the kernel. Big little CPUs have been supported in kvm and thus linux for quite some time now and overcommiting definitely should not be an issue. We run 16x overcommit on cpus in our cloud with any issues. Even if all threads light up and not all VMs get the configured amount of cpu time, for the end user it might feel slow, but nothing will crash or anything
I use Proxmox on NUC 13 i7-1360P (4P/8E/16TH) without any issue. But I didn't try to fully load system with benchmarks.
i5 13500 has been solid in unraid so far Haven't noticed any issues with E and P cores
I got an Erying QXZR ES 0000 i5-12XXX board with proxmox installed it works fine for few hours but then frezzes and i have to power cycle it to boot again. unfortunately its unusable this way.