VFIO is Back! Back AGAIN! Our Killer VFIO Build + Setup Guide
Вставка
- Опубліковано 16 сер 2023
- Two Windows VMs go round the outside, round the outside, round the outside.
Linux video/giveaway: • GPU Pass-Through - VFI...
Enter the giveaway here!
forum.level1techs.com/t/vfio-...
**********************************
Check us out online at the following places!
bio.link/level1techs
IMPORTANT Any email lacking “level1techs.com” should be ignored and immediately reported to Queries@level1techs.com.
-------------------------------------------------------------------------------------------------------------
Intro and Outro Music By: Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 3.0 License
creativecommons.org/licenses/b... - Наука та технологія
![[demo] VFIO in 2024 // windows + linux](http://i.ytimg.com/vi/6SoteC1FM14/mqdefault.jpg)
We're doing a VFIO PC giveaway!!
Info here on Level1Linux
+ ua-cam.com/video/HO_8liPirns/v-deo.html
Enter on the forum!
+ forum.level1techs.com/t/vfio-2023-radeon-7000-edition-wip/199252
~ Amber
Seggs with Wendell
Fully penetrative seggs with wendell
Multi vm with one card is the single most desired feature for desktop gpus for me right now. I'd take a 50% performance cut at the same price just for sriov support on a consumer card. One day we'll get it I hope.
step one, buy a down to 10 series nvidia GPU, step two, run vgpu unlocker, done
@@malloot9224 driver?
@@Infinitex9 what do you mean?
pReach! :D
Same, except I want the performance and I'd like dynamic scaling to work so that I can overprovision.
That way I can be doing work on AI or something for awhile then switch over to a gaming VM without having to stop the VM and reallocate the GPU.
mad props to the editor for that thumbnail, holy crap 😂
Super cool
That would be a Grant special ;P ~ Amber
THIS is what I come to your channel for. NOBODY else can match it.
I can't tell you how much I appreciate you carrying the VFIO torch. I've been playing with this forever but it's really hard to find updated info which is frustrating considering it is the future! It influenced the parts of my latest personal rig and I'm also spec-ing out a new forbidden everything box(maybe with a dash of llama?) and this video came out at the perfect time.
I need to dig into the forums as I'm sure there's more info there but any additional info on AM4 options or other less (wallet)bleeding edge hardware would be swell.
Thank you so much for all the work you do for this!
Using a single GPU for multiple VMs would be excellent for hardware accelerated video encoding / decoding. This would be super useful when running CCTV and video transcoding on a home server. Such as Jellyfin and Frigate.
I agree. Even if I personally would prefer using multiple physical GPUs for different VMs, for lighter workloads such as video transcoding and other GPU accelerated tasks, it makes more sense to have one GPU running all of those home server type workloads.
Although I'm still annoyed by the lack of PCIe expansion slots on a lot of newer motherboards. Even $300 AM5 motherboards will still only have 2 PCIe slots, with the second slot only being at 4x speed, not even 8x. And it's a shame that it's the Intel motherboards have this be somewhat more common since there you're giving up platform longevity that AM5 will (hopefully) destroy it in.
Depending on your comfort level, you can get a used H11SSL-i with a half decent Epyc for $350. MB comes with 3x16 and 3x8 PCIe 3.0's and fits into an ATX case. (don't forget the torque torx driver!)@@jamesbuckwas6575
All of those applications can run on a docker container with GPU acceleration
Thankful for your channel. Top quality content and its always fun to watch.
Amazing what you are doing please dont ever stop this VFIO work!
Thank you for your work, This will be epic one day!
Thank you!! Haven't been running vfio, but on my daily driver thanks to inspiration from you I've been running pci pass through running on poxmox. 3955WX in a super micro mobo. 6600 XT running arch as my primary desktop; P4000 on windows; P4000 on another debian box. And a P24 being used for some ML testing. All to different VMs.
This is golden content. Hope you push through with this. And I agree that this is the future.
This is just... the coolest thing. I cant even express how awesome this is.
Game DRM is what forced me to switch to a separate dedicated machine for gaming (I wasn't using VFIO but multiple GPU's passed through). I'd love to go back to virtualized but until that's sorted it's a tough sell.
What games didn’t work?
Anti cheat systems also don't like virtualization. I can kind of understand why, but it prevents you from playing most multiplayer games.
@@Tuxm. At the time it was R6 Siege but anything that uses BattleEye doesn't currently allow virtualization
@@r0ckr87 respectfully your doing it wrong only games that don’t work are ones with kernel level anti cheat which is few so far
@@Tuxm. True, it's mostly high level competitive games that do this. Some DRM too I think?
Thanks for updating VFIO! I want to be able to share my GPU across many virtual machines similar to how you can share the nvidia GPU with docket containers. They share the same scheduler and have access to all the GPU memory
This is the content I've been waiting for.
foshizzle -Wendell
Thx for all you do for the community
I had a crappy day, and watching this made me feel better. Wendell, you are Computer Bob Ross, and my blood pressure is lower because of it. Thank you!
Always great to see a video that's not the weekly show. Although I do love links with friends too!
super cool. this is a dream build for passthrough
Spot on, I have that same wood handle screwdriver at mark 0:57.
I have built my vfio set up with proxmox as my host OS, then I have a Debian VM as my primary OS, and Win10 for gaming. I use a kvm to keep everything using the 4 monitors, and have recently used looking glass as well. Works flawlessly.
This was a great video.
It's got me looking into building another machine again. Winning the Giveaway would be amazing thought, LOL.
Wendell I would like to start by thanking you so much for your time and dedication to VFIO. I couldn't agree more, this is the single biggest feature missing from modern day consumer hardware! The security features alone should being manufactured to release it, especially with the spotlight on cyber security in the last few years.
Sounds good :D
Thank you. Vm multi GPU instant for 2 gpus... I had more enjoyment with vista usage. Then it getting the most basic vm usage
That case is HUGE! and I want it!!! Lol
Hell yea!
I'm mostly in the VFIO game for Hackintoshing, though I do have sandboxed Windows and Linux VMs. My ASRock 6800XT PG is working great with no reset issues.
It would be absolutely fantastic if I could run multiple VMs simultaneously on the 6800XT -- it has plenty of resources to potentially run 2 VMs with decently high power for each, and even 4 VMs if GPU needs aren't terribly intensive.
I can't imagine having a system that can't do VFIO anymore. I'm doing it since 2021 and it has so many advantages that they're impossible to count. For example I'm not a fan of the whole TPM/secure boot thing that MS did with Windows 11. Well I do have a Win 11 gaming VM though. I'm just emulating the TPM 2.0 chip without actually using my hardware one. It enables me to be truly free in terms of software to the point that I dont have to care what big corps are pulling off. Oh you're requiring chip/feature XYZ now? Wait let me just slap it on my VM by choosing it from the list in virt-manager. In a day and age where our software and mainstream OSs are more and more ... externally controlled, this is really important.
And you Wendell were my main inspiration to do all this back then, thank you.
Great!
I used to have a VFIO setup on my old R7 3800x system. I had a GT 1030 for the host and a Vega 56 that a passed through to the Windows VM with Ubuntu as the base system. Took a couple weeks to fully work out but was really fun. I'd love to do it again at some point with Arch this time and I've got a much better CPU to have some fun with (7800X3D)
Dude Wendel it would rock to win your rig! N make my year!!!🎉🎉🎉
Did Wendell just quote Arrested Development!! Lmao
The only time I got PCI-E passthrough to work was in my forbidden router :) And even then I am really, really skeptical about missing something and accidentally spilling my LAN into the internet.
0:27 love the startrek reference
Been running VFIO for years at this point. A VM for work, shut it down and then pass all the same hardware through to the gaming VM. Has it's niggles but the setup works for me. As others have said though, DRM is a bit of a kicker, some games even with a windows VM didn't work properly vs native windows.
Is it hard to run this vfio on something like a Intel nuc? Wanting to try proxmox on an extra nuc I have
@@rudysal1429 depends on which model and if it supports vt-d.
Wow, just a day after I left a comment asking for an update on what's currently the best VFIO build!
that thumbnail! 10/10
(Here's one for the algorithm) Thank you, Wendell...🇺🇸 😎👍☕
Excellent news. I really want to be able turn my 5950x/3090 into 3 PCs for use at LAN events. I figure 8GB of VRAM, 21GB of RAM and 10 threads per VM, that should be enough to run mid tier games at 1080p.
I'm running freebsd as my host. No fix for Navi reset bug, but other than that, managing VMs with first class zfs support has been incredible.
This is definitely Level Two content...😏
Love the thumbnail 😂
I feel like a starving man looking through a restaurant window with my decade old GPU and this video.
I want to follow this but I dont have and can't afford the required hardware. Gonna watch anyways and I REALLY hope I win the setup so I can follow along!
I love that thumbnail more than I should
I was actualy thinking around for a similar project and was looking for states of the art (I try every 2-3 years).
Glad I can read fresh and proper how to.
I will try to also use vGPU since I am still on Turing line.
I wonder if it is possible to configure both a Hook and a vGPU and choose between vGPU or hook when starting the VM depending on what we want to do in it ?
Interesting mobo choice, I was considering it but ended up with the MSI MEG ACE - I feel like it has better separation of I/O on the ports. It also has one USB-C controller on an isolated pcie port, which allows me to passthrough the whole port.
10g Ethernet controller allows for no-compromise in speeds when doing the virt nic.
While SATA controller is also on the passthrough list - I should probably share the config on l1forums.
There are some quirks - running an older bios. In theory, it has the and patch and the new memory support so I can run 5600mhz with 192gb (yup, it works!).
Your Linux GPU has to be on the first slot (or use the usb-c GPU, but I had a few minor glitches, besides it wasn't doing 4k).
Great timing Wendell! Just picked up a second gpu and VFIO is top of my list for things to mess around with. Never had the benefit of 2 working gpus before to try it properly.
Me: Maybe I'll pass on Ryzen 7000.
Wendell: Look at this VFIO build!
Me: I'm listening...
Hi, I have an amd fx9590 cpu along with an amd rx 6600 and a gtx 1070 gpu. In proxmox i was able to vgpu unlock the gtx 1070 to make it have the sriov like features. i then use looking glass to capture 2 vm's using the gtx 1070 and display them on a linux vm using the rx 6600. i would like to show it off on the level1tech forum or on youtube for fun.
After years with the idea in the horizon but never seeing it viable to implement I was hoping to finally hop in with the performance the 7950X3D gives, this series could not have come at a better time!
It's also great having AMD involved, I hope this will bring SR-VIO closer to the desktop!
I want this specific setup for myself, and I want to run cables from the basement to the bedroom for monitor/keyboard/mouse/sound peripherals. Our bedroom gets SO HOT and LOUD and makes it hard to sleep. This would be the dream, it would also keep pet fur out of my machines...
After that GN Kiddy Drama good Content is so refreshing to see =D Very Interresting
this is the holy grail / ideal setup in my mind, but it's such an edge case in the world :(
NICE!
Need a new cpu now :)
*dozens of us*! Yep, maybe. But what a dedicated bunch.
10/10 for the thumbnail. 😅
I was looking into setting up a VM with VFIO recently for gaming, but ended up dismissing the idea because of concerns about anti-cheat software. I've heard that some anti-cheat software will ban an account if it detects that it's running in a VM environment. This can supposedly be bypassed by spoofing hardware IDs, but that would look more suspicious IMO. Do you have any thoughts on this use case?
Cool
I had pass through setup with gen 1 thread ripper and two 1070s. One bios update and boom I’ve never been able to get it to work properly since.
I like the concept of making spyware obsolete by making telemetry pointless. You even could randomize application virtual hardware resources a little to fool any tracking software.
And I can set it up in under 30 minutes, with a few automated scripts for near bare metal performance. it really has become much easier to just get up and running
I do this every day for years now, with a 9900K, iGPU for the host (Ubuntu), a GTX1660 for Win10 vm, and a GT710 for a Linux vm, this connected to a crt display (for emulators and old stuff). I pin the second thread of each core to the vms (6 to the win10 one and 2 to the linux one, in a reverse order, cpu #15 is the cpu #0 of the win10 vm), and leave the first thread to the host. Mitigations off. I got no problems with antipiracy, but I use only single player games and I use the hidden kvm option (tested GTA V, RDR2, ROTTR, SOTTR, AC Origins, Death Stranding, Flight Simulator, Hitman, Alien Isolation and others I can't remember).
This VFIO series could not have been timed better.
I had my GPU die a couple weeks ago and picked up a 7600 to get the system back up and running. Did not really take the time to consider if it would handle VFIO properly and was basically just looking for something reasonably affordable. Eventually got the VM back up and running with passthrough, but I have been consistently plagued by reset issues on this particular card.
Might also jump on the forums to try and sus out the cause of the reset issue :)
I don't have any AMD gpus but I have an Intel ARC that's been waiting for me to get a bit of time to set this up.
0:20 I cannot agree more....
0:32 well of course due to exploring the galaxy and all that
Perhaps place a link to the "guide on the level1 forums" in the video description.
Just did an ITX vfio build with a 7700x and using the iGPU on the host. The biggest landmine I found was that kernels 6.2 and 6.3 didnt support display port 2.0 with the built in ryzen 7000 igpu. I was only able to output 4k 90hz on a 144hz panel. My solution was to just stick with debian bookworm
Having a second GPU compatible (or that I can be reasonably sure I can make work with my limited linux experience) with VFIO is the only thing I'm waiting to switch to linux.
My ASUS Pro WS X570-ACE is just waiting there with its 128GB of ECC RAM wondering when I will install my R9 5950X in it.
I wouldn't go NVidia as they are way too scummy to my taste (that sounds, wrong ...) but I was afraid that any GPU newer than my aging RX 5700XT would cause me too much issues for me to resolve down the line for me to comfortably go for VFIO.
Thanks you Wendell for your work and dedication to that technology.
Two years ago I managed to get a single Xeon 14core something and 2 GTX1060s to run like two physical PCs. Getting the two GPUs dedicated each to their own VMware instance was difficult, but getting USB devices reliably assigned to their dedicated VM was a pain. PAIN!!! Getting more than one USB sound source to work was impossible. Fortunately one of my displays supported HDMI sound, which worked flawlessly. Thanks nVidia. Interesting project to work on, but scrapped it half a year later due to USB constantly pissing me off.
Have y'all thought of different costumes to do for October?
I've been dreaming a similar setup (but based on a threadripper instead) for quite a while, but it'll have to wait since single-gpu passthrough can be pretty iffy…
Curious about how 256GB or 192GB DDR5 on AM5 is going with the recent BIOS updates
You know what would be really cool? Vfio passthrough with memory sharing via memballoon or a similar mechanism. If your guest VM doesn't use all its memory it's so wasteful to allocate it the entire range and not give it back to the host.
Would love to be able to spinup a windows 7 instance to get my itens on the japanese lolpso2 server for some nostalgia.
Sure as heck not going to dual boot to do it. Sure as heck doesn't work in limmux.
Wendell, greetings from CHIAPAS México.
I have a question, can I enter the giveaway even though I've never used VFIO in my life?
Amber the thumbnail and editing o this video, you nailed it.
Sure
@@Level1Techs Thank you for answering. Greetings from Chiapas México.
My knowledge with this sort of thing is very entry level. What would be the main difference between this setup and just running TrueNas as host and separating the cores and assigning GPUs to each instance that way?
I'm guessing TrueNas doesn't let you use the one GPU for two OS at the same time.
LOL, that Eminem look!!!
Been running for years, even updating the guides. Unfortunately not on the latest HW.
This is such valuable content. Am looking into doing this to use Windows only hardware for music production, if possible, and dual booting is just not a viable option.
We're in the same boat!
In general it should work. What I've learned, is that you shouldnt passthrough the device directly unless it is a pci device. If it's a usb device pass through the usb controller etc. That way I've managed to get blu ray drm to work.
You think with this system (7950x3d and same taichi x670e mb etc) that the igpu from the cpu can be passedthrough? I have been trying for about 2 weeks now and 0 progress what so ever. the standard PCI card works fine, just not the igpu, to the point where now I just want to know if it is even possible so I don't waste more days on this.
I've been using VMs on windows with VMware just because it has support for 3d acceleration with Nvidia cards. Would like to use KVM based VMs but needing separate GPUs for VMs makes no sense for my use case. I thought this was showcasing virtualizing a single GPU for 2 VMs but looks like you're just using PCIE-Passthrough and 2 GPUs?
Virtual Function IO Passthrough, with a giveaway! Would you ship that to Australia?
Mmmmm specialized content
dunno wtf a vfio is, but that thumbnail got my click. guess it's time to learn.
😂 I feel the same, that Eminem edit is WILD!
That thumbnail got me too 😂
I wanna just spin up a Windows VM on my xcp-ng host and play a game every now and then.
AsRock is still putting sticker on the memory dimm slots.😅
Can you reset a crashed GPU by sending the host system into S3 Sleep?
Work wont let me run mixed VMs, some with GPU acceleration and some without. I have to have a separate host with dedicated VDI. Made me sad. Maybe I'm just thick?
Linus had made a video few months back about some Win 11 feature which allows us to share the same gpu to vm's. Has anyone tried that?
I've become a real fan of Asrock recently. But then, I'm a homelabber, not a gamer.
Would it be possible to get VFIO to work with Intel igpu for Linux and then a powerful GPU for gaming on windows? I'd like to try tinkering with it on a SFF PC
I'm right there with you, hopefully more people respond. I have a decent nuc type system with an Intel gpu I want to use for transcoding and what not for jellyfin.
@@rudysal1429 Tons of people have igpus I think it would be a great way to popularize VFIO and Linux since it would be easier to have a system that has the benefits of both windows and Linux
You can use Intel iGPU for Linux host, and pass the pcie GPU to a vm (linux or windows or whatever you want)
Running games in a vm do not introduce more input lag and display lag? There are some games that require low input/display lag ; mainly fighting games 😮💨
Can I try this on AM4?.
Maybe the 5700G paired with a 5000 series card? (RX 5700XT comes to mind)
Can work, but you might get some problems with the 5000 series RX. Some have the reset bug, some don't. IOMMU as a bios feature has been around since a decade, so chances are that you mainboard supports it in general.
@@Spicysauced I've got an ASRock B450 Steel Series. I'll give it a try, otherwise I'll try to get a 6000 series GPU then
Tbh Ive never had good luck with AMD cards, two Polaris and one Vega failed to work, even with vendor-reset and all bells and whistles. In contrast, Nvidia just works since they explicitly allow passthrough for consumer cards in their driver. I've had a 1660 Super and now a 3060ti and I didnt have to patch kernels, fiddle with ROMs, drivers etc., it just always worked. @@AlfieMakes
AMD what happened to MxGPU, RocM and of course SRIOV? Shouldn’t that become the(!) way to compartmentalize, share (“consumer”) GPUs and secure your system, without having to buy server/workstation GPU?
Thanks to Wendell the Great aka “PC janitor” (of the century) and @Level1Techs for this wonderful giveaway and good luck to all who can be part of it.
Cheers out of the old world of Europe,
Pepe
lmao the thumbnail, wendeminen