this setup for two gamers at once would be sick. just do 32GB of ram to save some cash and it is solid. I can imagine a techy dad building something like this for their two kids or something to play games.
could run the VMs from a pair of Pi400s and secondhand monitors or cheap tvs. keep the budget real low and still have a pretty solid gaming experience.
@@catalystguitarguy maybe use the mandatory school chromebooks depending on where you live. You could also probably realistically use phones and plug them in a monitor.
The kraken g12 aio bracket is a great cooling solution for these cards, I left the backplate with the thermal pads on for my m40 and didn't even have to use heatsinks on the vrm. Runs super cool with no problems
Jeff, Great video, to get the M40's to be quieter, I would suggest trying to use a fan controller with thermal probes such as NZXT's....I use one on my M40 12GB and it's very Vega 56 like versus being loud 24/7
I did a very similar build, but for a dedicated AI Rig, and you were about the only person using P40's, so thanks for the videos. They really helped! My rig ran only about ~$810 all in and I just repurposed my old T5500 (was about $90 on eBay at the time barebone with poor CPU & RAM), 192GB RAM ($20), Xeon X5690 ($10), 80 Plus 800W PSU ($0, included), 2x Nvidia P40's ($390) + Nvidia 1030 ($20) for Graphics output, 1TB SSD (hardware doesn't support NVME boot drive $35), 18TB HDD ($220), and adapters, fans, misc. ($25). The tower is interesting as it supports up to 4 GPU's if you can squeeze them in. Also, it supports adding a riser card for another physical CPU + 6 DIMM's of RAM. I opted not to do this, and just use the extra CPU 4-pin for the second P40 GPU (power). Another options would be the T5810 that supports the E5-2600 series CPU's (at about 2x performance) and supports up to 256GB of DD4 Memory (but based on my quick check that would cost $1/GB+, which is considerably more than the dirt cheap ebay bundle I got). And I designed and 3D printed a dual Shroud and used one 92mm x 38mm fan to cool both and I have been impressed by the performance for the cost, and the sound. I used some rubber and spacing for noise dampening on it, and for how much air it moves it's amazing, and was only ~$16 on Amazon. While the CPU is outdated, for my purposes, it honestly seems to run like a champ.
@RandomShart Yes, I was able to get this working with all VRAM but I modified the low profile 1030 and removed the fan (because it was a tight fit, sandwiched in-between the two P40s, and I wanted the airflow to have somewhere to go, and not just into the other card). As for using all the VRAM, I think it depends more on the software you are using. I've used LM Studio, Stable Diffusion and a few others without problems (other than your normal debug and setup stuff).
@RandomShart I'm not sure how to answer, because it'll likely be specific to your system. I'm running on Ubuntu Mate myself. The 1030 is just a generic display and I believe the driver I'm using is the the Data Center / Tesla Driver For Linux 64-bit (latest) with CUDA toolkit/bindings. If on Windows I wouldn't be able to tell you but I'd probably recommend getting and extracting the drivers only (without all of the other NVIDIA stuff) and pointing your device to it (in Device Manager to install the specific drivers).
@@TomiWebPro Off-lease ECC server "6pc lot ~ Samsung 16GB 4Rx4 DDR3 PC3L-8500R RAM". A little on the slow side by today's standard, but can't beat the price. And looking back at the listing, my bad it was $21.64 and for 96GB RAM. I had a bunch of RAM left over from a server running the same Xeon CPU and already had the PC (intentionally had matching hardware). I maxed it out at 192GB but half I already had (and without checking, I had thought it was from the same listing).
@@michealkinney6205 Wow this is insane value, great find! I brought 4 pairs of ddr4 2133 16GB recc ram for 12 buck each, and somehow price gone even higher now I thought I might just upgrade down the line🥲
I hate broccoli. You said to leave our hate in the comments. Love the video, been wondering how well similar combo would work (add some cheap laptops and android boxes for remote). Would really love it if it wasn't so darn loud.
For the fan mods on the Tesla cards, do a push/pull. one in front of the case as you mentioned, and another outside the case at the back, as there are no vidoe ports.
i went super aggressive . took the aluminum fins and separated each individual apart slightly then ziptied two 120mm fans to the front . temps are great
would like to see this in a server case to rack mount. I would also install SSD Drive for each VM, and passthrough disks to the VM. Want to build something like to but in NZ.
Solid VM content. Excited about the tutorial. I wish the gaming giants were not so picky about emulation, would love to see a few of these used for retro consoles. The 960-1050 level GPU is perfect match even for some 1080p Gamecube/Wii and even some WiiU
Nice project. I would have taken a different motherboard. Like the Huananzhi X99-F8. You have a extra PCI X16 slots that runs 4x, but you can install a 10Gbit card which improves latency. Also it supports quad channel memory, I'm not sure if this Jinghsha does, it looks like dual channel memory. You can also a variation with 1 Tesla GPU and use the 2nd slot for a PCIe to 4x M2 adapter. Then you can run 2 VM's with their own SSD's.
Great build! I'd use something like this to set gaming machines up for my nephews to run from their chromebooks since we don't have a lot of room for full desktops in our place at the moment.
I haven’t looked at my M40 12GB for a while, my feeling was that the memory profiles were the problem. I don’t think I’d seen the newer script or hadn’t appreciated that it had more granular control if I had. This is going to be the ideal end of month project.
Yes, the issue with the M40 in vGPU has always been the 8GB memory cap for Maxwell cards. You couldn't allocate more than 8GB before. VGPU-Unlock-RS allows you to customize the profiles, and use all available memory.
What I would do for cooling is get some acrylic or polystyrene sheet and pipe air from two fans in the front (something like a nidec beta 5 TA450 which can have great static pressure but also can run silent may be my choice)
Spent a few weekends building out something similar. Worked great until I realized most anti cheat systems don’t allow you to play games in vms and will ban you for “trying to work around it”. Sadly had to go back to a dedicated gaming pc…
I think you may be able use a non virtualized session and use something like splashtop with a virtual monitor, or any indirect display driver and steamlink.
Yay, the first video with a cameraman! There is some echo though in that part so you might want to ask your little helper to add some sound dampening panels
Thanks! I ordered a couple 12GB cards so my kids can play with their neighborhood friends. I'm going to put them in a PowerEdge R720 with a pair of e5-2667 v2 processors. Hopefully, the case will keep them cool enough not to require a shroud as I don't have a 3D printer.
4 gamers 1 HDD sounds rough with that random IO.. I know it wouldnt be applicable in this use case, but it would be cool if the windows spanned partition had a version meant for performance, where, yes each whole file only goes to one hard drive, but instead of filling up the first drive before moving to the second, it instead filled the drives up evenly, and then did some optane style optimisation later based how often the files are used together
afaik windows can do software raid0 too either with the old interface or with Storage Spaces, although not on boot drive, because screw that, nobody needs raid on boot drive ever
One thing I'd like to see is all the other things required for this type of setup. Would it work well on 1GB ethernet or would you need to add cards and hardware to run 2.5GB+? What kind of upload speed would you need at home to make this a viable experience away from home? I know these things have been mentioned before, but I don't have time to go back and find it. Would be nice to have a short section for all of these videos with little details like that. Basically, how fast does my network need to be to handle the different loads? 1GB should work fine for 1 VM, but what about 2? 4? At what point would I need to upgrade the networking? At 4 VMs? 6? Even just a breakdown of bandwidth requirements for a single connection and we can do the math from there. Love the content, just moving and trying to figure out what infrastructure I'd need to run in order for something like this to work.
Nice setup; you can try to mount the 120mm fans where the PSU sits to circulate the air in the whole case. since hot air is normally pushed out via the rear part of the case; I believe that should help
there's another way to cool those cards , if u disassemble the shroud you can remove the clear part and get a expansion slot cooler it makes it alot quieter but takes up a extra 1-1.5 slots, its what im doing rn
Yes, you can do this with the Tesla M40 as well. In fact, that's what my previous video on the card focused on: ua-cam.com/video/TQ9B67T5qWs/v-deo.html
I'd be curious to see if you could create enough positive pressure in the case to cool the GPU's. Even loaded up with Noctua industrial 3000 rpm fans running full boar would still sound way sexier than those blowers. Seal every gap except the GPU vents and put all fans pulling air into the case. Please and thank you
I was considering building a dual xeon 10 core 20 thread system and then break it out to 4 VM's each that can be used as a video editor/render machine running DeVinci. Any suggestions, comments or thoughts are always welcome.
I run my first Gen EPYC system with a Tesla M60, and use this same setup for remote video editing. The M40s would also be a great option, especially the 24GB models from this video. I'd spring for a pair of 12-Core or 18-core chips... E5-2690v3 or 2696v3. That would give you plenty of power for 4x editing VMs.
for those teslas, i think the most practical way to cool them in a desktop case is just remove the original shroud they have, and adapt a 3d printed one that can hold 2 case fans, maybe 92mm ones? i dont remember the size but yeah, you could even 3d print a new shroud for them to hold those fans
@@CraftComputing they do have closed fins, yes, but the fins are easy to bend down with some pliers/flathead screwdriver and allow top-down cooling. It's not pretty unless you take your time with it. I used this method with a firepro s7000, an arctic pwm P8 fan, and CA glue. Temps were out of control previously, but with the modification and fan at 50%, temps didn't go over 72°C.
@@CraftComputing well, nothing some scissors cant fix, but for real tho i think Myconst's friend did something like that taking out the top fins, is not elegant but it can work
Amazing video, Jeff! A while back, I was working on a silent cooling solution for Tesla K80s in desktop PCs. I left off at two Noctua 60mm hooked up to a Y-connection splitter, so that it could be controlled by a single PWM header. You can have them run at 100% load, and still be quieter than the CPU cooler on idle. A proof-of-concept is on my channel (card is not powered, but fans are at 100% load). Happy to send you the STL file if you wanted to print it and give it a shot. In hindsight, I would have actually just scrapped the concept of using a blower-style cooler configuration, and instead opt for an open-air configuration. Some desktop Tesla users found that slapping on one or two fans directly on the heatsinks could dissipate heat just as effectively and much quieter than using a blower. For further improvement in thermals, you might also want to consider repasting the cards, as many Teslas were manufactured ~10 years ago (thermal paste is likely underperforming by now). The other alternative is watercooling, and there are a few CNC-machined waterblocks on Ebay for the Tesla K80/M40. I haven't tested those, but they seem like a promising (albeit a bit costly) solution for quieter cooling.
@@mitlanderson As a quick hack, might be worth just slapping 2 noctua 80mm fans, straight onto the heat fins (can unscrew the top cover pretty easily, held by a few hex bolts on the side), and see if that can provide any feasible amount of thermal dissipation. Unfortunately, I gave my K80 to a colleague, but that would have been my next idea for a noctua-based cooler
Thank for your video to pass-trough a GPU to multiple Hyper-V's, I also have success with passthrough a displaylink DL3900 adapter with USB Eltima gateway. I think multiple USB3.0 docks will also be possible. No GPU hdmi/dp is needed because the docks have them self. Also no GPU encoding.
Jeff, it seems the M40 has the same layout as the GTX 980 ti. A friend of mine fitted a 3rd party cooler with 3 fans from that card and fitted it on the M40. It works great, full load he gets max 75°c.
I wonder if you can use LVM copy on write so you only have to keep one instance updated for all the installs to save time and storage, even better than deduplication
Unfortunately, as far as I can tell, the mainboard isn't available in EU at all. Possibly from aliexpress within 1-2 months, but not anywhere local that I could find. It not having a proper model name/number does make searching kinda hard though.
JEFF could you do an UPDATED installation on the 7.3-x version its a little different than some of your video's for a Noob like me i have found. Some optimization and things just seem alittle different
I'd make UA-cam videos about it lol I'm glad you clearly mentioned the cons. Seems some eBay sellers get stuck on some UA-camr saying how "amazing" something is and he pricing gets stupid. I think the M40 shouldn't go for even $100. I did buy one to try to sort out using it for a local machine, unfortunately it died with plenty of other parts when 12 volts arced across a USB connection. Thanks to a past video of yours, I used a virtual UB monitor to get games running that wouldn't before on a headless machine and streamed through Steam. Was mostly fun as a hobby experiment. A couple games I play would run without too much hassle, but vsync is typically broken, often can't change resolution (sometimes the resolution shows as "undefined" etc.), and games that seemed to get updates to run better were apparently just fixed by game ready drivers that the Teslas don't get without the setup fuss that makes these a harder sellllll. Neat/cool stuff, but yeah...
to be fair, in the current GPU climate, 980Ti/1070 level performance for $140 is a legitimate bargain when new 1050 Tis are still being pushed for $200 and 1650s for $300. I suppose it has entirely to do with the complete lack of video ports and active cooling that really prevent these cards from being readily repurposed by desperate gamers
@@bojinglebells Cooling is easy (if loud is fine), display out is THE problem, the end. If they had even one, then it'd be fine... and people would probably try getting $250-300 or maybe even more for them.
Is it possible to remove the gpu shrouds completely and jerry-rig a tower cooler to them to run with a normal fan? Or possibly water cooling with a custom block?
While I find your use case interesting and plan on experimenting with VM GPU sharing, mostly I want to use these older cards like the M40 for things like stable diffusion. Appreciate your content.
@@CraftComputing That is pritty cool. I figured it would be possibe but just wasn't sure. I guess hypervisors are light enough it wont really cause issues. I wonder how a vm would work for streaming to a quest via airlink. Would be intresting for a house with vr.
I'd probably set it up with 3 1660supers I have sitting around and give each vm a full pass-through gpu. I could have all 3 of my gremlins gaming on one cpu.
I've been looking for a multi PC in one. Running 2 FO76, 1 or 2 klipper printers, and a 3d model PC. Can this be the system I'm looking for? Just don't understand the video cards w/o outputs.
I found that if you have a Corsair commander fan hub, you can connect the blower fans to it and use the iCue software to tune the fan curve for those turbos and assign them to the gpu temp sensors as well, it’s a lot quieter during light video tasks; however I just need to know if someone have validated whether these m40s can work in sli mode or similar with the igpu output from motherboard?
I would love to setup something similar to this, one VM for my personal gaming desktop and another for a friend to use, although i doubt this would work over the internet even with some networking wizardry, at least until i get real fibre with 1Gbps upload.
Please explore GPU partitioning for AMD GPUs if you come across a way to do that, since that will allow virtualizing MacOS and Windows together with just 1 GPU.
Awesome, so jealous of this server. I would have built one if there was a super easy way to use "Remote Desktop" like streaming, like power on R-PI and it will boot and auto-connect to a windows desktop instead of per-game streaming
That's the cool part... You CAN use a Pi as a client and set it to auto connect to one of the Windows VMs. It's a full desktop connection, not per-game selection like Steam's game stream.
@@CraftComputing What software to use? I have not found any good software for that, that can handle 3D graphics like blender or start a game as if I was in front of the actual windows machine.
Random somewhat related question. Is there any issue with (or any way to allocate) multiple CPU boards and this kind of setup? We have some HP Z840's at work that are being replaced and they are dual CPU Workstations and i would love to set up one of them like this for my kids!
certain pcie lanes and i/o only go to certain parts of each cpu .thats about the main issue . just results in less sharing. so youd have one gpu and a few ports per cpu. Can only split the cpus and gpus together
I'm going to sound like a total n00b, but that's because I am... so, after you build the system physically, what's the next step to get this set up? Like, how did you install the software without any video out and what did you install? A video on that would be cool (you might already have one, I don't know) or just some links to some tutorials would be much appreciated. 😃 As always, you're Jeff, and thanks for the awesome content!
No idea how he did it, but Linus Tech Tips have used a separate gpu with video out to set everything up (including getting it to run without a monitor hooked up), then removing the temp gpu and voilà! Not much help if you don’t have a gpu with video out lying around though.
Great vid Jeff, It would be great to see better games used during benchmarks. The problem with games like you are showing is that theyre gpu bound rather than CPU bound. It would be great to see games such as CS:GO, League of legends or even Valorant. These are some of the most played games currently and are heavily cpu bound to get high fps
Hey. I know I’m late to the party on this topic, but can you use the p40 for streaming and gpu work loads? I wanted to use it with a 3060 and I want to know if it’s a good investment
If the cooling could be made quieter, I wonder if I could put this in my kubuntu box, give it to a Windows VM and game like that. Can one assign all the power to a single profile?
How does it sound with the side panels on? does it dampen noise in any significant way? also this is so awesome. I may try to do something similar on a dell precision dual socket system with some v4 xeons.
How did you get this to work? I assume ProxMox for the Hypervisor, but wouldn't any hypervisor require physical hardware to run? Curious as I have some hardware I'd like to use to turn into a dual-gaming pc
Hello, I have a dell precision t5600 with Nvidia quadro 6000 (6gb) the mother bridge is x79 my question is: the only way to use this card as gamming card is using virtual machine?
Cool, now I just need 3 more friends!
Hi fran
Hi brodi
I was thinking the same thing 🤣🤣🤣
On the upside you got a gaming PC for 1000
I'd be happy with 1
this setup for two gamers at once would be sick. just do 32GB of ram to save some cash and it is solid. I can imagine a techy dad building something like this for their two kids or something to play games.
could run the VMs from a pair of Pi400s and secondhand monitors or cheap tvs. keep the budget real low and still have a pretty solid gaming experience.
@@catalystguitarguy maybe use the mandatory school chromebooks depending on where you live.
You could also probably realistically use phones and plug them in a monitor.
Dad here, I want to do that (like a thin client), how do I connect to the VMs?
@@aliasname5518 look into parsec, it was designed for remote gaming, and should work well for a setup like this.
Keep in mind that if you do that, you still want four sticks of Ram. Losing quad channel on that set up would be a huge problem
The kraken g12 aio bracket is a great cooling solution for these cards, I left the backplate with the thermal pads on for my m40 and didn't even have to use heatsinks on the vrm. Runs super cool with no problems
Best solution. I can’t take noise anymore. Good to see someone already tried a quieter method :)
Fantastic camera work. Oh and it doesn’t matter how you apply thermal paste in a video, it’s always wrong. 😂
Jeff, Great video, to get the M40's to be quieter, I would suggest trying to use a fan controller with thermal probes such as NZXT's....I use one on my M40 12GB and it's very Vega 56 like versus being loud 24/7
Would you happen to have a little write up on this?
Vega 56? What?? It's way more powerful than the Vega 56.
@@Austin-cx2xe They likely meant sound-wise
I did a very similar build, but for a dedicated AI Rig, and you were about the only person using P40's, so thanks for the videos. They really helped! My rig ran only about ~$810 all in and I just repurposed my old T5500 (was about $90 on eBay at the time barebone with poor CPU & RAM), 192GB RAM ($20), Xeon X5690 ($10), 80 Plus 800W PSU ($0, included), 2x Nvidia P40's ($390) + Nvidia 1030 ($20) for Graphics output, 1TB SSD (hardware doesn't support NVME boot drive $35), 18TB HDD ($220), and adapters, fans, misc. ($25).
The tower is interesting as it supports up to 4 GPU's if you can squeeze them in. Also, it supports adding a riser card for another physical CPU + 6 DIMM's of RAM. I opted not to do this, and just use the extra CPU 4-pin for the second P40 GPU (power).
Another options would be the T5810 that supports the E5-2600 series CPU's (at about 2x performance) and supports up to 256GB of DD4 Memory (but based on my quick check that would cost $1/GB+, which is considerably more than the dirt cheap ebay bundle I got).
And I designed and 3D printed a dual Shroud and used one 92mm x 38mm fan to cool both and I have been impressed by the performance for the cost, and the sound. I used some rubber and spacing for noise dampening on it, and for how much air it moves it's amazing, and was only ~$16 on Amazon.
While the CPU is outdated, for my purposes, it honestly seems to run like a champ.
@RandomShart Yes, I was able to get this working with all VRAM but I modified the low profile 1030 and removed the fan (because it was a tight fit, sandwiched in-between the two P40s, and I wanted the airflow to have somewhere to go, and not just into the other card). As for using all the VRAM, I think it depends more on the software you are using. I've used LM Studio, Stable Diffusion and a few others without problems (other than your normal debug and setup stuff).
@RandomShart I'm not sure how to answer, because it'll likely be specific to your system. I'm running on Ubuntu Mate myself. The 1030 is just a generic display and I believe the driver I'm using is the the Data Center / Tesla Driver For Linux 64-bit (latest) with CUDA toolkit/bindings. If on Windows I wouldn't be able to tell you but I'd probably recommend getting and extracting the drivers only (without all of the other NVIDIA stuff) and pointing your device to it (in Device Manager to install the specific drivers).
What ram was 20 buck for 192GB? DDR2?
@@TomiWebPro Off-lease ECC server "6pc lot ~ Samsung 16GB 4Rx4 DDR3 PC3L-8500R RAM". A little on the slow side by today's standard, but can't beat the price. And looking back at the listing, my bad it was $21.64 and for 96GB RAM. I had a bunch of RAM left over from a server running the same Xeon CPU and already had the PC (intentionally had matching hardware). I maxed it out at 192GB but half I already had (and without checking, I had thought it was from the same listing).
@@michealkinney6205 Wow this is insane value, great find! I brought 4 pairs of ddr4 2133 16GB recc ram for 12 buck each, and somehow price gone even higher now I thought I might just upgrade down the line🥲
I hate broccoli. You said to leave our hate in the comments. Love the video, been wondering how well similar combo would work (add some cheap laptops and android boxes for remote). Would really love it if it wasn't so darn loud.
For the fan mods on the Tesla cards, do a push/pull. one in front of the case as you mentioned, and another outside the case at the back, as there are no vidoe ports.
i went super aggressive . took the aluminum fins and separated each individual apart slightly then ziptied two 120mm fans to the front . temps are great
would like to see this in a server case to rack mount. I would also install SSD Drive for each VM, and passthrough disks to the VM. Want to build something like to but in NZ.
Solid VM content. Excited about the tutorial. I wish the gaming giants were not so picky about emulation, would love to see a few of these used for retro consoles. The 960-1050 level GPU is perfect match even for some 1080p Gamecube/Wii and even some WiiU
The thing is you can use even lower end GPUs for gamecube and wii emulation. My htpc has a 750ti and its more than capable of 1080p emulation
Absolutely fantastic build, Jeff. Your videos are always excellent and informative. Keep up the awesome work.
hey Jeff, can you make a video on how you did the configuration of multiple VMs. A step by step guide would be great.
That's literally next 😉
Nice project. I would have taken a different motherboard. Like the Huananzhi X99-F8. You have a extra PCI X16 slots that runs 4x, but you can install a 10Gbit card which improves latency. Also it supports quad channel memory, I'm not sure if this Jinghsha does, it looks like dual channel memory.
You can also a variation with 1 Tesla GPU and use the 2nd slot for a PCIe to 4x M2 adapter. Then you can run 2 VM's with their own SSD's.
you know it's gonna be a good video when there's a chinese xeon board & tesla cards in the thumbnail
Great build! I'd use something like this to set gaming machines up for my nephews to run from their chromebooks since we don't have a lot of room for full desktops in our place at the moment.
Very cool project And i approve of the custom shroud for the 2 cards. With a high static pressure 140mm fan you could be good if its sealed up good.
I haven’t looked at my M40 12GB for a while, my feeling was that the memory profiles were the problem. I don’t think I’d seen the newer script or hadn’t appreciated that it had more granular control if I had. This is going to be the ideal end of month project.
Yes, the issue with the M40 in vGPU has always been the 8GB memory cap for Maxwell cards. You couldn't allocate more than 8GB before. VGPU-Unlock-RS allows you to customize the profiles, and use all available memory.
Great video, love all the Bequiet fans, and immediately overpowered by blowymatron.
Also 2nd shirt is fantastic.
What I would do for cooling is get some acrylic or polystyrene sheet and pipe air from two fans in the front (something like a nidec beta 5 TA450 which can have great static pressure but also can run silent may be my choice)
Spent a few weekends building out something similar. Worked great until I realized most anti cheat systems don’t allow you to play games in vms and will ban you for “trying to work around it”. Sadly had to go back to a dedicated gaming pc…
I think you may be able use a non virtualized session and use something like splashtop with a virtual monitor, or any indirect display driver and steamlink.
@@satibel hi can u please explain a little more, how will I be able to run different instances of the game on one PC
@@imnitin97 the game needs to allow multi instancing, otherwise you can only play on different games.
Yay, the first video with a cameraman! There is some echo though in that part so you might want to ask your little helper to add some sound dampening panels
Thanks! I ordered a couple 12GB cards so my kids can play with their neighborhood friends. I'm going to put them in a PowerEdge R720 with a pair of e5-2667 v2 processors. Hopefully, the case will keep them cool enough not to require a shroud as I don't have a 3D printer.
That's basically the setup I run here for neighborhood friends :-)
Yup, they will stay cool enough, but the idrac fan controller will jump to noisy levels by default unless you override it with ipmi commands.
4 gamers 1 HDD sounds rough with that random IO.. I know it wouldnt be applicable in this use case, but it would be cool if the windows spanned partition had a version meant for performance, where, yes each whole file only goes to one hard drive, but instead of filling up the first drive before moving to the second, it instead filled the drives up evenly, and then did some optane style optimisation later based how often the files are used together
afaik windows can do software raid0 too either with the old interface or with Storage Spaces, although not on boot drive, because screw that, nobody needs raid on boot drive ever
@@marcogenovesi8570 RAID1 for boot drive is perfection.
Well done with an amazing Build. it makes me drool and at same time gives me tears
It works! Great to see this setup working well. Awesome for a cave setup for small lan parties
Nice video....the format of live assembly is very interesting!
One thing I'd like to see is all the other things required for this type of setup. Would it work well on 1GB ethernet or would you need to add cards and hardware to run 2.5GB+? What kind of upload speed would you need at home to make this a viable experience away from home? I know these things have been mentioned before, but I don't have time to go back and find it. Would be nice to have a short section for all of these videos with little details like that. Basically, how fast does my network need to be to handle the different loads? 1GB should work fine for 1 VM, but what about 2? 4?
At what point would I need to upgrade the networking? At 4 VMs? 6?
Even just a breakdown of bandwidth requirements for a single connection and we can do the math from there.
Love the content, just moving and trying to figure out what infrastructure I'd need to run in order for something like this to work.
Nice setup; you can try to mount the 120mm fans where the PSU sits to circulate the air in the whole case. since hot air is normally pushed out via the rear part of the case; I believe that should help
there's another way to cool those cards , if u disassemble the shroud you can remove the clear part and get a expansion slot cooler it makes it alot quieter but takes up a extra 1-1.5 slots, its what im doing rn
great video! Wonder if it's possible to pass the GPU through windows the same way laptops do, with all of it going through the iGPU.
Yes, you can do this with the Tesla M40 as well. In fact, that's what my previous video on the card focused on: ua-cam.com/video/TQ9B67T5qWs/v-deo.html
This would be the perfect use for zfs deduplication, because every game would be installed 4x
I'd be curious to see if you could create enough positive pressure in the case to cool the GPU's. Even loaded up with Noctua industrial 3000 rpm fans running full boar would still sound way sexier than those blowers. Seal every gap except the GPU vents and put all fans pulling air into the case. Please and thank you
Need a dedicated section for craft beer! Thank you for the review!🍻
Would love to see a build like this with a newer Xeon Platinum 8124m with how cheap they have gotten
Now find a cheap LGA3647 motherboard :)
Always on the lookout for one 😉
I was considering building a dual xeon 10 core 20 thread system and then break it out to 4 VM's each that can be used as a video editor/render machine running DeVinci. Any suggestions, comments or thoughts are always welcome.
I run my first Gen EPYC system with a Tesla M60, and use this same setup for remote video editing.
The M40s would also be a great option, especially the 24GB models from this video. I'd spring for a pair of 12-Core or 18-core chips... E5-2690v3 or 2696v3. That would give you plenty of power for 4x editing VMs.
Fantastic video Jeff!
This sounds like quite a good beast for a games room, I'd be keen to play some multiplayer games on this and see how it goes.
for those teslas, i think the most practical way to cool them in a desktop case is just remove the original shroud they have, and adapt a 3d printed one that can hold 2 case fans, maybe 92mm ones? i dont remember the size but yeah, you could even 3d print a new shroud for them to hold those fans
That works for the K80s. Unfortunately the Maxwell and Pascal Tesla cards have closed fins on top, so you can't cool them 'top-down'.
@@CraftComputing they do have closed fins, yes, but the fins are easy to bend down with some pliers/flathead screwdriver and allow top-down cooling. It's not pretty unless you take your time with it. I used this method with a firepro s7000, an arctic pwm P8 fan, and CA glue. Temps were out of control previously, but with the modification and fan at 50%, temps didn't go over 72°C.
@@CraftComputing well, nothing some scissors cant fix, but for real tho i think Myconst's friend did something like that taking out the top fins, is not elegant but it can work
Amazing video, Jeff! A while back, I was working on a silent cooling solution for Tesla K80s in desktop PCs. I left off at two Noctua 60mm hooked up to a Y-connection splitter, so that it could be controlled by a single PWM header. You can have them run at 100% load, and still be quieter than the CPU cooler on idle. A proof-of-concept is on my channel (card is not powered, but fans are at 100% load). Happy to send you the STL file if you wanted to print it and give it a shot.
In hindsight, I would have actually just scrapped the concept of using a blower-style cooler configuration, and instead opt for an open-air configuration. Some desktop Tesla users found that slapping on one or two fans directly on the heatsinks could dissipate heat just as effectively and much quieter than using a blower. For further improvement in thermals, you might also want to consider repasting the cards, as many Teslas were manufactured ~10 years ago (thermal paste is likely underperforming by now).
The other alternative is watercooling, and there are a few CNC-machined waterblocks on Ebay for the Tesla K80/M40. I haven't tested those, but they seem like a promising (albeit a bit costly) solution for quieter cooling.
I'd be interested. I got myself a k80 a while back but haven't figured out a way to cool it without going deaf
@@mitlanderson As a quick hack, might be worth just slapping 2 noctua 80mm fans, straight onto the heat fins (can unscrew the top cover pretty easily, held by a few hex bolts on the side), and see if that can provide any feasible amount of thermal dissipation. Unfortunately, I gave my K80 to a colleague, but that would have been my next idea for a noctua-based cooler
Thank for your video to pass-trough a GPU to multiple Hyper-V's, I also have success with passthrough a displaylink DL3900 adapter with USB Eltima gateway. I think multiple USB3.0 docks will also be possible. No GPU hdmi/dp is needed because the docks have them self. Also no GPU encoding.
Jeff, it seems the M40 has the same layout as the GTX 980 ti. A friend of mine fitted a 3rd party cooler with 3 fans from that card and fitted it on the M40. It works great, full load he gets max 75°c.
Could you use the Noctua industrial fans as front intake to remove blowymatron?
I wonder if you can use LVM copy on write so you only have to keep one instance updated for all the installs to save time and storage, even better than deduplication
Also take the next step, have this server serve TFTP boot configuration for thinclients that automatically open the session from the VM on boot!
The e5 2690V3 can be run at all core tubo without generating much extra heat, if the X99 you have is bios modable
Unfortunately, as far as I can tell, the mainboard isn't available in EU at all. Possibly from aliexpress within 1-2 months, but not anywhere local that I could find. It not having a proper model name/number does make searching kinda hard though.
Hi. Can you do a performance test on davinci resolve over the virtualization? I wonder if it would be better than my 1060. Thanks.
Someone needs a height adjustable desk. My back hurts just by watching this video. Nice video, god job.
JEFF could you do an UPDATED installation on the 7.3-x version its a little different than some of your video's for a Noob like me i have found. Some optimization and things just seem alittle different
I'd make UA-cam videos about it lol I'm glad you clearly mentioned the cons. Seems some eBay sellers get stuck on some UA-camr saying how "amazing" something is and he pricing gets stupid. I think the M40 shouldn't go for even $100. I did buy one to try to sort out using it for a local machine, unfortunately it died with plenty of other parts when 12 volts arced across a USB connection. Thanks to a past video of yours, I used a virtual UB monitor to get games running that wouldn't before on a headless machine and streamed through Steam. Was mostly fun as a hobby experiment. A couple games I play would run without too much hassle, but vsync is typically broken, often can't change resolution (sometimes the resolution shows as "undefined" etc.), and games that seemed to get updates to run better were apparently just fixed by game ready drivers that the Teslas don't get without the setup fuss that makes these a harder sellllll. Neat/cool stuff, but yeah...
to be fair, in the current GPU climate, 980Ti/1070 level performance for $140 is a legitimate bargain when new 1050 Tis are still being pushed for $200 and 1650s for $300. I suppose it has entirely to do with the complete lack of video ports and active cooling that really prevent these cards from being readily repurposed by desperate gamers
@@bojinglebells well yes, ure right but 1650s and 1050tis are pushed due to not avaibility and demand for em
@@bojinglebells Cooling is easy (if loud is fine), display out is THE problem, the end. If they had even one, then it'd be fine... and people would probably try getting $250-300 or maybe even more for them.
Nice build...how many KASM instances could I run on it?
Music was too loud when you were talking in the intro. (Juat a heads up)
I love these multiple virtual gaming machines with gpu passthrough videos.
Is it possible to remove the gpu shrouds completely and jerry-rig a tower cooler to them to run with a normal fan? Or possibly water cooling with a custom block?
Was wondering if those GPU cores would OC under water cooling. If I can find a block i will try
Could you do 4 VR headsets on 2 Tesla P40s? How would that setup look like?
Hey Jeff, please is it possible to pass through one vgpu to a lxc container running jellyfin or Plex for hardware transcoding
Amazing - just amazing. Looking forward to see some Radeon/ryzen variation. Subscribed
Jeff to your knowledge do most those Chinese motherboards have above 4g decoding? Or at least the x99 ones.
This setup is sick Jeff!
Can't you use one of those in-line resistors that will slow down the turbine fan?
While I find your use case interesting and plan on experimenting with VM GPU sharing, mostly I want to use these older cards like the M40 for things like stable diffusion. Appreciate your content.
Remote VR game station, but problem with Oculus software is that it's very much doesn't want to work in a VM =\
did you ever do the video you said you'd link at 12:16?
If I had a spare 1.2k USD, I would make a similar settup for split screen Destiny 2 gaming. I even have a C1 65 inch OLED supporting VVR.
Thanks for the video, Goodnight!
I didn't catch how you partitioned the cores when your running 4 vms.
3/6 would be cool but than what is dedicated to the hyper visor?
I configured each VM with 3/6. The hypervisor doesn't need dedicated cores, as CPU scheduling takes care of any overhead needs.
@@CraftComputing That is pritty cool.
I figured it would be possibe but just wasn't sure. I guess hypervisors are light enough it wont really cause issues.
I wonder how a vm would work for streaming to a quest via airlink.
Would be intresting for a house with vr.
How you managed video outputs? How use it how wire it, how separate peripherials?
Not sure how to spell it but Rett is doing a fantastic job if I do say so myself.
And Jeff your killin it as always bud.
Keep these gems coming!
I wonder if a VM with this Tesla card can run Lumion renderer, a graphic card with enought VRAM is unobtanium
I'd probably set it up with 3 1660supers I have sitting around and give each vm a full pass-through gpu.
I could have all 3 of my gremlins gaming on one cpu.
I like your thermal paste application. I'm gonna follow suit ☮️❤️
I've been looking for a multi PC in one. Running 2 FO76, 1 or 2 klipper printers, and a 3d model PC. Can this be the system I'm looking for?
Just don't understand the video cards w/o outputs.
I’ve been wanting to do a sick lan rig. Hoping I can do this but but with output on the cards
Interesting build! Would be a cool AI/ML workstation
I found that if you have a Corsair commander fan hub, you can connect the blower fans to it and use the iCue software to tune the fan curve for those turbos and assign them to the gpu temp sensors as well, it’s a lot quieter during light video tasks; however I just need to know if someone have validated whether these m40s can work in sli mode or similar with the igpu output from motherboard?
When will you do a rendering focused build?
Nice server Jeff, yeah like it. If it were mine I'd use it for folding at home on its down time.
I would love to setup something similar to this, one VM for my personal gaming desktop and another for a friend to use, although i doubt this would work over the internet even with some networking wizardry, at least until i get real fibre with 1Gbps upload.
Please explore GPU partitioning for AMD GPUs if you come across a way to do that, since that will allow virtualizing MacOS and Windows together with just 1 GPU.
which virtualization platform did you use? XCP/Proxmox/ESXI? I haven't had luck trying to virtualize Nvidia GPUs on XCP
This is Proxmox. This method does not work on ESXi or XCP/Xen. It requires KVM/QEMU.
Love the gpu virtualization videos. But isn’t playing via Parsec always just a bit laggy?
Awesome, so jealous of this server. I would have built one if there was a super easy way to use "Remote Desktop" like streaming, like power on R-PI and it will boot and auto-connect to a windows desktop instead of per-game streaming
That's the cool part... You CAN use a Pi as a client and set it to auto connect to one of the Windows VMs. It's a full desktop connection, not per-game selection like Steam's game stream.
@@CraftComputing What software to use? I have not found any good software for that, that can handle 3D graphics like blender or start a game as if I was in front of the actual windows machine.
Awesome video, one question though, after the split do you still need the Nvidia License Server and vGrid CC License?
Random somewhat related question. Is there any issue with (or any way to allocate) multiple CPU boards and this kind of setup? We have some HP Z840's at work that are being replaced and they are dual CPU Workstations and i would love to set up one of them like this for my kids!
certain pcie lanes and i/o only go to certain parts of each cpu .thats about the main issue . just results in less sharing. so youd have one gpu and a few ports per cpu. Can only split the cpus and gpus together
My only bummer is there is too much latency for real time games using parsec. I do enjoy using my Tesla cards for video editing.
I'm going to sound like a total n00b, but that's because I am... so, after you build the system physically, what's the next step to get this set up? Like, how did you install the software without any video out and what did you install? A video on that would be cool (you might already have one, I don't know) or just some links to some tutorials would be much appreciated. 😃 As always, you're Jeff, and thanks for the awesome content!
No idea how he did it, but Linus Tech Tips have used a separate gpu with video out to set everything up (including getting it to run without a monitor hooked up), then removing the temp gpu and voilà! Not much help if you don’t have a gpu with video out lying around though.
Have you tried a P40 yet? I am curious if you can do the same workarounds that you can do with the M40 to get it working as a gaming card.
Bbbeeeaaaauuuutttiiiifffffuuuuulllllll!
Disclaimer: I may be biased.
Great vid Jeff, It would be great to see better games used during benchmarks. The problem with games like you are showing is that theyre gpu bound rather than CPU bound. It would be great to see games such as CS:GO, League of legends or even Valorant. These are some of the most played games currently and are heavily cpu bound to get high fps
Valorant won't run in a VM, but CS:GO runs just nearly as a fast in a VM as it does natively, even split up, it doesn't have alot of threads
Hey. I know I’m late to the party on this topic, but can you use the p40 for streaming and gpu work loads? I wanted to use it with a 3060 and I want to know if it’s a good investment
It's a Pure Base 500 (NO 5.25") not the 600 (2x5.25"), still a very good case !
If the cooling could be made quieter, I wonder if I could put this in my kubuntu box, give it to a Windows VM and game like that.
Can one assign all the power to a single profile?
Yes, all power can be assigned to a single VM. And I'm working on a quieter solution...
How would you send the virtual display images to 4 different computers?
How does it sound with the side panels on? does it dampen noise in any significant way? also this is so awesome. I may try to do something similar on a dell precision dual socket system with some v4 xeons.
How did you get this to work? I assume ProxMox for the Hypervisor, but wouldn't any hypervisor require physical hardware to run? Curious as I have some hardware I'd like to use to turn into a dual-gaming pc
Planning to build a headless gaming server with lower midrange hardware… any suggestions
So, can I team up 2 of these on a single VM for rendering, modelling, etc?
Is it possible to use two Intel Arc A770’s? They like 10th gen or higher.
Does a MB even exist for such a feat?
If I had a system like this I would use it for Touchdesigner or Magic Music Visuals art installations.
Hello, I have a dell precision t5600 with Nvidia quadro 6000 (6gb) the mother bridge is x79 my question is: the only way to use this card as gamming card is using virtual machine?