So many times a client will ask if we can do something, and I think to myself "yes, that's totally doable, and shouldn't be too expensive to implement." And usually, the base software is NOT. It's just the licensing for all of the useful features that gets piled on over top that kills you. So you do your research and go back to the client, and then they look at you like you have 3 heads when you quote them about 100x more than what they were hoping to pay. Enterprise software is a bitch.
its intentional. They know the people using this in a business are willing to "expense" a bunch of money for stupid shit. This "license to death" model will have to change, given that windows server 2016's default "license" allows only 16 cores and a single CPU. Got a Zen+ based EPYC CPU with 64 cores? you need to buy 4 license keys.
Chubbysumo Is the windows license based on physical or logical cores. I’m only asking because I wouldn’t be surprised if they count logical cores which would basically require double the amount of licenses needed.
@@chubbysumo2230 you can license more than 16 cores. The minimum you can license is 16. So if you buy a server with 8 physical cores you still buy 16. After 16 you can buy as many as needed if you buy datacenter. Standard works the same except when talking about VMs. Datacenter can run any number of VMs as long as you have enough core licenses. While Standard only covers 2 VMs and the host(hyperv). If you have 4 VM and 16 cores you need a license for 32 Standard core or 16 Datacenter cores. Now the price difference is significant. At 16 cores it takes like 10+ VMs before datacenter makes sense.
We have actually used these NVidia cards to run Catia CAD as VDI infrastructure on it because the old Catia certified IBM PowerPC workstations got too rare and so in turn too expensive to keep going for the users
Good video series.... is it weird to say that I like watching you do projects so that I will NEVER have to do it myself. Love the beer reviews too and I don't drink.
Looking forward to your video on SR-IOV with the Firepro. After some testing with RX580 and an old GTX660 using Proxmox and PCIe passthrough, I'm thinking that a Firepro and SR-IOV might be the way to go. Letting you do the groundwork on that seems like the smart move😉
We're almost 100% virtual, using tesla M10's within VMWare. This video series revived the part of my soul that has been crushed by the experience, solely for the purpose of crushing it again.
Again as a technical achievement I really enjoyed this build. It makes me sad that some of the biggest failings preventing most from trying this is being bound to additional licensing. But it is enterprise gear after all. The company I work for stood up a Grid K2 farm for our engineering dept. a few years ago and it really worked out well. I assume it still is working well as i've not heard any of the engineers complain about shortcomings. And the setup is appreciated even more now that all of us are working from home until at least Jan 2021.
Jeff, I wanted to say thanks for covering this sort of info. I just ordered a Xeon e5 2669 v3 to put in my old ASRock X99 board. I'm pairing it with 24TB's of SAS storage, 128GB of RAM and will be ordering an M40 GPU. This will be a Plex, cloud gaming, and NAS for my home. I'm super excited! You make this sort of thing possible for basics like myself! PS, my name is also Jeff!
@@CraftComputing I might just use an old 1070, but we'll see once the other hardware shows up. As for my username, I've had that since the old AIM instant messenger days, and am actually from Cleveland lol. The Browns weren't around when I was a little kid, and picked the Packers because green was my favorite color.
a friend has been doing VDI setups for quite a few years. the "value" side of it was in buying and paying employees to support (lets say 1000, though some of his were into the 10s of thousands) modestly powerful office desktops. In a real world usage scenario its highly unusual for all of the VMs to be running at or anywhere near 100% utilization. the most recent ones he did were 64x VMs (with 1GB vram) per server with 72 cores & 512gb ram hosts or 72 cores & 768gb ram hosts for the higher ram spec VMs. its one of those things where the value is seen at scale (and at scale, the licensing costs are "flexible" enough that you are getting a decent discount).
I've been working on a couple different gaming servers with different methods. Thanks for sharing. Haven't tried this yet and now I know why I shouldn't 😅 your loss was our gain
Kind of expected that they [M60s] would be terrible, but I'm amazed that you got them working! They're really made for CUDA workloads like neural networks and other machine learning... Great storage and great software solutions you figured out! The pricing for licensing on both VMWare and NVidia make it more worth while to passthrough cheap GPUs like GTX 1050s or GTX 1660s!!
I really want to know what software companies like Shadow run in their servers. Yes, Linus had an actual Shadow server and played games off of it, but that was with a local install of Windows, so it was pretty pointless (in my opinion). Virtualized gaming has always been something i have wanted to do/try, so I have loved this series! I'm glad that you have been able to push through all that BS licensing to bring us such awesome content!
We use virtual workstations at work for people on laptops, temporary placements and sometimes tablets, but we’re using a custom made 4u enclosure that has 10 mini itx computers in it with a gpu each. Very cool stuff.
We use these at a school district. It allows us to deliver rendering capabilities to technology and art students remotely. Each student is handed a Chromebook for their schooling. It todays day and age this has come in handy.
Your persistence of this project reminds me of how I got 2x 3TB SAS HDDs in RAID-0 to run on my consumer board with an i7-6700k. I love channels that go deep into us trying out industrial use hardware. It's more times than not a cool and nerdy gem. Sometimes hits a dead end, like with your license issue or my first IBM cockblocks at the hardware level, despite on paper being %100 capable -_-. These manufacturers lock in companies and it hurts us consumer grade people. SAS 3tb 6gb/s is $23. SAS reads and writes at a single time while SATA3 is one or the other at a single time. Should make a video on that stuff. If not I'll beat ya. 200TB @6gbs/sec for $200 isn't uncommon. Thanks for keeping it real ! Tesla K80 GPUs are $110 now. Brought me here. That vram alone to sink with an i-GPU is potential for me for novelty lols.
I've been tryingto do the same since I saw your first video with Grid K2. I finally ended up to replace the dell R520 with R730xd and quaddro M4000 in passthrough for 1 VM with esxi 6.5. With horizon client I can play with 30 fps 😕 . With parsec got 60. I even conneted the output of the gpu to my monitor and used it as normal pc. But weird stuff happen when i do that. Monitor counts get confused etc... Once again great educational video... Keep the good job
These also work on KVM (we used them on ovirt during testing) But ultimately we settled with using intel graphics using VCA/VCA2 cards. Just needed a little 3D and no licensing was nicer. Not to mention with intel GVT/KVM you can still use the built in SPICE/VNC device for display. We also tried the AMD mxgpu stuff, good luck, you'll need it.
Snap! Firepro! I have been buying a few smaller ones just for easy plug and play solutions for my work systems but i just bought a much larger Firepro, the S9050 (has 12 GB of GDDR5 on it!) for a mad scientist project. Only paid $70 bucks for it and just had to make my own cooling system for it. I have seen videos where others pair a different Firepro with it to make a super computer, some pair it with another S9050, while others paired it with a consumer card that shared the same Tahiti chips on it. Something like a HD 7870 XT (i didn't understand why, since i thought the 2gb card would lower the 12gb card). I have an older FX 8150 with 32 GB or ram and the 990x mobo does support dual cards on the shelf, i just want to make a recycled inexpensive AMD version of parts I already have and what you are trying to accomplish (few of my friends are strictly Intel and this would be funny to see them using AMD), which will allow me to play some mid/low tier games with my friends. (Super Blood hockey, Battle Toads, Scorched Earth, Super Arcade Racing, Mutant Assault) 20 year's ago, we used to LAN party and sometimes we would have our battles outside in the driveway when it was 30 degree's outside. We would have to put little space heaters under the tables and some of them would game with blankets, but it was awesome to game under a full moon. Now pushing 40, would like to do that again...
At my old job we used servers with this kind of technology for architects that went from the giant P-series Lenovo laptops to the much smaller T4xxS laptops, using the VM's to do all the work
Funny enough I went down this rabbit hole as well but in a low tier version of this. Funny enough I ran into the same problems using vmsphere and unraid no matter how hard I tried to get passed some of the limitations of both I couldn't get good fps on any games. The only way I got decent fps was using proxmox and a 1050 ti with a dummy plug to trick the VM into a higher resolution. This works pretty nicely but for what I use it for the server was drawing to much power and I was limited on that since I also have a VM running a Plex server. Still haven't given up since my plan for this extra VM is to play emulators on it like ps2 and Wii with my friends. I think the path is long but is gonna be worth it. Can't wait to see what your gonna do with those amd cards.
So my question is how would the vmug licenses work if i remember correctly vcenter is included with it as a 365 day evaluation license, considering is 200 usd per year. Also my second question is what is blast extreme on this configuration when connected to a thin client.
I posted on your last video about QEMU/KVM could you utilize his cards in that type of setup or do specifically only work with grid? just asking because I have a R720 with two empty GPU slots. while I have vfio set up on my main rig using parsec the latency would be too high I think for the server for esports
I might not fully understand this setup. Is it possible to deticate one card to game graphics and the second for video encoding? Cut out the number of instances, but boost performance.
Really interesting video. I just bought a Dell R730 that when I looked up the service tag was spec'd up with the GPU kit, 512GB of Ram, 2 E5-2695 and 2 nVidia Grid K2's. Although it looks like the CPUs have been replaced with E5-2630L versions now. Just before I bought the server I found a Quadro K2000 on ebay and snapped it up but the Dell R730 won't even complete post with the K2000 installed?? . Anyway do the AMD cards have all the associated licensing costs of the nVidia Tesla and Grid. Looking at the AMD card you mentioned in the Video now although for my use I will prob give up at this stage.
I have a friend that does some remote data science work for a major Canadian bank, and he was given access to a virtual machine instead of a physical workstation. One of the rationale being that all data and computation should be kept on the bank's servers, within a VPN for security reasons. I have a feeling that this kind of technology is what was used.
I'm using a 1U with two RX580s. Never going to have more than 2 people playing at a time my house so it works perfect. Used it at a LAN party to play Bro Force on the TV in the house we rented using a Steam Link.
Can you please give me some details about the water cooling for the Telsa M60? Me card is running too hot and I'd like to replace the 2x 40mm fans (Noctua, silent but not enough airflow). Thanks!
Nvidia vGPU doesn't dedicate Vida Cores for vGPU instances from everything I know about it. Only the vRAM (Video RAM) is dedicated. The Nvidia vGPU CLI Tools allow you to configure the mode the Cuda Core sharing uses between 2 (again, from memory).
Wondering if a Mobo that has a bunch of x8 electrical/x16 physical slots, and then P2000 or similar 1slot Cards in pass thru mode would perhaps be better/easier and eliminate all the software license restrictions.
Finally it works 🙌 I am happy for you man. But try making a server with consumer hardware for 2 gamers maximum on a Budget. It will be a great video as well.
I've sticked an AMD S7150x2 in an HP DL380 Gen 9 for CAD VDI. It's basically a dual chip R9 290X and has a driver stack based on SR-IOV. Implementation was totally free aside from usual VMware and Win Server licenses (probably because AMD is lagging in the market) These cards can be found pretty cheap on ebay.
Wow I've left the comment before noticing you have one! Looking forward to see it's gaming performance, didn't really have an opportunity to test beyond Specviewperf
Last comment, look into Teradici Cloud Access which I'm using for high end media clients and unlike previous Teradici solutions, doesn't require a hardware offload card.
Could you modify it so it was budget friendly? Say use an X99 or X79 xeon and two dedicated GPU's (plus a low end graphic adapter for GUI access to the machine) use PCI passthrough and throw on a 1080p dummy plug so you can always have 1080p output for each VM?
@@Gastell0 I get my question was out of the scope of this video, I just wanted to verify someone had done it before I, you know.. Did it. I don't wanna blow $800 on hardware just for me to find out it ain't gonna work. But thank you for confirming for me! :)
@@justanotherhuman-d6l Yea, you can go on the UnRaid forums to see which motherboards and GPUs are easier to pass-through. When any of my GPUs are plugged in the main X16 slot, they require the use of OVMF BIOS on Q35 AND a custom ROM to work properly. Otherwise, when they are inserted in the secondary X16 slot they work without any custom rom. But these settings can differ per motherboard / GPU / slot combination, so you might need to try out different permutations before you find out the one that works for your specific setup.
Hi Jeff, nice video, very useful, but I would like to ask you, how do two Nvidia Tesla K80 cards go together. I know that they cannot work in SLI and that the typical memory is divided by a maximum of 12 GB... if I install two identical cards on a PC, with an AMD EPYC processor, to be able to create renders, what happens? Is the work divided between the two video cards and the processor? Do I need to change the software? Thanks for the attention.
Does the the n'viddya license server let you run multiple instances? I'm wondering (and no, I'm not proposing this is worth the effort) if you can put them behind a TCP load balancer and make things slightly less terrible.
I thought about pairing a Tesla K80 with my GTX 760 to do some data crunching for science (I figured they were both Kepler so why not). But it seems like the K80 wouldn't be a simple plug and play kind of situation, especially when needing something better than passive cooling.
Would it be possible to pass through a single gpu to a vm, i.e. one of the four, or would you be limited to passing on a per card basis? Then using registry edits on a per vm basis to get them to act as the equivalent of a high performance laptop card. If this is possible, I'm not sure whether you'd need some gpu to act as the igpu or whether it'd be possible to get by with just parsec. Potentially this could mean one could avoid the pricy drivers at the cost of being limited to as many vms as individual gpus. Could be interesting to try out.
Hi dear. How you does to conected and run two tesla at the same time. What is Your mother board? i put two tesla on an eth79-5x 5 pci-e 8x gen 2 motherboard. when i use only one tesla m40 , the motherboard works, but when i put the second one the cpu does not start. Could you let me know what could be happening? I
I find your videos really great and well explained. I am currently doing a project similar to yours. I have two NVIDIA Tesla M60s installed in an HP ProLiant DL380 Gen9 server and Windows 10 pure without virtualization. Then I installed several drivers from the NVIDIA driver page. Unfortunately, none of the graphics cards show up in the performance tab of Task Manger, but they all show up via GPU-Z. When I run a benchmark, it runs on the Intel GPU and not on the Nvidia GPUs. I also can't access it via Parsec because the host encoder can't be initialized. Can you maybe give me some tips on what I did wrong or forgot? Thanks already.
Did you ever try qemu-kvm with vGPU ? This might at least get rid of some licensing woes. Though based on the nvidia documentation it might not be officially supported besides with OpenStack?
Is is possible to pass the GPU power through the integrated grfx?. Much like a mid range gaming laptops work with discrete grfx. Also keen to see the AMD card stuff :)
You’d better use VMware’s Blast Extreme protocol instead of Parsec. Just install VMware Horizon Agent in your VM and VMware Horizon Client on your Mac, PC, etc. Connect directly to the VM from your desktop client without having to deploy the full blown Horizon Suite. BTW, using VFIO under Linux on consumer cards will provide a much cheaper scenario... and way better performance too!
He dunked on that idea (GPU passthrough). Many have tried this concept before, to varying levels of success in the end. And they keep trying, it's good content.
I have a Dell Precision Rack 7910 with a 2070 Super and a Teradici 2 Remote workstation Card in the basement. I use it for gaming with a Dell 7030 Zero client. Works great
Hey Jeff, love the channel and awesome content and guides...and the beer. Have you looked at a possible CPU bottleneck with the lower Epyc speeds? I just built a dual E5-2690 with Proxmox pcie passthrough of a RX5700XT and I’m getting bottlenecked on certain games. Still debugging it but seeing the Epyc speeds made me think of my situation. I’m going to try to pin CPU cores next to the VM to see if I get a boost.
So anyone that knows this technology knows the entire design and use cases are not meant for a single user/gamer. What you did is what we do for homelabbing for testing and learning VMware technologies. Those cards aren’t meant for “gaming” they are meant for an enterprise environment with multiple users needing gpu like someone described below. Cool project and I like the water cooling approach cause goddamn do Tesla cards get 🥵. I’ve been playing with a Tesla t4 in my homelab and you can cook eggs on it
This painfully convoluted journey has almost wrapped up and my opinion still stands that it's cheaper and less challenging to build individual computers than to use a Big Computer Many Terminal approach
Nicolas Ballet I know sadly there wasn’t a lot they did better on in the last years. I also do wonder what’s there strategy for CDNA e.g if they allow the “pass through” on there RDNA cards too as of these game streaming services. The other thing is that we will see especially with the consoles better optimized games for pc on the AMD side, hopefully and maybe also some cards that don’t suck?
given the bad performance, ¿what are the real use cases for this?, GPGPU crunching only i take it(but in those cases you wouldn't split the GPU into vGPUs to several VMs, but running them on bare metal, right?). Let's say you'd want to do graphic work, are these drivers certified for CAD/CAM like Quadro drivers?, do they accelerate professional CAD workload?
These are fully certified for CAD, just like Quadro drivers. In fact, that is the most common use case. But these were definitely limited by the GPUs, as I was only running one VM per GPU core, so each had the full 2048 CUDA available to it.
@@CraftComputing still quite expensive, but i guess it would allow remote work to a powerful workstation with no much issues. then again without remote work a local workstation would probably end up cheaper i think
Those prices seem high. I'm covered by a VMware enterprise licensing agreement so I think I get a price break. How much I'm not sure. Its all negotiated by our corporate office. If only I had a Tesla card to test with 🤔 Can you get the feature in VMware remote office as that is more economical. Then again your paying per VM instead of per server.
And I'm glad I waited for the end... I was going to ask if there was an AMD solution that was possibly more budget friendly. Now I know a video on that might be incoming :)
Not at all for the numbers I got. Single core speeds on the 7601 are similar to the Xeon 2670, which had no problem driving GTX 1070 and higher at well over 120FPS in these games.
This is made for thin client environments. Its so the client session isn't hitting the CPU super hard trying to do training videos or UA-cam. So if your running citrix the user experience isn't a choppy mess trying to do everyday tasks.
Once again the World of Enterprise Solutions shows how to go from °technically feasible° to °financially excruciating° with the Magic of Licensing!
That is so true
So many times a client will ask if we can do something, and I think to myself "yes, that's totally doable, and shouldn't be too expensive to implement." And usually, the base software is NOT. It's just the licensing for all of the useful features that gets piled on over top that kills you. So you do your research and go back to the client, and then they look at you like you have 3 heads when you quote them about 100x more than what they were hoping to pay. Enterprise software is a bitch.
its intentional. They know the people using this in a business are willing to "expense" a bunch of money for stupid shit. This "license to death" model will have to change, given that windows server 2016's default "license" allows only 16 cores and a single CPU. Got a Zen+ based EPYC CPU with 64 cores? you need to buy 4 license keys.
Chubbysumo Is the windows license based on physical or logical cores. I’m only asking because I wouldn’t be surprised if they count logical cores which would basically require double the amount of licenses needed.
@@chubbysumo2230 you can license more than 16 cores. The minimum you can license is 16. So if you buy a server with 8 physical cores you still buy 16. After 16 you can buy as many as needed if you buy datacenter.
Standard works the same except when talking about VMs. Datacenter can run any number of VMs as long as you have enough core licenses. While Standard only covers 2 VMs and the host(hyperv). If you have 4 VM and 16 cores you need a license for 32 Standard core or 16 Datacenter cores.
Now the price difference is significant. At 16 cores it takes like 10+ VMs before datacenter makes sense.
This process sounds terrrrrible :P
Sounds like so much fun to me. Ya I am sys admin...
It was both awesome and terrible. So much cool hardware, so much stress, not enough performance 😂😂😂
THE ADMIN😮😮
Lol you know how many people ask "can you game on it"
We have actually used these NVidia cards to run Catia CAD as VDI infrastructure on it because the old Catia certified IBM PowerPC workstations got too rare and so in turn too expensive to keep going for the users
Good video series.... is it weird to say that I like watching you do projects so that I will NEVER have to do it myself. Love the beer reviews too and I don't drink.
Looking forward to your video on SR-IOV with the Firepro. After some testing with RX580 and an old GTX660 using Proxmox and PCIe passthrough, I'm thinking that a Firepro and SR-IOV might be the way to go.
Letting you do the groundwork on that seems like the smart move😉
You couldn't have posted this at a better time. I'm still looking into this myself.
Thank You!
We're almost 100% virtual, using tesla M10's within VMWare. This video series revived the part of my soul that has been crushed by the experience, solely for the purpose of crushing it again.
When you pulled out the Firepro - "Aw yeah"
Again as a technical achievement I really enjoyed this build. It makes me sad that some of the biggest failings preventing most from trying this is being bound to additional licensing. But it is enterprise gear after all. The company I work for stood up a Grid K2 farm for our engineering dept. a few years ago and it really worked out well. I assume it still is working well as i've not heard any of the engineers complain about shortcomings. And the setup is appreciated even more now that all of us are working from home until at least Jan 2021.
Jeff, I wanted to say thanks for covering this sort of info. I just ordered a Xeon e5 2669 v3 to put in my old ASRock X99 board. I'm pairing it with 24TB's of SAS storage, 128GB of RAM and will be ordering an M40 GPU. This will be a Plex, cloud gaming, and NAS for my home. I'm super excited! You make this sort of thing possible for basics like myself!
PS, my name is also Jeff!
Awesome! Hope it works well for you.
We're both Jeff, but your user name offends me.
FTP 😉
@@CraftComputing I might just use an old 1070, but we'll see once the other hardware shows up. As for my username, I've had that since the old AIM instant messenger days, and am actually from Cleveland lol. The Browns weren't around when I was a little kid, and picked the Packers because green was my favorite color.
I come for the tech, but I stay for the beer review
I like the concept of Jeff doing all the hard work with those unicorn builds so we don't have to :D Keep it going sir, your videos are great :D
Thank you so much for your level of detail. Excellent video. Looking forward to the Firepro project.
a friend has been doing VDI setups for quite a few years. the "value" side of it was in buying and paying employees to support (lets say 1000, though some of his were into the 10s of thousands) modestly powerful office desktops. In a real world usage scenario its highly unusual for all of the VMs to be running at or anywhere near 100% utilization. the most recent ones he did were 64x VMs (with 1GB vram) per server with 72 cores & 512gb ram hosts or 72 cores & 768gb ram hosts for the higher ram spec VMs. its one of those things where the value is seen at scale (and at scale, the licensing costs are "flexible" enough that you are getting a decent discount).
"you with me so far"
I mean, no, but I like your style
I've been working on a couple different gaming servers with different methods. Thanks for sharing. Haven't tried this yet and now I know why I shouldn't 😅 your loss was our gain
Kind of expected that they [M60s] would be terrible, but I'm amazed that you got them working! They're really made for CUDA workloads like neural networks and other machine learning... Great storage and great software solutions you figured out! The pricing for licensing on both VMWare and NVidia make it more worth while to passthrough cheap GPUs like GTX 1050s or GTX 1660s!!
After some time as an enterprise administrator, I feel like their pricing should just be listed as "bite the pillow, I'm going in dry"
100%
I really want to know what software companies like Shadow run in their servers. Yes, Linus had an actual Shadow server and played games off of it, but that was with a local install of Windows, so it was pretty pointless (in my opinion).
Virtualized gaming has always been something i have wanted to do/try, so I have loved this series! I'm glad that you have been able to push through all that BS licensing to bring us such awesome content!
Most of the Times they run Quadro cards using GPU passthrough.
@@tubefulable My base tier shadow runs a Quadro P5000
I was curious about this. Thank you for doing this and identifying all of the licensing issues.
We use virtual workstations at work for people on laptops, temporary placements and sometimes tablets, but we’re using a custom made 4u enclosure that has 10 mini itx computers in it with a gpu each. Very cool stuff.
We use these at a school district. It allows us to deliver rendering capabilities to technology and art students remotely. Each student is handed a Chromebook for their schooling. It todays day and age this has come in handy.
Your persistence of this project reminds me of how I got 2x 3TB SAS HDDs in RAID-0 to run on my consumer board with an i7-6700k.
I love channels that go deep into us trying out industrial use hardware. It's more times than not a cool and nerdy gem. Sometimes hits a dead end, like with your license issue or my first IBM cockblocks at the hardware level, despite on paper being %100 capable -_-.
These manufacturers lock in companies and it hurts us consumer grade people.
SAS 3tb 6gb/s is $23.
SAS reads and writes at a single time while SATA3 is one or the other at a single time.
Should make a video on that stuff.
If not I'll beat ya.
200TB @6gbs/sec for $200 isn't uncommon.
Thanks for keeping it real !
Tesla K80 GPUs are $110 now.
Brought me here. That vram alone to sink with an i-GPU is potential for me for novelty lols.
I've been tryingto do the same since I saw your first video with Grid K2.
I finally ended up to replace the dell R520 with R730xd and quaddro M4000 in passthrough for 1 VM with esxi 6.5.
With horizon client I can play with 30 fps 😕 . With parsec got 60. I even conneted the output of the gpu to my monitor and used it as normal pc. But weird stuff happen when i do that. Monitor counts get confused etc...
Once again great educational video... Keep the good job
This has been a fascinating trip, at least it looks cool. Good luck with the red build.
These also work on KVM (we used them on ovirt during testing) But ultimately we settled with using intel graphics using VCA/VCA2 cards. Just needed a little 3D and no licensing was nicer. Not to mention with intel GVT/KVM you can still use the built in SPICE/VNC device for display. We also tried the AMD mxgpu stuff, good luck, you'll need it.
Great video! Sorry to hear all that work was for naught though! But at least I hope it gave you a lot of views! :)
Snap! Firepro! I have been buying a few smaller ones just for easy plug and play solutions for my work systems but i just bought a much larger Firepro, the S9050 (has 12 GB of GDDR5 on it!) for a mad scientist project. Only paid $70 bucks for it and just had to make my own cooling system for it. I have seen videos where others pair a different Firepro with it to make a super computer, some pair it with another S9050, while others paired it with a consumer card that shared the same Tahiti chips on it. Something like a HD 7870 XT (i didn't understand why, since i thought the 2gb card would lower the 12gb card). I have an older FX 8150 with 32 GB or ram and the 990x mobo does support dual cards on the shelf, i just want to make a recycled inexpensive AMD version of parts I already have and what you are trying to accomplish (few of my friends are strictly Intel and this would be funny to see them using AMD), which will allow me to play some mid/low tier games with my friends. (Super Blood hockey, Battle Toads, Scorched Earth, Super Arcade Racing, Mutant Assault) 20 year's ago, we used to LAN party and sometimes we would have our battles outside in the driveway when it was 30 degree's outside. We would have to put little space heaters under the tables and some of them would game with blankets, but it was awesome to game under a full moon. Now pushing 40, would like to do that again...
That's a nice cliffhanger RE: the AMD card. Really cool to see this come together, despite being underwhelming at best.
AMD’s SRIOV cards are cheaper indeed... but performance is awful!!
At my old job we used servers with this kind of technology for architects that went from the giant P-series Lenovo laptops to the much smaller T4xxS laptops, using the VM's to do all the work
What sort of idle power consumption were you seeing?
I did not expect the Beer knowledge at the end.
Huge respect for your choice of beermat. Nice.
You can get ESXi Enterprise Plus and vCenter Standard for 200 a year in VMUG Advantage (Personal use only). You get enough licenses for 6 Sockets.
Can't wait for the S7150x2 version of this build!
Insert "What year is it meme" =)) good job in making it work.
Nice work! I'm still in for some games!
Funny enough I went down this rabbit hole as well but in a low tier version of this. Funny enough I ran into the same problems using vmsphere and unraid no matter how hard I tried to get passed some of the limitations of both I couldn't get good fps on any games. The only way I got decent fps was using proxmox and a 1050 ti with a dummy plug to trick the VM into a higher resolution. This works pretty nicely but for what I use it for the server was drawing to much power and I was limited on that since I also have a VM running a Plex server. Still haven't given up since my plan for this extra VM is to play emulators on it like ps2 and Wii with my friends. I think the path is long but is gonna be worth it. Can't wait to see what your gonna do with those amd cards.
You should look up the VMUG (VMware users group) licensing for VMware products.
You can get a majority of the enterprise grade licenses for 250$
I'm actually shocked Jeff doesn't already have this...
Do you plan to use Vcenter again for the Firepro card, Or are you planning to do some LibvirtD+QEMU fun?
So this has been a gripping enthralling journey... congrats :-) I think they maybe it needs a trailer and playlist for itself like a suspense film.
So my question is how would the vmug licenses work if i remember correctly vcenter is included with it as a 365 day evaluation license, considering is 200 usd per year.
Also my second question is what is blast extreme on this configuration when connected to a thin client.
You said AMD ‘epic’ but I heard ‘fx’ and was like wat 😂
I posted on your last video about QEMU/KVM
could you utilize his cards in that type of setup or do specifically only work with grid?
just asking because I have a R720
with two empty GPU slots.
while I have vfio set up on my main rig using parsec the latency would be too high I think for the server for esports
If you used GPU Passthrough on a tesla card I believe you would only be able to use them for compute as the physical GPUs have no video outputs.
@@nathancreates ahhh thanks!
I might not fully understand this setup.
Is it possible to deticate one card to game graphics and the second for video encoding? Cut out the number of instances, but boost performance.
Really interesting video. I just bought a Dell R730 that when I looked up the service tag was spec'd up with the GPU kit, 512GB of Ram, 2 E5-2695 and 2 nVidia Grid K2's. Although it looks like the CPUs have been replaced with E5-2630L versions now. Just before I bought the server I found a Quadro K2000 on ebay and snapped it up but the Dell R730 won't even complete post with the K2000 installed?? . Anyway do the AMD cards have all the associated licensing costs of the nVidia Tesla and Grid. Looking at the AMD card you mentioned in the Video now although for my use I will prob give up at this stage.
Thanks for saving me from myself on this one. I was looking to HW accel security camera transcodes, VDI, media transcodes etc.
I have a friend that does some remote data science work for a major Canadian bank, and he was given access to a virtual machine instead of a physical workstation. One of the rationale being that all data and computation should be kept on the bank's servers, within a VPN for security reasons. I have a feeling that this kind of technology is what was used.
I can't wait for him to test the Firepro card!
I'm using a 1U with two RX580s. Never going to have more than 2 people playing at a time my house so it works perfect. Used it at a LAN party to play Bro Force on the TV in the house we rented using a Steam Link.
Can you please give me some details about the water cooling for the Telsa M60? Me card is running too hot and I'd like to replace the 2x 40mm fans (Noctua, silent but not enough airflow). Thanks!
Nvidia vGPU doesn't dedicate Vida Cores for vGPU instances from everything I know about it.
Only the vRAM (Video RAM) is dedicated.
The Nvidia vGPU CLI Tools allow you to configure the mode the Cuda Core sharing uses between 2 (again, from memory).
Wondering if a Mobo that has a bunch of x8 electrical/x16 physical slots, and then P2000 or similar 1slot Cards in pass thru mode would perhaps be better/easier and eliminate all the software license restrictions.
Finally it works 🙌 I am happy for you man. But try making a server with consumer hardware for 2 gamers maximum on a Budget. It will be a great video as well.
I've sticked an AMD S7150x2 in an HP DL380 Gen 9 for CAD VDI. It's basically a dual chip R9 290X and has a driver stack based on SR-IOV. Implementation was totally free aside from usual VMware and Win Server licenses (probably because AMD is lagging in the market) These cards can be found pretty cheap on ebay.
Wow I've left the comment before noticing you have one! Looking forward to see it's gaming performance, didn't really have an opportunity to test beyond Specviewperf
Last comment, look into Teradici Cloud Access which I'm using for high end media clients and unlike previous Teradici solutions, doesn't require a hardware offload card.
I thought via GeForce Now, as long as the Host and the intended platform is connected on the same network that you can still game?
How do you avoid knocking your beer over? If it was Mei would have done so about 30 seconds in. Great vid btw!
Wow well explained the price number hurt my head the green goblin looks aggressive
Could you modify it so it was budget friendly? Say use an X99 or X79 xeon and two dedicated GPU's (plus a low end graphic adapter for GUI access to the machine) use PCI passthrough and throw on a 1080p dummy plug so you can always have 1080p output for each VM?
Yes, everyone already did that, I have it running. This is not he point though, VGPU/GRID is what's in the scope of interest of this video
@@Gastell0 I get my question was out of the scope of this video, I just wanted to verify someone had done it before I, you know.. Did it. I don't wanna blow $800 on hardware just for me to find out it ain't gonna work. But thank you for confirming for me! :)
@@justanotherhuman-d6l Yea, you can go on the UnRaid forums to see which motherboards and GPUs are easier to pass-through. When any of my GPUs are plugged in the main X16 slot, they require the use of OVMF BIOS on Q35 AND a custom ROM to work properly. Otherwise, when they are inserted in the secondary X16 slot they work without any custom rom. But these settings can differ per motherboard / GPU / slot combination, so you might need to try out different permutations before you find out the one that works for your specific setup.
Is the bottle always going to be the LAN, HDMI 2.0 transfers data at 40G.
Hi Jeff, nice video, very useful, but I would like to ask you, how do two Nvidia Tesla K80 cards go together. I know that they cannot work in SLI and that the typical memory is divided by a maximum of 12 GB... if I install two identical cards on a PC, with an AMD EPYC processor, to be able to create renders, what happens? Is the work divided between the two video cards and the processor? Do I need to change the software?
Thanks for the attention.
Why not just forgo the nvidia cards if they are being an issue and go something like the radeon pro duo polaris or just the pro duo?
Does the the n'viddya license server let you run multiple instances? I'm wondering (and no, I'm not proposing this is worth the effort) if you can put them behind a TCP load balancer and make things slightly less terrible.
I thought about pairing a Tesla K80 with my GTX 760 to do some data crunching for science (I figured they were both Kepler so why not). But it seems like the K80 wouldn't be a simple plug and play kind of situation, especially when needing something better than passive cooling.
Would it be possible to pass through a single gpu to a vm, i.e. one of the four, or would you be limited to passing on a per card basis? Then using registry edits on a per vm basis to get them to act as the equivalent of a high performance laptop card. If this is possible, I'm not sure whether you'd need some gpu to act as the igpu or whether it'd be possible to get by with just parsec. Potentially this could mean one could avoid the pricy drivers at the cost of being limited to as many vms as individual gpus. Could be interesting to try out.
Hi dear. How you does to conected and run two tesla at the same time. What is Your mother board?
i put two tesla on an eth79-5x 5 pci-e 8x gen 2 motherboard. when i use only one tesla m40 , the motherboard works, but when i put the second one the cpu does not start. Could you let me know what could be happening?
I
Enabled “above 4g decoding” in bios?
I find your videos really great and well explained. I am currently doing a project similar to yours. I have two NVIDIA Tesla M60s installed in an HP ProLiant DL380 Gen9 server and Windows 10 pure without virtualization. Then I installed several drivers from the NVIDIA driver page. Unfortunately, none of the graphics cards show up in the performance tab of Task Manger, but they all show up via GPU-Z. When I run a benchmark, it runs on the Intel GPU and not on the Nvidia GPUs. I also can't access it via Parsec because the host encoder can't be initialized. Can you maybe give me some tips on what I did wrong or forgot? Thanks already.
Did you ever try qemu-kvm with vGPU ? This might at least get rid of some licensing woes. Though based on the nvidia documentation it might not be officially supported besides with OpenStack?
Does the tesla on a regular install run much better than a og 980? I would guess the extra ram would be quite helpful no?
Hope you will make more detailed guide for s7150x2 ... cant make mine working =(
What do you like Lager wise? I am not big on hops so IPA's aren't really my thing
Why do you have to do virtual machine, there's got to be some work around to get it working through regular windows
Is is possible to pass the GPU power through the integrated grfx?. Much like a mid range gaming laptops work with discrete grfx.
Also keen to see the AMD card stuff :)
You’d better use VMware’s Blast Extreme protocol instead of Parsec. Just install VMware Horizon Agent in your VM and VMware Horizon Client on your Mac, PC, etc. Connect directly to the VM from your desktop client without having to deploy the full blown Horizon Suite. BTW, using VFIO under Linux on consumer cards will provide a much cheaper scenario... and way better performance too!
He dunked on that idea (GPU passthrough). Many have tried this concept before, to varying levels of success in the end. And they keep trying, it's good content.
Hi love the series....with all that licenced software what else would you need to comercialy use this server and start connecting customers
I have a Dell Precision Rack 7910 with a 2070 Super and a Teradici 2 Remote workstation Card in the basement. I use it for gaming with a Dell 7030 Zero client. Works great
I'm confused on how your water-cooling is setup looks like you have your inlet and outlet on each car t'd to one line......
Me 2. I am still looking for a water cooling block. Have you found something?
Van you play vr games in the virtual machines?
Virtual reality in a virtual machine :p
Hey Jeff, love the channel and awesome content and guides...and the beer.
Have you looked at a possible CPU bottleneck with the lower Epyc speeds? I just built a dual E5-2690 with Proxmox pcie passthrough of a RX5700XT and I’m getting bottlenecked on certain games. Still debugging it but seeing the Epyc speeds made me think of my situation. I’m going to try to pin CPU cores next to the VM to see if I get a boost.
How many VURTURE MURCHURNS are you running?
So anyone that knows this technology knows the entire design and use cases are not meant for a single user/gamer. What you did is what we do for homelabbing for testing and learning VMware technologies. Those cards aren’t meant for “gaming” they are meant for an enterprise environment with multiple users needing gpu like someone described below. Cool project and I like the water cooling approach cause goddamn do Tesla cards get 🥵. I’ve been playing with a Tesla t4 in my homelab and you can cook eggs on it
do shadow cloud gaming and so on also work similar to that? but with a better setup of course.
This painfully convoluted journey has almost wrapped up and my opinion still stands that it's cheaper and less challenging to build individual computers than to use a Big Computer Many Terminal approach
You should do a video on the firepro s9300 x2
How about passthrough 6 of Quadro P1000 or P2200 ?
Firepro! We choose you!
did you passthough SSD drive thought to VM. Would running on a VHD decrease prefomace. My ESXi VM Gaming build has a nvme drive passthough.
I ran a TrueNAS server in a VM, passed through the controller, and connected to the VM via iSCSI. Got ~1200MB/s read and 550MB/s write.
Is there a Virtual GPU solution from AMD also? I do have a Amd R9 SSG Card here.
AMD’s SRIOV cards are cheaper indeed... but performance is awful!!
Nicolas Ballet I know sadly there wasn’t a lot they did better on in the last years. I also do wonder what’s there strategy for CDNA e.g if they allow the “pass through” on there RDNA cards too as of these game streaming services. The other thing is that we will see especially with the consoles better optimized games for pc on the AMD side, hopefully and maybe also some cards that don’t suck?
I wonder how much craft beer you had to drink to get this headache to run...
Always cool to build stuff
given the bad performance, ¿what are the real use cases for this?, GPGPU crunching only i take it(but in those cases you wouldn't split the GPU into vGPUs to several VMs, but running them on bare metal, right?).
Let's say you'd want to do graphic work, are these drivers certified for CAD/CAM like Quadro drivers?, do they accelerate professional CAD workload?
These are fully certified for CAD, just like Quadro drivers. In fact, that is the most common use case. But these were definitely limited by the GPUs, as I was only running one VM per GPU core, so each had the full 2048 CUDA available to it.
@@CraftComputing still quite expensive, but i guess it would allow remote work to a powerful workstation with no much issues.
then again without remote work a local workstation would probably end up cheaper i think
i cant even get the drivers to install for these things
help? manual nvidia driver install no go. geforce experience no go.. maybe linux?
"virture machine"🤣 sorry man I love the videos but it makes me laugh
I was going to go with "shoulda benched virtua fighter on your virtua workstation".
Literally had to stop watching because of this 😂
Is the future. Gaming on the cloud have a similar concept loading your game on a remote server
Hey #CraftComputing, what is SRIOV?
Those prices seem high. I'm covered by a VMware enterprise licensing agreement so I think I get a price break. How much I'm not sure. Its all negotiated by our corporate office. If only I had a Tesla card to test with 🤔
Can you get the feature in VMware remote office as that is more economical. Then again your paying per VM instead of per server.
And I'm glad I waited for the end... I was going to ask if there was an AMD solution that was possibly more budget friendly. Now I know a video on that might be incoming :)
Could some of the performance issues be cpu limits as they are only 2.2Ghz?
Not at all for the numbers I got. Single core speeds on the 7601 are similar to the Xeon 2670, which had no problem driving GTX 1070 and higher at well over 120FPS in these games.
I had these setup these to run a Public Safety application remotely at 4K+ with GIS maps for locations. It worked well, but not running games.
This is made for thin client environments. Its so the client session isn't hitting the CPU super hard trying to do training videos or UA-cam. So if your running citrix the user experience isn't a choppy mess trying to do everyday tasks.