Passing a GPU through to a Proxmox container
Вставка
- Опубліковано 12 вер 2024
- We will pass a GPU through to a proxmox container and utilize it for video transcoding.
In the shell of the node
Check to see what kernel version you are running
uname -r
Install headers
apt-cache search pve-header
apt install pve-headers-*.*.*-*-pve
Blacklist Nouveau
nano /etc/modprobe.d/blacklist.conf
blacklist nouveau
update-initramfs -u
Reboot
Install Dependencies
apt install build-essential
Download Drivers
wget (YOURDRIVERS)
Nvidia driver site: www.nvidia.com...
Make driver file executable
chmod +x (YOURDRIVERFILE)
Install the drivers
./(YOURDRIVERFILE)
Make sure drivers load when restarted
nano /etc/modules-load.d/modules.conf
Add these lines
Nvidia modules
nvidia
nvidia-modeset
nvidia_uvm
Update initramfs
update-initramfs -u
Create udev rules
nano /etc/udev/rules.d/70-nvidia.rules
Add lines
KERNEL=="nvidia", RUN+="/bin/bash -c '/usr/bin/nvidia-smi -L && /bin/chmod 666 /dev/nvidia*'"
KERNEL=="nvidia_modeset", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -m && /bin/chmod 666 /dev/nvidia-modeset*'"
KERNEL=="nvidia_uvm", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -u && /bin/chmod 666 /dev/nvidia-uvm*'"
Reboot
Check that the drivers are running
nvidia-smi
Edit the conf container you want to passthrough too
nano /etc/pve/lxc/(YOURCONTAINERID).conf
Add these lines but make sure the path (numbers) are correct
ls -l /dev/nv*
Allow cgroup access
lxc.cgroup2.devices.allow = c 195:0 rw
lxc.cgroup2.devices.allow = c 195:255 rw
lxc.cgroup2.devices.allow = c 195:254 rw
lxc.cgroup2.devices.allow = c 509:0 rw
lxc.cgroup2.devices.allow = c 509:1 rw
lxc.cgroup2.devices.allow = c 10:144 rw
Pass through device files
lxc.mount.entry = /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry = /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry = /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry = /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry = /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry = /dev/nvram dev/nvram none bind,optional,create=file
Start the container
apt update && apt upgrade -y
Download Drivers in the container this time
wget (YOURDRIVERS)
Nvidia driver site: www.nvidia.com...
Make driver file executable
chmod +x (YOURDRIVERFILE)
Run the drivers file with the extension
./(YOURDRIVERFILE) --no-kernel-module
Reboot
nvidia-smi
Install Plex
wget downloads.plex...
dpkg -i plexmediaserver*.deb
systemctl enable plexmediaserver.service
systemctl start plexmediaserver.service
Access Plex
(YOURIPADDRESS):32400/web
Number of transcodes for your card: developer.nvid...
Quadro K620 Specs: www.techpoweru....
Ebay seller I used: www.ebay.com/i...
Level1Tech article and youtube channel:
Article: theorangeone.n...
UA-cam Channel: / level1techs
#loresdiy
My Channel: / loresdiy
I really appreciate this guy, but Pro Tip for all my fellow Linux Sysadmins: Run this video at 1.75 speed ;-)
I've been trying for the last 3 days to get hardware transcoding working in jellyfin and this tutorial got it up and running in 30 minutes. Thanks so much for this man.
Glad I could help
Just the video I needed for jellyfin as container! Followed all the steps and just swapped to jelly at the end. Was getting tired of not getting the pass through to work but I think it’s finally complete! Nvidia-smi doesn’t show processes but it definitely brought down the cpu usage from 90% to 4%, and can see its temp and usage go up. Thanks so much!
its the year 2024 and this tutorial is very much relevant
probably the tutorial with less friction i followed in a very long time
nice job
I'm a vmware boomer and switched to proxmox. You have solved 90% of my issues by explaining in the first few minutes to switch to the non liscenced repo. Fixed all of my issues related to pci passthrough right there. Fantastic video haha
TELEPHONE NUMBERS!! Not one guide except yours, explains the reference between the numbers and the lineup. Thanks so much for this!
Been working on this all day, so frustrating. You solved my issues! Thanks a lot. And by the way, love the mid-west accent! Lived in Nebraska for couple years! Let it out! Much appreciation from Amsterdam, Netherlands
You sir are a GODSEND! I have tried so many video tutorials, including your previous one's on this topic, and this is finally the only one that has work for my amd cpu and RTX2060 combo. Excellent tutorial amigo!!!
Great to hear!
Thanks for the awesome vid, got me past a point where i was stuck.
for anyone wondering initramfs is pronounced "init ram fs" and stands for initial ram file system. on proxmox boot up, you will see a step that states Loading initial ramdisk. it is this ramdisk that we are updating with update-initramfs command
Thanks for sharing you knowledge.
Cheers
This is by far the best gpu passthrough tutorial! Nice work!!!
This was the best video I found on this subject. Follow these instructions exactly and pay attention, you will get your passthrough sorted with this guide.
Thank you for this! I had this working at one point but it was fiddly and I had to re-do it. The second go around it was kicking my butt until this video/write up took me through it.
Out of all the tutorials that I have tried yours is the only one that actually worked, well done and thank you very much! Well it lasts for a little while at least, HW conks out after a while.
thanks mate, .. you helped me a lot, the "cgroup access and Pass through device files" explaination was spot on... got it up and running...
Thank you so much. I've spent 2 days following NVIDIA's official instructions and stuff scattered all over the interwebs with no success, however your instructions worked first go! I had no idea how complicated this could be, thanks for breaking it down for us!
Glad I could help!
Cheers
I don't think I've ever commented on a UA-cam video before; I have to on this one. Thank you so much for this tutorial as it was the only one that got my hardware transcoding to work on my Proxmox server. Thank you!!!!
Great to hear. Glad it helped tou out.
Thank you so much! this really helped me, I've been struggling to get it working for a while now!
This is still relevant in 2024.
Just got my Asus GTX 1660 Super in my homelab server transcoding for my Proxmox Plex LXC. This worked great!
I did run across a couple of issues, one of them being the Nouveau driver disable not working, but installing the NV driver prompts to disable that, add the information to the correct files, and even run the update initramfs. It does recommend you abort the driver install, reboot, then install again.
So, then I needed to re-run the apt-install pve header command after reboot.
Beyond that, the rest of it went great with Proxmox 8.1, NV 550.78 drivers, GTX 1660 Super 6GB in a PCI-e x2 slot. I am currently transcoding two 4K streams at once. One to transcoded to 720P one transcoded to 1080P. Both show (hw) in the monitoring. I've been a Plex Pass Lifetime owner since 2015, and I was able to select either Any or my TU116(GeForce GTX 1660 SUPER), so I chose the latter. I haven't messed with the transcode settings yet.
I purchased the 1660 SUPER because of its price point and ability to handle several transcodes at once with its 6GB RAM. From my research, and so far tests, being in a slower/smaller PCI-e slot (also older gen, PCI-e 2 on an older Supermicro board), but transcoding is working great.
My nvidia-smi results aren't showing the transcodes in the LXC, but the temp, load, RAM wattage usage, etc. results are all good.
Sun Apr 28 19:16:35 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.78 Driver Version: 550.78 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce GTX 1660 ... Off | 00000000:04:00.0 Off | N/A |
| 41% 60C P2 42W / 125W | 716MiB / 6144MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+
Thanks for this video! This was one of the last steps after migrating to Proxmox from Hyper-V and adding hardware transcoding to the mix as well. Project completed! :)
Thank you for this video.
Worked like a charm (well...mostly).
Had a little issue where when I blacklisted the kernel modules on the Proxmox host itself, I also needed to blacklist the vfio-pci kernel module as well before I was able to install the Nvidia driver for my RTX A2000.
(You can find out what kernel modules are using the video card with the command "lspci -k".)
Beside that, the rest of the instructions worked.
Wow,,, you are hitting the mark,,,, must have viewed 10 videos on this,,,, and your step by step was smooth,,,, also having all the commands posted above in your description makes the tutorial so complete,,,, speeds things up on my side so I can follow along fast.... i appreciate BIOS mode,,, never had much use for UEFI yet....
Glad they have been helpful. Cheers
Fantastic Video. Thanks to you, I now have my emby server running in a container using a GTX 1030 with NFS mounts from my OMV NAS.
Awesome, glad it helped.
Excellent video sir. I have been at this for a couple of weekends and you have saved my life. Keep up the great work!!
Just followed this on Proxmox 8.1, worked like a charm! Massive thanks for the tutorial.
"It's an older card sir but it checks out"
You won the internet with this.
Made my day!
Great video, thank you. Trying to make it work with both nvidia & intel passthrough on my end and this is proving real useful.
This is the best video on GPU passthough to Proxmox LXC, not one of the best, THE best.
Awesome video. So easy to follow! By the way, you pronounce 'Nouveau' like you're starting to say 'New Volkswagen' but stop after 'New Vo---'.
VROOM VROOM
Awesome video... exactly what I was looking for - I wish I found this hours ago!
Fantastic video, very clear explanations, can exactly see what your doing, and got it working with no hiccups, Thanks!
Good to hear.
Cheers
Thanks for the video, planning out a proxmox Plex install. Let's Go Blues :)
Thank you!!! you are amazing and they way you explain everything is awesome
Awsome video. I tried to follow on the NVIDA website...impossible. Great Job.
Didn't get me all the way there but was very useful, thanks!
These instructions only pass the compute capabilities, not the entire GPU. A real test of passing a GPU to a container is to run a 3d accelerated desktop environment (example, install gnome and run glxgears or davinci resolve), which requires also the graphics capabilities of a GPU not just the compute capabilities.
Thank you, this video helped me a lot :)
508509😄. thanks man for explaining and patience !
NP
Cheers
My second time following through this, thank you again!!
Your Welcome.
I switched from using a Quadro 6000 to Quadro K2200 simply cos I couldn't play 4k videos on the 6000. Somehow, plex still found ways of consuming the CPU, meanwhile Jellyfin on a different container was using the 6000 just fine. I thought having more vram 6g was going to give me so much power to achieve a lot.
I'm yet to test playing 4k videos but hopefully something good comes of it. Used this tutorial and switching a gpu wasn't so much of a pain. If anything, I'll update.
Thanks man you made my day!!!
Great video! Question, does this setup also works when passing GPU to VMs?
You are a life saver! Thank you very much for this! 👍
Happy to help!
Great video, still works in 2023
Good to hear!
Cheers
8:57 skipping ESP sync .... so you have proxmox installed in bios mode not UEFI. It is useful for videos like passthrough this info to be known. Other paths and files to edit when using UEFI than BIOS. Everyone forgets to mention it.
What changed and made the procedure with more steps than before. In my notes passing through the gpu to the LXC doesn t require so much steps like installing drivers for the OS and then for the LXC as well without the kernel. What is that out of a sudden?
I installed Proxmox with the kde desktop using a nvidia quadro k4000 with one monitor (I use as video editing workstation), and a few vm's with one in passthrough to an AMD Radeon HD 6350 with 2 monitors attached... it works very well for general office and light graphic stuff.
KDE on top of Proxmox?? You sir are a genius!
You just gave me an idea. Instead of installing kde-desktop on top of Proxmox, I'm gonna install Spiral Linux KDE (essentialy Debian 11), then add the Proxmox Repositories and then install Proxmox on it.
Well, thank's for this video. I have one question, - how monitoring gpu loading? When using
proprietary nvidia gpu driver, i not using nvtop for see gpu load or no.
Thank You so much for your Videos
You solved 90% of my setup
Happy to help
how would you do this on an unprivileged container, getting permission denied after chmod +x within the container
Can confirm these instructions work for pve 8.0.3 with a nvidia p400. The only other thing was I installed build-essentials (may of missed this instruction) as it comes up with a compile error when installing the nvidia drivers.
Also I'm still testing but notice that when I run nvidia-smi only shows jelly-fin process working on the proxmox host, not the vm. Could be an issue...
Excellent tutorial, very clear and informative.
Glad you liked it
Great Video
Your videos are gold
Awesome thanks
I got this working and passed through a old GTX 660 card. When I enable hw transcode and change to a lower resolution, I do get the status that it is indeed using hw and cpu usage is very low. I'm getting lots of buffering after about a minute or so of playback in the stream to the point where it is unwatchable. Did you encounter this as well? Any ideas on what to check? Thanks for the video and love the channel. Assume you are in the St. Louis area. I am as well.
Works great! Thanks!
I used to use the passbe "GPU (Nvidia) passthrough on Proxmox LXC container" guide but i think yours keeps up to date.
The only thing i want to know if you get any errors after update proxmox becouse "driver version mismatch".
And if i want to update my nvidia driver must i download the new file from their website and remake all the job?
Thanks!
This is fantastic, thanks. Have you ever had a situation where the "phone numbers" suddenly change? This happened to me recently and the hw transcoding stopped working. I was able to fix it by re-doing a couple of steps from your video but I wonder if you know how or why that might happen so I can avoid it in the future. Thanks again.
So far in my adventure to get this working, I've read on a couple different sources online that driver updates can cause them to change.
Yeah they changed for me recently after rebooting proxmox. I got it working again by correcting the "phone numbers".
espectacular, es el unico tutorial que de verdad me a funcionado. muchas gracias!
Contenta de que haya ayudado
you literally saved my ass thank you!!
NP Cheers
Very helpful! Thanks!
Cheers
hmmm not working....getting "Failed to initialize NVML: Unknown Error" in the container
Does this method work for sharing the GPU with mutliple containers?
Thanks for the video, I do have a challenge though. On my previous server (on dell 7559) with limited resources, I was able to setup gtx 960M which run smoothly with plex. This new setup on a more powerful server works too. Somehow, with 8 cores and over 16gb assigned to plex lxc alone, it is not able to do "convert automatically" 4k videos but rather throws Error code: s1001 (Network). Now I wonder if the problem is with the smb mount or if it is a transcoding issue, which is what I actually suspect.
Been trying to play a 4k version of Zack Synerder's justice league and haven't been able to. I noticed plex uses the CPU heavily than the GPU.
I tried to do the same but I have two Nvidia cards and the nvidia-smi only shows one. Do you know why?
will this mean that the gpu can only be used by the container? Im hoping to create a Plex server with hardware encoding/decoding. I have a 5900x which does not have integrated graphics and an Intel Arc A380. Will passing through the gpu work for me even though i only have 1 gpu?
thank you for this!
You're so welcome!
Any chance you can do an update for Proxmox 8.0.3?
nvidia-smi works fine on nodehead
hm.. when you ran the LS command your number for uvm was 508 but when you looked in the file your number was 509.
For anyone.. these numbers change but you can just add more than one
What about passing a ntfs partitioned hard drive from old windows plex server that is now physically installed in my proxmox system and passing it thru to a container such as Linux turnkey file server?
It's showing up currently as sdc
great video as always, could i ask i have passed through a gpu to ubuntu vm on proxmox problem is if i remote connect to that vm using xrdp the gpu is not there but if i connect a hdmi cable to the gpu ive passed through it works just fine but that means i have to be sitting beside the server ant ideas thanks pal, john
Did you set the gpu as the primary for the vm?
Excelent video bro ! . I have installed to lxc sharing one GPU, but one of these containers take all the GPU memory and there isnt enought memory for the other lxc. I have a GTX 1050 ti with 4GB. Is there any solution ? . Thaks in advance.
Thankyou for a nice video. I jave a quadro 2000 that i want to passthru.
@Lo-Res DIY I'm having issues with Proxmox 7.2-11. I'm running a Plex server under Debian 11 (bullseye) and I cannot get the host to run nividia-smi. I get an error that the driver is not running. I have the blacklist in place but it looks like the nouveau driver is still loading. Any assistance would be awesome!\
Been having the same issue. I believe it is down to the newer kernel not being supported by cuda. Plan to install pve 7.0/7.1 and test again and prevent kernel updates going forward
Is it possible to do this with an AMD Ryzen 9 4900H and Renoir gfx ?
no matter what i try installer wont install for me.. did everything to blacklist and remove.. purge etc the nvidia drivers in proxmox
ARG lol got it installed now it freezes when i run nvidia-smi
I got this error Failed to initialize NVML: Unknown Error. Does anyone have fix solution.
Thanks for the guide. Weirdly nvidia-smi does not show a process and my card appears but in plex it shows transcode (hw) file is 4k going to 720p and cpu is barely getting stretched. so i guess its working any ideas on what else to check?
actually this is checking on the container. if i check from the node it shows it working.
Sounds like you got it. Cheers
Are you able to pass the gpu to a docker container running inside the lxc? Using Nvidia-docker2 runtime
🤷♂️
Massive help.... I'm still working on my conf file. My ls -l pulled up 14 different lines/files for my setup (1070ti). I'll grab an IPA and put on my readers to knock that out tonight.
Cheers
don't work in proxmox 8.1.x
can't install nvidia driver in proxmox.
Works for me in proxmox 8.1.4
when are gonna do it without a cap on? never?
Never ever
Hello Sir Plz Make Video For Pci Passthrough For Elgato game Capture Card In Proxmox VM Windows Sir in You Tube i dont find any video plz sir help me Thank you. Umar From Greece
Could you do steps by steps promox/ubutu server plex gpu passthrough?
? Ｐｒⓞм???
sorry for the question but I don't understand English very well, do you necessarily need the vT-D or iommu? unfortunately my motherboard does not support this function and I hope that if I do not use a vm but a container it is not necessary to pass the gpu to the container but if I take it from the host system.
Thanks 😘 🦿
Thx for that video, one lil question though, in this section something weird going on with syntax, take a look at "nvidia_uvm" and "nvidia-modeset" it's kinda confusing you keep changing back and forth "-" with "_":
Add these lines
# Nvidia modules
nvidia
nvidia-modeset
nvidia_uvm
Update initramfs
update-initramfs -u
Create udev rules
nano /etc/udev/rules.d/70-nvidia.rules
Add lines
KERNEL=="nvidia", RUN+="/bin/bash -c '/usr/bin/nvidia-smi -L && /bin/chmod 666 /dev/nvidia*'"
KERNEL=="nvidia_modeset", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -m && /bin/chmod 666 /dev/nvidia-modeset*'"
KERNEL=="nvidia_uvm", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -u && /bin/chmod 666 /dev/nvidia-uvm*'"
Great video - works like a charm. However, I noticed recently, that when I reboot Proxmox, these two numbers which we get from `ls -l /dev/nv*` change: /dev/nvidia-uvm and /dev/nvidia-uvm-tools. Sometimes, they are 507, sometimes 509, 511... and then I have to update the lxc or otherwise GPU passthrough stops working. Any help would be appreciated.
Help!! when following your guide I get a response saying that the install option of not loading the kernel module is unknown!?
root@docker-frigate:~# ./cuda_12.3.2_545.23.08_linux.run.3 --no-kernel-module --tmpdir=/DUMP
Unknown option: --no-kernel-module
@EdgeRowing. Did you get this sorted, if so what was the fix?