For anyone else struggling to determine which GPU is which, run `ls -l /dev/dri/by-path`, and cross reference the addresses in that output with the output of `lspci`, which will also list the full GPU name.
If you're deploying Jellyfin in a Portainer LXC on Proxmox then you will need to create a Custom Template the in Portainer WebUI and add the yaml contents to the Docker-Compose section. This will create the container and deploy in the Stack. From here, changes can be made to the docker-compose yaml and the container can be updated as necessary. I am still new to Linux, Docker, and Portainer so I am sure there is an easier way to do this but after having a difficult time of finding the yaml for Jellyfin in the LXC this is what worked for me. All other steps from the video worked like a charm!
Spent the last couple of weeks trying to achieve this myself and couldn't - had to stick with a privileged container. This worked perfectly first time, thank you Jim!
I just used this guide to get hardware encoding working in an unprivileged Immich lxc container, through docker compose. ( After a lot of work) Thank you so much for your great and comprehensive guides.
My weekend project right here. I run unraid in a VM with some docker containers running in it. I want to move all containers outside the unraid VM. Now I can test this and also sharing the iGPU!!! Not straight put to a single VM. NICE!
Great video. I did a similar thing ages ago to passthrough a couple of printers to an lxc unprivileged cups printer server! Was a headache to figure everything out at the time hehehe
thank you for this tutorial. It works. just want to let you know, it works also with handbrake docker container, just in the compose file add GROUP_ID=107 in the environment section. intel qsv is detected!
I struggled with an AMD igpu pass through with Jellyfin and you were very kind to help , in my case it did not work with a regular VM , but with this it was a breeze to setup Jellyfin with HW transcoding , the only hiccup was the lxc image of Debian 12 did not work , but Ubuntu did , latest proxmox fully updated , thanks again your walkthroughs are really helpful thanks !
@@codexclusiveNLfollow the steps that Jim laid out, some films depending on the setup will work depending on the browser or device also, for me even the groups that Jim has in the video are the same , y setup is a beeliknk mini pc with an AMD 5650
Congrats, good stuff. You may want to check out how to run docker images as LXC containers, since they are OCI compliant. It would remove an abstraction layer, but instead of compose it would be set up with ansible.
@@Jims-Garage and @gamermerijn I was wondering about this as well. What's the added benefit of running Jellyfin in Docker when you can just install it in the LXC directly?
@@RudyBleeker mainly security and density. VMs use a separate kernel to the host, especially good for internet facing containers. Furthermore, 1 backup of the VM and all of my containers are covered (albeit you can install docker in an LXC).
@@Jims-Garage I know all about the difference between VMs and LXC. But in the video you installed Jellyfin in Docker, that Docker runs in LXC, on a bare metal Proxmox host, correct? There was no talk about VM's, or did I miss something? So I'm curious why you chose to install the Docker runtime in LXC and run Jellyfin through that, instead of installing the Jellyfin packages in the LXC directly using Jellyfin's official repos. Adding Docker into the mix just introduces another layer of (very minimal) overhead and complexity if you'd ask me.
@@Jims-Garage You've already figured out the hard part. 13:34 in practice by the way it doesn't matter. As long as the host is newer or the same and you load any kernel modules you might need. Linux mostly adds new functionality, as Linus always says: "don't break user space". I was able to run Debian 2/Hamm LXC container on a modern Linux kernel aka Debian 12. Not like I've never done this before. I was running Linux containers before LXC existed, before I ever touched VMs. On Debian Woody with Linux-VServer.
Hardware passthrough to LXC is definitely something I want to explore. I have a few services running in an Alpine QEMU and the footprint is small but I would prefer to have one LXC per service
2:40 actually it's for some intel gpus possible to split between vms. but didn't do any benchmark on it and had no use, so i went for priviledged lxc at the time i was setting up mine. but now i'm considering redoing it unpriviledged, thanks for the video!
@@Jims-Garage right, there are lots of tiny differences on intel gpus. had it running with an 7700k about a year ago, i think this still would work today if the hw supports it (?) also played around with a DVA xpenology vm, unfortunately the 7700 igpu is too new for that
@@Jims-Garage as long as you have older GPU it works but it is quite limited. On mini PC with i5-10500T I was able to split iGpu into 2 GVT devices. Interesting part is that even if you assign vGPU to VMs you can still use real iGPU in LXCs. Of course the performance will suffer this way but in case of load like transcoding it is perfectly fine. I suggest you give it a try.
I just discovered your series and it is amazing. I am Trying to do something similar on my homelab since a year ago but still failed. I already had some id maps in place for my mounts (more in my next comment on that video) but you essentially solved for me what I was struggeling and nearly given up. Now Jellyfin is HW rranscoding on my NUC Lab host and I am so happy with it :D One more thing that I am currently struggeling with - and you might have an idea / solution / future video: Docker swarm seems not to work inside an lxc container. Containers get deployed but are not accessible via the ingress network. Anyways thanks again I am looking forward to the new videos while watching the back catalog.
@@Jims-Garage Thank you - I got it working on a debian 12 lxc container. Some of the IDs needed to be different but now it is merged with my lxc mounts and everything is working. If i now only could get docker swarm to work. (but this a known problem in LXC - works fine in VM).
I tested this out and Jellyfin worked great in a Proxmox LXC container also with Intel A380 passthrough. Can you please make a guide on how to get it running on Plex? I could not get Plex working with Hardware Acceleration for the life of me.
Impressive, I wonder if its as simple with an AMD igpu, with an xcp-ng hypervisor, probably not. But it is amazing to share an igpu like this, multiple graphic cards is rediculous. Seems like this solves gpu sharing in general 🤔
It should work on Proxmox with an iGPU in almost exactly the same way, I've no experience with xcp-ng though... SR-IOV is also another way to do it but consumer devices don't typically support it.
What you say 6 minutes into the video about the /etc/subgid file is wrong. These entries are not mappings but ranges of gid's. It's a start gid and a count. I'm still trying to get my head dialled in on the lxc.idmap entries in the .conf file. Getting closer. Thanks for the video.
You do have an error in your github notes. After carefully following the directions and c/p from your notes I thought it odd when no /etc/subguid could be found. Still I proceeded but the container wouldn't start. After looking around a bit I noticed that /etc/subguid should have been /etc/subgid. After fixing the issue the container started just fine. Regardless, great video and you gained a new sub. Thanks..
@@Szala89r I haven't as I don't have a card to test it with unfortunately. My understanding is that you wouldn't have graphical output though, just GPU accelerated workloads
Ty for sharing your knowledge Two questions if you may know the answer? 1. Can Proxmox install Nvidia linux drivers over Nouveau and still share the video card? 2. If one adds a newer headless GPU like the Nvidia L4, can you use this as a secondary or even primary video card in a VM or CT?
only issue with this is mounting NFS shares. I have yet to find a way to mount an NFS share into a unprivileged LXC then recreate the container onto another node that has a GPU.
Hello Jim, thanks for your instructions . I really appreciate you sharing your valuvable knowledge, I did follow your instructions, but the sharing does not work at the end with a permission error message, and whenever I do upadte the configuration file for the LXC with the line (inside my /etc/pve/lxc/1xx.conf) lxc.mount.entry%3A /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file , any help please
At 16:03 is the group for renderD129 supposed to be _ssh? It's the same on my system and in Emby I'm getting a permission denied error, so I'm wondering if those 2 are related
Not working for me the Proxmox LXC - Docker is not taking the IGPU no activtiy on the engines while play diffrent content :( i do not know what i am doning wrong.
Hey tried to follow exactly, but I get stuck after editing the config file. Afterwards when I spin back up the container, I get nothing on console. So as you mention it might be a permission issue, but I think I did everything!
How does it work with dedicated GPUs? Do I need to install the driver on the Proxmox Host or in the LXC? Do I need to specify the card in the docker compose or is the ID enough? Do I need the Container Toolkit for Docker? I really like your content, one of the best channels right now about selfhosting, but haven't found a solution to this.
The video is using a dedicated intel arc a380 GPU. For Nvidia you should be able to follow the same process. I believe most modern OS will have drivers but you might need to add them.
amazing. use for my iGPU. are there any other devices apart of the GPU in addition to video and render? can i not pass all the functions to the LXC or virtual machine? On my system it says the iGPU is the same IOMMU group as the USB controllers and such. So i can't pass it through the the VM, would it be possible the share the iPU among VMs?
Is there a way to do the opposite? As in consolidate multiple GPUs, RAM etc. into one server? I have 2 laptops and an external GPU I want to connect together to combine their compute to then be able to redistribute it out to multiple devices similar to this video. Is it possible?
Great guide. I just got this working for two LXC and Jellyfin. I am trying to use Plex in a Docker container but can't get the hardware transcoding to work. Can anyone help?
@@Jims-Garage great video! I got it working with Jellyfin just like in your video, but under Plex(using the link you provided) I get "No VA display found for device /dev/dri/renderD128" in the Plex transcoder settings it recognizes the iGPU, "lshw" in the container also sees the iGPU. any ideas you can share would be a big help. thanks!
Hey Jim, thanks for your thorough tutorials. Most of my home lab setup is done with the help of your videos. Following this specific tutorial a question arises: since the Jellyfin CT is using the host's hardware, should I enable the GPU passthrough prior to sharing my iGPU to the Jellyfin CT, or doing the groups mapping trick is enough? By 'sharing' the iGPU does it mean I can still use it for the proxmox host (if I have to connect a display straight to my server and access the proxmox CLI)?
@@Jims-Garage mapped the uid and gid values as you did; 1. I am able to see my renderD128 device in the LXC: crw-rw---- 1 nobody _ssh 226, 128 Jul 4 07:53 renderD128; 2. lspc icommand in the LXC shows: 00:02.0 VGA compatible controller: Intel Corporation CometLake-S GT2 [UHD Graphics 630] (rev 03) 3. Installed intel gpu tools and ran intel_gpu_top - Failed to initialize PMU! (Permission denied) 4. Installed Jellyfin and ffmpeg6, then proceeded to enabling transcoding, following the official Jellyfin guide. when trying to check the supported QSV / VA-API codecs gives me the following output: root@Jellyfin:~# /usr/lib/jellyfin-ffmpeg/vainfo --display drm --device /dev/dri/renderD128 Trying display: drm Failed to open the given device! I will appreciate any help.
@@Jims-Garage nevermind. I had mistake in my gid mappings. In your video you map gid 107 (LXC) to 104 (Host). In my case i had to map 106 to 104. Slightly changed my mappings and everything works as a charm now! Thank you once again for the sublime tutorial
It's possible do the same thind with an external disk drive and an unprivileged LXC? I try to do a Proxmox Backup Server in this scenario and backup on a USB external disk drive, I managed to install with no problems PBS, but failed multiple time the USB hard drive passthrough... If enyone has some useful hints it will be nice...
@@Jims-Garage I changed the mapping as per your guide. But, I do not have the drivers installed on either proxmox host and LXC as well. Sorry, if this is required to make it work.
@@durgeshkshirsagar5160 for intel this video works (drivers are installed in the kernel). For Nvidia you need to install drivers on Proxmox host and inside the LXC
Is there a chance this setup can be broken with a future update? That being said is safer to pass through gpu and hdd to a vm so you won’t have to worry about your pass through hardware from not being pass through.
@@Jims-Garage are you referring to the hardware acceleration script? If yes I am reading the script and correct if I am wrong but I believe the script requires the container to be a privileged container which is not a good thing.
How exactly do i get this running with a different user other than root? You said that you could do this through somewhere that you mentioned in the start of the tutorial, but i cant seem to figure it out. Pls help hahaha
I did not catch this quite right -so is this a way that works only with many LXC+Docker inside or many LXC+ anything inside. That is - can i run, say, 4 LXC debian containers and in each one of them, one Windows 10 VM? If so - it is interesting and great! Otherwise (LXC+Docker)... isn't it already possible to share GPU with every docker container after installing nvidia cuda docker, and pass -gpu all
Unfortunately you cannot have a windows LXC. You could use this for a Linux desktop though with GPU acceleration. E.g., you could have a Linux gaming remote client
@@Jims-Garageso, you are saying: yes, it is not exclusive for LXC+Docker, but anything running in LXC can access gpu? If so, what would one get just for sake of having it: proxmox > LXC (debian with gpu) > cockpit > windows VM > gpu intensive app like game or cad software?
Three questions: 1) Have you tried gaming with this, simultaneously? 2) Have you tested this method using either an AMD GPU and/or a NVIDIA GPU? 3) Do you ever run into a situation where the first container "hangs on" to the Intel Arc A380 and wouldn't let go of it such that the other containers aren't able to access said Intel Arc A380 anymore? I am asking because I am running into this problem right now with my NVIDIA RTX A2000 where the first container sees it and even WITHOUT the container being started and in a running state -- my second container (Plex) -- when I try to run "nvidia-smi", it says: "Failed to initialize NVML: Unknown Error". But if I remove my first container, than the second container is able to "get" the RTX A2000 passed through to it without any issues.
1. No, not sure how I'd test it. Would have to be Linux desktop environment I assume. 2. No, but the process should be identical, it's not intel specific. 3. No, haven't seen that issue. As per the video I created 4 and all had access and survived reboots etc
@@Jims-Garage 1. I would think that if you ran "apt install -y xfce4 xfce4-goodies xorg dbus-x11 x11-xserver-utils xfce4-terminal xrdp", you should be able to at least install the desktop environment that you can then remote into and install Steam (for example) and then test it with like League of Legends or something like that -- something that wouldn't be too graphically demanding for the Arc A380, no? 2. The numbers for the cgroup2 stuff that you have to add to the .conf changes depending on whether it's an Intel (i)GPU (or dGPU) vs. NVIDIA. i.e. with my Nvidia RTX A2000, I don't have that RenderD128 option or whatever it is that it corresponds to. 3. Are you able to test passing the same GPU between from a CT to a VM and back? This is the issue that I am running into right now with my A2000 where my VM won't release the GPU, even after the VM has been stopped. The CT will report back (when I try to run "nvidia-smi") "Failed to initialize NVML: Unknown Error". However, prior to shutting down my LXC container and starting the VM, the CT is able to "see" and use said A2000 (as reported by "nvidia-smi") when I am running a GPU accelerated CFD application. Shut down the CT, start the VM, run the same GPU accelerated CFD application, shut down the VM, and start the CT again -- that same GPU accelerated CFD application now won't load/utilize said A2000 and "nvidia-smi" will give me that error. So I am curious if you're running into the same thing, if you were to try and pass the GPU back and forth between VM CT.
@@ewenchan1239 I could do that by installing a desktop or game I think. I think the issue you're facing is that because you're using a VM for passthrough you're likely blacklisting devices and drivers. This would stop the host being able to share the GPU with the LXC
@@Jims-Garage "I think the issue you're facing is that because you're using a VM for passthrough you're likely blacklisting devices and drivers. This would stop the host being able to share the GPU with the LXC" But you would think that when the VM is stopped, it would release the GPU back to the host, so that you can use it for something else, e.g. a LXC.
@@Jims-Garage Thanks. Might try the unprivileged method...just seems like a rather complicated process which would be simplified in the privileged scenario. Do realize the security implications.
@@Jims-Garage Thanks! I'm planning to use an Nvidia Tesla P4 card. I tried following a craft computing tutorial, but some of the repositories arent available anymore. im bummed...
Jim in your video why is it after you edit the conf file and boot up the 104 container that when you run ls -l /dev/dri the render is showing group ssh 226, 129, shouldnt it be render 226, 129
on my CT the render group is 106 but when I try to edit the conf file and use lxc.idmap: u 0 100000 65536 lxc.idmap: g 0 100000 44 lxc.idmap: g 44 44 1 lxc.idmap: g 45 100045 62 lxc.idmap: g 106 104 1 lxc.idmap: g 107 100107 65428 it fails to boot. it only works if I use lxc.idmap: u 0 100000 65536 lxc.idmap: g 0 100000 44 lxc.idmap: g 44 44 1 lxc.idmap: g 45 100045 62 lxc.idmap: g 107 104 1 lxc.idmap: g 108 100108 65428 but again its showing the /dev/dri is in group _ssh for me instead of render on my CT do we need to edit the conf file before the first boot to have render grouped to 107?
Just wanted to add that there is a more simple way of doing this by placing the following in the .conf file instead of the other lines including idmap lines: dev0: /dev/dri/card0,gid=44,uid=0 and dev1: /dev/dri/renderD128,gid=105,uid=0
Can you shed some light on why the container would start but then not show anything on the console but a black screen? In my case I'm running an unmanic container on Debian 12 and followed the guide and while I don't get any errors, my Console is just a black screen (but the container shows up on my network - it's unreachable though)
@@Jims-Garage Alright so strangely enough it must just take a very long time to startup because going back to the container console after a while does present a prompt. However, now I cannot access the app interface (in this case - unmanic) via the IP:PORT but the IP does show up on the network
@@Jims-Garage I have 2 GPUs a 4090 for my gaming VM and a Quadro P2000 for my LXCs for Plex, Frigate etc.. will the driver on the host cause me issues with my 4090 and VM passthrough?
@@sohail579 you should be fine, drivers are device specific. I imagine the same driver will be used for both of those cards. A 4090 is wild for a gaming VM. Jealous!
@@Jims-Garage Thanks it used to be a 3090ti but found a deal I couldn't refuse so sold the 3090 and took the plunge on the 4090 and I have plenty of cores with my TR Pro 5975wx its prob not the best thing but I have 1 box which does it all as im only home-lab-ing - and let me tell you that you have been a god send I have really been ramping up what my server does since i came accross your channel you explain so well keep up the good work.. now to go figure out the NVIDIA drivers
Has anyone gotten this working without running JF in Docker? Or is there anyone who can point me to documentation for commands/configurations for JF for the "group add" and "devices" variables from the yaml for docker compose?
After days of struggling between guides on the internet I was able to install the NVIDIA drivers on the host. I have tried to install the drivers in the lxc without success. How did you get yours to work? Thank you for the answer, and thank you for the awesome guide.
@@Jims-Garage Thanks but no thanks, that app needs my phone number for some reason, I don't want to get robocalls from them. Do you have anything better set up, like Slack?
Not sure if I should just edit my comment, but... I'm just dumber than I thought. I had a gpu passthrough to a vm. I just removed the gpu from the hardware of that vm and shut it down. But since it's been a while I forgot that I actually had to edit GRUB so proxmox won't load/use the GPU itself. i just removed the extra stuff from this line from /etc/default/grub: GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off" so it would be back at GRUB_CMDLINE_LINUX_DEFAULT="quiet"
@@zabu1458 Very Interesting, I have the same problem. When I change my grub, and restart, it won't add the /dri folder in /dev, but my frigate lxc won't start... I effectively tried multiple tutos to do gpu passthrougt... may i have to do another action to see again the /dri?
i forget a an important one of the vms is a windows vm and this pc is under my tv can i accses the gpu with hdmi and play directly from it or not and thanks for the respond @@Jims-Garage
Hmmm, I'm not sure what I'm doing wrong here. I followed all the steps and can see renderD128 showing up with ls -l /dev/dri command in my LXC. I can also see that if I run the same command in the client within Portainer. But I when I enable hardware acceleration in Jellyfin then I get a fatal error. The only thing that puzzles me is when I was looking for the group ID for my GPU, it was 65534 rather than 107. If anyone has any thought's I'd be grateful. It's a big change from running CasaOS on a Pi 😂
@@Jims-Garage Does it provide any advantage of containerising insider of a container ? Don't get me wrong I have docker containers running on unraid, which is running on proxmox....But my reason is: I made a mistake putting my storage on unraid and shifting from unraid is going to cost 000s.
So I've been playing around with this some more, and found that if I deleted the VM, and was ONLY running LXC containers (right now, I am using all privileged containers -- haven't tested with unprivileged containers yet) -- I am able to have multiple LXC containers do different things with my RTX A2000. Going to be testing with gaming next, so we'll see. But yeah - it would appear that I can't have both VMs and CTs on the same host, sharing a GPU. I can either have ONE VM using the GPU at a time, or I can have NO VMs (at all, on the host, that uses the GPU), and at least a few LXC containers, sharing the one GPU.
@@Jims-Garage But the crazy thing is that you would think that when the VM ISN'T running, that the LXC should be or ought to be able to use the "free" GPU that isn't being used/tied to a VM anymore. That doesn't appear to be the case. It wasn't until I removed said VM, did it "release" the GPU back over to the LXC containers.
@@ewenchan1239 I could be wrong but it sounds like you aren't blacklisting the drivers and device completely. To my knowledge the LXC wouldn't work with hardware passthrough if you were as the host won't be loading drivers
@@Jims-Garage "I could be wrong but it sounds like you aren't blacklisting the drivers and device completely." I'm at work right now, so I'll have to pull my config files later, when I get back home. *edit* Here are the config files: /etc/modprobe.d/nvidia.conf blacklist nvidia blacklist nouveau blacklist vfio-pci /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream nofb nomodeset initcall_blacklist=sysfb_init video=vesafbff,efifbff vfio-pci.ids=10de:2531,10de:228e disable_vga=1" /etc/modprobe.d/vfio.conf options vfio-pci ids=10de:2531,10de:228e disable_vga=1 /etc/modprobe.d/kvm.conf options kvm ignore_msrs=1 /etc/modprobe.d/iommu_unsafe_interrupts.conf options vfio_iommu_type1 allow_unsafe_interrupts=1 /etc/modprobe.d/pve-blacklist.conf blackllist nvidiafb blacklist nvidia blacklist nouveau blacklist radeon /etc/modules vfio vfio_iommu_type1 vfio_pci vfio_virqfd nvidia nvidia-modeset nvidia_uvm Yeah...so that's what I have, in my config files. As far as I can tell, it's complete (because it works for both VMs and CTs, just not being able to pass the GPU back and forth between said VM(s) and CT(s)). But between CTs, not a problem.
@@Jims-Garage "To my knowledge the LXC wouldn't work with hardware passthrough if you were as the host won't be loading drivers" Updated my previous comment. With the config information that I just shared, it works for both VMs and CTs - just not when they exist on the same host, at the same time.
update: i fix it... its was my mistake root:44:1 rooot:104:1 :/ lxc_map_ids: 245 newgidmap failed to write mapping "newgidmap: gid range [107-108) -> [104-105) not allowed": newgidmap 228560 0 100000 44 44 44 1 45 100045 62 107 104 1 108 100108 65428 lxc_spawn: 1795 Failed to set up id mapping.
You have solved just one of my little problems , I've moved jellyfin form one server to another and frigate VA worked , but jellyfin was giving me a error . Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_amf)) Stream #0:1 -> #0:1 (aac (native) -> aac (libfdk_aac)) Press [q] to stop, [?] for help [h264_amf @ 0x557e719b81c0] DLL libamfrt64.so.1 failed to open double free or corruption (fasttop) Could not work it out it was, it was from a backup so the same configs etc , look at your notes and there was a OOPs forgot to the. usermod -aG render,video root Now all working again .
[hevc @ 0x614d20d42dc0] Failed to create decode context: 2 (resource allocation failed). [hevc @ 0x614d20d42dc0] Failed setup for format vaapi: hwaccel initialisation returned error. Impossible to convert between the formats supported by the filter 'Parsed_setparams_0' and the filter 'auto_scale_0' Error reinitializing filters! Tried to fresh install of proxmox 8.2.4, Debian 12 lxc , jellyfin in docker and i dont know where i faild, using Arc A380. In LXC ls -l /dev/dri total 0 crw-rw---- 1 nobody _ssh 226, 128 Aug 2 19:45 renderD128
does not work for me FFmpeg gives this error [AVHWDeviceContext @ 0x642ff9562240] No VA display found for device /dev/dri/renderD128. Device creation failed: -22. [h264 @ 0x642ff954c540] No device available for decoder: device type vaapi needed for codec h264. Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_vaapi)) Stream #0:2 -> #0:1 (aac (native) -> aac (native)) Device setup failed for decoder on input stream #0:0 : Invalid argument
@@Jims-Garage I tried passing through the iGPU without success. I then attempted it with a privileged container, and it works. I installed Jellyfin directly in the LXC without Docker. Probably there is an issue with the permissions.
with the help of ChatGPT i figured out the config that works for me lxc.cgroup2.devices.allow: c 226:0 rwm lxc.cgroup2.devices.allow: c 226:128 rwm lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file lxc.idmap: u 0 100000 65536 lxc.idmap: g 0 100000 44 lxc.idmap: g 44 44 1 lxc.idmap: g 45 100045 59 lxc.idmap: g 104 104 1 lxc.idmap: g 105 100105 65431
For anyone else struggling to determine which GPU is which, run `ls -l /dev/dri/by-path`, and cross reference the addresses in that output with the output of `lspci`, which will also list the full GPU name.
I've been rewatching the video twice trying to figure this out - your comment saved me a lot of headaches - thanks a lot !
i dont have a dri folder :(
This is quickly becoming my favorite channel to watch :D Great stuff! Can't wait to see what you have for us next!
Haha, thanks for the feedback. Next step is network shares on LXC. Then onto clusters on LXC with GPU shared.
The “eat like an American…” wall hanging got me. The content is secondary.
If you're deploying Jellyfin in a Portainer LXC on Proxmox then you will need to create a Custom Template the in Portainer WebUI and add the yaml contents to the Docker-Compose section. This will create the container and deploy in the Stack. From here, changes can be made to the docker-compose yaml and the container can be updated as necessary. I am still new to Linux, Docker, and Portainer so I am sure there is an easier way to do this but after having a difficult time of finding the yaml for Jellyfin in the LXC this is what worked for me. All other steps from the video worked like a charm!
Spent the last couple of weeks trying to achieve this myself and couldn't - had to stick with a privileged container. This worked perfectly first time, thank you Jim!
Nice work! Enjoy the added security :)
I just used this guide to get hardware encoding working in an unprivileged Immich lxc container, through docker compose. ( After a lot of work) Thank you so much for your great and comprehensive guides.
Great stuff, well done ✅
Thank you. Your tutorials are some of the best, very well explained and functional.
You're very welcome!
My weekend project right here. I run unraid in a VM with some docker containers running in it. I want to move all containers outside the unraid VM. Now I can test this and also sharing the iGPU!!! Not straight put to a single VM. NICE!
Absolutely, it's pretty huge being able to share the iGPU between LXCs
Came from the Selfhosted Newsletter a few days ago and I am loving it. Great video, and I will definetly try that as soon as I have time
Awesome! Thank you!
Nice Jim 😀. You keep making great content👌🤌
Thanks 👍
Great video. I did a similar thing ages ago to passthrough a couple of printers to an lxc unprivileged cups printer server! Was a headache to figure everything out at the time hehehe
Ooh, that's a great use case. I like it.
Your github is a pot of gold. TY sir
Thanks 👍
excellent video, thanks!
@@robertodepetro1996 thanks 👍
Great video of Proxmox outside.Thank you Sir!
You're welcome 😁
Finally Subscribed to your channel! Thank YOU!
Thanks for subbing!
thank you for this tutorial. It works. just want to let you know, it works also with handbrake docker container, just in the compose file add GROUP_ID=107 in the environment section. intel qsv is detected!
Awesome, thanks for letting me know
Great video. I am going to play with this this week so both Jellyfin and Plex have access to the GPU. Maybe other stuff eventually.
Great stuff 👍
I really love your channel Jim. I learn(ed) a lot from you !!
I would love to see how to get the low power encoding working 🙂
Coming soon!
I struggled with an AMD igpu pass through with Jellyfin and you were very kind to help , in my case it did not work with a regular VM , but with this it was a breeze to setup Jellyfin with HW transcoding , the only hiccup was the lxc image of Debian 12 did not work , but Ubuntu did , latest proxmox fully updated , thanks again your walkthroughs are really helpful thanks !
That's great, good job 👍
Could you help me out?
@@codexclusiveNLfollow the steps that Jim laid out, some films depending on the setup will work depending on the browser or device also, for me even the groups that Jim has in the video are the same , y setup is a beeliknk mini pc with an AMD 5650
Congrats, good stuff. You may want to check out how to run docker images as LXC containers, since they are OCI compliant. It would remove an abstraction layer, but instead of compose it would be set up with ansible.
Good suggestion, something I can check out later. Thanks
@@Jims-Garage and @gamermerijn I was wondering about this as well. What's the added benefit of running Jellyfin in Docker when you can just install it in the LXC directly?
@@RudyBleeker mainly security and density. VMs use a separate kernel to the host, especially good for internet facing containers. Furthermore, 1 backup of the VM and all of my containers are covered (albeit you can install docker in an LXC).
@@Jims-Garage I know all about the difference between VMs and LXC. But in the video you installed Jellyfin in Docker, that Docker runs in LXC, on a bare metal Proxmox host, correct? There was no talk about VM's, or did I miss something?
So I'm curious why you chose to install the Docker runtime in LXC and run Jellyfin through that, instead of installing the Jellyfin packages in the LXC directly using Jellyfin's official repos. Adding Docker into the mix just introduces another layer of (very minimal) overhead and complexity if you'd ask me.
Great Content! Would be nice if you elaborated more on the low power encoder in one of your next videos.
Noted!
I also run my Kubernetes test env. in LXC on my laptop, makes a lot of sense.
That's great. I'm hoping to do similar for GPU sharing.
@@Jims-Garage You've already figured out the hard part.
13:34 in practice by the way it doesn't matter. As long as the host is newer or the same and you load any kernel modules you might need. Linux mostly adds new functionality, as Linus always says: "don't break user space". I was able to run Debian 2/Hamm LXC container on a modern Linux kernel aka Debian 12. Not like I've never done this before. I was running Linux containers before LXC existed, before I ever touched VMs. On Debian Woody with Linux-VServer.
@@autohmae wow, that's impressive. Thanks for sharing
@@Jims-Garage well, it's supposed to work 🙂
Hardware passthrough to LXC is definitely something I want to explore. I have a few services running in an Alpine QEMU and the footprint is small but I would prefer to have one LXC per service
I started down the hardware passthrough rabbithole with CUPS. Network printing is another use case
This is great stuff! Thanks Jim :-)
Thanks. It's a really good feature of LXCs.
2:40 actually it's for some intel gpus possible to split between vms. but didn't do any benchmark on it and had no use, so i went for priviledged lxc at the time i was setting up mine. but now i'm considering redoing it unpriviledged, thanks for the video!
It was. Unfortunately it's now discontinued...
@@Jims-Garage right, there are lots of tiny differences on intel gpus. had it running with an 7700k about a year ago, i think this still would work today if the hw supports it (?)
also played around with a DVA xpenology vm, unfortunately the 7700 igpu is too new for that
@@sku2007 my understanding is that you have to use sr-iov now.
@@Jims-Garage as long as you have older GPU it works but it is quite limited. On mini PC with i5-10500T I was able to split iGpu into 2 GVT devices. Interesting part is that even if you assign vGPU to VMs you can still use real iGPU in LXCs. Of course the performance will suffer this way but in case of load like transcoding it is perfectly fine.
I suggest you give it a try.
There is a video where a passthrough nvidia GPU is split between vms.
this is way cool ! LXC all the way
Thanks, it's a great tool to have.
Great tutorial.
Thanks 👍
I just discovered your series and it is amazing. I am Trying to do something similar on my homelab since a year ago but still failed. I already had some id maps in place for my mounts (more in my next comment on that video) but you essentially solved for me what I was struggeling and nearly given up.
Now Jellyfin is HW rranscoding on my NUC Lab host and I am so happy with it :D
One more thing that I am currently struggeling with - and you might have an idea / solution / future video:
Docker swarm seems not to work inside an lxc container. Containers get deployed but are not accessible via the ingress network.
Anyways thanks again I am looking forward to the new videos while watching the back catalog.
Great work 👍
Firstly, don't use the KVM image, use a standard cloud image (there's an issue). Let me know if that solves it.
@@Jims-Garage Thank you - I got it working on a debian 12 lxc container.
Some of the IDs needed to be different but now it is merged with my lxc mounts and everything is working.
If i now only could get docker swarm to work. (but this a known problem in LXC - works fine in VM).
I dont belive I see this, you are the best
Excellent, thanks! Was able to get it working with the iGPU. Is the process the same for a NVIDIA GPU?
It should be, just change the mappings to suit. Thanks a lot for the donation!
I tested this out and Jellyfin worked great in a Proxmox LXC container also with Intel A380 passthrough. Can you please make a guide on how to get it running on Plex? I could not get Plex working with Hardware Acceleration for the life of me.
Do you make your own thumbnails? Yours are top tier!!!
Impressive, I wonder if its as simple with an AMD igpu, with an xcp-ng hypervisor, probably not. But it is amazing to share an igpu like this, multiple graphic cards is rediculous. Seems like this solves gpu sharing in general 🤔
It should work on Proxmox with an iGPU in almost exactly the same way, I've no experience with xcp-ng though... SR-IOV is also another way to do it but consumer devices don't typically support it.
What you say 6 minutes into the video about the /etc/subgid file is wrong. These entries are not mappings but ranges of gid's. It's a start gid and a count.
I'm still trying to get my head dialled in on the lxc.idmap entries in the .conf file. Getting closer. Thanks for the video.
The subguid is a moot point if you're running as root and can be skipped
@@Jims-Garage, what about adding root to the video and render groups on the host (@12:30)...is that necessary? This is a weird step to me.
You do have an error in your github notes. After carefully following the directions and c/p from your notes I thought it odd when no /etc/subguid could be found. Still I proceeded but the container wouldn't start. After looking around a bit I noticed that /etc/subguid should have been /etc/subgid. After fixing the issue the container started just fine. Regardless, great video and you gained a new sub. Thanks..
OMG! Thank you for that! I've been pulling my hair out.
Thanks! I will fix this now.
Hi @Jim's Garage, have you managed to run X11 with nvidia/intel acceleration on it? Something that would allow to run e.g. steam games?
@@Szala89r I haven't as I don't have a card to test it with unfortunately. My understanding is that you wouldn't have graphical output though, just GPU accelerated workloads
Ty for sharing your knowledge
Two questions if you may know the answer?
1. Can Proxmox install Nvidia linux drivers over Nouveau and still share the video card?
2. If one adds a newer headless GPU like the Nvidia L4, can you use this as a secondary or even primary video card in a VM or CT?
Yes to both. Follow the same procedure and mount the additional GPU.
only issue with this is mounting NFS shares. I have yet to find a way to mount an NFS share into a unprivileged LXC then recreate the container onto another node that has a GPU.
Hello Jim, thanks for your instructions . I really appreciate you sharing your valuvable knowledge, I did follow your instructions, but the sharing does not work at the end with a permission error message, and whenever I do upadte the configuration file for the LXC with the line (inside my /etc/pve/lxc/1xx.conf) lxc.mount.entry%3A /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file , any help please
Really awesome! But how is this working on the technical level without GPU virtualization at all?
The LXC is sharing access with the host's GPU
At 16:03 is the group for renderD129 supposed to be _ssh? It's the same on my system and in Emby I'm getting a permission denied error, so I'm wondering if those 2 are related
Not working for me the Proxmox LXC - Docker is not taking the IGPU no activtiy on the engines while play diffrent content :( i do not know what i am doning wrong.
@@weysinchathamazight9956 probably the groups, I used a dGPU, change it to match iGPU
Hey tried to follow exactly, but I get stuck after editing the config file. Afterwards when I spin back up the container, I get nothing on console. So as you mention it might be a permission issue, but I think I did everything!
Great video! How does this work with nvidia drivers with a GPU? Does the driver need to be installed on the host and then in each LXC?
Yes it does
Seems like a powerful feature. How do you do this in the UI?
@@imorganmarshall I don't think you have access to all of the necessary options (might be wrong)
@@Jims-Garage 8.2 does
@@Scott-f9 ahh that's great news. I'll perhaps revisit this with a gui version
What is the command to get the gid or uid when you mention LXC namspace or host namespace?
Greate video i hope this help me to solve the HWA.
How does it work with dedicated GPUs? Do I need to install the driver on the Proxmox Host or in the LXC? Do I need to specify the card in the docker compose or is the ID enough? Do I need the Container Toolkit for Docker? I really like your content, one of the best channels right now about selfhosting, but haven't found a solution to this.
The video is using a dedicated intel arc a380 GPU. For Nvidia you should be able to follow the same process. I believe most modern OS will have drivers but you might need to add them.
@@Jims-Garage Thank you for the answer. I'll try it.
amazing. use for my iGPU. are there any other devices apart of the GPU in addition to video and render? can i not pass all the functions to the LXC or virtual machine? On my system it says the iGPU is the same IOMMU group as the USB controllers and such. So i can't pass it through the the VM, would it be possible the share the iPU among VMs?
how can we do this for an NFS share on an Unprivileged LXC?
Thanks a lot! Don't need to activate iommu in grub?
@@Danilo_TI no, because it's simply sharing the host's hardware, not passing it through.
@@Jims-Garage Can i do same thing for USB storage ?
@@Danilo_TI you should be able to share it yes
Is there a way to do the opposite? As in consolidate multiple GPUs, RAM etc. into one server? I have 2 laptops and an external GPU I want to connect together to combine their compute to then be able to redistribute it out to multiple devices similar to this video. Is it possible?
I don't think so. The closest I could imagine is pooling the resources into a Kubernetes cluster or docker swarm.
Great guide. I just got this working for two LXC and Jellyfin. I am trying to use Plex in a Docker container but can't get the hardware transcoding to work. Can anyone help?
Check the docs here, it's what I use. Almost identical: github.com/linuxserver/docker-plex
@@Jims-Garage great video! I got it working with Jellyfin just like in your video, but under Plex(using the link you provided) I get "No VA display found for device /dev/dri/renderD128" in the Plex transcoder settings it recognizes the iGPU, "lshw" in the container also sees the iGPU. any ideas you can share would be a big help. thanks!
@@narkelo It's likely t o be permissions with the Plex user. Try running as root then dial it back if that works.
Hey Jim, thanks for your thorough tutorials. Most of my home lab setup is done with the help of your videos. Following this specific tutorial a question arises: since the Jellyfin CT is using the host's hardware, should I enable the GPU passthrough prior to sharing my iGPU to the Jellyfin CT, or doing the groups mapping trick is enough? By 'sharing' the iGPU does it mean I can still use it for the proxmox host (if I have to connect a display straight to my server and access the proxmox CLI)?
By following this video the host and the CTs can use the iGPU. If you do a passthrough then only that VM can use it (the host cannot).
That's awesome! And thanks for the swift response. In fact, thank you for all the awesome work you do!
@@Jims-Garage mapped the uid and gid values as you did;
1. I am able to see my renderD128 device in the LXC: crw-rw---- 1 nobody _ssh 226, 128 Jul 4 07:53 renderD128;
2. lspc icommand in the LXC shows: 00:02.0 VGA compatible controller: Intel Corporation CometLake-S GT2 [UHD Graphics 630] (rev 03)
3. Installed intel gpu tools and ran intel_gpu_top - Failed to initialize PMU! (Permission denied)
4. Installed Jellyfin and ffmpeg6, then proceeded to enabling transcoding, following the official Jellyfin guide.
when trying to check the supported QSV / VA-API codecs gives me the following output:
root@Jellyfin:~# /usr/lib/jellyfin-ffmpeg/vainfo --display drm --device /dev/dri/renderD128
Trying display: drm
Failed to open the given device!
I will appreciate any help.
@@Jims-Garage nevermind. I had mistake in my gid mappings. In your video you map gid 107 (LXC) to 104 (Host). In my case i had to map 106 to 104. Slightly changed my mappings and everything works as a charm now! Thank you once again for the sublime tutorial
@@stanislavtodorov8705 you're most welcome. Perhaps the difference between iGPU and dGPU
Hello, it's possible on the same PVE node to have a split gpu for LXC and for VM ? Thanks for this good video.
Not possible with the same GPU as VM requires the GPU is not loaded by the host. Dual GPU would work.
@@Jims-Garage that was what it seemed to me, thanks. (I am French and you are easy to understand)
It's possible do the same thind with an external disk drive and an unprivileged LXC?
I try to do a Proxmox Backup Server in this scenario and backup on a USB external disk drive, I managed to install with no problems PBS, but failed multiple time the USB hard drive passthrough...
If enyone has some useful hints it will be nice...
I followed you guide but when I played video, it said fatal error. My processor is 4670s and gpu is GTX 950.
@@durgeshkshirsagar5160 did you make sure you changed the mapping for your Nvidia GPU and that each machine has the same drivers?
@@Jims-Garage I changed the mapping as per your guide. But, I do not have the drivers installed on either proxmox host and LXC as well. Sorry, if this is required to make it work.
@@durgeshkshirsagar5160 for intel this video works (drivers are installed in the kernel). For Nvidia you need to install drivers on Proxmox host and inside the LXC
@@Jims-Garage Thank you for the help.
Is there a chance this setup can be broken with a future update? That being said is safer to pass through gpu and hdd to a vm so you won’t have to worry about your pass through hardware from not being pass through.
Yes, kernel updates can break this without following specific procedures. VMs don't have that problem.
@@Jims-Garagedo you have the specific procedures so it won’t break when there’s a kernel update?
@@mg3299 there's a handy script here, but do take time to understand it. github.com/tteck/Proxmox
@@Jims-Garage are you referring to the hardware acceleration script? If yes I am reading the script and correct if I am wrong but I believe the script requires the container to be a privileged container which is not a good thing.
How exactly do i get this running with a different user other than root? You said that you could do this through somewhere that you mentioned in the start of the tutorial, but i cant seem to figure it out. Pls help hahaha
I did not catch this quite right -so is this a way that works only with many LXC+Docker inside or many LXC+ anything inside. That is - can i run, say, 4 LXC debian containers and in each one of them, one Windows 10 VM? If so - it is interesting and great! Otherwise (LXC+Docker)... isn't it already possible to share GPU with every docker container after installing nvidia cuda docker, and pass -gpu all
Unfortunately you cannot have a windows LXC. You could use this for a Linux desktop though with GPU acceleration. E.g., you could have a Linux gaming remote client
@@Jims-Garageso, you are saying: yes, it is not exclusive for LXC+Docker, but anything running in LXC can access gpu? If so, what would one get just for sake of having it: proxmox > LXC (debian with gpu) > cockpit > windows VM > gpu intensive app like game or cad software?
Three questions:
1) Have you tried gaming with this, simultaneously?
2) Have you tested this method using either an AMD GPU and/or a NVIDIA GPU?
3) Do you ever run into a situation where the first container "hangs on" to the Intel Arc A380 and wouldn't let go of it such that the other containers aren't able to access said Intel Arc A380 anymore?
I am asking because I am running into this problem right now with my NVIDIA RTX A2000 where the first container sees it and even WITHOUT the container being started and in a running state -- my second container (Plex) -- when I try to run "nvidia-smi", it says: "Failed to initialize NVML: Unknown Error".
But if I remove my first container, than the second container is able to "get" the RTX A2000 passed through to it without any issues.
1. No, not sure how I'd test it. Would have to be Linux desktop environment I assume.
2. No, but the process should be identical, it's not intel specific.
3. No, haven't seen that issue. As per the video I created 4 and all had access and survived reboots etc
@@Jims-Garage
1. I would think that if you ran "apt install -y xfce4 xfce4-goodies xorg dbus-x11 x11-xserver-utils xfce4-terminal xrdp", you should be able to at least install the desktop environment that you can then remote into and install Steam (for example) and then test it with like League of Legends or something like that -- something that wouldn't be too graphically demanding for the Arc A380, no?
2. The numbers for the cgroup2 stuff that you have to add to the .conf changes depending on whether it's an Intel (i)GPU (or dGPU) vs. NVIDIA.
i.e. with my Nvidia RTX A2000, I don't have that RenderD128 option or whatever it is that it corresponds to.
3. Are you able to test passing the same GPU between from a CT to a VM and back?
This is the issue that I am running into right now with my A2000 where my VM won't release the GPU, even after the VM has been stopped.
The CT will report back (when I try to run "nvidia-smi") "Failed to initialize NVML: Unknown Error".
However, prior to shutting down my LXC container and starting the VM, the CT is able to "see" and use said A2000 (as reported by "nvidia-smi") when I am running a GPU accelerated CFD application.
Shut down the CT, start the VM, run the same GPU accelerated CFD application, shut down the VM, and start the CT again -- that same GPU accelerated CFD application now won't load/utilize said A2000 and "nvidia-smi" will give me that error.
So I am curious if you're running into the same thing, if you were to try and pass the GPU back and forth between VM CT.
@@ewenchan1239 I could do that by installing a desktop or game I think.
I think the issue you're facing is that because you're using a VM for passthrough you're likely blacklisting devices and drivers. This would stop the host being able to share the GPU with the LXC
@@Jims-Garage
"I think the issue you're facing is that because you're using a VM for passthrough you're likely blacklisting devices and drivers. This would stop the host being able to share the GPU with the LXC"
But you would think that when the VM is stopped, it would release the GPU back to the host, so that you can use it for something else, e.g. a LXC.
Is this simplified if one were to go with a privileged container?
A privilegeled LXC doesn't require the idmap, you can simply mount
@@Jims-Garage Thanks. Might try the unprivileged method...just seems like a rather complicated process which would be simplified in the privileged scenario. Do realize the security implications.
@@ckthmpson if it's simply for internal applications you're probably okay
How can this be modified to split the GPU between VMs?
You can't with consumer gear. You either need an old GPU with a driver hack or an intel iGPU with gpu-vt (but it's deprecated AFAIK)
@@Jims-Garage Thanks! I'm planning to use an Nvidia Tesla P4 card. I tried following a craft computing tutorial, but some of the repositories arent available anymore. im bummed...
@@pnwscitech1589 Tesla can be split from what I know, grid or something. I don't have one to test though unfortunately
is it possible to do this on multiple windows or macos vms?
No, unfortunately this method only works on Linux.
@@Jims-Garage thankyou
Jim in your video why is it after you edit the conf file and boot up the 104 container that when you run ls -l /dev/dri the render is showing group ssh 226, 129, shouldnt it be render 226, 129
on my CT the render group is 106 but when I try to edit the conf file and use
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 44
lxc.idmap: g 44 44 1
lxc.idmap: g 45 100045 62
lxc.idmap: g 106 104 1
lxc.idmap: g 107 100107 65428
it fails to boot.
it only works if I use
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 44
lxc.idmap: g 44 44 1
lxc.idmap: g 45 100045 62
lxc.idmap: g 107 104 1
lxc.idmap: g 108 100108 65428
but again its showing the /dev/dri is in group _ssh for me instead of render on my CT
do we need to edit the conf file before the first boot to have render grouped to 107?
hello, have you figure it out? how to change the group of renderd128 to render?
Just wanted to add that there is a more simple way of doing this by placing the following in the .conf file instead of the other lines including idmap lines: dev0: /dev/dri/card0,gid=44,uid=0 and dev1: /dev/dri/renderD128,gid=105,uid=0
@@dfleiva thanks, I'll test that out
Can you shed some light on why the container would start but then not show anything on the console but a black screen? In my case I'm running an unmanic container on Debian 12 and followed the guide and while I don't get any errors, my Console is just a black screen (but the container shows up on my network - it's unreachable though)
Can you SSH?
@@Jims-Garage Alright so strangely enough it must just take a very long time to startup because going back to the container console after a while does present a prompt.
However, now I cannot access the app interface (in this case - unmanic) via the IP:PORT but the IP does show up on the network
If I am using an nvidia card do i need to install he drivers for it on the Proxmox host first?
You'll need the same drivers on the host and LXC.
@@Jims-Garage I have 2 GPUs a 4090 for my gaming VM and a Quadro P2000 for my LXCs for Plex, Frigate etc.. will the driver on the host cause me issues with my 4090 and VM passthrough?
@@sohail579 you should be fine, drivers are device specific. I imagine the same driver will be used for both of those cards. A 4090 is wild for a gaming VM. Jealous!
@@Jims-Garage Thanks it used to be a 3090ti but found a deal I couldn't refuse so sold the 3090 and took the plunge on the 4090 and I have plenty of cores with my TR Pro 5975wx its prob not the best thing but I have 1 box which does it all as im only home-lab-ing - and let me tell you that you have been a god send I have really been ramping up what my server does since i came accross your channel you explain so well keep up the good work.. now to go figure out the NVIDIA drivers
Has anyone gotten this working without running JF in Docker? Or is there anyone who can point me to documentation for commands/configurations for JF for the "group add" and "devices" variables from the yaml for docker compose?
What if I want to add multiple GPUs?
That should be possible, you'd need to follow the same process and add the other devices. I haven't ever done it though (perhaps in future).
After days of struggling between guides on the internet I was able to install the NVIDIA drivers on the host. I have tried to install the drivers in the lxc without success. How did you get yours to work?
Thank you for the answer, and thank you for the awesome guide.
I'm using an intel arc a380 GPU. The drivers are baked into the OS. It's definitely possible with Nvidia though, I'll try to find some instructions.
I think this might need updating. I followed this exactly but didn't work.
Worth hopping onto Discord, this still works for me.
@@Jims-Garage Thanks but no thanks, that app needs my phone number for some reason, I don't want to get robocalls from them. Do you have anything better set up, like Slack?
Did I miss a previous step? I have no /dri folder under /dev "ls: cannot access '/dev/dri': No such file or directory"
Not sure if I should just edit my comment, but... I'm just dumber than I thought. I had a gpu passthrough to a vm. I just removed the gpu from the hardware of that vm and shut it down. But since it's been a while I forgot that I actually had to edit GRUB so proxmox won't load/use the GPU itself.
i just removed the extra stuff from this line from /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off"
so it would be back at
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
@@zabu1458 Thanks for taking your time and shareing this. It helped me revolve mine :)
@@zabu1458 Very Interesting, I have the same problem. When I change my grub, and restart, it won't add the /dri folder in /dev, but my frigate lxc won't start... I effectively tried multiple tutos to do gpu passthrougt... may i have to do another action to see again the /dri?
Found ! had to edit /etc/modules and remove :
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
Is this possible with LXD?
I lost SSH and terminal login access after updating my container. How do I get it back?
me too. how did you solve it?
can i share my gtx 1650 between couple of vms or not ?
Yes, there is a hack for it using vGPU. For an LXC you can follow this video (but it's Linux only).
i forget a an important one of the vms is a windows vm and this pc is under my tv can i accses the gpu with hdmi and play directly from it or not and thanks for the respond @@Jims-Garage
Hmmm, I'm not sure what I'm doing wrong here.
I followed all the steps and can see renderD128 showing up with ls -l /dev/dri command in my LXC. I can also see that if I run the same command in the client within Portainer. But I when I enable hardware acceleration in Jellyfin then I get a fatal error.
The only thing that puzzles me is when I was looking for the group ID for my GPU, it was 65534 rather than 107.
If anyone has any thought's I'd be grateful. It's a big change from running CasaOS on a Pi 😂
I fixed it. Just needed to reboot proxmox to get the permissions to work. I was then getting the group 107 as expected
Awesome, glad you fixed it!
Would there be a use case for a higher end card like a spare RYX 3070?
This solution is GPU agnostic, you can use whatever you want.
Can i ask why you're running docker inside of a lxc container ?
Why not? Simplifies deployment as I have all of the compose files ready. You could do it manually.
@@Jims-Garage Does it provide any advantage of containerising insider of a container ? Don't get me wrong I have docker containers running on unraid, which is running on proxmox....But my reason is: I made a mistake putting my storage on unraid and shifting from unraid is going to cost 000s.
@@basdfgwe think of the LXC as a virtual machine. It's the same as running a standard docker instance.
What would this look like with a high end GPU like a GTX 3070?
I do the same as Jim and it makes perfectly sense (to me). As a starting point, you could see docker as app containers and lxc as OS containers.
Is Jim ai generated?
"No, he is real" - JimBotv2.0
So I've been playing around with this some more, and found that if I deleted the VM, and was ONLY running LXC containers (right now, I am using all privileged containers -- haven't tested with unprivileged containers yet) -- I am able to have multiple LXC containers do different things with my RTX A2000.
Going to be testing with gaming next, so we'll see.
But yeah - it would appear that I can't have both VMs and CTs on the same host, sharing a GPU.
I can either have ONE VM using the GPU at a time, or I can have NO VMs (at all, on the host, that uses the GPU), and at least a few LXC containers, sharing the one GPU.
Yes, makes sense as a VM requires isolation of the hardware, a LXC doesn't.
@@Jims-Garage
But the crazy thing is that you would think that when the VM ISN'T running, that the LXC should be or ought to be able to use the "free" GPU that isn't being used/tied to a VM anymore.
That doesn't appear to be the case.
It wasn't until I removed said VM, did it "release" the GPU back over to the LXC containers.
@@ewenchan1239 I could be wrong but it sounds like you aren't blacklisting the drivers and device completely. To my knowledge the LXC wouldn't work with hardware passthrough if you were as the host won't be loading drivers
@@Jims-Garage
"I could be wrong but it sounds like you aren't blacklisting the drivers and device completely."
I'm at work right now, so I'll have to pull my config files later, when I get back home.
*edit*
Here are the config files:
/etc/modprobe.d/nvidia.conf
blacklist nvidia
blacklist nouveau
blacklist vfio-pci
/etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream nofb nomodeset initcall_blacklist=sysfb_init video=vesafbff,efifbff vfio-pci.ids=10de:2531,10de:228e disable_vga=1"
/etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2531,10de:228e disable_vga=1
/etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1
/etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
/etc/modprobe.d/pve-blacklist.conf
blackllist nvidiafb
blacklist nvidia
blacklist nouveau
blacklist radeon
/etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
nvidia
nvidia-modeset
nvidia_uvm
Yeah...so that's what I have, in my config files.
As far as I can tell, it's complete (because it works for both VMs and CTs, just not being able to pass the GPU back and forth between said VM(s) and CT(s)). But between CTs, not a problem.
@@Jims-Garage
"To my knowledge the LXC wouldn't work with hardware passthrough if you were as the host won't be loading drivers"
Updated my previous comment.
With the config information that I just shared, it works for both VMs and CTs - just not when they exist on the same host, at the same time.
update: i fix it... its was my mistake
root:44:1
rooot:104:1
:/
lxc_map_ids: 245 newgidmap failed to write mapping "newgidmap: gid range [107-108) -> [104-105) not allowed": newgidmap 228560 0 100000 44 44 44 1 45 100045 62 107 104 1 108 100108 65428
lxc_spawn: 1795 Failed to set up id mapping.
You have solved just one of my little problems , I've moved jellyfin form one server to another and frigate VA worked , but jellyfin was giving me a error .
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_amf))
Stream #0:1 -> #0:1 (aac (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[h264_amf @ 0x557e719b81c0] DLL libamfrt64.so.1 failed to open
double free or corruption (fasttop)
Could not work it out it was, it was from a backup so the same configs etc , look at your notes and there was a OOPs forgot to the. usermod -aG render,video root
Now all working again .
Awesome, glad it's fixed
[hevc @ 0x614d20d42dc0] Failed to create decode context: 2 (resource allocation failed).
[hevc @ 0x614d20d42dc0] Failed setup for format vaapi: hwaccel initialisation returned error.
Impossible to convert between the formats supported by the filter 'Parsed_setparams_0' and the filter 'auto_scale_0'
Error reinitializing filters!
Tried to fresh install of proxmox 8.2.4, Debian 12 lxc , jellyfin in docker and i dont know where i faild, using Arc A380.
In LXC
ls -l /dev/dri
total 0
crw-rw---- 1 nobody _ssh 226, 128 Aug 2 19:45 renderD128
does not work for me FFmpeg gives this error [AVHWDeviceContext @ 0x642ff9562240] No VA display found for device /dev/dri/renderD128.
Device creation failed: -22.
[h264 @ 0x642ff954c540] No device available for decoder: device type vaapi needed for codec h264.
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_vaapi))
Stream #0:2 -> #0:1 (aac (native) -> aac (native))
Device setup failed for decoder on input stream #0:0 : Invalid argument
What are you trying to pass through?
@@Jims-Garage I tried passing through the iGPU without success. I then attempted it with a privileged container, and it works. I installed Jellyfin directly in the LXC without Docker. Probably there is an issue with the permissions.
with the help of ChatGPT i figured out the config that works for me lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 44
lxc.idmap: g 44 44 1
lxc.idmap: g 45 100045 59
lxc.idmap: g 104 104 1
lxc.idmap: g 105 100105 65431
@@ziozzot What did you ask ChatGPT to do? 😂 I've tried a couple thing to help with permissions and I can't see where I'm going wrong