I ran my homelab rig on Scale for a week. It was the longest month of my life. I wish them well and hope they can solve it, but I've stuck with Debian+KVM for good reason. Everything just works for me there.
When was this? Were the issues mostly revolving around the older apps system they're now replacing with proper docker in the next release? I don't have any pcie passthrough for my vm's, but I do have a ton of things running and its been great. It even runs my ansi text environment linux shell host for retro terminals lol.
@@VeronicaExplains my impression has been. TrueNas for storage and docker apps, and the occasional VM work. Pretty much the same with UNRAID. But serious VM work and containers should be proxmox, or other hypervisor, with VM running docker apps. ONly have the basics myself but that is just what I keep seeing from people.
Debian/Qemu/KVM is about as bulletproof as it comes to be honest. You can also do that without proxmox at all, if you're cool with managing everything via CLI (or some other, inferior gui). That said, you could probably Python your way to a decent enough gui without much trouble if you're just looking to do some specific things.
Was curious on the topic, ended up running TrueNAS on Proxmox, have to admit it feels a bit iffy should things go south. First try and all, both on VM and NAS 🤣
It has a GPLv3 Affero license, be sure to set up a mirror of the git repos, just in case. While the license is a great choice to keep it open, BUT the project does demand a Contributor License Agreement, which is a big red flag.
VLAN support for VMs is there since some years. You create a VLAN in the network section and then attach that to the VM in NIC settings. It works fine for me. noVNC was fully replaced by Spice due to security issues in the VNC protocol, but the usability is still bad. However, using virt-viewer should work much better compared to the web client. TrueNAS 24.10-RC.2 is already out and the full release of 24.10 will be out by the end of the month. There are multiple fixes in that version. For example: The Spice VM password is correctly hidden when entering it on 24.10. The 8 length password limitation issue is also resolved. Also, for the disk latency test, was the virtual disk passed through via AHCI or VirtIO? 24.10 might also improve that, but I haven't tested it. Just a guess.
I've been running TrueNas Scale as my sole Homeserver/NAS/Hypervisor for over half a year now. It has worked perfectly fine for running a Reverse Proxy VM, my Nextcloud VM (+ related services), a VM for Jellyfin and Handbrake (with GPU passthrough) along with the occasional Test VM for random stuff i come across. I've standardized on OpenSUSE MicroOS (which is extremely easy to provision in a wide variety of different hosting environments) and the only thing I found lacking were the janky VM Display viewer and Snapshots not including the VM configuration. I tested both Proxmox and XCP-ng for about 6 months each before (and even some more exotic solutions like a Hyperconverged Ceph cluster based on RHEL derivatives) and switched over to Scale, because I just didn't need the extra features. For me the better functionality and UI for handeling Storage (Block device and ZFS options), User management and Share Permissions along with the Performance and Efficiency increase, as well as the complexity decrease of not messing around with a virtualized NAS appliance was worth the usability tradeoff in the Virtualization department.
I agree. I just built a new home server with the latest release of TrueNAS Scale and I'm perfectly fine with the results. I have a few containers running, and a couple linux VMs... nothing super complicated, but it all just works. Yes, the Spice VRT sucks and is the one thing I'd change... nothing is perfect. I looked at Proxmox & xcp-ng and favored the latter, but in the end, wanted my NAS OS to have direct PCIx access to all drives without question. Yes, you can virtualize and pass the HBA through, but many experts including TrueNAS state to not virtualize for production use cases. TrueNAS Scale is not the best pick for enterprise solutions IMO. But for a small, single, do-everything home server... I have no complaints. If I was doing this at work, sure, I'd buy 3-6 hosts, break it all apart with clusters, etc... but for home where energy consumption is factor - no thanks.
I also have truenas as my sole homeserver. it runs 4 vm 1 homeassistaint 2 win 11 vm to remote in and mess around when i am at work 3 xubuntu for the same reason 4 ubuntu server runin docker instances (after wendell's level1tech guide), bc truenas built in docker solution is barbone, for my 20+ docker image setup. i soent 10 minutes to create the vm-s i never touched that part of truenas since, it runs for me falwlessly
I am running 2 Proxmox machines and was looking for a way to run a Proxmox Backup Server and a NAS. When I got an older Server for free I decided to run TrueNas bare metal and virtualize PBS on top of it, since the NAS performance was more critical to me than that of the backup server. It works, but TrueNas is nowhere near Proxmox as a Hypervisor. I felt restricted in a lot of things I wanted to do. So, for me Proxmox is my go-to for hypervisors and TrueNas Scale my favorite NAS solution. Both great in their area of expertise but no competition for each other.
That's the way I'm running things now and was hoping to migrate my Proxmox VMs over to TrueNas Scale. This video (and the se comments) have pretty much convinced me not to change anything as everything is working reliably as it is. Thanks for all the confirmation!!!!!
I really wanted to see the as well now that its in the RC stage. I think that its a good place to start as home lab for testing as long as its not mission critical services.
I have a proxmox cluster for my compute and a truenas server that's just for storage. I've considered running some docker containers in truenas once that feature is released, but never did any VMs there.
@@ravenssettle I've done that to run some LXC's before, and to help make sure I'll always have a quorum incase any of my other nodes go down. Never had any problems running it that way. It ran fine with 128GB of storage and 8GB of RAM.
That was the previously recommended way to do it actually, truenas for pooled, shared storage, proxmox for compute. Nowadays you lose so little by virtualizing that it makes sense to just keep it all under one roof. Also, it's very likely you can just throw proxmox on your truenas host instead, and then pass through your HBA's to a truenas VM and import your pools and be back in action with very little nonsense. You can probably even get away with backing up your truenas config and just importing it, assuming the versions match.
@@jttech44 You can 100% flip-flop from a VM to bare metal install by donwloading/uploading the config file! I've done it a few times in my testing and TrueNAS is none the wiser to the changes.
So.. time for a new and updated Proxmox guide? :) EDIT: It should be noted that i am a huge fan of Jeffs guides, i have spend many hours learning and installing stuff, that i don't really need, but just for the sake of it. I mean, nerdy and all. But it's been a while since he took our hand and walked us through a complete Proxmox guide, with window install, Truenas Setup, GPU pass-through on a consumer card, backup etc etc.. Yeah i know, months of content/work.. sorry Jeff
In my opinion, if you run truenas bare metal and run vms in it it's like using an ISP router that is wifi/switch/toaster, in my opinion you only run truenas for a NAS software. And a hypervisor as a hypervisor.And you can run truenas as a VM too.
I'm running TrueNas Scale and I'm running a basic debian VM as an application host. It's only running docker with a dozen containers. Works fine for that!
I use truenas scale to host my caddy based reverse proxy VM. It is very simple and I can rebuild it in just a few minutes if I have a problem. I would never host any serious VM on truenas. The hypervisor feature is just not finished and not trustworthy. I have had much better success with the apps feature. I am super excited to have that switch to docker in this next release which will make it even easier to use for me.
Once I tried running Truenas Scale as you, because in the store I could install docker containers such us jellyfin, and Radarr... File permissions were too confusing and I ended installing a VM instead of using those apps
I've found myself not using a VM these days. The jailmaker script works really well for making systemd-nspawn containers and you can choose from a pretty good list of LXC images.
Yep, I’ll stay with TrueNAS running inside ProxMox - doing just the NAS/ZFS stuff it’s really good at. All other tasks in other PM containers or VMs. That has been running close to four years now, shutting it down only for thunderstorms… 😅
The apps are hot grabage too. Upgrades end up locking up and failing to restart if you look at them sideways. Have to reboot server. I'll stick to my xcp-ng for virtualisation
You finding bugs in TrueNAS is just normal. I've never been able to use TrueNAS's GUI for an afternoon without running into something showstopping. It's also confusing because they explicitly recommend you use the GUI and not the console.
They recommend you use the GUI, until they don't, and then the documentation is... occasionally sparse. The situation really is that they don't recommend CLI for modifying settings that are in the GUI, for obvious reasons tbh, but if you never use the GUI, that's not going to be a problem either.
Yep. I had to drop to the command line to turn a single disk zpool into a mirrored zpool. So dumb. It's an easy job, but I would expect it to be an available task in the GUI
You tested TrueNAS's VM and found it lacking. I have been told that ProxMox can do NAS. Do you plan on doing a video for that, or have you already done that recently? It seems like it would be a good compliment to this video.... Thank you for the video, and thank you for your time.
Thanks one big aspect this review seemed to gloss over. You can say TrueNAS Scale's VM features suck, but Proxmox as a NAS OS sucks even more... unless you virtualize TrueNAS under Proxmox... which is then going against a number of experts that warn not to do that for long term production use... including TrueNAS themselves. If cost isn't a consideration, then seperate and isolate your NAS from your VM infra and have the best of both worlds. But most of us homelabbers need an all-in-one... and that means compromise in one direction or the other.
@@bw3388 I'm still learning in my homelab, so this isn't a snarky/rhetorical question, but why? Windows servers are generally always run in VMs now and perform fine when storage controllers are passed through to them. Why does TrueNAS not recommend going the same route?
@@maurvir3197 I'm not an expert, but I believe many have seen unstable vt-d (pci-x pass thru) on some motherboards, SMART data not making it to TrueNAS, something going wrong and having to deal with the VM aspects add more headaches, etc. It's also a bit hard to argue when Scale has basic virtualization and containers as part of it's base OS, so do you really need the complexity of adding in xcp-ng or proxmox as a base layer? Again, I don't claim to be an expert. But running Scale on bare iron with a few linux VMs and a few containers is working just fine for me for a few months now.
Great video Jeff, always informative and helpful, I would like to see you do a video on CEPH, on setting it up, running it, etc, and also your thoughts of course. Cheers keep up the great videos.
Well, I'm glad I watched this and was thinking of redoing my entire vm host OS, thinking TrueNAS would have been much more flushed out by now. Thanks, Jeff, for saving me hours of time.
I've been using it temporarily but once you are used to all the features/functions of esx/proxmox then this feels like stone age :D Better to run truenas as a vm with hdd passthrough and having a proper hypervisor.
How hard is it to do ZFS disk storage/configuration/permissions directly in Proxmox without setting up a truenas VM? Hate to have to allocate all those resources to a VM for something so basic to ZFS.
Thanks for suffering so i don't have to, Jeff! I'll be like you and stick with Proxmox while using TrueNAS in a VM. What a beer to pick for when this video went live. Maiden played at the Moda last night, what a great show!
I run truenas on bare metal and run containers instead of using it as a hypervisor. I have a beefier server running as a hypervisor instead, however, I can't wait for the switch from k8's to docker later this year, and getting access to containers and apps like Authentik and a better implementation of Tailscale. Right now, I'm just running NextCloud and my NGINX proxy manager with TrueNas and it seems to work really well.
OpenMediaVault with the Proxmox kernel, Docker and Compose work well for me as a NAS first, virtualisation second setup. Although, my virtualisation needs are modest.
i'm running TrueNAS scale as a one stop solution with two VMs. One for jellyfin and plex with an a310 passed through. the other one for docker and running a printserver with an usb pcie card passed through. afair the setup was a bit tricky and i ended up doing the passthrough config manual but aside from that all is running perfectly stable. i had the same problems using spice but it worked well eough to set up ssh.
Checked mine and the vnc password is definitely obfuscated as you type it in 24.04.2. and the character limit is not 8. What version are you using in this demo? I will admit using this as a hypervisor is not the greatest experience but I only have one or two machines that I need to run as VMs instead of containers and for me it OK enough.
Thanks for this video was debating between Truenas and Unraid for a new NAS build. I don't like the chunkiness of virtualizing truenas in proxmox and found that unraid may not give me the fastest read write speeds but does everything well enough in my lab in an easy setup without sacrificing a drive bay for an os.
Hey Jeff. Great video, as usual! I figured I'd chime in with my thoughts. Albeit, in a personal-use/small software dev env. TrueNAS's virtualization really is clunky. It's not refined and can be confusing with how some things are implemented (i.e. over-provisioning memory & such). I've also experienced most of the things you mention in the video. However, I really only run Linux virtuals to simulate a fail-safe database environment for software development purposes. That being said, VNC really isn't a thing for me. It's all pretty much SSH and if I need direct console access, I just use the Serial Shell utility in the VM's drop-down. I also haven't experienced too much gripe with GPU pass-through. The steps to set that up aren't very intuitive but once you do it, it's done. I can say that once you do get it all working and it's doing its thing, you really don't ever have to touch it again. This isn't to say that "it just works" because it really is finicky but, I haven't had any issues with it since I initially set it up. All this to say... TrueNAS is storage-centric. Proxmox is virtualization-centric. If TrueNAS wants to realistically compete with Proxmox's hypervisor capabilities and features, iX would most likely need to release a product with virtualization as its core. When people ask me about this type of stuff I tell them the same thing you mentioned in the video, choose the technologies you need to accomplish your goals. I'm a true believer that a storage solution should be a storage solution and a compute solution should be a compute solution. Mix-n-match to fit your use case.
Hey Jeff, On the Minisforum BDI790i... I'm assuming you can get to the m.2 wifi card and remove it? I thought that if we wanted to virtualize, say a OPNSense router... we could swap the wi-fi card out for a second 2.5gig nic, say a m.2 2230 2.5gig A+E key and then just install the nic port in one of the case slots. Thoughts?
I have used truenas for hosting a couple of headless Linux VMs when doing some full environment dev but didn't want it on my other machines. And I totally agree, it's a flawed implementation of a hypervisor that's trying to be vmware esxi but doesn't know how to.
Trooper is an awesome all around pub food beer. Goes great with bar pizza, tacos, fish & chips, the whole gamut. Just an easy drinking, smooth, British lager as you said.
I immediately saw the Iron Maiden "Trooper" beer on the desk and cannot sufficiently express my enthusiasm for the band. Saw them recently in Melbourne and tried the beer at a pop up bar here near the venue too. Also recently got a "Powerslave" tattoo on my forearm which has been on the wishlist for a long time.
Thank you for calling out these issues. I actually do use Truenas Scale for virtualization but I have definitely found it as buggy. I will say that I think it is a bug that your password is being shown. It is definitely hidden when I do it. The Spice console is definitely infuriating though.
What about running truenas (core or scale) as storage and separate server with proxmox as your primary VM? What would be your network requirements to make that workable?
It is stated that there is a cap on ZFS cache space with RAM in virtualized TrueNAS but the limit is not stated. As I have been dabbling with Proxmox on an i3-7100 dual core with 16GB of memory (yes it is brutal) - while my NAS is a 16 core X99 with 256GB of registered ECC, completely overkill for just a home NAS. Oh and for giggles - it's got a 25gig connectx-4 as does my desktop. I cannot saturate the 25gig link but I definitely can go past 10gig. That is as long as the file is in the cache already. Something I'd also like to understand more about is iSCSI and how it interacts with the ZFS cache. Does the entire iSCSI volume need to fit in the ZFS cache or can it reference parts of it, ect.
I can agree, TrueNAS as seems a bit lacking for a full hypervisor. I run XCP-NG but host a few vms on TrueNAS to run some automations against my actually hypervisor but I am anxiously awaiting the docker update for TrueNAS to retire those vms.
TrueNAS for storage, hosting Proxmox as a VM, hosting all services on SSD with pinned CPU threads in TrueNAS. Seems to be fine, but I know I could use some different hardware to push the limits.
To add detail: i5-10400 64GB DDR4 3200 ZFS On TrueNAS: 2 WD Gold 10TB (Deep long term) Mirror 2 WD Blue 8 TB (Not as important) Mirror 1 Barrracuda 6 TB (Downloads and ISO's) Span 1 Samsung EVO 1 TB Span Running: Proxmox as VM Jellyfin as App Damselfly as App PiHole as 2nd DNS App Proxmox is running on a few pinned CPU threads, allocated 32gbs of memory, CPU Passthrough. It is serving: PiHole as Primary DNS Wireguard Mumble Jitsi Subsonic I have some issues with stuttering video which isn't easy to track back. Maybe this is related to how fast the data is retrieved, relayed. However, this doesn't happen all the time. Otherwise, the system is stable and serves NFS/SMB at NIC Speed.
Thanks for this Jeff, I played with VM's on Scale (no experience with any other hypervisors) and found it laggy and buggy, thought it was my hardware or my lack of competence. I'm wondering if the Debian+KVM is the easiest to use or if Proxmox is a better bet? For storage I've been very comfortable on Core for about 4 years now, while I do have a VM with that, it's only for TailScale. VM's are a curiosity, not so much a necessity! 🙂
Great vid thanks. I was wrestling with this exact question for a new server. TrueNAS VM on Proxmox or TrueNAS as the host with other stuff running in VMs/containers.
Thanks for the very straightforward and honest review Jeff. It's good to know that there are some areas TrueNAS shines in, but others where you're probably better off with some other solution. On the 'British beers' front (Scots and Orcadians are probably going to hate me for lumping them in that category), but have you tried any from the Orkney Brewery? I've enjoyed Puffin and Dark Islands, and just found out that their website tells you the hops and malts used, as well as where it sits on the bitter to sweetness scale.
In regards to the VNC Password, when setup during the wizard with a 16 character password, did you try to just use the first 8 characters later on to access the VNC session. I wonder if the wizard just omits the last 8 characters and the Password field also only uses the first 8 characters. I have seen this happen on other products that act like you mentioned it.
I feel perhaps silly asking this, why do you not use the integrated graphics on the CPU to do transcoding? Is it because you can’t pass that to a VM like you can with a dGPU? Is it because of software support, or performance?
As much as I love TrueNAS this is pretty much the conclusion I've come to as far as using it as a VM host goes. I run TrueNAS Core on bare metal on one box and Proxmox on another and have been pretty happy with that setup for some time now. That said I'm pretty sure VLAN's do work and the new version of TrueNAS Scale should be coming soon and should address some of the issues you saw.
Timely video. Just set up a TNScale and had the exact same experience. I thought I'd just made all the mistakes possible: I'm now vindicated. As a NAS, works great. I'll keep my Proxmox for VMs for now. Will see what the next update holds with reworking containerization.
I would love a similar one about the new Docker support in Scale. Like : can TrueNas replace Debian as a docker host now? Always wanted to bring my containers closer to the storage.
I currently use Truenas CORE, and soon looking to switch to Proxmox - Truenas for me was easy and a first step onto the ladder but I experienced odd issues with VNC (noVNC) for VMs where it wouldn't always work or pass through the correct key commands according to my keyboard layout. Other strange issues were that with statistics graphs for quite some time only worked in Firefox but that was eventually resolved.
I too found TN's virtualization problematic, so went with proxmox. You talked about virtualizing TrueNas in a VM under ProxMox. how well does it handle serving a datastore from TN to a separate VM running docker containers?
Once ran a VM (simple CentOS installation) to serve as a Plex front end -- No hardware decoding. It worked without a fuzz, but it hadn't that much work to do. I set up the VM with auto patching and left it running. The only thing special I had going on was a an in-VM mount of my multimedia datapool.
Been running Truenas Scale since Angelfish, It has been a learning experience thats for sure. While, yes yes you can attach a VLAN to a VM as others have mentioned here. It is not super intuitive in any capacity. I have also experienced most of the issues you did as well especially with SPICE VNC. But I have successfully gotten my Homeassistant OS Supervised in a stable VM setup (Not the recommended installation method btw). I am running most of my services through the apps available in their catalog though. Jellyfin, Octoprint, Frigate, OLlama, Pi Hole and Tailscale. All stable and working pretty well. Edit: btw Im a welder, I dont code. Just a tech nerd and have most of the wonderful Truenas community to blame? for my success. Lol
Having had truenas apps drop out on me while migrating and boot issues on updates to scale in the past I’m 100% back to running truenas inside proxmox for remote management still being available in maintenance
I am running truenas scale on bare metal with a win10 vm that has been rock solid for at least a year. Nothing crazy, but I can’t complain. It does the one small task I need it to do
I currently have an old Dell R710 as a TrueNAS server. My "main" system that I was using for ProxMox is currently not running as I plan on transplanting the innards to a 1U rack. I plan on using an iSCSI disk on the Dell to store VM's in ProxMox. I would also like to know how to get some of the beer you drink! :)
Hoi Jeff , I am following your videos running proxmox myself what kind off 10 gb nics can you recommend running 8.2 are the intel x540 suitable please advice greetings from the netherlands
I've noticed that with the VNC window that if it is left alone with no input for 3-5min (even with the window still active) it will become unresponsive to your inputs. It seems to be some sort of timeout when there is no input from the user.
I've tried it before but ended up with a small Lenovo PC as a dedicated virtualization host and a Ugreen NAS as a TrueNAS Scale NAS. The only service I run on TrueNAS now other than the file services themselves is Plex out of laziness and for the direct drive access. I would consider TrueNAS with VMs for small "deployments" at for example a family members place where the primary need is storage and where containers and VMs are just an added bonus on top.
would be great to see if you would have same issues with nvme and MCIO connector which generally are horizontally placed rather than vertically like the one you failed with
you can get m.2 4x pcie risers - you put a normal lsi 9400-8i or whatever into a physical pcie slot on the case (that has no slot), you run the riser between the card edge connector to the m.2 slot.
The issue is that, if I remember correctly, the HL-8 only supports mini itx motherboards and consequently only has one PCIe slot, so even if you were to use a riser you still need a place to mount it
I run TrueNAS as the "bare metal" OS and I've used it for a couple of light virtualization tasks--and it's always worked perfectly fine for me, albeit a bit limited. BUT, this was the older "Bhyve" virtualization stack. I've never tried the newer KVM-based stack.
I currently have a TrueNAS scale server acting as the KVM at a church I attend. They have very basic needs when it comes to a hypervisor, but it is working and has not really had any issues.
Tried TrueNAS as a VM host about a year ago and it went pretty bad. It was a great NAS, but yeah VMs were really unstable. I decided to just use two machines - one with Proxmox for VMs and a separate TrueNAS machine. It's been great so far, and I would definitely recommend that over virtualizing TrueNAS if you can. I also use the NAS for VM backups with rsync as a CRON job for offsite copies.
Honestly, I'll never run anything except purely ephemeral workloads on trueNAS ever since an update wiped everyones data. Sure, we had backups, but this kind of breach of trust takes over 10 years to earn back in my book.
Hardware Haven did a video recently on updating his HP SFF server with modernish hardware. In that video he too was using m.2 to sas adapter cards but his had the ports lying horizontal on the card itself and not vertical like the card you tried
im using scale mostly for apps works pretty well for me. I only have one actual VM though with 1 measly gig of ram and its running home assistant. had no issues. but also no real demands.
I saw this hoping you'd say TrueNAS Scale was great, but apparently not! Thank you for the details, as your complaints, particularly around networking and pci pass-through, are directly applicable to my use cases too. I have a supermicro nas box I was weighing using proxmox and loading truenas into as I do this with my firewall today, but wasn't sure how pci pass-through would behave for passing the sata controller for the 4x spinners on it. I saw this thinking maybe I wouldn't have to, but as you said proxmox base with truenas in vm sounds like the way to go. Thanks again!
Given the way THIS server is configured, it isn't really an option to run Proxmox and then run TrueNAS as a VM (which would require PCIe passing through the HBA for disk health monitoring - which is a critical requirement for a NAS, not ECC memory). One critical thing to note is that TrueNAS runs ZFS and even in 2024, there is still a bug with ZFS that prevents proper isolcpu functionalities (i.e. reserving some CPU cores for VM and preventing ZFS from using those cores doesn't really work). So running a VM on TrueNAS means you WILL run into high latency during high IO because ZFS will use whatever cores it wants, at any time it wants. This happens EVEN WHEN your VM exclusively uses passed-through NVMe drives. Running TrueNAS VM under Proxmox will mitigate this bug, at least to some extent. Under high IO, your isolated VM will 100% still be isolated from TrueNAS. There will still be some latency if your Proxmox storage is also on ZFS but presumably that would be on a fast NVMe (or at least a SATA SSD) so it will be order of magnitude less than when your VM having to wait for TrueNAS ZFS having to wait for a HDD to respond.
Pretty much nailed it. TrueNAS is fine as a storage host and not much else. I pretty much do what you suggest. Proxmox host, TrueNAS VM and other VMs doing fun things inside proxmox.
I've run VMs on Truenas Scale, they were a pain to set up in earlier versions, but its been getting slowly better. I never tried to run anything that needed passthrough. I ran Proxmox Backup server on it for a long time as it felt better to me to have the backup VM running on a storage machine.
i have been running a dietpi (running pihole) VM and an Mint build flawlessly on boot for over a year. im wondering if you are having issues with the actual CPU config, are you doing passthrough. password obfuscated for me, yes, the focus loss is annoying, thats why i run other remote SW for longer duration stuff. i do need to test the passthrough
I use both setups: 1. Main server: Proxmox is the hypervisor, and TrueNAS is a VM with passed-through storage. I have ~10 VMs in Proxmox. 2. Secondary server for backups only: TrueNAS is the primary OS, and only one VM - Proxmox Backup Server. Why? TrueNAS is easy to recover/migrate to a new server, so I always can restore it w/the Proxmox Backup Server. From here, I can restore my Primary server.
I also run Proxmox with Truenas Scale as a VM with the HBA passed through. I wish I could have a simpler stack, but after managing VMs with Proxmox I won't even try using Truenas for that with its current UI and features. I am looking forward to Electric Eel to hopefully simplify my containers. I'm really tired of Proxmox Hypervisor -> TrueNAS VM -> NFS/SMB to 2nd VM running Docker. That's not even considering the downsides of this, like no inotify for shared folders for Syncthing and Jellyfin.
i experienced all of the issues he mentions. I knew i was in for a rough time when Spice K&M was barely functional, but the pcie gpu passthrough that just ended up not working properly did it for me. Just looking to have my NAS as its own separate box at this point
And all this time I thought it was just my inexperience! I gave up on VM's in TrueNAS because I assumed I was using a mature system with a proven wizard, and I was just too dense to figure it out. Thanks for clarifying the situation, and at the same time making me feel a little less like an idiot!
This turned out wonderfully, though, part of me was hoping for more hackery, like a PCI switch or something. I suppose actually having it work and be reliable is... nice, so, 45D's solution is no doubt better. That said you could probably find an M.2 card that'd support the SFF cables well enough, likely with right angle cables. Like you said though, more than enough for the spinners you'll be using. If you wanted all flash... well, there are likely better options there anyway.
It's one of the reasons why I run TrueNAS on a dedicated hardware strictly for storage as it was made for in the first place. I am running Core and will soon move over to Scale. I use ProxMox for everything else.
Good video. I would revisit this with TrueNas Scale 24.10 as that version is getting some major changes including doing away with kubernetes and moving to the adopted docker app system along with other changes. It may be better with the changes being made. I moved to TrueNas Scale from Unraid as I was building my server cabinet out and wanted a little more freedom. I was comp templating the change partially based on the upcoming 24.10 update and working on a steam/game cache that worked better.
The truenas VM situation is painful. I have switched everything to docker containers (using docker compose) in a jail. The next version (electric eel) is supposed to integrate docker compose functionality within the main OS. However, my home lab is still in development and I may eventually need a VM. Hopefully things will have improved by then, but if not I may have to go promox with a truenas VM. Electric eel me be addressing some of the VM issues, but if so I have not been following them.
I am currently runing true nas in a vm with sata controler passthrough and 10GB Virtual function network adapter and it take me a day to find out that first: cfgbev driver is not loaded bvy default and even if I load it you still can not connect using tcp because of broken check sumes which is realy hard to find out since for example you have pin but no tcp connection.
Also tried trunas scale Hypervisor, ran into similar issues, one time a vm would not stop, in no way could i delete it, stop it or acces it, had to force reboot Truenas to stop it. after that i just done the proxmox as a Hypervisor and truenas as a vm thing and that worked great
i recently built a pc, to run TrueNAS and then stumbled across Proxmox. as im new to both, i've been swapping between the 2 as the main installed OS and learning as i go along. I like the flexibility of Proxmox to run different VM's and containers, but having to passthrough the hhd's for TrueNAS to use and then not being able to take snapshots (proxmox errors out), is a bit of a fail. but then when TrueNAS is the OS, running apps and VM's, can be a right pain to set up and get running. if only the there was a way to dual boot Proxmox to run the VM's etc and TrueNAS as a dedicated NAS as the same time.
I guess I look at truenas to do one thing for me: host storage and the services directly related to storage. It isn't what I'd look for to be my virtual host. Would I like that? Sure, I could have a couple use cases for that. But I wouldn't *trust* it for that even without the problems you encountered. I trust it for storage.
thx for the video ! for me VMs only Proxmox , and File Sharing etc with TrueNAS. i had many problems with vms and apps in TrueNAS Scale , and on the latest version many more.
Right off the bat my number one hangup with hosting VMs on TrueNAS Scale at home is that it limits the hardware I can use. I frequently build these homelab systems from older parts, and TrueNAS doesn't let me run older drivers when needed for compatibility. At least not without a lot of extra effort. Love TrueNAS Scale for many things, but not as a primary VM host.
I have had nothing but problems trying to get a GPU passed through properly on this platform. The NAS function works great as well as the kubernetes feature.
I like to keep copies of my VMs on alternate servers (rsync) so if my main server dies I can start it on a secondary server, this was impossible on Truenas when I tried it about a year ago. I run an Unraid server and a Proxmox server (running Truenas in a vm solely for NAS purposes). I can rsync VMs back and forth no problem because they are stored in a standard folder in both OSs.
About the instability, maybe it is because you used ballooning and passthrough? On proxmox as far as I understand ballooning doesn't work with passthrough, because it requires DMA. Or maybe it works on trueNAS somehow?
I ran my homelab rig on Scale for a week. It was the longest month of my life.
I wish them well and hope they can solve it, but I've stuck with Debian+KVM for good reason. Everything just works for me there.
When was this? Were the issues mostly revolving around the older apps system they're now replacing with proper docker in the next release?
I don't have any pcie passthrough for my vm's, but I do have a ton of things running and its been great. It even runs my ansi text environment linux shell host for retro terminals lol.
@@KyleMahaney Timeline was about a year ago. And I didn't use their app system, I mainly run VMs. At the time I had no need to test containers.
@@VeronicaExplains my impression has been. TrueNas for storage and docker apps, and the occasional VM work. Pretty much the same with UNRAID. But serious VM work and containers should be proxmox, or other hypervisor, with VM running docker apps. ONly have the basics myself but that is just what I keep seeing from people.
Debian/Qemu/KVM is about as bulletproof as it comes to be honest. You can also do that without proxmox at all, if you're cool with managing everything via CLI (or some other, inferior gui).
That said, you could probably Python your way to a decent enough gui without much trouble if you're just looking to do some specific things.
Was curious on the topic, ended up running TrueNAS on Proxmox, have to admit it feels a bit iffy should things go south.
First try and all, both on VM and NAS 🤣
In short: No.
Long: NooOoOOOoOoooOooo!
Hahahahahah!!!
This is what I came to say. Truenas is great for storage but awful at just about everything else.
Twenty One Long Minutes Later... :) Ok Twenty Long Minutes the two on beer was useful.
A+; no notes.
You can pry Proxmox from my cold dead fingers LMAO
I also have found some uses for Hyper-V depending on hardware and Hardware needs
Truth
Challenge accepted.
It has a GPLv3 Affero license, be sure to set up a mirror of the git repos, just in case.
While the license is a great choice to keep it open, BUT the project does demand a Contributor License Agreement, which is a big red flag.
For VM's it's the beesknees.... but storage and dockers are cantankerous AF.
VLAN support for VMs is there since some years. You create a VLAN in the network section and then attach that to the VM in NIC settings. It works fine for me.
noVNC was fully replaced by Spice due to security issues in the VNC protocol, but the usability is still bad. However, using virt-viewer should work much better compared to the web client.
TrueNAS 24.10-RC.2 is already out and the full release of 24.10 will be out by the end of the month. There are multiple fixes in that version. For example:
The Spice VM password is correctly hidden when entering it on 24.10. The 8 length password limitation issue is also resolved.
Also, for the disk latency test, was the virtual disk passed through via AHCI or VirtIO? 24.10 might also improve that, but I haven't tested it. Just a guess.
9:23 he used AHCI, that's probably the issue here
I used virtIO and my single NVME drive VDEV performance is TERRIBLE
I've been running TrueNas Scale as my sole Homeserver/NAS/Hypervisor for over half a year now. It has worked perfectly fine for running a Reverse Proxy VM, my Nextcloud VM (+ related services), a VM for Jellyfin and Handbrake (with GPU passthrough) along with the occasional Test VM for random stuff i come across.
I've standardized on OpenSUSE MicroOS (which is extremely easy to provision in a wide variety of different hosting environments) and the only thing I found lacking were the janky VM Display viewer and Snapshots not including the VM configuration.
I tested both Proxmox and XCP-ng for about 6 months each before (and even some more exotic solutions like a Hyperconverged Ceph cluster based on RHEL derivatives) and switched over to Scale, because I just didn't need the extra features. For me the better functionality and UI for handeling Storage (Block device and ZFS options), User management and Share Permissions along with the Performance and Efficiency increase, as well as the complexity decrease of not messing around with a virtualized NAS appliance was worth the usability tradeoff in the Virtualization department.
I agree. I just built a new home server with the latest release of TrueNAS Scale and I'm perfectly fine with the results. I have a few containers running, and a couple linux VMs... nothing super complicated, but it all just works. Yes, the Spice VRT sucks and is the one thing I'd change... nothing is perfect.
I looked at Proxmox & xcp-ng and favored the latter, but in the end, wanted my NAS OS to have direct PCIx access to all drives without question. Yes, you can virtualize and pass the HBA through, but many experts including TrueNAS state to not virtualize for production use cases.
TrueNAS Scale is not the best pick for enterprise solutions IMO. But for a small, single, do-everything home server... I have no complaints. If I was doing this at work, sure, I'd buy 3-6 hosts, break it all apart with clusters, etc... but for home where energy consumption is factor - no thanks.
I also have truenas as my sole homeserver.
it runs 4 vm
1 homeassistaint
2 win 11 vm to remote in and mess around when i am at work
3 xubuntu for the same reason
4 ubuntu server runin docker instances (after wendell's level1tech guide), bc truenas built in docker solution is barbone, for my 20+ docker image setup.
i soent 10 minutes to create the vm-s i never touched that part of truenas since, it runs for me falwlessly
I am running 2 Proxmox machines and was looking for a way to run a Proxmox Backup Server and a NAS. When I got an older Server for free I decided to run TrueNas bare metal and virtualize PBS on top of it, since the NAS performance was more critical to me than that of the backup server.
It works, but TrueNas is nowhere near Proxmox as a Hypervisor. I felt restricted in a lot of things I wanted to do.
So, for me Proxmox is my go-to for hypervisors and TrueNas Scale my favorite NAS solution. Both great in their area of expertise but no competition for each other.
That's the way I'm running things now and was hoping to migrate my Proxmox VMs over to TrueNas Scale. This video (and the se comments) have pretty much convinced me not to change anything as everything is working reliably as it is. Thanks for all the confirmation!!!!!
I would be really interested if you could retry this experiment with the Electric Eel version of TrueNAS.
I was hoping he used EE, it had some noticeable (according to patch notes) improvements
I really wanted to see the as well now that its in the RC stage. I think that its a good place to start as home lab for testing as long as its not mission critical services.
I am running EE, Spice is still hot garbage.
Unfortunately, still a mess on the UI side... rsync module is also broken when trying to use as a Synology backup target - works fine in 24.04.2.3
I have a proxmox cluster for my compute and a truenas server that's just for storage. I've considered running some docker containers in truenas once that feature is released, but never did any VMs there.
Should I think about virtualizing truenas and putting proxmox on it? 🤔🤔
@@ravenssettle I've done that to run some LXC's before, and to help make sure I'll always have a quorum incase any of my other nodes go down. Never had any problems running it that way. It ran fine with 128GB of storage and 8GB of RAM.
That was the previously recommended way to do it actually, truenas for pooled, shared storage, proxmox for compute. Nowadays you lose so little by virtualizing that it makes sense to just keep it all under one roof.
Also, it's very likely you can just throw proxmox on your truenas host instead, and then pass through your HBA's to a truenas VM and import your pools and be back in action with very little nonsense. You can probably even get away with backing up your truenas config and just importing it, assuming the versions match.
@@jttech44 You can 100% flip-flop from a VM to bare metal install by donwloading/uploading the config file! I've done it a few times in my testing and TrueNAS is none the wiser to the changes.
So.. time for a new and updated Proxmox guide? :)
EDIT: It should be noted that i am a huge fan of Jeffs guides, i have spend many hours learning and installing stuff, that i don't really need, but just for the sake of it. I mean, nerdy and all.
But it's been a while since he took our hand and walked us through a complete Proxmox guide, with window install, Truenas Setup, GPU pass-through on a consumer card, backup etc etc..
Yeah i know, months of content/work.. sorry Jeff
Please
I second this motion.
I third this motion
Yeah, all the people leaving VMware need help!
*In Darth Sideous' voice:* DO IT.
In my opinion, if you run truenas bare metal and run vms in it it's like using an ISP router that is wifi/switch/toaster, in my opinion you only run truenas for a NAS software. And a hypervisor as a hypervisor.And you can run truenas as a VM too.
The problem with that is that many systems these days leave a lot of power and utility unused in that scenario.
Assuming you can pass thru your HDD controller yes. TrueNAS is a very basic hypervisor but it is there.
Thank you Peter! One should use the right tool for the job...
"Trying to outsmart the room" is the modus operandi of this entire channel :P
I'm running TrueNas Scale and I'm running a basic debian VM as an application host. It's only running docker with a dozen containers. Works fine for that!
Same here, I’m running 2 VMs one for Docker and another for WordPress. Runs smoothly!
I use truenas scale to host my caddy based reverse proxy VM. It is very simple and I can rebuild it in just a few minutes if I have a problem. I would never host any serious VM on truenas. The hypervisor feature is just not finished and not trustworthy. I have had much better success with the apps feature. I am super excited to have that switch to docker in this next release which will make it even easier to use for me.
Once I tried running Truenas Scale as you, because in the store I could install docker containers such us jellyfin, and Radarr... File permissions were too confusing and I ended installing a VM instead of using those apps
I've found myself not using a VM these days. The jailmaker script works really well for making systemd-nspawn containers and you can choose from a pretty good list of LXC images.
Yep, I’ll stay with TrueNAS running inside ProxMox - doing just the NAS/ZFS stuff it’s really good at. All other tasks in other PM containers or VMs.
That has been running close to four years now, shutting it down only for thunderstorms… 😅
The apps are hot grabage too. Upgrades end up locking up and failing to restart if you look at them sideways. Have to reboot server. I'll stick to my xcp-ng for virtualisation
You finding bugs in TrueNAS is just normal. I've never been able to use TrueNAS's GUI for an afternoon without running into something showstopping. It's also confusing because they explicitly recommend you use the GUI and not the console.
They recommend you use the GUI, until they don't, and then the documentation is... occasionally sparse.
The situation really is that they don't recommend CLI for modifying settings that are in the GUI, for obvious reasons tbh, but if you never use the GUI, that's not going to be a problem either.
Yep. I had to drop to the command line to turn a single disk zpool into a mirrored zpool. So dumb. It's an easy job, but I would expect it to be an available task in the GUI
how about open media vault?
You tested TrueNAS's VM and found it lacking. I have been told that ProxMox can do NAS. Do you plan on doing a video for that, or have you already done that recently? It seems like it would be a good compliment to this video....
Thank you for the video, and thank you for your time.
Thanks one big aspect this review seemed to gloss over. You can say TrueNAS Scale's VM features suck, but Proxmox as a NAS OS sucks even more... unless you virtualize TrueNAS under Proxmox... which is then going against a number of experts that warn not to do that for long term production use... including TrueNAS themselves.
If cost isn't a consideration, then seperate and isolate your NAS from your VM infra and have the best of both worlds. But most of us homelabbers need an all-in-one... and that means compromise in one direction or the other.
@@bw3388 I'm still learning in my homelab, so this isn't a snarky/rhetorical question, but why? Windows servers are generally always run in VMs now and perform fine when storage controllers are passed through to them. Why does TrueNAS not recommend going the same route?
@@bw3388 From what I can tell ixsystems says it's fine to run TrueNAS in a VM, as long as you pass through the entire storage controller.
@@maurvir3197 I'm not an expert, but I believe many have seen unstable vt-d (pci-x pass thru) on some motherboards, SMART data not making it to TrueNAS, something going wrong and having to deal with the VM aspects add more headaches, etc.
It's also a bit hard to argue when Scale has basic virtualization and containers as part of it's base OS, so do you really need the complexity of adding in xcp-ng or proxmox as a base layer?
Again, I don't claim to be an expert. But running Scale on bare iron with a few linux VMs and a few containers is working just fine for me for a few months now.
@@bw3388 TRUE SAY
The timing of this video is impeccable, thanks!
Great video Jeff, always informative and helpful, I would like to see you do a video on CEPH, on setting it up, running it, etc, and also your thoughts of course. Cheers keep up the great videos.
Well, I'm glad I watched this and was thinking of redoing my entire vm host OS, thinking TrueNAS would have been much more flushed out by now. Thanks, Jeff, for saving me hours of time.
I've been using it temporarily but once you are used to all the features/functions of esx/proxmox then this feels like stone age :D
Better to run truenas as a vm with hdd passthrough and having a proper hypervisor.
How hard is it to do ZFS disk storage/configuration/permissions directly in Proxmox without setting up a truenas VM? Hate to have to allocate all those resources to a VM for something so basic to ZFS.
@@marcosv8426 honestly don't know since i've never yet used proxmox, only vsphere/esx but am going to delve in it this fall when I get a new server
I would be interested to see if when you use Proxmox as hypervisor and do the truenas-VM thing, is the performance of the VM the same as in your test?
Thanks for suffering so i don't have to, Jeff! I'll be like you and stick with Proxmox while using TrueNAS in a VM. What a beer to pick for when this video went live. Maiden played at the Moda last night, what a great show!
I run truenas on bare metal and run containers instead of using it as a hypervisor. I have a beefier server running as a hypervisor instead, however, I can't wait for the switch from k8's to docker later this year, and getting access to containers and apps like Authentik and a better implementation of Tailscale. Right now, I'm just running NextCloud and my NGINX proxy manager with TrueNas and it seems to work really well.
OpenMediaVault with the Proxmox kernel, Docker and Compose work well for me as a NAS first, virtualisation second setup. Although, my virtualisation needs are modest.
Also what I run. I even installed cockpit for better control of VMs. OpenMediaVault is underrated.
@@Boburto Agreed, OMV has been doing the Debian based NAS for a lot longer than iX Systems. I have been using it for more than a decade now.
i'm running TrueNAS scale as a one stop solution with two VMs. One for jellyfin and plex with an a310 passed through. the other one for docker and running a printserver with an usb pcie card passed through. afair the setup was a bit tricky and i ended up doing the passthrough config manual but aside from that all is running perfectly stable. i had the same problems using spice but it worked well eough to set up ssh.
Checked mine and the vnc password is definitely obfuscated as you type it in 24.04.2. and the character limit is not 8.
What version are you using in this demo?
I will admit using this as a hypervisor is not the greatest experience but I only have one or two machines that I need to run as VMs instead of containers and for me it OK enough.
Thanks for this video was debating between Truenas and Unraid for a new NAS build. I don't like the chunkiness of virtualizing truenas in proxmox and found that unraid may not give me the fastest read write speeds but does everything well enough in my lab in an easy setup without sacrificing a drive bay for an os.
Hey Jeff. Great video, as usual! I figured I'd chime in with my thoughts. Albeit, in a personal-use/small software dev env.
TrueNAS's virtualization really is clunky. It's not refined and can be confusing with how some things are implemented (i.e. over-provisioning memory & such). I've also experienced most of the things you mention in the video. However, I really only run Linux virtuals to simulate a fail-safe database environment for software development purposes. That being said, VNC really isn't a thing for me. It's all pretty much SSH and if I need direct console access, I just use the Serial Shell utility in the VM's drop-down. I also haven't experienced too much gripe with GPU pass-through. The steps to set that up aren't very intuitive but once you do it, it's done. I can say that once you do get it all working and it's doing its thing, you really don't ever have to touch it again. This isn't to say that "it just works" because it really is finicky but, I haven't had any issues with it since I initially set it up.
All this to say...
TrueNAS is storage-centric. Proxmox is virtualization-centric. If TrueNAS wants to realistically compete with Proxmox's hypervisor capabilities and features, iX would most likely need to release a product with virtualization as its core. When people ask me about this type of stuff I tell them the same thing you mentioned in the video, choose the technologies you need to accomplish your goals. I'm a true believer that a storage solution should be a storage solution and a compute solution should be a compute solution. Mix-n-match to fit your use case.
Hey Jeff, On the Minisforum BDI790i... I'm assuming you can get to the m.2 wifi card and remove it? I thought that if we wanted to virtualize, say a OPNSense router... we could swap the wi-fi card out for a second 2.5gig nic, say a m.2 2230 2.5gig A+E key and then just install the nic port in one of the case slots. Thoughts?
I have used truenas for hosting a couple of headless Linux VMs when doing some full environment dev but didn't want it on my other machines. And I totally agree, it's a flawed implementation of a hypervisor that's trying to be vmware esxi but doesn't know how to.
Trooper is an awesome all around pub food beer. Goes great with bar pizza, tacos, fish & chips, the whole gamut. Just an easy drinking, smooth, British lager as you said.
I immediately saw the Iron Maiden "Trooper" beer on the desk and cannot sufficiently express my enthusiasm for the band. Saw them recently in Melbourne and tried the beer at a pop up bar here near the venue too. Also recently got a "Powerslave" tattoo on my forearm which has been on the wishlist for a long time.
A true fan !
Thank you for calling out these issues. I actually do use Truenas Scale for virtualization but I have definitely found it as buggy. I will say that I think it is a bug that your password is being shown. It is definitely hidden when I do it. The Spice console is definitely infuriating though.
What about running truenas (core or scale) as storage and separate server with proxmox as your primary VM? What would be your network requirements to make that workable?
It is stated that there is a cap on ZFS cache space with RAM in virtualized TrueNAS but the limit is not stated. As I have been dabbling with Proxmox on an i3-7100 dual core with 16GB of memory (yes it is brutal) - while my NAS is a 16 core X99 with 256GB of registered ECC, completely overkill for just a home NAS. Oh and for giggles - it's got a 25gig connectx-4 as does my desktop. I cannot saturate the 25gig link but I definitely can go past 10gig. That is as long as the file is in the cache already. Something I'd also like to understand more about is iSCSI and how it interacts with the ZFS cache. Does the entire iSCSI volume need to fit in the ZFS cache or can it reference parts of it, ect.
I can agree, TrueNAS as seems a bit lacking for a full hypervisor. I run XCP-NG but host a few vms on TrueNAS to run some automations against my actually hypervisor but I am anxiously awaiting the docker update for TrueNAS to retire those vms.
TrueNAS for storage, hosting Proxmox as a VM, hosting all services on SSD with pinned CPU threads in TrueNAS. Seems to be fine, but I know I could use some different hardware to push the limits.
To add detail:
i5-10400
64GB DDR4 3200
ZFS On TrueNAS:
2 WD Gold 10TB (Deep long term) Mirror
2 WD Blue 8 TB (Not as important) Mirror
1 Barrracuda 6 TB (Downloads and ISO's) Span
1 Samsung EVO 1 TB Span
Running:
Proxmox as VM
Jellyfin as App
Damselfly as App
PiHole as 2nd DNS App
Proxmox is running on a few pinned CPU threads, allocated 32gbs of memory, CPU Passthrough.
It is serving:
PiHole as Primary DNS
Wireguard
Mumble
Jitsi
Subsonic
I have some issues with stuttering video which isn't easy to track back. Maybe this is related to how fast the data is retrieved, relayed. However, this doesn't happen all the time. Otherwise, the system is stable and serves NFS/SMB at NIC Speed.
All of these problems are exactly why im keeping truenas strictly as a NAS and switching all hypervisor tasks over to proxmox.
Thanks for this Jeff, I played with VM's on Scale (no experience with any other hypervisors) and found it laggy and buggy, thought it was my hardware or my lack of competence. I'm wondering if the Debian+KVM is the easiest to use or if Proxmox is a better bet? For storage I've been very comfortable on Core for about 4 years now, while I do have a VM with that, it's only for TailScale. VM's are a curiosity, not so much a necessity! 🙂
Great vid thanks. I was wrestling with this exact question for a new server. TrueNAS VM on Proxmox or TrueNAS as the host with other stuff running in VMs/containers.
Thanks for the very straightforward and honest review Jeff. It's good to know that there are some areas TrueNAS shines in, but others where you're probably better off with some other solution.
On the 'British beers' front (Scots and Orcadians are probably going to hate me for lumping them in that category), but have you tried any from the Orkney Brewery? I've enjoyed Puffin and Dark Islands, and just found out that their website tells you the hops and malts used, as well as where it sits on the bitter to sweetness scale.
In regards to the VNC Password, when setup during the wizard with a 16 character password, did you try to just use the first 8 characters later on to access the VNC session. I wonder if the wizard just omits the last 8 characters and the Password field also only uses the first 8 characters. I have seen this happen on other products that act like you mentioned it.
I feel perhaps silly asking this, why do you not use the integrated graphics on the CPU to do transcoding? Is it because you can’t pass that to a VM like you can with a dGPU? Is it because of software support, or performance?
As much as I love TrueNAS this is pretty much the conclusion I've come to as far as using it as a VM host goes. I run TrueNAS Core on bare metal on one box and Proxmox on another and have been pretty happy with that setup for some time now. That said I'm pretty sure VLAN's do work and the new version of TrueNAS Scale should be coming soon and should address some of the issues you saw.
Timely video. Just set up a TNScale and had the exact same experience. I thought I'd just made all the mistakes possible: I'm now vindicated. As a NAS, works great. I'll keep my Proxmox for VMs for now. Will see what the next update holds with reworking containerization.
I would love a similar one about the new Docker support in Scale. Like : can TrueNas replace Debian as a docker host now? Always wanted to bring my containers closer to the storage.
I currently use Truenas CORE, and soon looking to switch to Proxmox - Truenas for me was easy and a first step onto the ladder but I experienced odd issues with VNC (noVNC) for VMs where it wouldn't always work or pass through the correct key commands according to my keyboard layout.
Other strange issues were that with statistics graphs for quite some time only worked in Firefox but that was eventually resolved.
I too found TN's virtualization problematic, so went with proxmox. You talked about virtualizing TrueNas in a VM under ProxMox. how well does it handle serving a datastore from TN to a separate VM running docker containers?
Great update. How would you have configured if you did not need the additional Intel GPU card?
Once ran a VM (simple CentOS installation) to serve as a Plex front end -- No hardware decoding. It worked without a fuzz, but it hadn't that much work to do. I set up the VM with auto patching and left it running.
The only thing special I had going on was a an in-VM mount of my multimedia datapool.
@CraftComputing I'm curious as to which version did you use?
Been running Truenas Scale since Angelfish, It has been a learning experience thats for sure. While, yes yes you can attach a VLAN to a VM as others have mentioned here. It is not super intuitive in any capacity. I have also experienced most of the issues you did as well especially with SPICE VNC. But I have successfully gotten my Homeassistant OS Supervised in a stable VM setup (Not the recommended installation method btw). I am running most of my services through the apps available in their catalog though. Jellyfin, Octoprint, Frigate, OLlama, Pi Hole and Tailscale. All stable and working pretty well. Edit: btw Im a welder, I dont code. Just a tech nerd and have most of the wonderful Truenas community to blame? for my success. Lol
Having had truenas apps drop out on me while migrating and boot issues on updates to scale in the past I’m 100% back to running truenas inside proxmox for remote management still being available in maintenance
I am running truenas scale on bare metal with a win10 vm that has been rock solid for at least a year. Nothing crazy, but I can’t complain. It does the one small task I need it to do
Are you sure your 8+ character passwords are not getting silently truncated to 8 for vnc?
I currently have an old Dell R710 as a TrueNAS server.
My "main" system that I was using for ProxMox is currently not running as I plan on transplanting the innards to a 1U rack.
I plan on using an iSCSI disk on the Dell to store VM's in ProxMox.
I would also like to know how to get some of the beer you drink! :)
@level1tech I summon thee
Edit: I just grabbed the BD790i SE can't wait to build with it. The SE slightly cut down version of the regular board.
Hoi Jeff , I am following your videos running proxmox myself what kind off 10 gb nics can you recommend running 8.2 are the intel x540 suitable please advice greetings from the netherlands
Have your tried Unraid before? It's pretty basic. I think it's good with VM and something I'd recommend to new to doing home labs
I've noticed that with the VNC window that if it is left alone with no input for 3-5min (even with the window still active) it will become unresponsive to your inputs. It seems to be some sort of timeout when there is no input from the user.
I've tried it before but ended up with a small Lenovo PC as a dedicated virtualization host and a Ugreen NAS as a TrueNAS Scale NAS. The only service I run on TrueNAS now other than the file services themselves is Plex out of laziness and for the direct drive access.
I would consider TrueNAS with VMs for small "deployments" at for example a family members place where the primary need is storage and where containers and VMs are just an added bonus on top.
would be great to see if you would have same issues with nvme and MCIO connector which generally are horizontally placed rather than vertically like the one you failed with
you can get m.2 4x pcie risers - you put a normal lsi 9400-8i or whatever into a physical pcie slot on the case (that has no slot), you run the riser between the card edge connector to the m.2 slot.
The issue is that, if I remember correctly, the HL-8 only supports mini itx motherboards and consequently only has one PCIe slot, so even if you were to use a riser you still need a place to mount it
Regarding the Sata Expansion Card, what is the difference between this Silverstone Card and any other 6 Port ASM1166 cards available out there?
I run TrueNAS as the "bare metal" OS and I've used it for a couple of light virtualization tasks--and it's always worked perfectly fine for me, albeit a bit limited. BUT, this was the older "Bhyve" virtualization stack. I've never tried the newer KVM-based stack.
I currently have a TrueNAS scale server acting as the KVM at a church I attend. They have very basic needs when it comes to a hypervisor, but it is working and has not really had any issues.
Tried TrueNAS as a VM host about a year ago and it went pretty bad. It was a great NAS, but yeah VMs were really unstable. I decided to just use two machines - one with Proxmox for VMs and a separate TrueNAS machine. It's been great so far, and I would definitely recommend that over virtualizing TrueNAS if you can. I also use the NAS for VM backups with rsync as a CRON job for offsite copies.
Honestly, I'll never run anything except purely ephemeral workloads on trueNAS ever since an update wiped everyones data. Sure, we had backups, but this kind of breach of trust takes over 10 years to earn back in my book.
Hardware Haven did a video recently on updating his HP SFF server with modernish hardware. In that video he too was using m.2 to sas adapter cards but his had the ports lying horizontal on the card itself and not vertical like the card you tried
im using scale mostly for apps works pretty well for me. I only have one actual VM though with 1 measly gig of ram and its running home assistant. had no issues. but also no real demands.
Everyone has their own use cases and budgets to manage, but I lean on the Unix philosophy when it comes to NAS - let it do that one thing well.
I saw this hoping you'd say TrueNAS Scale was great, but apparently not! Thank you for the details, as your complaints, particularly around networking and pci pass-through, are directly applicable to my use cases too.
I have a supermicro nas box I was weighing using proxmox and loading truenas into as I do this with my firewall today, but wasn't sure how pci pass-through would behave for passing the sata controller for the 4x spinners on it. I saw this thinking maybe I wouldn't have to, but as you said proxmox base with truenas in vm sounds like the way to go. Thanks again!
Given the way THIS server is configured, it isn't really an option to run Proxmox and then run TrueNAS as a VM (which would require PCIe passing through the HBA for disk health monitoring - which is a critical requirement for a NAS, not ECC memory).
One critical thing to note is that TrueNAS runs ZFS and even in 2024, there is still a bug with ZFS that prevents proper isolcpu functionalities (i.e. reserving some CPU cores for VM and preventing ZFS from using those cores doesn't really work). So running a VM on TrueNAS means you WILL run into high latency during high IO because ZFS will use whatever cores it wants, at any time it wants. This happens EVEN WHEN your VM exclusively uses passed-through NVMe drives.
Running TrueNAS VM under Proxmox will mitigate this bug, at least to some extent. Under high IO, your isolated VM will 100% still be isolated from TrueNAS. There will still be some latency if your Proxmox storage is also on ZFS but presumably that would be on a fast NVMe (or at least a SATA SSD) so it will be order of magnitude less than when your VM having to wait for TrueNAS ZFS having to wait for a HDD to respond.
Pretty much nailed it. TrueNAS is fine as a storage host and not much else. I pretty much do what you suggest. Proxmox host, TrueNAS VM and other VMs doing fun things inside proxmox.
I've run VMs on Truenas Scale, they were a pain to set up in earlier versions, but its been getting slowly better. I never tried to run anything that needed passthrough. I ran Proxmox Backup server on it for a long time as it felt better to me to have the backup VM running on a storage machine.
i have been running a dietpi (running pihole) VM and an Mint build flawlessly on boot for over a year. im wondering if you are having issues with the actual CPU config, are you doing passthrough.
password obfuscated for me, yes, the focus loss is annoying, thats why i run other remote SW for longer duration stuff. i do need to test the passthrough
I use both setups:
1. Main server: Proxmox is the hypervisor, and TrueNAS is a VM with passed-through storage. I have ~10 VMs in Proxmox.
2. Secondary server for backups only: TrueNAS is the primary OS, and only one VM - Proxmox Backup Server.
Why?
TrueNAS is easy to recover/migrate to a new server, so I always can restore it w/the Proxmox Backup Server. From here, I can restore my Primary server.
I also run Proxmox with Truenas Scale as a VM with the HBA passed through. I wish I could have a simpler stack, but after managing VMs with Proxmox I won't even try using Truenas for that with its current UI and features.
I am looking forward to Electric Eel to hopefully simplify my containers. I'm really tired of Proxmox Hypervisor -> TrueNAS VM -> NFS/SMB to 2nd VM running Docker. That's not even considering the downsides of this, like no inotify for shared folders for Syncthing and Jellyfin.
i experienced all of the issues he mentions. I knew i was in for a rough time when Spice K&M was barely functional, but the pcie gpu passthrough that just ended up not working properly did it for me. Just looking to have my NAS as its own separate box at this point
For an AiO solution try Unraid. Great Hypervisor, Full Docker Support, ZFS Pools or for power saving purposes the good old unraid Array.
And all this time I thought it was just my inexperience! I gave up on VM's in TrueNAS because I assumed I was using a mature system with a proven wizard, and I was just too dense to figure it out. Thanks for clarifying the situation, and at the same time making me feel a little less like an idiot!
This turned out wonderfully, though, part of me was hoping for more hackery, like a PCI switch or something.
I suppose actually having it work and be reliable is... nice, so, 45D's solution is no doubt better.
That said you could probably find an M.2 card that'd support the SFF cables well enough, likely with right angle cables. Like you said though, more than enough for the spinners you'll be using. If you wanted all flash... well, there are likely better options there anyway.
I use a Delock M.2 to PCIe Adapter with an LSI HBA plugged into it (use pcie riser cable if needed)
It's one of the reasons why I run TrueNAS on a dedicated hardware strictly for storage as it was made for in the first place. I am running Core and will soon move over to Scale. I use ProxMox for everything else.
Good video. I would revisit this with TrueNas Scale 24.10 as that version is getting some major changes including doing away with kubernetes and moving to the adopted docker app system along with other changes. It may be better with the changes being made. I moved to TrueNas Scale from Unraid as I was building my server cabinet out and wanted a little more freedom. I was comp templating the change partially based on the upcoming 24.10 update and working on a steam/game cache that worked better.
what about virtualizing proxmox backup server in TrueNas Scale for backing up a proxmox cluster to the TrueNas server?
The truenas VM situation is painful. I have switched everything to docker containers (using docker compose) in a jail. The next version (electric eel) is supposed to integrate docker compose functionality within the main OS. However, my home lab is still in development and I may eventually need a VM. Hopefully things will have improved by then, but if not I may have to go promox with a truenas VM. Electric eel me be addressing some of the VM issues, but if so I have not been following them.
I am currently runing true nas in a vm with sata controler passthrough and 10GB Virtual function network adapter and it take me a day to find out that first:
cfgbev driver is not loaded bvy default and even if I load it you still can not connect using tcp because of broken check sumes which is realy hard to find out since for example you have pin but no tcp connection.
Also tried trunas scale Hypervisor, ran into similar issues, one time a vm would not stop, in no way could i delete it, stop it or acces it, had to force reboot Truenas to stop it. after that i just done the proxmox as a Hypervisor and truenas as a vm thing and that worked great
i recently built a pc, to run TrueNAS and then stumbled across Proxmox. as im new to both, i've been swapping between the 2 as the main installed OS and learning as i go along. I like the flexibility of Proxmox to run different VM's and containers, but having to passthrough the hhd's for TrueNAS to use and then not being able to take snapshots (proxmox errors out), is a bit of a fail. but then when TrueNAS is the OS, running apps and VM's, can be a right pain to set up and get running. if only the there was a way to dual boot Proxmox to run the VM's etc and TrueNAS as a dedicated NAS as the same time.
I have had no problem doing vm on my truenas scale just the spice, but I did a remote viewer to the vm and no problems
I guess I look at truenas to do one thing for me: host storage and the services directly related to storage. It isn't what I'd look for to be my virtual host. Would I like that? Sure, I could have a couple use cases for that. But I wouldn't *trust* it for that even without the problems you encountered. I trust it for storage.
thx for the video ! for me VMs only Proxmox , and File Sharing etc with TrueNAS. i had many problems with vms and apps in TrueNAS Scale , and on the latest version many more.
Right off the bat my number one hangup with hosting VMs on TrueNAS Scale at home is that it limits the hardware I can use. I frequently build these homelab systems from older parts, and TrueNAS doesn't let me run older drivers when needed for compatibility. At least not without a lot of extra effort.
Love TrueNAS Scale for many things, but not as a primary VM host.
I have had nothing but problems trying to get a GPU passed through properly on this platform. The NAS function works great as well as the kubernetes feature.
I like to keep copies of my VMs on alternate servers (rsync) so if my main server dies I can start it on a secondary server, this was impossible on Truenas when I tried it about a year ago. I run an Unraid server and a Proxmox server (running Truenas in a vm solely for NAS purposes). I can rsync VMs back and forth no problem because they are stored in a standard folder in both OSs.
About the instability, maybe it is because you used ballooning and passthrough? On proxmox as far as I understand ballooning doesn't work with passthrough, because it requires DMA. Or maybe it works on trueNAS somehow?