LXCs vs VMs - What Was My Rationale?
Вставка
- Опубліковано 29 вер 2024
- So in my last video, we took a look at what I've got running in my home lab back here, and we briefly discussed the hardware.
Last video: • What Am I Self-Hosting...
Then we talked about the two Proxmox servers I've got for production and the one I've got for testing and development. I don't develop, but it's for testing and tutorials, that sort of thing. I guess that's the better way to word that.
In the comment section of the last video I released about all of this stuff, a couple of people asked what was my rationale in deploying LXCs or Linux containers for each of my individual services rather than throwing up a couple of VMs, installing Docker, and then putting all of my services in those couple of VMs.
And I think that's a great question.
We discuss my rationale of my setup in this video.
Blog post/transcript:
dbtechreviews....
/=========================================/
Additional info on LXCs and VMs:
✅ LXC Pros:
- Lightweight: LXCs share the host system’s kernel, making them much faster to start and use fewer resources than VMs.
✅ LXC Cons:
- Limited Isolation: They provide some isolation, but since they share the kernel, a security issue in one container could potentially impact others.
-Linux-only: LXCs can only run Linux distributions that are compatible with the host’s kernel.
✅ Use for VMs:
- Running applications that require a specific operating system not compatible with the host.
- Situations where a high degree of security and isolation is critical.
- Running software that requires direct access to hardware resources.
/=========================================/
Get early, ad-free access to new content by becoming a channel member, or a Patron!
✅ / dbtech
✅ / @dbtechyt
All My Social Links:
✅ dbt3.ch/@dbtech
Join Discord!
✅ / discord
Good points for using LXC’s over VM’s! I’ve seen multiple ways to passthrough hardware to LXC’s. I’d love to see a definitive video on hardware passthrough to LXC’s! Especially for unprivileged LXC’s
I'm planning on doing some videos on passthroughs and that sort of thing when I get my new HL15. I've already got hardware waiting to be put in it specifically for passthrough content :)
Whilst I completely agree with your rationale Dave and do things the same way (and get flamed on Reddit for it), you could restore a single container on a VM. If you go into the interface of PBS and click on the backup of that machine you want to use, you can actually access to the folder structure of that VM and can download folders directly to your desktop, where you could just scp it back in to your VM.
I don’t myself, as I prefer the simplicity of LXC containers and splitting all my services out.
I guess i wasn't aware of that specific method of doing things, but that seems WAY more time consuming and tedious than doing it this way :)
Just make regular backups of your docker volumes.
This works for me and it's all automated. You do what works for you
I had the same dilemma with my home server.
The reason I use VMs on some mission critical services is due to HA and Backups.
Since I use my NAS's SSD pool as storage via NFS, if 1 node shuts down, the HA manager can migrate without any issues. If I use local-lvm and my node powers off for any reason, HA cannot migrate since the storage is on the offline PVE node. If I use LXC with NFS as storage, backups will fail.
So It's mix and match for me until I upgrade my hardware and move to CEPH, this is my setup.
I am also a total fan of LXC. Fast, small and uses little hard disk space.
The reason why you install Docker in a VM (should not have to) is the security of the Horst server.
We visited the Proxmox developers.
When I asked why not Docker in LXC?
We were shown how. With one LXC (docker ) you can bring down the entire Proxmox Horst.
Nothing happened in a VM.
VM containers use more resources, but prevent a total failure of the server.
The backup strategy for VMs is snapshots + a weekly full backup (stop mode) in individual operation.
Proxmox Backup Server & Cluster Nodes is a separate topic.
It was a very impressive seminar with the developer.
I apologize for my English, writing in a foreign language on a cell phone is a disaster.
Greetings from Germany
That's good info! Thanks for sharing :)
I'm planning to move my Swarm and Kubernetes clusters to LXC. However, I'm curious about what you saw that could potentially crash the Proxmox host. What specifically was shown to you?
@@jbauson1 No instructions on how to crash the software, that is bad for business.
For this great software that we can all use privately.
So let's change the subject. The question is why in the LXC and not in a VM ?
@@bps3374 Well, why not? I'm not using Proxmox for critical workloads-it's more for personal projects like Plex, file servers, and simple experiments that can afford to be offline briefly. For serious production environments, I rely on AWS, Azure, and other cloud providers. I'm not looking for step-by-step instructions, but rather understanding what might cause a crash. Since you mentioned that this issue was presented by Proxmox developers, I'm interested in what they found generally causes crashes. Given that these are Proxmox developers, I assume they have the proper setup and permissions in place for both the LXC containers and the Proxmox installation itself.
I had this exact question while watching the video. I like, and would almost say that I prefer, LXC containers to a VM running multiple Docker containers but think there's a balance to be struck between the two. I think I saw/read from another UA-camr I follow (can't remember who it was at the moment) that hosted LXC's for internal-only applications and did a VM with Docker for anything that was exposed to the internet, which probably makes sense from a threat surface standpoint. Great to hear your rationale!
Thanks!! I appreciate you! :)
I like the new yellow backgrounds you are using in your last two videos, it really does make them stand out from the crowd.
Great video and I completely agree with you. I have had one of the containers within a proxmox VM get corrupted by an update, and had to revert to an earlier backup of the whole VM to recover, but that meant I was restoring to an older version of everything else in that VM which was annoying as well. So now I strategically choose what can be in the same VM and what I split across multiple VMs.
That sucks!! I'm glad you've got a strategy that works for you :)
Only the ignorant would “need” convinced!
But the TRUE geeks, it’s just another FAD for the WEAK!
If you can containerise it then it should be containerised, the real question I keep going back and forth between docker and lxc. Half my services are on LXC and the other half are on docker.
Totally you are right 👍👍👍
Thanks for watching and commenting!! Much appreciated :)
I wish I had that "monkey brain" Great video, man! ❤
Thanks so much!! :)
@db tech just curious what are the machines specs for these LXC containers for the single services? Just started setting up my home lab so looking for some recommendations. Thanks
I usually set them to either:
1 core and 512mb of ram
Or
2 cores and 1gb of ram
And then I adjust as needed
Man! That's starts like a soapy opera 2.30 min same 6 words different order
2:27 'isn't a VM' ???, thought it was, or not ?. I must dip out here before total confusion sets in as I am a very new Proxmox user and fear taking the wrong road in the early days. But thks for the attempt at the compare but I guess unless you already know what you are doing I feel this vid will confuse, I need a clear and concise vid to get me through this, better luck next time, thks for the effort and Kind Rgds.
This wasn't really meant to be a tutorial as much as an explanation of why I did what I did in a previous video. Thanks for watching and commenting :)
Makes complete sense. I may be wrong, but an LXC is essentially a container just running in proxmox natively instead of something like portainer on a VM. Least that's how I understand them
I have two points. 1. I do agree with you on LXC vs Docker. Of course, majority of us do the easy docker-compose templates mixing quite stagnant configuration files with databases (mysql, postgress...). The recovery dilemma could be solved by keeping databases in separate dedicated LXC/VMs replicated at least to one extra instance. There is no reason to create separate mysql installation for every application. They could be combined, which long term simplifies backups and recoveries (IMHO).
2. The issue of having multiple instances of DNS (or homepage) apps. Implementing vip failover with Keepalived solves it beautify. Quick and easy installation, but the benefit of losing DNS resolution for 1 second is just priceless. My pain Pihole runs as LXC container on Proxmox, and its failover backup (updated by gravity-sync every 15 mins) on a raspberry pi 3.
Also on question: why don't your production Proxmox server run in a cluster? Just wondering.
So... Let's see... I disagree with you on the "no reason to create a separate mysql installation for every application". If someone manages sql injection on one application, that would, in turn compromise the other applications using that database. I always have a dedicated database for each application. I used to work as a web developer and had issues with hackers doing ignorant stuff and really screwing us over. I've been overly cautious since then about this kind of stuff.
As far as why I'm not running my 2 production servers in a cluster is fairly simple: I used to have a 3 node HA Proxmox cluster and it was fine. But it never actually did anything for me. It was "neat" that I could see all the info from one dashboard and whatnot, but it just didn't do much more than that for me. So I migrated everything over to a single node that was just as, if not more, powerful than the 3 nodes I previously had set up. Recently I decided to set up a second node specifically for a VM I wanted to run that would have overloaded the first unit when added to everything else. So I just quickly set up that node and started deploying stuff on it. I never felt the need to cluster them because I've got new hardware coming that will eliminate the need for multiple nodes entirely.
As I've said in a few other replies, "the great part about this hobby is that everyone can do things the way that makes sense to them. (as long as it actually works)" :)
Love the plan. But I moved away from proxmox because of kernel error which crashed my system. Backup was on the same disk so that was quite a nightmare for me. Maybe another video for us if you have a backup plan for this scenario?
I think the thing to be learned from this is that your backups shouldn't be on the same drive. I'm not trying to be mean. I just can't imagine a scenario where having backups on the same device is a good idea.
Loved the rational, especially the snapshot restore. I guess my two lazy points are what would keep from doing it.
1. Manually updating by logging in and pulling images, etc.
2. Having to think about resources for each lxc.
My favorite thing about docker in a VM is I don't have to care if one container uses more resources than another. Only have to monitor the overall VM system usage.
Either way though, going to move pi hole to an lxc for sure!
As I've said in a few other replies, "the great part about this hobby is that everyone can do things the way that makes sense to them. (as long as it actually works)" :)
Point 2. Can be an advantage too, you CAN set resources on each LXC, especially a RAM and CPU limit which you can also adjust dynamically without the need to reboot to LXC so some LXC can consume more resources that others.
But for me the biggest advantage is with LXC you get a proper network with a real dedicated IP address for each container and that network can be mapped to a Proxmox bridge interface with proper VLAN tagging. I have LXC on several different VLANS for example one VLAN goes through a privacy VPN and is UK based, another is on a more secure network such as my password manager (I keep that firewalled from any other containers as you don't know what code or vulnerabilities are in Docker Hun images. I know you "can" do most of this is Docker with MACVLAN and VLAN interfaces but its far simpler with LXC's
Good thing about LXC you can run docker inside of it as well. Outside of running desktops, security onion, and a few rdp managers, VM not really needed.
💯
You can run xrdp in an LXC. But I would not run docker under lxc, there have been reports of breakage in the past after updating Proxmox at the host level.
I hope those are all unprivelaged lxcs.
For security you could run one VM and all the services on their own LXC in that VM and backup each lxc separately, like you do, and just save backups to a smb share. Backing up method would be a bit different, but you could probably make it work for you. Passing through, or different backup solution that produces same backup file type, or something.
Fun challenge, if nothing else.
I always run my LXCs as unprivileged.
@@DBTechYT exxxxcellent
hi, the way you suggested doing things in terms of easy backup also exist for a monolithic docker host too. most of the ways people do it (like me) are to make use of BTRFS snapshotting and also to make use of docker compose bindmounts. so you'd create a directory like /appdata/utility_name and then you'd have timeshift or snapper or btrfs-progs or whatever similar utility then snapshot that /appdata/ at regular intervals (either on another separate disk or as part of a mirror) and should something bad happen to any specific docker container then its as simple as just going into that specific subvolume (snapshot) for the files and yoinking them out
That's a cool strategy!
What template/OS do you use for your LXCs?
I use Debian 12 Turnkey Core for my templates. I'm planning on making a video showing the process I go through to set all that up :)
@@DBTechYT I use the same template, so it is nice to get some reassurance! Thanks
So very happy that you are back to posting videos. Missed ya man!
Thank you so much. It's been a process (and still is to a point) but I'm really excited to get back into creating new content more often :)
I go for an LXC out of the gate because my lab doesn't have a lot of horsepower and I like how lightweight they are. I tried getting AWX running in an LXC and couldn't make it work. It's a hobby for me and getting frustrated makes me stop before pulling my hair out so i don't completely give up on it from burnout. I'd be curious to see if anyone has won that battle and what their process was.
I use LXC's exclusively too. So much better.
Great point never thought of it like that. Can you do a video on how you created your lxc template and how you map external storage to it please?
good to see you back :)
also just realised i've been spruiking your vids and i wasn't even subbed 🤦♀
well, that is very interesting but can i go one step further? I run my services in docker on ubuntu server. what would be the difference from that to running docker inside an lxc. Sounds to me like an inception of containers... I mean there might be a reason why people recommend doing it that way, but I just can't get the reason why.
The difference for me, as I said in the video, is further isolating the different services from each other. And then having the LXCs fully backed up allows me to restore a single service without impacting the rest of my services on the same machine.
Again, the great part about this hobby is that everyone can do things the way that makes sense to them. (as long as it actually works)
Makes sense to me. Thanks for sharing.
As always, thanks for watching and commenting! :)
As per usual - Really great info.
Thanks!! I appreciate you :)
I think I was with you until 5:35. From my understanding, since the data (shouldn't) be in the VM itself, just mounted, it's just the configurations that needs to be restored. And even if you don't have something like Anisble, Terraform, or something like that for auto-deployments, even just making a backup of the docker compose file used to setup the docker in the VM should be good enough. Now, this only doesn't make sense if you don't update your docker compose file each time any updates are done, and I would encourage you to do that, if you don't already. Of course, the other reason why this approach wouldn't work is that you're storing your VM data directly on the VM, instead of stored outside, to be mounted inside. Depending on your setup, this may the best way, and then your rational sorta makes sense, but even then, not really. Because data compromise is data compromise, regardless of what manner of container you're using. And you're already using Proxmox, so setting up your storage in Proxmox (NFS, ISCSI) shouldn't be that much more difficult than your current setup.
Your rational at 7:30 is probably the only one that is objectively true. Memory usage should also decrease slightly using LXCs vs VMs, which allows you to have much more LXCs at the same time running than that amount of VMs. At this point, I must ask: What's the point of using Docker if you're just running 1-2 apps per LXC? Shave down the resources more and just run them 'baremetal' on the LXC.
I'm not currently using NFS or iSCSI for my storage. All of my data is being stored locally. I used to have my data stored on a different device via NFS and the amount of shit I took for doing that was unreal. I left it like that (using NFS) until I decided to rebuild my setup and simplify things a bit.
Again, the great part about this hobby is that everyone can do things the way that makes sense to them.
@@DBTechYT No, what I meant is that in your homelab explanation video, you stated that each mini pc had 2 ssds in them. One for OS, one for I can't remember, but probably storage. You can setup NFS or ISCSI on Proxmox for local use, and then just mount them into each VM/LXC. That way, the data lives on even if the VM/LXC disappears, all on the same machine. That's what I mean about the data storage not in the VM/LXC.
> I used to have my data stored on a different device via NFS and the amount of shit I took for doing that was unreal.
F them. That's what I'm doing, and even worse, gonna do it over 1gbe until I get a job to get 2.5gbe to usb cables for faster connection. (Unfortunately for me, no 2.5gbe or 10gbe for HP mini g2 or g3, which is what I have.) Using 3 synology machines for it, to boot. (2 of app storage, 1 for backups).
I wasn' trying to attack your setup, just don't necessarily agree with your rationale. But only academically. Way too old to get into arguments on the internet, especially about stuff that doesn't affect me, and even more when it works for someone else. Just felt I had to point one one avenue I think you missed.
Gotcha. I see what you're saying. I have an nvme drive in each unit for boot and an SSD for storage. I have it set up via ZFS. No offense taken on anything. My Proxmox setup works for me and my backup solution has saved me a few different times. I think my concern with having persistent data via NFS like you're saying is this:
Let's say someone compromises your system and you just restore the Docker setup. That compromised data will still be in the NFS data pool. By restoring everything (the docker container AND the data), there's less of a chance that the compromised files will be restored.
I use this as an example as I worked for a web development company years ago and hackers infiltrated some of our sites, compromised them with backdoors and malware, and then waited a month or 2 before they actually launched their attack. No matter how many times we restored the sites, we just kept restoring compromised files and never removed their access as a result.
We had to take a completely different approach to undoing what they did, so I always like to do a full restore of everything whenever possible as a result.
I build my (fixed image container) server with ansible then periodically backup just the persistence directories and sql db's off site. Then should the site get popped I run the playbook this time it pulls the backup data during the build. What do you think? I'm v new to all this. Maybe there is a better/simpler way I've not seen?
@@50_Pence sounds like you've got a good solution in place. I've only played with Ansible a little bit so I don't have much to say on it at this point, but I'm planning on learning at least a bit of it because it really does fascinate me.
Sad that your main reason is “backups”!
If so, you are only one full generation BEHIND!
(Backups are NOT an issue for “some”).