Here is an excellent post by Olivier Lambert who runs Vates, the company behind XCP-NG & Xen Orchestra covering some very important details regarding the history and architecture of the hypervisors forums.lawrencesystems.com/t/xen-vs-xenserver-vs-kvm-vs-proxmox/14256/2 awrence Systems Forum Post forums.lawrencesystems.com/t/xcp-ng-vs-proxmox-2022/14200 Lawrence Systems tutorials on XCP-NG lawrence.technology/xcp-ng-and-xen-orchestra-tutorials/ Learn Linux TV Proxmox course www.learnlinux.tv/proxmox-virtual-environment-full-course-class-1-getting-started/ Learn Linux TV Proxmox Backup www.learnlinux.tv/proxmox-backup-server-full-getting-started-guide/ ⏱ Timestamps ⏱ 00:00 XCP-NG VS Proxmox 2022 02:30 Features Comparison 03:09 Explanining Xen Orchestra 08:15 Max Support VM Disk 11:30 Proxmox interface 13:57 XCP-NG Xen Orchstra Interface
I also have used both and both are very powerful. However, I have mostly used Proxmox and have used it for the last 13 year in large scale and it really is a solid enterprise solution, especially together with CEPH. I know every corner of it, it can’t surprise me with anything I can’t do or fix. I believe Lawrence have the same knowledge about XCP, there is nothing that will take him by surprise that he can't do or fix, and this is really the right reason to choose one or the other. Choose one and stick with it, be a god with it and it will never fail you.
I have used both and settled on Proxmox. I really didn't care for the XO interface, and the VM console is very limited. I want to be able to break out the console to its own window, and Proxmox can use SPICE which is vastly superior to what XCP-NG offers. Granted, I haven't used XCP-NG for over a year so maybe things have changed in that regard.
I've used both also. XO was cripple-ware when I last used it (certain features were turned off unless you paid). Proxmox has ZFS, SPICE, it's 100% free, has replication, ZFS snapshots, HA, etc. Really no comparison. Proxmox is the way to go for sure.
@@SteveWebbbb More advanced ZFS support and the fact that Proxmox has its own web interface integrated into every host while being lightweight at the same time is also more important to me than having drag-and-drop feature for the boot order on a VM or a toggle button for Windows guests. Not to mention the container support in Proxmox...
XCP NG looks beautiful, I'd love to delve into it if I had the need but for my small enterprise Proxmox + TrueNAS iSCSI is bulletproof. It's easy to train my staff to use it as its paradigm mirrors real hardware and snapshots and replication are painless...
I have a question for you regarding Proxmox and TrueNAS iSCSI setup. I ran into an issue with getting proxmox to connect my iSCSI storage on both hosts (2 nodes). Each node connects to the TrueNAS server via a direct cable connection and on a different ip that is on a different subnet, no switch as I have dual 10GB nics in each host (proxmox and TruNAS systems). What I found was I can add the storage but only input one of the two ip addresses the TrueNAS server is answering on. The fist proxmox host sees the storage but the second hos sees the storage as off line. I did some research into multipath configuration but that didn't work for me. Any thoughts?
@@bdunigan80 You can only have 1 default gateway on TrueNAS and it is usually for the management interface. All other NICs (for iSCSI/NFS) on TrueNAS for sharing are gateway-less as your servers that are consuming them have NICs on the same (dedicated) network segment. Maybe you just have a misconfigured NIC on either TrueNAS or servers that prevent you font seeing it?
If you are using two different subnets check that you have a portal on each subnet. Or get a 10Gb switch and keep both servers on the same subnet and portal. Good luck.
Great video that answered quite a few questions about the two systems, with the VM ESXI drama going on right now, I bet this will start to get more views as people start looking for other options.
Tom is right. It depends on use case and your familiarity to the particular tool you intend to use. Once you master any of the tool, you can invent the work around its shortcomings.
Thank you for putting this comparison together. I've been looking at the two platforms, and I think that one of the ways for me to experience it would just be to try it. I think that it IS important to mention though, that Proxmox does not have an easy/default way to import OVA appliances (so if you are say, migrating from VirtualBox or some other virtualisation ecosystem -- I would mention that). Also, I would mention, in your comparison chart, that XOSAN is a ***PAID*** feature. Therefore; if someone wants to use hyperconverged features and Proxmox allows you to do it for free (even if the hyperconvergence feature set is limited with the free option), that might also be worthwhile to note for some of your audience members.
@@wayland7150 I don't think that I can import OVAs into Proxmox easily/readily like I am supposed to be able to with XCP-ng/XenOrchestra. But I guess that I'll find out! We shall see.
I love the comparison, so thank you Tom, and this gives some really terrific aspects of XO and XCP-NG. I'd love to find out if XCP-NG plans to support LXC / LXD Containerization int he future. It's just a no brainer to want to use those for much of the VM work these days due to the far lower resource needs for a ton of the use cases out there.
Containers would seem to be out of scope for XCP-NG, which I think is trying to just be a solid, easily administered secure hypervisor. Proxmox is meant more as a home lab all in one server, so host OS containers make sense, but I expect that a "correct" implementation of LXC on XCP would be inside a guest OS.
Its kinda silly that people think just because you dont use something its bad. The main reason apart from the second is i like the XO interface look and if im spinning up stuff all the time i at least want the ui to look sleek but on a different note for me XCP-ng turned out to fit my use case because of the meany to one relationship my lab scales up and down systems get added and tour down so having the signal pain of glass setup were i can just add servers and remove live or shutdown migration of VMs is really powerful
I'd like to see a vSphere vs Prpxymox comparison, it seem that proxmox has a reasonable attempt at the feel of vSphere. Mind you it has been two years since so perhaps I'm mistakened.
Proxmox can somewhat do the tagging features with their pools. You add resources to a pool and you can have some actions target everything in the pool.
I use pools in ProxMox as well to help figure out which group of VMs are using TrueNAS storage so I can shut those VMs down before rebooting it due to updates.
This is informational and really well done, Tom. I've worked with so many Hypervisers through the years I couldn't tell you my opinion of each of them; when it comes to Citrix (the heart of where Xen is) I always remember going back to Citrix terminal servers in the late 90s off of NT4.0 to enable dumb terminals at a hospital client.. I digress. Most of those I work with are VMWare, or, in some cases, Microsoft Hyper-V under a setup where the root server cannot be virtualized due to specific hardware passthroughs/key security issues. In this sense, I've always preferred VMWare. Once VMWare 7 came about, it put a stop to a lot of the things I really liked about VMWare, and I've looked at other options. Their are a few things here; I am not opposed to cost; I have always preferred as a backup solution Acronis for varying reasons - most recently native endpoint web management and full security modes, but also easy export to review for test conversion (because if you aren't testing a backup on a schedule it's worthless) This is where I have issues with XCP. I like the core concept of it; but the way that XO is structured really boils down to a "pay to play" $85/mo*12 year for features included in ProxMox, which has a newer, and more frequently updated Hyperviser in KVM, and Xen is absolutely showing it's aging when I look at the IOPs I see... I think what makes XCP really solid is that XO connects to multisite in a way that really only Microsoft's manager compares to in node control for Hyper-V, in a different way; but if you're using Hyper-V it is is, again, often for transition purposes, specific hardware pass-throughs, or for hardware supports. If you're a homelab, ProxMox might be more likely to provide you exactly what you are after; the interface is instantly there, many of the items that are removed from XO without $ are present (backup), and easy management capability works. XO's handling of XOSAN and backups is a great feature, but it is very important to mention they are -not- free or included by default, as people who walk through the default install will find they can't get there. Your videos are really informational. Every hyper viser I've used has it's strengths and weaknesses.. It boils down to what is the end goal of the Hyperviser and what am I trying to accomplish
This was pretty nice, but I think it would be much more interesting to have a proper comparison of strengths and weaknesses between XCP NG and Proxmox with maybe some guest youtuber who knows Proxmox well. Would be even nicer if you could do some more advanced scenarios such as Ceph vs. XOSAN. Looking at discussion forums after Broadcom's announcement to buy VMWare, sysadmins' choice to replace VMWare seem to be leaning more toward Proxmox while XCP NG isn't often even mentioned. Might be because Proxmox has been around much longer than XCP NG though.
Nice video and a good try to do the VS thing but 2.5 mins Proxmox against the rest of the video ( 10+ mins ) isn't really a Versus video. I understand you are more at home with XCP and therefor can demonstrate more features. This would be the reverse with someone using Proxmox daily. Some clarification though: Backups do not need the Proxmox backup application. There is backup build-in and has options to include/exclude disks, so big disks do not have to be an issue.
Agreed. I know that this isn't a priority for xcp-ng and that there's a myriad of alternative options you can pursue, but there's something just _nice_ about having your VMs and CTs 'colocated' in the same workspace/GUI. Currently using xcp-ng but considering moving back to Proxmox for this reason (wasted memory by having docker running in VMs)
@@transatlant1c Yes. It is quite nice. The only thing Proxmox doesn't have is native docker containers. But that is mostly because it is supposed to run LXC CTs. It is possible to run Docker cts inside a VM or even a n LXC CT, but it is not trivial, it will have another virtualization layer and it can be somewhat buggy. I try to avoid docker into proxmox.
@@gustavinus yep that’s right, when I was going down that path I found setting the CT as unrestricted and enabling nesting, along with a turnkey Linux image was successful but results may vary!
@@transatlant1c yep... with those settings it runs pretty much natively on the Proxmox kernel. I tend to use Debian for those CTs because Proxmox is also base on Debian.
One imporant fact: XCP-NG is based on Xen vs. Proxmox on KVM. The former is gradually losing the battle against the latter in the commercial virtualization market. This may be not quite noticeable to a home lab user but personally I would be more inclined to stand for the future.
For small team and self managed VM server, I feel like proxmox has more online videos and tutorials to learn from, XCP-NG seems to be much less. Thus, I consider proxmox has a easier learning curve. Not sure if my impression is right.
Two factors at play. Learning curve and system capabilities. Obviously you want an easy learning curve to get started but if the thing does not have the features you need then it gets increasingly difficult as you implement workarounds..
-- I've tried both, and while Proxmox looks nice for a one-server student lab, it doesn't scale nearly as well as Xen/XCP-ng/XO. For large-scale deployments, the Xen/ng/XO stack is fairly reliable and the Xen Hypervisor is a lot more secure and quicker
Firstly great video not less from you Tom. I am newbie to OpenSource applications. Been to long brain washed with those high end applications. A friend introduce me to Proxmox I have installed it on 3 server and clustered them as well. Yes was really easy install and setup. I am looking at migrating a number of VM from Hyper-V to Proxmox or XCP NG. I have not installed XCP NG but after this video I will be formating the Promox lab setup. My goal is provide easy management for my staff knowing who has done what but also key factor migrating from Hyper-V.
Thanks for this comparison. A few typos in your matrix: 'suported', 'stoage', 'packacge'. I'm waiting on a Beelink GTR5 to spin up a virtualization lab and looked at XCP-NG but I don't like it as compared to Proxmox. I'll be running quite a few Linux VMs so the LXC container support in Proxmos will really come in handy.
Good comparison at a glance as a good starting point, thank you. Slight typo in the "Shared Storage", for Proxmox you put "CefRBD" instead of CephRBD. Also "stroage" and "packacge".
great video!! thanks for the comparison. I think I will still keep my proxmox setup, due to the LXC containers I have. And I wont have so many VMs haha. Also, I was not able to find much information about GPU/PCI passthrough for xcpng, and I use the VMs for gaming. thanks again!
I tried both, but I like proxmox more. The Webinterface is IMO far more intuitive, it runs perfeclty stable, most people already know kvm/qemu and I don't need a VM just to have a webinterface. But that doesn't mean XCP NG is bad. Also, at least for Samba and NFS the used storage graph works fine
Who says you hate Proxmox ? Dude i've found out about Proxmox 5.X from you and i've been watching your channel when you switched from Proxmox to xcp-ng. I love your videos about Proxmox, XCP-NG, pfSense, TrueNAS. (not so much Synology and Unifi)
Many thanks for the excellent video. I have to confess I'm a Proxmox fan and love that system, but I was very impressed with XCP-NG in your video. It has a very nice web UI, unlike vSphere which looks nice but is buggy, and seems more capable at a very large scale.
I love your open. I use in my home lab libvirtd on openSUSE, Proxmox and VMWare ESXi, if I had another server I would tried some thing. I don't use Citrix Xen because there is a limit on the free versions. Great Video.
with the container management I had asked during a live stream and they will be implementing RunX which should allow for similar container management on the hypervisor.
To me, the real issue is Xen vs Qemu/KVM. Xen is pretty old and Qemu/KVM is clearly setting up to be the dominant Linux hypervisor of the future. Even AWS switched over to Qemu/KVM from Xen... I do like a lot of the feature-set of Xcp-Ng and even Xen Orchestra, but if one is choosing to standardize around Qemu/KVM, then unfortunately both are really non starters. 😔
@@LAWRENCESYSTEMS All new AWS instance types released from ~2017 forwards (ie anything on AWS' Nitro system) are using KVM, including all Graviton products. So... Unless you're sure the majority of currently running AWS' instances are using older instance types, that may not be a factually correct statement. 😉 Not trying to give you a hard time... But enterprise-scale cloud DevOps is my day to day, so it's an area of the market I track quite closely. Xen will no doubt be around for quite some time; speaking of AWS they even built a compatibility layer for it, so older Xen-only OS deployments don't break with KVM-based instances. But if you look at market share trajectory over time, feature velocity, and enterprise backing, it's clear KVM will eventually become the standard for HV in the Linux space.
@@shammyh Linode is also using Qemu/KVM in their infrastructure. I've created a few test Linux VMs on their platform and took me a few mins to figure out what their host systems are running which are in fact using Qemu/KVM. Wouldn't be surprised if they are using ProxMox to manage the VMs and Host as it can be fully scripted to provision the VMs.
The most important question is whether understanding Xen can be an asset on a professional level. I'm not worried about Proxmox because KVM is the most widely used hypervisor on the market along with Hyper-V and ESXI but Xen, there aren't many job postings about it.
Just a bit of clarification that was not evidently clearbinnthe video. Hyperconverged storage using XOSAN is not available with Xen Orchestra from sources.
When you do things at scale, you do infrastructure as code ;-) For that reason, oVirt's terraform provider is pretty compelling, integrated with CI/CD of sorts like drone, pxe/matchbox and ansible (rundeck), you then get ability to manage infrastructure of basically any scale, and that's on-prem, in the cloud and in hybrid environment - simple and great for lab or bigger production project.
I use some (older) Infiniband hardware for my storage backbone and having the OS not hinder my ability to install what I need to get it working is kind of a big deal. For me the "custom", and rather limited, when it comes to hardware support, CentOS image was a big strike against XCP-NG. Similar issue with TrueNAS Scale where they actively block the user from installing additional packages. Doesn't necessarily matter for professional use-case, but it does for homelabbing.
Yeah I also had some older motherboards where xcp-NG would just panic and crash, the same motherboards work fine on Windows or with a modern Linux system (or proxmox/Ovirt)
I'm using Proxmox since 2 weeks installed it on i7 875K (2010 plarform) with 16Gb RAM and I love it. I run Openmediavault RAID5 as VM (HDD passthrough) and 2 minecraft servers - one in VM docker on Debian Stable second in LXC archlinux container. Perfomance is stability are great. I had problem tho with True nas and ZFS RAIDZ but I gues it was too demanding for that PC to handle.
I run a Storj node as a VM in one of my servers. The virtual disk attached to it is 27TB right now (I expand it as it gets filled up). I also used over 2TB virtual disks when I was mining chia and I have a 3TB virtual disk on a VM that is backing up my other physical machines. The 2TB limit is quite small to be honest, I know some VMs our clients have that have more than 2TB virtual disks. Of course, with a limited virtual disk size it would still be possible to just attach multiple virtual disks to the VM and then combine them using zfs or software RAID inside the VM, but that feels like more complicated setup for no real gain. I like backing up using zfs snapshots (on the host) and doing "incremental forever" with only one "full" backup.
As I read, both have issues with iSCSI multipath. That's reason why I would probably move to VMware, even it is most expensive. In small business, I need a solution which is easy to administer and maintain.
Even more interesting is the new kids on the block. OpenShift/OKD Virtualization and Harvester. Leveraging Kubernetes and kube-virt. Red Hat is stopping support for RHV/oVirt (which XCP reminds me of) in favor of OpenShift Virtualization.
@@levskilevov4888 I didn't say that oVirt itself was stopping. Just that one of their biggest contributors is stopping support for them. That already small community is about to get even smaller.
Personally I cannot get over the ackward XOA web gui interface.. It just is very unintuitive (to me). XOA-center isn't really an option since it only has limited features. Proxmox has a lot of problems in that area too. However I temporarily switched back to proxmox because XOA kept throwing errors at me which made no sense.... basically the log kept filling up with erors. It seemed every 1 out of 5 actions I tried to perform, gave me errors.. I've been around IT systems for 40 years and they don't make sense.... anyways... Of course all of this is very subjective to personal taste. One thing that bothers me about xen is that the opensource guest tools are developed/maintained by 1 overburdoned guy and the offical ones are being a login wall now.. (new ones anyway). Without those guest tools you might as well go back to T2 virtualization..
Funny because I find xen orchestra to have the superior ui compared to proxmox but to each their own. Also Having only 1 guy responsible for the open source guest tool is alarming though.
Yeah... I am now running XCP-NG inside a proxmox VM and I have to say... It may also be just a bit of knowing how to do things the right way.. Some of the gripes I had with it are probably mostly due to my own inept skills with it... Keep on learning :) Thanks to Tom's videos I've learned more about virtualization and firewalls than in the 20 years before that :) btw @tom ... I've just been watching some of your 2016 on-site videos and they are amazing... Thorougly impressed with what you did there and are still doing..
Ok, I hate you 😜 On VMware and now Azure we have big machines, some disks around 16TB, some machines up to 36TB, and we back them up ok - although with huge disks you really must use CBT (change block tracking). But yes, best to avoid hugh disks/VMs as restoring them could be challenging. We did use VMware SRM, and now Azure ASR, so possible to failover to another replicated copy, in another location, quickly, within your RPO and RTO. Just installed Proxmox, great video as always, I will have a look at XCP.
In xen-orchestra from source you will have not all features. Example is Proxy. It is not available in XO from source. XOSAN not available in XO from source
I too settled on ProxmoxVE after using both. PVE feels solid and mature. I find this video a bit biased so I advice newcomers to research more. Also, the argument regarding the low attack surface is misleading. PVE is basically debian linux which already runs on the majority of internet servers. Lastly, XCP-ng has some grey areas, e.g. installing guest tools
XCP guest tools is the one thing I really dont like about XCP - Ive had major headaches with large MSSQL workloads where the network just drops. Still evaluating both but on the Proxmox side I think it has a terrible backup solution unless you move to PBS. That then requires another separate CIFS share for file backups from Windows.
I'm using proxmox for a mont or two. I have my work pc converted to proxmox and installed windows on a vm with hardware pass through (RTX 3060, 3 monitors few usbs, pcie pass nvme disk). Problems I had so far: proxmox doesn't do a great job of passing error messages from kvm. PCI-passthrough will sometimes hang and proxmox won't know what the problem is. Once you hit the right settings it works fine. However I had problems with usb devices disconnecting. One time the whole VM powered off after plugging in passed-through usb device. Later I checked - there is nothing to show the reason for that. It's like I unplugged the virtual power cable. One could argue that I'm using the free version, which is not production ready, but if that doesn't work well I wouldn't pay for it. With vmware I had much less problems with virtual machines in general. I think proxmox has some polishing to do. I'm thinking of trying xen for a while. It's nice to try different options.
I am a fan of XCP-NG but it does have a gap in enterprise environments, or even a home lab. Try to load/import enterprise VMs such as Palo Alto firewalls, Pulse Secure and a many others. They will not work. These will run fine on VMware and Proxmox. Palo Alto does support Xen ( have virtuals deployed in AWS on their customized Xen ) but the Palo Alto Xen image can't be imported into XCP. So, if you only need Windows/Linux/BSD VM support, this product works really well. For other enterprise solutions, you may have to stick with VMware/KVM/HyperV.
As far as I'm concern Proxmox is now more stable then few years ago. KVM seems to be faster then XEN too. But 3-2-1 backup with it's backup solution is atrociously inflexible. Orchestra is not perfect either as it not support compressed incremental backups, but still is far more intuitive and simple in that regard. I still find disk image limitation in XCP-ng a problem, as unfortunately I do have some images that big.
@@JustSomeGuy009 No, no, no... Proxmox backup is more of a storage solution. It enables server do all that staff. It cannot create disk image out of backup, cannot extract files form backup and basically is useless without a Server itself. That a problem if you keep backup on remote location with network as bottleneck. Moreover if I lose my online infrastructure due to unforeseen circumstances I need to rebuild it as it was. I know that would not be possible in short time. Meanwhile offline off-premise backup will be useless to me in that period. Very inflexible backup solution when it comes to 3-2-1.
@@wielkiptok For your use-case i would recommend to install both PVE and PBS on your off-premise server. It will give you all features you mentioned and an ability to run some important VMs off-site during disaster recovery.
@@mvs3246 When server is on the same site as backup, then backup is no longer offsite. But I get what you mean. I just do not have enough redundant infrastructure offsite for that to work (and money is the factor). In that kind situation I prefer that backup solution would operate on disk images not unknown chunks, that all. It makes stuff lot easier and would let me panic in more organized fashion.
Hi Tom, Thanks for your video, I think unfortunately you skip very easily and elegantly over proxmox, it is very clear that you do not really use Proxmox, many of the things you show in XCP-NG can also be done on proxmox, maybe not quite as nice and elegant, I have been using proxmox for 12 years now and thought it was a bit exciting to see something other than proxmox. I thought your video would have been better if you could have found a super user who knows proxmox in and out, and then could have set up some scenarios, with how it works on both platforms, with many of the things you showed on XCP- NG.
@UC53kDE601V2xOx1hRVI4v8Q I fully understand the choice of XCP-NG, and as I wrote, I thought it was exciting to see a platform I am not familiar with. I'm even wondering if I should try XCP-NG in my LAB, and thought there are many nice things that XCP-NG can do. And of course, this is not a 1: 1 feature comparison video. But it would have been very nice if in e.g. 2-3 minutes could have been shown how e.g. a proxmox user with many VM's would handle similar. scenario as Tom describes he does with XCP-NG. We are dealing with 2 very different systems that can solve some similar tasks in their own way, and it was very clear to see that Tom feels at home in XCP-NG. If a new video is to be made about exactly XCP-NG and Proxmox, I wish there could be a little more how can the 2 systems be administered easily in something similar to everyday tasks, and then compare the workflow on these 2. And I think this is perhaps best done if the proxmox part is then displayed by someone who knows and uses proxmox. Without having experience with, or trying XCP-NG myself, my thought is also quite clear that if you like proxmox as I do, then proxmox is perhaps best for single host, But I have a belief in XCP-NG, can easily be best choice, when setup with multiple hosts. I manage several systems with proxmox as single hosts, and a few clusters in operation with up to 4-5 hosts.
I have been running proxmox for a year now for my homelab workflows. Never had any issues with it but I am loving the xen orchestra interface! It’s well thought out and has a better user interface compared to proxmox. Proxmox team please take note.
I couldn't possibly care less about how pretty the UI is... I just want it to work, and work correctly and intuitively (eg following the principle of least astonishment). I will admit the different trees Proxmox puts things in is a bit wacky... and they should just abandon that and stick to one that just works.
Neither is great with terraform, first I chose xen because it seemed more fleshed out. Having tested both the proxmox provider have worked much better in reality. There are definitely very frustrating issues still however.
Nice overview, but what about GPU pass through for example ? We use that a lot in our work vms and proxmox has been able to do it (albiet not flawlessly)
Personally I hate the XCP-NG interface but then again I'm more familiar with VMWare so Proxmox is way easier to use. I mean its kind of like Linux GUI's the main reason why so many people like KDE is because it looks like windows. Also in most use cases where scale mattered that much they should probably use VMWare or consider a Cloud solution. That being said I used Proxmox on my home lab and VMWare at work. I just viewed Proxmox as a less bloated version VMWare especially if you only have a 1 or 2 cluster setup.
Great video on a topic that is now pretty much at a junction of which way to move off ESXi. One question I have is - If I were to run XCP-NG free edition with no support for say my development or "not so important" workloads then could I still manage these nodes with a paid version of XOA?
Used XCP-NG for the past 2 years. Lost entire environment when some kind of file corruption was introduced and it would NOT let me export VM's. Wiped entire environment, started over. Was experimenting with backing stuff up, tried to export VM's so I could do extensive testing. Exporting failed so early on it didn't even create the config files for the VM's. 100% unable to export VM's again. Wiped environment *again* and went bare-metal. I can't keep having to wipe my whole environment because XCP-NG was failing with 2 different fresh builds to export VM's. Any VM. Windows, Linux, didn't matter. No errors thrown in GUI.
@@LAWRENCESYSTEMS I can understand the file corruption - but not having file corruption and being unable to export VM's was a show-stopper for me. I wasn't too crazy about Proxmox, and, seeing as how all of my hosting was in docker containers, I decided to go bare-metal. I had to give up my PXE boot server though - I'm not too comfortable yet with building that in a docker container.
@@hiddenfromyourview Have also experienced corruption on XCP where we copied a VM to a different pool and on trying to start it, it wouldn't come up. The most frustrating thing was that even the original VM wouldn't come up. It had a disk size of 1TB and on checking on the storage itself the parent vhd file was merely 3.2MBs
The only benefit I see in XCP is tiny extra performance of Xen over KVM and maybe hosting only one running web-UI. But we are talking tiny advantage here that you just won't notice in either production or homelab imo. Beyond that, Proxmox has every feature I think I've heard you mention in most video's. But Proxmox is more of a "install and go" experience. I want to like Xen stuff because it's part of Linux Foundation Project. But it's just makes me think.... "yeah and.... so me something special". This said, I'm currently running on Cockpit, but it's like going backwards by a decade. so heading back to Proxmox
If I follow the guide & compile a full copy of Xen Orchestra myself.. will updates via web console work or will I have to compile another version each time?
2TB disk size is a deal killer. Backup solutions are virtualized now a days... and say I have a Veeam backup server backing up several machines. 2TB gets burned up really quick. Even if you're on shared storage such as a SAN, 2TB limit is horribly low.
@@wayland7150 Veeam is the backup server. VMware supports up to 62TB per disk. Say you wanted to backup a 2TB file server, which contains large engineering files. Your backup server is Veeam as VM. With this statement, Veeam wouldn’t be able to backup that 2TB file server.
Excellent video Tom. You clearly demenstrated why XCP-NG is/should be the choice for big installations. The interface and options are much more polished and versitile. Proxmox is a good solution for home-hobby/small environments that can deal with more hands-on approach as it allows the LXC containers to be deployed(like myself). The features of tagging is just a killer for production support folks. Proxmox would be well adviced to do so in the future.
Can Zen Orchestra be hosted on the same hardware as the XCP-NG hypervisor? That is have a VM that runs that stack, or even install the package within the Debian environment it is built on? I have a small home lab with only one node that does all the things.
I've been running Proxmox for about 5 years now, and for my homelab its been great. I would however love a solid solution that's FreeBSD and Byhve based.
Proxmox is arguably better it functions more like esxi and vcenter than Xen Orchestra. The one thing that works better on XenOrchestra is clustering and cross cluster transfers. Proxmox is limited by being HA all the time.
I love proxmox but I run into a challenge performing differential/incremental backups on proxmox. It performed full backups each time. If I recall correctly this affected windows VMs most. However on XCP-NG the backups were flawless. Both differential and full. Has anyone experienced such an issue. I am strongly considering reverting to XCP-NG because of this
@Astrocat 3D I get that, but if you're not familiar enough then you have to approach the issue differently, not as a comparison because you wouldn't be able to provide a fair comparison. I manage an XCP-ng pool with very "enterprisey" things, but even myself who works daily with this system, have tried Proxmox for different scenarios and trust me, it's WAY easier than what Tom makes it to be. And a lot of advanced things can be done much easier in proxmox than in XCP-ng. It's just a matter of understanding that you will not do things "the xen way" when dealing with proxmox. It's like trying to run the latest AAA games or MS office on Linux. It can be done, but you will not download a .exe and next, next, next. You HAVE to understand that it's a different system, and yoy have to learn the system to understand how to do things. It's not that it can't be done or it's more limited, it's just different. In short, yes, I fully understand that Tom is very at home with XCP-ng/XO, and I agree, it's a very solid platform with a lot of good things about it. But if the format of the video is a comparison, then bring Jay as a guest so he can properly explain how to achieve the same in proxomox as Tom is not an expert in that regard. Otherwise just refer to his channel and don't make a comparison at all. It's misleading AF and leads people who are learning into the very much not needed complexity of XCP-ng when Proxmox is way beyond what they'll ever need, and let me tell you from experience. Proxmox is by a long shot easier to repair and diagnose when things go south. New people learning virtualization platforms better keep off XCP-ng, once you are knowledgeable enough, sure go ahead. But XCP-ng in production is rock solid, until it fails and there's no easy way to fix it! So yeah, not so easy as to say "Tom said he's no expert, go to Jay's channel for proxmox". That's not how it works, by doing so you're providing a biases, unfair comparison and misleading people who don't know better. There's a responsibility that lays on you once you become influential.
@Astrocat 3D move on then, good luck in life with that attention span though! Tells a lot when you react dismissive as soon as someone brings logic that contradicts your opinion. But to each their own. I respect that you just don't care, thank you for letting me know, that's fair. Have a nice day 😉👍
@@KnightRiderOfVoid He explains why he chose it. It is not a direct comparison of every single feature. You chose yours for your own reasons, lol. It’s a preference.
@@farmeunit I know what he said, I watched the video too. The thing is, even if he didn't mean to make it a comparison, it ended up being a comparison format. And as a comparison it was a really bad one. Sure, you could say he was not intending to do so, but he did make an unfair comparison. Better off avoiding the topic details if you're not going to actively learn and give a balanced and fair comparison no? And don't get me wrong. I know exactly why Tom uses XCP-ng instead of proxmox, and trust me it's not just because he's more familiar with it, there's simply a couple scenarios where he's absolutely right going that route instead of Proxmox. But I'm not criticising that. I support that! The thing is whether he intended or not, he ended up making a comparison, and a very biased, unfair, misinformed and overall poor and misleading comparison. He knows better, I follow his channel not to troll and be the devil's advocate. I respect the man and I know for a fact he knows better. That's why I'm very firm and vocal about this. It's not a "naive mistake", he just didn't think of the responsibility he bears to not mislead less experienced people with his influence. That's all. Edit: way too much typos, hate mobile keyboards! 😅
@Lawrence Systems Kindly advise me how I can compile xen orchestra to have the version of the Premium so I can play around with all that features. Thank you in advance.
Bonjour, je me pose la question quel est le mieux, est-ce VMware Workstation Pro ou Proxmox. Le but est d'utiliser aussi la carte graphique et de faire facilement un copier/coller, pour installer des jeux ou programmes.
Both VMWare and Proxmox support GPU passthrough, so you shouldn't have any issues with either platform. For disk management, it depends on what you want to achieve. If you want to allow multiple people to play the same games simultaneously, you might be butter off with a multiseat solution such as Windows MultiPoint Server (sadly defunct as of 2012), MultiPoint Services for Windows Server 2015 or later (sadly also defunct as of 2018), or Remote Desktop Services for Windows Server 2016 or later. If you want to run Linux instead of Windows, search for apalrd's video on setting up a Linux terminal server, and/or look at the Linux Terminal Server Project.
It would be nice if you have the chance to share how do you migrate the machines from esxi, for example I'm able to export to ova/ovf but when I Import them on XPC it just says importing then says nothing and do nothing, I tested very small vm's with no luck. Best regards
I do have an old Dell server that is incompatible with VMware so it is running on Proxmox but I will try XCP-NG to see if it is compatible and probably move the 4 servers I have to XCP-NG if it is compatible as you can have all servers at one place. Right now VMware you have to open the GUI to be accessible with VCenter. I seen both videos of Proxmox and VMware comparison with XCP-NG and the Proxi looks promising. I do really love to see each videos you do but I do really think I need to look foward for a move to have all my servers to be accessible at the same place. The big test as I said will be my old Dell R320 that is old but still working good for some applications. I did moved him from KVM to Proxmox as KVM was a pain in the ass for windows VM still if the VM vas Light. Did any one tryed to give a GPU dedicated directly to a VM in XCP-NG ? The best way for me was VMware with modification in configuration. Thanks for all the infos!!!
What backup tools do people use with these open source solutions, or does it require running agents on all the virtual machines, vs using something like Veeam backing up or Replicating the clients on the Hypervisors directly,
Hey Tom, what are your thoughts on Harvester? It's a super simple ha cloud native hyper converged cluster. Been using instead of proxmox and xcp and it's nice!
Does XPC NG or Proxmox support NVMeof or NVMe-IP? I know with VMWare and Pure Storage you can enable NVMe over fiber to improve IO speeds. Also I saw some forum conversations and blogs on XCP-ng which talked about being able to use LVM command line to extend a volume beyond the 2TB limit, was wondering what the pros/cons of that are.
@@rreveler6705 thanks for the reply. i have an old "Sansa" and you need to load it with the MP3 files. i have tried my Samsung phone but only screwed up the phone after a few times. i think i should stick with the "sansa".thanks.
Here is an excellent post by Olivier Lambert who runs Vates, the company behind XCP-NG & Xen Orchestra covering some very important details regarding the history and architecture of the hypervisors
forums.lawrencesystems.com/t/xen-vs-xenserver-vs-kvm-vs-proxmox/14256/2
awrence Systems Forum Post
forums.lawrencesystems.com/t/xcp-ng-vs-proxmox-2022/14200
Lawrence Systems tutorials on XCP-NG
lawrence.technology/xcp-ng-and-xen-orchestra-tutorials/
Learn Linux TV Proxmox course
www.learnlinux.tv/proxmox-virtual-environment-full-course-class-1-getting-started/
Learn Linux TV Proxmox Backup
www.learnlinux.tv/proxmox-backup-server-full-getting-started-guide/
⏱ Timestamps ⏱
00:00 XCP-NG VS Proxmox 2022
02:30 Features Comparison
03:09 Explanining Xen Orchestra
08:15 Max Support VM Disk
11:30 Proxmox interface
13:57 XCP-NG Xen Orchstra Interface
I also have used both and both are very powerful. However, I have mostly used Proxmox and have used it for the last 13 year in large scale and it really is a solid enterprise solution, especially together with CEPH. I know every corner of it, it can’t surprise me with anything I can’t do or fix. I believe Lawrence have the same knowledge about XCP, there is nothing that will take him by surprise that he can't do or fix, and this is really the right reason to choose one or the other. Choose one and stick with it, be a god with it and it will never fail you.
How/Did do you solve the IaC? Terraform? Ansible? Other?
How to get cheap addon for whmcs with Proxmox？
I find Proxmox interface much easier: this is just to reiterate it is a matter of preferences between two very good products.
Proxmox is simpler interface however I can see why Lawrence likes Orchstra.
@@wayland7150 prox is dated on the interface but it’s ok it works. I still prefer xcp
XEN Orchestra is payware, correct? Is Cheph the same way?
I have used both and settled on Proxmox. I really didn't care for the XO interface, and the VM console is very limited. I want to be able to break out the console to its own window, and Proxmox can use SPICE which is vastly superior to what XCP-NG offers. Granted, I haven't used XCP-NG for over a year so maybe things have changed in that regard.
I rarely ever use the console except to setup a VM. Once configured I use either ssh or RDP.
@@LAWRENCESYSTEMS For sure. I thought I saw Olivier say they were working on SPICE integration. That might be enough for me to try XCP-NG again.
+1
The lack of SPICE support is an argument for not choosing XCP-ng.
I've used both also. XO was cripple-ware when I last used it (certain features were turned off unless you paid). Proxmox has ZFS, SPICE, it's 100% free, has replication, ZFS snapshots, HA, etc. Really no comparison. Proxmox is the way to go for sure.
@@SteveWebbbb More advanced ZFS support and the fact that Proxmox has its own web interface integrated into every host while being lightweight at the same time is also more important to me than having drag-and-drop feature for the boot order on a VM or a toggle button for Windows guests.
Not to mention the container support in Proxmox...
Love that you have Jays Proxmox video's linked.
XCP NG looks beautiful, I'd love to delve into it if I had the need but for my small enterprise Proxmox + TrueNAS iSCSI is bulletproof. It's easy to train my staff to use it as its paradigm mirrors real hardware and snapshots and replication are painless...
I have a question for you regarding Proxmox and TrueNAS iSCSI setup. I ran into an issue with getting proxmox to connect my iSCSI storage on both hosts (2 nodes). Each node connects to the TrueNAS server via a direct cable connection and on a different ip that is on a different subnet, no switch as I have dual 10GB nics in each host (proxmox and TruNAS systems). What I found was I can add the storage but only input one of the two ip addresses the TrueNAS server is answering on. The fist proxmox host sees the storage but the second hos sees the storage as off line. I did some research into multipath configuration but that didn't work for me. Any thoughts?
@@bdunigan80 You can only have 1 default gateway on TrueNAS and it is usually for the management interface. All other NICs (for iSCSI/NFS) on TrueNAS for sharing are gateway-less as your servers that are consuming them have NICs on the same (dedicated) network segment. Maybe you just have a misconfigured NIC on either TrueNAS or servers that prevent you font seeing it?
If you are using two different subnets check that you have a portal on each subnet. Or get a 10Gb switch and keep both servers on the same subnet and portal. Good luck.
Certainly does look nice, for my homelab I'm probably steering to Proxmox, with iSCSI storage. However XCP-NG mmm...
The affero license is a non-starter. Anything that talks to proxmox becomes subject to it. Are you prepared to provide the source code for Windows?
Great video that answered quite a few questions about the two systems, with the VM ESXI drama going on right now, I bet this will start to get more views as people start looking for other options.
Tom is right. It depends on use case and your familiarity to the particular tool you intend to use.
Once you master any of the tool, you can invent the work around its shortcomings.
Thank you for putting this comparison together.
I've been looking at the two platforms, and I think that one of the ways for me to experience it would just be to try it.
I think that it IS important to mention though, that Proxmox does not have an easy/default way to import OVA appliances (so if you are say, migrating from VirtualBox or some other virtualisation ecosystem -- I would mention that).
Also, I would mention, in your comparison chart, that XOSAN is a ***PAID*** feature.
Therefore; if someone wants to use hyperconverged features and Proxmox allows you to do it for free (even if the hyperconvergence feature set is limited with the free option), that might also be worthwhile to note for some of your audience members.
Proxmox does OVM? Oh!
@@wayland7150
I don't think that I can import OVAs into Proxmox easily/readily like I am supposed to be able to with XCP-ng/XenOrchestra.
But I guess that I'll find out!
We shall see.
I love the comparison, so thank you Tom, and this gives some really terrific aspects of XO and XCP-NG. I'd love to find out if XCP-NG plans to support LXC / LXD Containerization int he future. It's just a no brainer to want to use those for much of the VM work these days due to the far lower resource needs for a ton of the use cases out there.
Containers would seem to be out of scope for XCP-NG, which I think is trying to just be a solid, easily administered secure hypervisor. Proxmox is meant more as a home lab all in one server, so host OS containers make sense, but I expect that a "correct" implementation of LXC on XCP would be inside a guest OS.
I love both options but for my homelab I went with proxmox because it has just worked so easy.
Its kinda silly that people think just because you dont use something its bad. The main reason apart from the second is i like the XO interface look and if im spinning up stuff all the time i at least want the ui to look sleek but on a different note for me XCP-ng turned out to fit my use case because of the meany to one relationship my lab scales up and down systems get added and tour down so having the signal pain of glass setup were i can just add servers and remove live or shutdown migration of VMs is really powerful
Thanks for this. It is helpful to see the comparisons.
I’m not rushing away from vSphere immediately but this was interesting. Thanks Tom.
I'd like to see a vSphere vs Prpxymox comparison, it seem that proxmox has a reasonable attempt at the feel of vSphere. Mind you it has been two years since so perhaps I'm mistakened.
Proxmox can somewhat do the tagging features with their pools. You add resources to a pool and you can have some actions target everything in the pool.
I use pools in ProxMox as well to help figure out which group of VMs are using TrueNAS storage so I can shut those VMs down before rebooting it due to updates.
This is informational and really well done, Tom. I've worked with so many Hypervisers through the years I couldn't tell you my opinion of each of them; when it comes to Citrix (the heart of where Xen is) I always remember going back to Citrix terminal servers in the late 90s off of NT4.0 to enable dumb terminals at a hospital client.. I digress. Most of those I work with are VMWare, or, in some cases, Microsoft Hyper-V under a setup where the root server cannot be virtualized due to specific hardware passthroughs/key security issues.
In this sense, I've always preferred VMWare. Once VMWare 7 came about, it put a stop to a lot of the things I really liked about VMWare, and I've looked at other options. Their are a few things here; I am not opposed to cost; I have always preferred as a backup solution Acronis for varying reasons - most recently native endpoint web management and full security modes, but also easy export to review for test conversion (because if you aren't testing a backup on a schedule it's worthless)
This is where I have issues with XCP. I like the core concept of it; but the way that XO is structured really boils down to a "pay to play" $85/mo*12 year for features included in ProxMox, which has a newer, and more frequently updated Hyperviser in KVM, and Xen is absolutely showing it's aging when I look at the IOPs I see...
I think what makes XCP really solid is that XO connects to multisite in a way that really only Microsoft's manager compares to in node control for Hyper-V, in a different way; but if you're using Hyper-V it is is, again, often for transition purposes, specific hardware pass-throughs, or for hardware supports.
If you're a homelab, ProxMox might be more likely to provide you exactly what you are after; the interface is instantly there, many of the items that are removed from XO without $ are present (backup), and easy management capability works. XO's handling of XOSAN and backups is a great feature, but it is very important to mention they are -not- free or included by default, as people who walk through the default install will find they can't get there.
Your videos are really informational. Every hyper viser I've used has it's strengths and weaknesses.. It boils down to what is the end goal of the Hyperviser and what am I trying to accomplish
Brief add on: screw my auto-correct; I do know grammar well enough to recognize the correct version, and it should be "THERE"..
This was pretty nice, but I think it would be much more interesting to have a proper comparison of strengths and weaknesses between XCP NG and Proxmox with maybe some guest youtuber who knows Proxmox well. Would be even nicer if you could do some more advanced scenarios such as Ceph vs. XOSAN.
Looking at discussion forums after Broadcom's announcement to buy VMWare, sysadmins' choice to replace VMWare seem to be leaning more toward Proxmox while XCP NG isn't often even mentioned. Might be because Proxmox has been around much longer than XCP NG though.
Would love to see the performance differences between Ceph and XOSAN. XOSANv2 is coming out soon so rather wait on that.
Nice video and a good try to do the VS thing but 2.5 mins Proxmox against the rest of the video ( 10+ mins ) isn't really a Versus video. I understand you are more at home with XCP and therefor can demonstrate more features. This would be the reverse with someone using Proxmox daily.
Some clarification though:
Backups do not need the Proxmox backup application. There is backup build-in and has options to include/exclude disks, so big disks do not have to be an issue.
Great video, I've been using clustered Proxmox but really like the XO interface you showed.
Excellent evidence-based presentation. Quality tech advice does not get much better than this.
Thank you
I use LXC containers a lot. Basically all my services run on Debian CTs. It works really well in Proxmox.
Agreed. I know that this isn't a priority for xcp-ng and that there's a myriad of alternative options you can pursue, but there's something just _nice_ about having your VMs and CTs 'colocated' in the same workspace/GUI. Currently using xcp-ng but considering moving back to Proxmox for this reason (wasted memory by having docker running in VMs)
@@transatlant1c Yes. It is quite nice. The only thing Proxmox doesn't have is native docker containers. But that is mostly because it is supposed to run LXC CTs. It is possible to run Docker cts inside a VM or even a n LXC CT, but it is not trivial, it will have another virtualization layer and it can be somewhat buggy. I try to avoid docker into proxmox.
@@gustavinus yep that’s right, when I was going down that path I found setting the CT as unrestricted and enabling nesting, along with a turnkey Linux image was successful but results may vary!
@@transatlant1c yep... with those settings it runs pretty much natively on the Proxmox kernel. I tend to use Debian for those CTs because Proxmox is also base on Debian.
One imporant fact: XCP-NG is based on Xen vs. Proxmox on KVM. The former is gradually losing the battle against the latter in the commercial virtualization market. This may be not quite noticeable to a home lab user but personally I would be more inclined to stand for the future.
Doesn't Citrix use Xen?
@@EarthStarzYes. XCP-ng was originally a fork of Citrix XenServer, which is now just called XenServer.
proxmox ceph storage is far better than xosan (very expensive) , but at the end overt is also interesting...and good.....
cannot agree more
settled on Proxmox, KVM is easy to set up when testing things like beta versions of MacOS or Windows, VMs are basically set and forget :)
Proxmox is what I've using. I haven't tried XP-NG or XO before. I will delve into it in the future.
An awesome teacher. This channel is a must watch for me and has long been in my toolbox.
Thank you Tom
For small team and self managed VM server, I feel like proxmox has more online videos and tutorials to learn from, XCP-NG seems to be much less. Thus, I consider proxmox has a easier learning curve. Not sure if my impression is right.
Two factors at play. Learning curve and system capabilities. Obviously you want an easy learning curve to get started but if the thing does not have the features you need then it gets increasingly difficult as you implement workarounds..
4:22 Get all features by downloading the source code. (Just has No Support added to the interface).
-- I've tried both, and while Proxmox looks nice for a one-server student lab, it doesn't scale nearly as well as Xen/XCP-ng/XO. For large-scale deployments, the Xen/ng/XO stack is fairly reliable and the Xen Hypervisor is a lot more secure and quicker
Firstly great video not less from you Tom. I am newbie to OpenSource applications. Been to long brain washed with those high end applications. A friend introduce me to Proxmox I have installed it on 3 server and clustered them as well. Yes was really easy install and setup. I am looking at migrating a number of VM from Hyper-V to Proxmox or XCP NG. I have not installed XCP NG but after this video I will be formating the Promox lab setup. My goal is provide easy management for my staff knowing who has done what but also key factor migrating from Hyper-V.
Thanks for this comparison. A few typos in your matrix: 'suported', 'stoage', 'packacge'. I'm waiting on a Beelink GTR5 to spin up a virtualization lab and looked at XCP-NG but I don't like it as compared to Proxmox. I'll be running quite a few Linux VMs so the LXC container support in Proxmos will really come in handy.
Good comparison at a glance as a good starting point, thank you. Slight typo in the "Shared Storage", for Proxmox you put "CefRBD" instead of CephRBD. Also "stroage" and "packacge".
Thanks, spelling is always a challenge for me, maybe I need to hire a proof reader.
great video!! thanks for the comparison. I think I will still keep my proxmox setup, due to the LXC containers I have. And I wont have so many VMs haha. Also, I was not able to find much information about GPU/PCI passthrough for xcpng, and I use the VMs for gaming. thanks again!
It is not supported through the UI but you can do pass through
@@LAWRENCESYSTEMS care to explain more?
I tried both, but I like proxmox more. The Webinterface is IMO far more intuitive, it runs perfeclty stable, most people already know kvm/qemu and I don't need a VM just to have a webinterface.
But that doesn't mean XCP NG is bad.
Also, at least for Samba and NFS the used storage graph works fine
Who says you hate Proxmox ?
Dude i've found out about Proxmox 5.X from you and i've been watching your channel when you switched from Proxmox to xcp-ng.
I love your videos about Proxmox, XCP-NG, pfSense, TrueNAS. (not so much Synology and Unifi)
Many thanks for the excellent video. I have to confess I'm a Proxmox fan and love that system, but I was very impressed with XCP-NG in your video. It has a very nice web UI, unlike vSphere which looks nice but is buggy, and seems more capable at a very large scale.
Great episode with interesting looks into both systems.
Proxmox is deffo my favorite.
I love your open. I use in my home lab libvirtd on openSUSE, Proxmox and VMWare ESXi, if I had another server I would tried some thing. I don't use Citrix Xen because there is a limit on the free versions. Great Video.
Thanks for the video, lots of great info. That guy Tom that keeps showing up in the logs seems suspicious though.
He's the biggest risk to having a stable system.
I’m getting off ESXI for my home environment and your videos lately have been leaning me more towards XCP, plus it’s gorgeous
with the container management I had asked during a live stream and they will be implementing RunX which should allow for similar container management on the hypervisor.
Excellent point about large storage volumes and VM's, although I am not sure it really matters. Worth a look, I think.
To me, the real issue is Xen vs Qemu/KVM. Xen is pretty old and Qemu/KVM is clearly setting up to be the dominant Linux hypervisor of the future. Even AWS switched over to Qemu/KVM from Xen...
I do like a lot of the feature-set of Xcp-Ng and even Xen Orchestra, but if one is choosing to standardize around Qemu/KVM, then unfortunately both are really non starters. 😔
The majority of AWS is running on Xenserver, it's part of the Linux Foundation, and it's still very actively being developed.
@@LAWRENCESYSTEMS All new AWS instance types released from ~2017 forwards (ie anything on AWS' Nitro system) are using KVM, including all Graviton products. So... Unless you're sure the majority of currently running AWS' instances are using older instance types, that may not be a factually correct statement. 😉
Not trying to give you a hard time... But enterprise-scale cloud DevOps is my day to day, so it's an area of the market I track quite closely.
Xen will no doubt be around for quite some time; speaking of AWS they even built a compatibility layer for it, so older Xen-only OS deployments don't break with KVM-based instances. But if you look at market share trajectory over time, feature velocity, and enterprise backing, it's clear KVM will eventually become the standard for HV in the Linux space.
@@shammyh Linode is also using Qemu/KVM in their infrastructure. I've created a few test Linux VMs on their platform and took me a few mins to figure out what their host systems are running which are in fact using Qemu/KVM. Wouldn't be surprised if they are using ProxMox to manage the VMs and Host as it can be fully scripted to provision the VMs.
The most important question is whether understanding Xen can be an asset on a professional level. I'm not worried about Proxmox because KVM is the most widely used hypervisor on the market along with Hyper-V and ESXI but Xen, there aren't many job postings about it.
That might be the most important question to you. It is not to me when choosing a platform.
Just a bit of clarification that was not evidently clearbinnthe video. Hyperconverged storage using XOSAN is not available with Xen Orchestra from sources.
When you do things at scale, you do infrastructure as code ;-) For that reason, oVirt's terraform provider is pretty compelling, integrated with CI/CD of sorts like drone, pxe/matchbox and ansible (rundeck), you then get ability to manage infrastructure of basically any scale, and that's on-prem, in the cloud and in hybrid environment - simple and great for lab or bigger production project.
Yes, that is why they added Terraform support to XCP-NG registry.terraform.io/providers/terra-farm/xenorchestra/latest/docs
I use some (older) Infiniband hardware for my storage backbone and having the OS not hinder my ability to install what I need to get it working is kind of a big deal.
For me the "custom", and rather limited, when it comes to hardware support, CentOS image was a big strike against XCP-NG.
Similar issue with TrueNAS Scale where they actively block the user from installing additional packages.
Doesn't necessarily matter for professional use-case, but it does for homelabbing.
Yeah I also had some older motherboards where xcp-NG would just panic and crash, the same motherboards work fine on Windows or with a modern Linux system (or proxmox/Ovirt)
I'm using Proxmox since 2 weeks installed it on i7 875K (2010 plarform) with 16Gb RAM and I love it. I run Openmediavault RAID5 as VM (HDD passthrough) and 2 minecraft servers - one in VM docker on Debian Stable second in LXC archlinux container. Perfomance is stability are great. I had problem tho with True nas and ZFS RAIDZ but I gues it was too demanding for that PC to handle.
I run a Storj node as a VM in one of my servers. The virtual disk attached to it is 27TB right now (I expand it as it gets filled up). I also used over 2TB virtual disks when I was mining chia and I have a 3TB virtual disk on a VM that is backing up my other physical machines.
The 2TB limit is quite small to be honest, I know some VMs our clients have that have more than 2TB virtual disks.
Of course, with a limited virtual disk size it would still be possible to just attach multiple virtual disks to the VM and then combine them using zfs or software RAID inside the VM, but that feels like more complicated setup for no real gain.
I like backing up using zfs snapshots (on the host) and doing "incremental forever" with only one "full" backup.
As I read, both have issues with iSCSI multipath. That's reason why I would probably move to VMware, even it is most expensive. In small business, I need a solution which is easy to administer and maintain.
I have never tried iSCSI multipath in Proxmox, but it works fine in XCP-NG
Very informative. Thank you Tom.
Even more interesting is the new kids on the block. OpenShift/OKD Virtualization and Harvester. Leveraging Kubernetes and kube-virt. Red Hat is stopping support for RHV/oVirt (which XCP reminds me of) in favor of OpenShift Virtualization.
Stopping? No. they are not stopping anything. Oracle/Rocky,Alma will support for many years ahead oVirt.
@@levskilevov4888 I didn't say that oVirt itself was stopping. Just that one of their biggest contributors is stopping support for them. That already small community is about to get even smaller.
Personally I cannot get over the ackward XOA web gui interface.. It just is very unintuitive (to me). XOA-center isn't really an option since it only has limited features. Proxmox has a lot of problems in that area too. However I temporarily switched back to proxmox because XOA kept throwing errors at me which made no sense.... basically the log kept filling up with erors. It seemed every 1 out of 5 actions I tried to perform, gave me errors.. I've been around IT systems for 40 years and they don't make sense.... anyways... Of course all of this is very subjective to personal taste.
One thing that bothers me about xen is that the opensource guest tools are developed/maintained by 1 overburdoned guy and the offical ones are being a login wall now.. (new ones anyway). Without those guest tools you might as well go back to T2 virtualization..
Funny because I find xen orchestra to have the superior ui compared to proxmox but to each their own. Also Having only 1 guy responsible for the open source guest tool is alarming though.
@@bthegawd8113 It is really a user preference. I myself actually find Proxmox UI more odd ball when comparing with Xen.
@@TanKianW79 yeah the proxmox ui looks like a first year college web design project lol.
Yeah... I am now running XCP-NG inside a proxmox VM and I have to say... It may also be just a bit of knowing how to do things the right way.. Some of the gripes I had with it are probably mostly due to my own inept skills with it... Keep on learning :) Thanks to Tom's videos I've learned more about virtualization and firewalls than in the 20 years before that :) btw @tom ... I've just been watching some of your 2016 on-site videos and they are amazing... Thorougly impressed with what you did there and are still doing..
I actually agree about the XOA interface.
For my homelab I still regularly use XCP-NG center (the former Xencenter app) on Windows.
Ok, I hate you 😜 On VMware and now Azure we have big machines, some disks around 16TB, some machines up to 36TB, and we back them up ok - although with huge disks you really must use CBT (change block tracking). But yes, best to avoid hugh disks/VMs as restoring them could be challenging. We did use VMware SRM, and now Azure ASR, so possible to failover to another replicated copy, in another location, quickly, within your RPO and RTO. Just installed Proxmox, great video as always, I will have a look at XCP.
In xen-orchestra from source you will have not all features. Example is Proxy. It is not available in XO from source. XOSAN not available in XO from source
I too settled on ProxmoxVE after using both. PVE feels solid and mature. I find this video a bit biased so I advice newcomers to research more. Also, the argument regarding the low attack surface is misleading. PVE is basically debian linux which already runs on the majority of internet servers. Lastly, XCP-ng has some grey areas, e.g. installing guest tools
XCP guest tools is the one thing I really dont like about XCP - Ive had major headaches with large MSSQL workloads where the network just drops. Still evaluating both but on the Proxmox side I think it has a terrible backup solution unless you move to PBS. That then requires another separate CIFS share for file backups from Windows.
I'm using proxmox for a mont or two. I have my work pc converted to proxmox and installed windows on a vm with hardware pass through (RTX 3060, 3 monitors few usbs, pcie pass nvme disk). Problems I had so far: proxmox doesn't do a great job of passing error messages from kvm. PCI-passthrough will sometimes hang and proxmox won't know what the problem is. Once you hit the right settings it works fine. However I had problems with usb devices disconnecting. One time the whole VM powered off after plugging in passed-through usb device. Later I checked - there is nothing to show the reason for that. It's like I unplugged the virtual power cable. One could argue that I'm using the free version, which is not production ready, but if that doesn't work well I wouldn't pay for it. With vmware I had much less problems with virtual machines in general. I think proxmox has some polishing to do. I'm thinking of trying xen for a while. It's nice to try different options.
I am a fan of XCP-NG but it does have a gap in enterprise environments, or even a home lab. Try to load/import enterprise VMs such as Palo Alto firewalls, Pulse Secure and a many others. They will not work. These will run fine on VMware and Proxmox. Palo Alto does support Xen ( have virtuals deployed in AWS on their customized Xen ) but the Palo Alto Xen image can't be imported into XCP. So, if you only need Windows/Linux/BSD VM support, this product works really well. For other enterprise solutions, you may have to stick with VMware/KVM/HyperV.
Would oyu consider doing a deeper dive into XOSAN?
Eventually
Does XCP-NG have any integration or offload for storage like VAAI?
As far as I'm concern Proxmox is now more stable then few years ago. KVM seems to be faster then XEN too. But 3-2-1 backup with it's backup solution is atrociously inflexible. Orchestra is not perfect either as it not support compressed incremental backups, but still is far more intuitive and simple in that regard.
I still find disk image limitation in XCP-ng a problem, as unfortunately I do have some images that big.
Just use proxmox backup server for backups. Works good, incremental, retention shcedules, etc
@@JustSomeGuy009 No, no, no... Proxmox backup is more of a storage solution. It enables server do all that staff. It cannot create disk image out of backup, cannot extract files form backup and basically is useless without a Server itself. That a problem if you keep backup on remote location with network as bottleneck.
Moreover if I lose my online infrastructure due to unforeseen circumstances I need to rebuild it as it was. I know that would not be possible in short time. Meanwhile offline off-premise backup will be useless to me in that period. Very inflexible backup solution when it comes to 3-2-1.
@@wielkiptok For your use-case i would recommend to install both PVE and PBS on your off-premise server. It will give you all features you mentioned and an ability to run some important VMs off-site during disaster recovery.
@@mvs3246 When server is on the same site as backup, then backup is no longer offsite. But I get what you mean. I just do not have enough redundant infrastructure offsite for that to work (and money is the factor).
In that kind situation I prefer that backup solution would operate on disk images not unknown chunks, that all. It makes stuff lot easier and would let me panic in more organized fashion.
Hi Tom, Thanks for your video, I think unfortunately you skip very easily and elegantly over proxmox, it is very clear that you do not really use Proxmox, many of the things you show in XCP-NG can also be done on proxmox, maybe not quite as nice and elegant, I have been using proxmox for 12 years now and thought it was a bit exciting to see something other than proxmox.
I thought your video would have been better if you could have found a super user who knows proxmox in and out, and then could have set up some scenarios, with how it works on both platforms, with many of the things you showed on XCP- NG.
@UC53kDE601V2xOx1hRVI4v8Q I fully understand the choice of XCP-NG, and as I wrote, I thought it was exciting to see a platform I am not familiar with.
I'm even wondering if I should try XCP-NG in my LAB, and thought there are many nice things that XCP-NG can do.
And of course, this is not a 1: 1 feature comparison video.
But it would have been very nice if in e.g. 2-3 minutes could have been shown how e.g. a proxmox user with many VM's would handle similar. scenario as Tom describes he does with XCP-NG.
We are dealing with 2 very different systems that can solve some similar tasks in their own way, and it was very clear to see that Tom feels at home in XCP-NG.
If a new video is to be made about exactly XCP-NG and Proxmox, I wish there could be a little more how can the 2 systems be administered easily in something similar to everyday tasks, and then compare the workflow on these 2.
And I think this is perhaps best done if the proxmox part is then displayed by someone who knows and uses proxmox.
Without having experience with, or trying XCP-NG myself, my thought is also quite clear that if you like proxmox as I do, then proxmox is perhaps best for single host,
But I have a belief in XCP-NG, can easily be best choice, when setup with multiple hosts.
I manage several systems with proxmox as single hosts, and a few clusters in operation with up to 4-5 hosts.
I have been running proxmox for a year now for my homelab workflows. Never had any issues with it but I am loving the xen orchestra interface! It’s well thought out and has a better user interface compared to proxmox. Proxmox team please take note.
I couldn't possibly care less about how pretty the UI is... I just want it to work, and work correctly and intuitively (eg following the principle of least astonishment). I will admit the different trees Proxmox puts things in is a bit wacky... and they should just abandon that and stick to one that just works.
Neither is great with terraform, first I chose xen because it seemed more fleshed out. Having tested both the proxmox provider have worked much better in reality. There are definitely very frustrating issues still however.
Thanks Tom.
Nice overview, but what about GPU pass through for example ? We use that a lot in our work vms and proxmox has been able to do it (albiet not flawlessly)
They can be done but not through the UI
IaC - It became standard approach with clouds. Why it's so neglected feature among hypervisors and their comparisons?
Personally I hate the XCP-NG interface but then again I'm more familiar with VMWare so Proxmox is way easier to use. I mean its kind of like Linux GUI's the main reason why so many people like KDE is because it looks like windows.
Also in most use cases where scale mattered that much they should probably use VMWare or consider a Cloud solution.
That being said I used Proxmox on my home lab and VMWare at work. I just viewed Proxmox as a less bloated version VMWare especially if you only have a 1 or 2 cluster setup.
Great video on a topic that is now pretty much at a junction of which way to move off ESXi. One question I have is - If I were to run XCP-NG free edition with no support for say my development or "not so important" workloads then could I still manage these nodes with a paid version of XOA?
Yes
Used XCP-NG for the past 2 years. Lost entire environment when some kind of file corruption was introduced and it would NOT let me export VM's. Wiped entire environment, started over. Was experimenting with backing stuff up, tried to export VM's so I could do extensive testing. Exporting failed so early on it didn't even create the config files for the VM's. 100% unable to export VM's again. Wiped environment *again* and went bare-metal. I can't keep having to wipe my whole environment because XCP-NG was failing with 2 different fresh builds to export VM's. Any VM. Windows, Linux, didn't matter. No errors thrown in GUI.
Not an issue I have encountered
@@LAWRENCESYSTEMS I can understand the file corruption - but not having file corruption and being unable to export VM's was a show-stopper for me. I wasn't too crazy about Proxmox, and, seeing as how all of my hosting was in docker containers, I decided to go bare-metal. I had to give up my PXE boot server though - I'm not too comfortable yet with building that in a docker container.
@@hiddenfromyourview Have also experienced corruption on XCP where we copied a VM to a different pool and on trying to start it, it wouldn't come up. The most frustrating thing was that even the original VM wouldn't come up. It had a disk size of 1TB and on checking on the storage itself the parent vhd file was merely 3.2MBs
The only benefit I see in XCP is tiny extra performance of Xen over KVM and maybe hosting only one running web-UI. But we are talking tiny advantage here that you just won't notice in either production or homelab imo.
Beyond that, Proxmox has every feature I think I've heard you mention in most video's. But Proxmox is more of a "install and go" experience. I want to like Xen stuff because it's part of Linux Foundation Project. But it's just makes me think.... "yeah and.... so me something special".
This said, I'm currently running on Cockpit, but it's like going backwards by a decade. so heading back to Proxmox
If I follow the guide & compile a full copy of Xen Orchestra myself.. will updates via web console work or will I have to compile another version each time?
2TB disk size is a deal killer. Backup solutions are virtualized now a days... and say I have a Veeam backup server backing up several machines. 2TB gets burned up really quick. Even if you're on shared storage such as a SAN, 2TB limit is horribly low.
I got the impression Lawrence was saying use a backup server rather than a 2TB attached hard drive. The server can be whatever capacity you need.
@@wayland7150 Veeam is the backup server. VMware supports up to 62TB per disk.
Say you wanted to backup a 2TB file server, which contains large engineering files. Your backup server is Veeam as VM. With this statement, Veeam wouldn’t be able to backup that 2TB file server.
Afaik you can also use tags in Proxmox, but I think they work a little different
Excellent video Tom. You clearly demenstrated why XCP-NG is/should be the choice for big installations. The interface and options are much more polished and versitile. Proxmox is a good solution for home-hobby/small environments that can deal with more hands-on approach as it allows the LXC containers to be deployed(like myself). The features of tagging is just a killer for production support folks. Proxmox would be well adviced to do so in the future.
Can Zen Orchestra be hosted on the same hardware as the XCP-NG hypervisor? That is have a VM that runs that stack, or even install the package within the Debian environment it is built on? I have a small home lab with only one node that does all the things.
Yes, you install it as a VM on the main host
wow, i really like the interface compared to proxmox, especially the tags. But also something simple as ip adresses, which i really mis in proxmox
Been waiting for this comparison! Also first lol.
What do you use for UPS graceful shutdown on XCP-NG? USB passthrough to a VM, or just install nut on the host?
I've been running Proxmox for about 5 years now, and for my homelab its been great. I would however love a solid solution that's FreeBSD and Byhve based.
But why? The last time I used Bhyve it was REALLY bad.
@@tobimai4843 I like FreeBSD. I find it more a real server OS. I found no performance issues when using Bhyve in FreeNAS/TrueNAS.
Can you cover how to backup or protect Xen Orchestra (HA diff sites) or just backup restore process.
I have all the tutorials for that and more here lawrence.technology/xcp-ng-and-xen-orchestra-tutorials/
Proxmox is arguably better it functions more like esxi and vcenter than Xen Orchestra. The one thing that works better on XenOrchestra is clustering and cross cluster transfers. Proxmox is limited by being HA all the time.
I use proxmox. But this is pretty cool.
I love proxmox but I run into a challenge performing differential/incremental backups on proxmox. It performed full backups each time. If I recall correctly this affected windows VMs most. However on XCP-NG the backups were flawless. Both differential and full. Has anyone experienced such an issue. I am strongly considering reverting to XCP-NG because of this
While I normally enjoy your relatively fair comparisons, this was far from the mark.
2.5mins of proxmox vs 9mins of xcp.
@Astrocat 3D I get that, but if you're not familiar enough then you have to approach the issue differently, not as a comparison because you wouldn't be able to provide a fair comparison.
I manage an XCP-ng pool with very "enterprisey" things, but even myself who works daily with this system, have tried Proxmox for different scenarios and trust me, it's WAY easier than what Tom makes it to be. And a lot of advanced things can be done much easier in proxmox than in XCP-ng. It's just a matter of understanding that you will not do things "the xen way" when dealing with proxmox.
It's like trying to run the latest AAA games or MS office on Linux. It can be done, but you will not download a .exe and next, next, next. You HAVE to understand that it's a different system, and yoy have to learn the system to understand how to do things. It's not that it can't be done or it's more limited, it's just different.
In short, yes, I fully understand that Tom is very at home with XCP-ng/XO, and I agree, it's a very solid platform with a lot of good things about it. But if the format of the video is a comparison, then bring Jay as a guest so he can properly explain how to achieve the same in proxomox as Tom is not an expert in that regard. Otherwise just refer to his channel and don't make a comparison at all. It's misleading AF and leads people who are learning into the very much not needed complexity of XCP-ng when Proxmox is way beyond what they'll ever need, and let me tell you from experience. Proxmox is by a long shot easier to repair and diagnose when things go south.
New people learning virtualization platforms better keep off XCP-ng, once you are knowledgeable enough, sure go ahead. But XCP-ng in production is rock solid, until it fails and there's no easy way to fix it!
So yeah, not so easy as to say "Tom said he's no expert, go to Jay's channel for proxmox". That's not how it works, by doing so you're providing a biases, unfair comparison and misleading people who don't know better. There's a responsibility that lays on you once you become influential.
@Astrocat 3D move on then, good luck in life with that attention span though! Tells a lot when you react dismissive as soon as someone brings logic that contradicts your opinion.
But to each their own. I respect that you just don't care, thank you for letting me know, that's fair. Have a nice day 😉👍
@@KnightRiderOfVoid awesome response. Thank you.
@@KnightRiderOfVoid He explains why he chose it. It is not a direct comparison of every single feature. You chose yours for your own reasons, lol. It’s a preference.
@@farmeunit I know what he said, I watched the video too. The thing is, even if he didn't mean to make it a comparison, it ended up being a comparison format. And as a comparison it was a really bad one.
Sure, you could say he was not intending to do so, but he did make an unfair comparison. Better off avoiding the topic details if you're not going to actively learn and give a balanced and fair comparison no?
And don't get me wrong. I know exactly why Tom uses XCP-ng instead of proxmox, and trust me it's not just because he's more familiar with it, there's simply a couple scenarios where he's absolutely right going that route instead of Proxmox. But I'm not criticising that. I support that!
The thing is whether he intended or not, he ended up making a comparison, and a very biased, unfair, misinformed and overall poor and misleading comparison. He knows better, I follow his channel not to troll and be the devil's advocate. I respect the man and I know for a fact he knows better. That's why I'm very firm and vocal about this. It's not a "naive mistake", he just didn't think of the responsibility he bears to not mislead less experienced people with his influence. That's all.
Edit: way too much typos, hate mobile keyboards! 😅
@Lawrence Systems Kindly advise me how I can compile xen orchestra to have the version of the Premium so I can play around with all that features. Thank you in advance.
The `tags` option is nice to have, on Proxmox also have it but only from console. no LXC support also is a disappointed
Bonjour, je me pose la question quel est le mieux, est-ce VMware Workstation Pro ou Proxmox. Le but est d'utiliser aussi la carte graphique et de faire facilement un copier/coller, pour installer des jeux ou programmes.
Both VMWare and Proxmox support GPU passthrough, so you shouldn't have any issues with either platform. For disk management, it depends on what you want to achieve. If you want to allow multiple people to play the same games simultaneously, you might be butter off with a multiseat solution such as Windows MultiPoint Server (sadly defunct as of 2012), MultiPoint Services for Windows Server 2015 or later (sadly also defunct as of 2018), or Remote Desktop Services for Windows Server 2016 or later.
If you want to run Linux instead of Windows, search for apalrd's video on setting up a Linux terminal server, and/or look at the Linux Terminal Server Project.
It would be nice if you have the chance to share how do you migrate the machines from esxi, for example I'm able to export to ova/ovf but when I Import them on XPC it just says importing then says nothing and do nothing, I tested very small vm's with no luck.
Best regards
Use clonezilla or your back up software of choice
Wonder why they wouldn't compile a version for each release.
I do have an old Dell server that is incompatible with VMware so it is running on Proxmox but I will try XCP-NG to see if it is compatible and probably move the 4 servers I have to XCP-NG if it is compatible as you can have all servers at one place. Right now VMware you have to open the GUI to be accessible with VCenter. I seen both videos of Proxmox and VMware comparison with XCP-NG and the Proxi looks promising. I do really love to see each videos you do but I do really think I need to look foward for a move to have all my servers to be accessible at the same place. The big test as I said will be my old Dell R320 that is old but still working good for some applications. I did moved him from KVM to Proxmox as KVM was a pain in the ass for windows VM still if the VM vas Light. Did any one tryed to give a GPU dedicated directly to a VM in XCP-NG ? The best way for me was VMware with modification in configuration. Thanks for all the infos!!!
What backup tools do people use with these open source solutions, or does it require running agents on all the virtual machines, vs using something like Veeam backing up or Replicating the clients on the Hypervisors directly,
With XCP-NG I use the backup system that comes built into Xen Orchestra.
Hey Tom, what are your thoughts on Harvester? It's a super simple ha cloud native hyper converged cluster. Been using instead of proxmox and xcp and it's nice!
Have not used it and not on my list to test
I was diagnosed with Hyper-V Syndrome🤦♂️🤣
I've got the Proxmox myself.
Does XPC NG or Proxmox support NVMeof or NVMe-IP? I know with VMWare and Pure Storage you can enable NVMe over fiber to improve IO speeds. Also I saw some forum conversations and blogs on XCP-ng which talked about being able to use LVM command line to extend a volume beyond the 2TB limit, was wondering what the pros/cons of that are.
NVMe-IP is not something I have tested.
Both look awesome but I think I'ma stay with Bhyve for now. :-P
What's the better backup solution IE delta snap etc for vms, proxmox or XO?
I prefer delta via XO.
Tom, have to go back to commuting. is there a way to convert the UA-cam video to MP3, so i can listen on the train? (Preferably a free converter)
I’m using UA-cam app on iPhone and there’s a Download button, but haven’t used/tried it.
@@rreveler6705 thanks for the reply. i have an old "Sansa" and you need to load it with the MP3 files. i have tried my Samsung phone but only screwed up the phone after a few times. i think i should stick with the "sansa".thanks.
The most missing is LXC