I'm a sysadmin at a decently sized European university. While our large faculty server is stuck with ESXi for the near future (which I'm hoping to change), our smaller servers all run PVE. This includes our storage/backup/mass-deployment server, a cluster of several machines that deal with high compute power for AI and machine learning, a cluster that runs *several hundred* concurrent VMs for networking and white hat hacking exercises, and smaller roles too numerous to mention. I've never had a single "I wish PVE had x..." moment. The only thing that's a bit difficult to work with is the user and permission management, but it's not a deal-breaker. As for LXCs - I'd rather have them than not. My home server has very little memory, and having my services run inside Debian or Alpine LXCs that each consume ~20 megabytes of RAM is pretty baller.
Also LXC could support the same almost-everything live migration if someone put in the work, because LXD supports it using CRIU and it works and uses the same APIs.
I'm basically the same at my company (which is ~100 employees but growing decently mostly at a main location but we have two satellite locations with 10-20 employees each). I put vcenter/esxi at the main office but when we expanded to the other locations and it looked like vmware might get sold I gambled on proxmox at those locations (I was using the free version at home in the home lab) and I haven't looked back. By the end of this year the main area will be transferred off esxi/vcenter to proxmox as well. Having the free version only minorly annoying reminded me at the right time and has turned into a few thousand a year in licence fees for them from my day job so it's basically the best advertising program they can have. The only thing I miss is the multi location management through a single pane you can do with vcenter and I'm hoping sometime in 2025 they release that feature, there is talk of them working on it on the forum on roughly that timeline.
I am a teacher at high school in Norway. My students that were new to virtualization, they prefer Proxmox over VMware when it comes to user friendliness. And LXC containers are very useful, when you're low on resources.
Hello, @Wampa842 , @barfnelson5967 , @2GuysTek I'm going to start building my minilab (especially for AI) and was looking at PVE. My first challange is to port a "custom" vmdk to PVE (to free up a server). The thing is that this one has: 1.bitlocker enabled (windows 10) 2.(virtual)TPM device present (Virtual Machine Settings->Hardware->Trusted Platform Module=Present) 3.VM is fully encrypted (Virtual Machine Settings->Options->Access Control=Encrypted) My question(s): a)Can I migrate it as is / how ? b) if not,do I really need to remove/disable all 3,2,1 ? Thanks!
"host" copies the motherboard CPU model. You have to be careful if you have different CPU models in a Proxmox Cluster and then do live migration. Therefore, the default version assigned is intended to secure against issues. If every CPU in the cluster is the same, then you can keep it at Host and in some cases (newest CPU's) you'll experience better performance 😊
8:05 "all management needs to be done via web gui" completely incorrect. Proxmox makes virtually everything and more available via the command line. "pct" program and "pve" prefixed commands, tons of them. You can enter hosts via the "pct enter hostID" etc in /etc/pve/ all machines are available for config edit as well
Agreed, it's not as neat as a menu where any ape can just navigate and press enter, but the functionality exists for sure. There are loads of scripts that you run in the terminal that does entire VM or CT setups.
@@watvannou Yea you can tell he didn't do much research and just spat out his video to get views in this high traffic time. The docs etc are filled with command line stuff to use, yet he claims it "NEEDS" to be done via GUI lol. Unfortunately (for those apes) there are still many things not even available in the GUI so sometimes you have to get into the cli and files. One day it'll probably all be in GUI though.
Sure you *can* go into the shell and manage things directly from there, but his perspective is talking about for general use and deployment. Would I trust a L1 or L2 tech to go down to the machine and navigate through the menu of XCP-ng to start a VM? Sure, absolutely. Would I trust one of them to go log into a shell and start running commands that directly manage the entire VM structure and config of the house without prior training? Probably not, no.
@@1armbikerI mean....how is, "qm start (insert VM machine number here)" something one of your tech's wouldn't be able to handle? Train them on the dark arts of --help. The comment section of this video has suddenly made a lot of past interactions make sense...lol
LXC containers are one of the main reasons I use Proxmox. They are extremely lightweight and fulfill all my requirements in my homelab. Compared to traditional VMs (Virtual Machines), their resource footprint is significantly lighter. I have only 16 threads available on my CPU, which means I need to be particularly cautious with thread reservation. LXC containers, on the other hand, offer great flexibility in terms of resource usage. I really love Proxmox and plan to continue using it in the future.
What proxmox devs are doing is little different, it is a place where all the opensource virtualization technology meets. Yes, it might not be general user intiuitive, but it provides well on core virtualization. And as a linux nerd I love it.
tbh proxmox's interface seems far *_far_* better to me. Yes, there is a lot, but it's all very clearly formatted and hierarchically displayed. Being able to see everything you can do means you'll be able to sumble across features that solve your problems, it keeps things organized, and it keeps things simple. The Windows route of submenus that open other submenus that do the job of that other submenu but are actually different and can also open this other submenu which is technically deprecated by that submenu over there but actually still lets you do this thing a lot easier, just isn't good. The only reason why people don't complain is they're used to it, but being used to something doesn't mean it isn't bad UX. FFS I literally had to google how to even empty my trash on windows because my windows partition was low on space and there isn't a clear option to do it anywhere. It takes several submenus deep in the system settings window to empty your trash. If your UI is so convoluted that I need to google how to do basic tasks, don't bother making a UI at all, it's more work than if you just made me google the commands to do the same thing. I haven't done much virtualization, (I've only really used virt-manager for local-virtualization, though I've always kept an eye on higher scale virtualization since I've been planning to build a bigger homelab long term) but without any experience, I'm picking the proxmox UI without question. I don't need to look for anything, it's just there, I can see it, it's there. If I have something I need to do I can know where it is without having to memorize it because there is a simple and consistent design philosophy.
Re proxmox management from the console: The gui available via the web browser is only a think overlay over the command line tools, available from the command prompt. All VM and container related functionality is available from the CLI, and in some specialized cases, use of the CLI is still required to complete the task. The big advantage of the GUI is that it is much easier to use. The "short" version of the help text listing only available command line options for "qm" (the VM management tool) is several hundred lines long, spanning many pages. Super intimidating for a beginner, but a life-saver for an advanced user, who wants to automate routine tasks.
I mentioned in the video something to the effect of how you can do everything you need to do from the command line, but the point is, that makes Proxmox less of an option for a homelabber or business that doesn't have that deep knowledge in linux command line. I think that when a company is considering their options in terms of their future replacement for VMware, they're going to have to take into account their IT staff and engineers and how much retraining/learning they're going to have to do, and I think for Proxmox, that's a concern.
@@2GuysTek8:04 You’re saying all management from the host has to happen from the GUI, that’s not at all true. Anything you could do from the GUI is possible from that prompt. You then followed it by saying that kind of access is useful when you have no network connectivity, but again, you have far more access to modify the HV config than you’re letting on. I’m fine with giving ESXi the win for ease of use, but the phrasing is bad dude. Completely blowing past the amount of power in that console because admins don’t know their tools is a cop out.
@@bigjaydogg3384 I don't think the phrasing is bad. He has a point. Perhaps what Proxmox can implement is a CLI menu similar to TrueNAS whether you can drop into Bash
@@zpariharThats the thing, you don’t need a menu option because you’re already there. Thats why the video is so wrong, its not that the prompt it drops you into is incapable of anything, its that the prompt is so open ended simple tasks require at least some level of Linux knowledge.
@@bigjaydogg3384missing the point. "can" do it is different to "how" do it. if i run a small business and nobody in it knows linux nor proxmox, it doesn't matter how much you tout that it can be done if we don't know how to do it, especially if it's an emergency and we need to do something from the console. if i want a linux expert, i need to also pay their salary. i've personally helped several small businesses without any in-house expert team to do these tasks and having a more layman useable interface could save them big bucks hiring a technician to do emergency maintenance on an hourly fee. how much work would it really be to have a TUI where you can change hostname and network options? for example, they could use NetworkManager and simply call nmtui for the user, which is a massive life saver.
I can understand the the immediate thought of Docker and kubernetes when it comes to containers, but remember that the container support in Proxmox actually started with support for OpenVZ containers, and moved to LXC even before docker even existed. Today I'm not sure that LXC makes that much sense anymore, but go back 10 years, and LXC provided significantly reduced overhead in contrast to a full VM. I've been using Proxmox since version 1.0, and with 3GB of ram on a server, shaving the memory usage by a few 100mb actually made a difference. As for the upgrade, it's actually not that hard to do. In fact I find it more unnerving to upgrade a vcenter, especially if NSX is involved.... Keeping my hands off the hypervisor and don't changing anything I've never had any issues upgrading a Proxmox host. That's not something I can say about upgrades of ESXi hosts...
I still do not fully understand the upgrade process on standalone ESXi, like why do I need do download some sort of file to start the update? Can't I just push a button or run a command?
I'm still on the fence as to which hypervisor to migrate to, so I appreciate the videos. However, I wanted to say that I appreciate the presentation style of these comparison videos. I've watched both, and they are fantastic examples of how to do an X vs Y comparison. Nice work!
Tom Lawrence (Lawrence Systems on UA-cam) has a new video on XCP-ng and Xenorchestra out today. Might take a look to see the Proxmox vs. XCP-ng differences (even if his video doesn't really demo that per se). I use Proxmox on "small systems," and XCP-ng on the big iron.
Been swinging between Proxmox and XCP-ng on my test system. XCP-ng feels more of an 'Enterprise-ready' but Proxmox is pretty strong and powerful out the box without over complicating things. I think for me the built-in LXC is going to make Proxmox the homelab solution. I'd probably want to do XCP-ng if we had to drop ESX at work though.
I haven´t made any experience on EXI because as a non professional user I don´t want to pay monthly for a Virtualization. I am running a mac mini but sometimes I have to use a Windows machine and I love Linux. Parallels for mac is so expensive and so a bought a mini PC for about 200 € and can now virtualize Windows and several Linux on that machine by using proxmox. Although I am 71 years old I was able to manage all this because there is a great community supporting proxmox users. And you will find big help on UA-cam also. I think proxmox is a good software and free to use.
would love comparisons with Proxmox and XPG-ng vs Virtual Center. Our VMware bill went from $132k/year to 1.225 mil/year and we need to find some alternative.
thank for the video. As a linux nerd I say yes to a full featured gui with all options but a fallback to a shell for scripting helpfull tools is a great thing. Converting VM to proxmox works with simple scripts but every Vm should be testet after a transfer.
As you just stated, the reason Proxmox CLI is minimal is the fact the CLI is debian with KVM enabled. I have Ubunutu servers with KVM installed making them TYPE1 hyperivosors and now I just manager them with their gui tools like Cockpit or Virt-Manager. yes I can get in to the hosts via ssh but I don't run additional services on the local host at this point and just manage VM's runing said services. KVM was designed to be open source and full customizable to your liking.
Not sure if this has been mentioned but you have multiple CLIs availabla: qm, pct, pvecm, etc, to launch, stop, enter, reboot, etc., vms, lcxs and managing the cluster. Also all the networking and virtual bridges (switches) can be managed from the linux terminal.
LXC have a lot of uses. It's basically for services that would best be run on bare metal, but you don't want to pollute host OS with it (samba server, nfs, ..)
Thank you for making these videos! My organization was just talking about this very thing, and this will help me explain options to my management team.
Good video my dude. I would like to see a follow up video on clustering, vim networking, vm storage, virtual disks, migration, fault tolerence and "DRS".
I'm interested in this as well with Fault Tolerance considering he didn't list it, only that it was feature parity was complete between the two and I do not think that is correct.
I migrated ESXi VMs to Proxmox using the OVFtool from VMware. It was quite easy and where are guides out there. Having used VMware (ESX) since it was running on top of Linux (ya, I'm old) professionally and personally, I've been quite happy with the switch to Proxmox. I switched when ESXi 7 came out and made it harder to run on the same hardware as 6.5 did. Learn Linux. You'll be glad you did.
Yeah, I upgraded via scripts and my machine ran fine. Then did a new install and was told my CPU had aged out. I shut it off and have used a KVM based machine ever since but am interested in ProxMox :) Large customers I work with are moving away from VMware as FAST as they can and the renewal costs I'm hearing about are eye watering.
Ubuntu microcloud, I think you are referring to this one the latest version the install fails, microovn doesn’t install and microceph bootstrap fails and you need to install all manually. Also the lxd project leader just left canonical and forked lxd into incus. Microstack bootstrap process fails more than it installs successfully.
I moved from Esx to HyperV then to Proxmox a few years ago. VM migration was a nightmare. However I’ve not regretted moving to Proxmox. It works well. Particularly good when you set up a Ceph cluster. There are easy ways to disable the nagging. I have a script that removes the nagging that executes on every boot. Not seen the nag in a very long time. The LXC containers don’t live migrate but they do migrate. They just shut down and start again on the destination host. They’re very useful for lightweight tasks such as running ddclient, letsencrypt, cloudflared etc. Generally, I’m glad I moved to Proxmox. Free HA, good for Linux and Windows VMs, an acceptable built-in VM backup system, Ceph for cluster storage. What’s not to like?
I started using proxmox last year and I have to say that it is clearly very good I agree concerning the anoying repository I must also add that to have the possibility to make snapshots of your vm you have to pay attention to the dis format of your VMs however up to now it is working like a charm
I believe the reason Proxmox has the VMware features, the pvscsi, vmxnet3, etc is because if installed Vmware Player on a Linux host, it has to compile kernel modules. So they may have the modules precompiled and ready, not to mention there is Open-VM-Tools, which is an open source version of VMware Guest Additions for Linux. When the Open VM Tools became standard, VMware pretty much dropped support for their own proprietary guest additions for linux, especially for newer kernels. Maybe having these for guests is for compatibility should you wish to migrate from ESXi or any other VMware product.
I´m not exactly sure how user an group management is in esxi, but have you noticed that you can also add other realms in proxmox like from LDAP/AD or even OIDC?
Should have really talked more about the ability to GPU and other pass through capabilities as well without paying an arm an a leg for Nvidia licensing. It's really nice for Plex or Jellyfin transcoding and running cheaper LLM/AI solutions. Also the Proxmox backup solution is truly amazing and probably needs to be talked about more. It doesn't just give you VM backups but it also does file backups as well. For me this was such a better and easier solution than going with Commvault or Veeam.
Ick. CommVault. What a pile of steaming crap that thing is. Thank God we got rid of that a few years ago. We went to Veeam, but someone else manages that, so I don't know how good or bad that is. But, yeah, backup capabilities being built into the platform vs 3rd party is so practical.
I'm sure you will hear linux ops folks talking about how great the cmd line is for admin'ing... and I don't disagree... IF you're doing it frequently enough to remember how to do everything. For the rest of us who don't admin linux machines, we waste a LOT of time looking up commands and that goes away with a basic (even text based like with esxi) ui. I just started playing with Proxmox as a potential migration path from vsphere for my roughly 500 core cluster, the new vGPU non-prod release gives me hope that by the time I need to migrate there will be production gpu support which would make proxmox fully capable of replacing my vsphere e+ environment. Good overview but again I'd like a deeper dive in to advanced features like backup... upgrade workflow, software defined network and so on. oh and yea I DID assume containers meant app containers too.
@@brodriguez11000 Hosts need more than an api... they need a capable diagnostics interface (that should allow admins to manage the local machine). An api only host would be a problem in certain disaster recovery scenarios. Could be console based, ssh or web... but should not rely on external custom software to work. (everything needed on the box).
One thing I noticed in your XCP-ng comparison was the mention of the limitation of VM pausing during snapshot creation if memory is included in the snapshot. However I don't remember hearing about Proxmox snapshots in this video. Does it have the same VM pausing issue as XCP-ng for memory-inclusive snapshots? It looks like the snapshot capability is dependent on a number of factors such as the virtual disk format or the file system used to hold VM images (so you need to use QCow virtual disk files if your VM storage is networked using NFS or Samba), and that would have been good to mention.
I can't give a comprehensive answer but the short answer for taking a snapshot with memory. Yes, the VM state is paused for the duration of taking the snapshot.
You can both pause or stop to get all the memory in snapshots. Those limitations are the differences between block and file storage. Not due to proxmox.
Since when can Proxmox automatically shift VMs to other hosts in the cluster to balance out workloads? It has HA so that in the event a host goes down, the VMs should power up on a host that's online.
That's kinda what I was hinting at in terms of being able to emulate down to the VM any number of different CPUs. EVC only allows back compatibility from the procs in your host, and when EVC is enabled, you can't migrate a live VM between hosts with different generation processors. It's entirely possible that choosing a different CPU ID in Proxmox is meaningless to KVM, but it doesn't feel _right_ to me, you know?
@@2GuysTek It is not meaningless. If you choose a cpu type that is the base common denominator such as qemu64 and (even older less performant) kvm64 you can plop it on pretty much anything and it will work. There are also the vulnerabilities that come with something that can run on anything. So the named models are a way to get close to the cpu types your system uses, add in vulnerabilities/mitigations and still be safe to live migrate to a new host, unlike cpu type pass through which would require the identical cpu on the host you are migrating onto. ABI Compatibility if you can google.
Wonder if the CPU type selection when creating a VM has to do with something similar to EVC on VMWare, where the VM is limited to a specific instruction set regardless of physical CPU, so you can move VM's around between hardware from different generations, so that if you set the CPU type to the lowest common denominator in a cluster of machines, it shouldn't have any issue moving a vm between those hosts.
Yes to being able to select a cpu type that is the lowest common denominator as well as the reverse case of utilizing something about the actual type of cpu(s) you are using on your hypervisors.
You pretty much hit the nail on the head. It does indeed choose a common instruction set so that you may do live migration from one node to another even if they have different physical CPUs. _(also enables high availability for the same situations.)_
You can still use ESXi, assuming you have kept the ISO's and your keys, which i have. I decided to give Proxmox a go last month an after being a VMware admin for 9 years i found it fairly easy to get to grips with it. There were a few things i had to google such as the ISO storage !. Personally i have found it very good and on comparable hardware im noticing lower CPU use and lower RAM use compared to ESXi running exactly the same VM's.
Any news on what they plan to do with VM Workstation Pro? I use Workstation Pro for testing and for keeping older OS's available so I can offer support for older OS's I have over 100 OS's from desktop to server to Appliances like pfsense Any word on those licenses or some sort of alternative?
I absolutely agree with all of the comments on the lxc containers in proxmox. The first time I started my proxmox virtual server, I started with VMs. However, I realized that it takes more memory and resources to run the virtual machines, so I changed to creating lxc containers in proxmox and it uses very little resources when loading a lxc container. I only use VMs if it calls for using them in proxmox. I am now getting ready to run a second proxmox VM because I love that they added the backup server to backup virtual machines. however, I do want to try other VMs, such as XCP-NG with Xen orchestra since I am very familiar with citrix, and also SUSE harvester. I want try VMware ESXI, but I am glad I did not go further since they got rid of the free edition.
I totally agree! If proxmox made the GUI a little easier, made the upgrade more seamless, and made it possible to migrate from vmware to proxmox smoothly, it would be a no brainer. But these points remains and a couple more you mentioned in the Video. Really hope to see these changes! Thanks for the video, good summary!
Just to add to this, Proxmox recently released a conversion tool to make migrating from VMware to Proxmox easier: news.itsfoss.com/proxmox-vmware-migration/
With VMware screwing the pooch I'm betting lots more dev is going to be pushed into KVM based stuff and OEMs of storage solutions are going to want to support it too. Hopefully it comes along!
I just wiped my drives last night from Vmware ESXI 7u03 after using it since 2022. I ended up going with proxmox for now because due to having way more support in terms of DevOps tools such as Ansible and Terraform as well as installing nested hypervisors. I tested xcp-ng but the support base is too small esp for DevOps and API automation stuff at the moment. Also the RAM usage is slightly higher.
thanks for the comparison, i noticed that with proxmox unlike esxi, you can't install a promox host to a proxmox cluster if that host has vm's on it.. even if they are shutdown.. the promox most MUST be clean.
The VM CPU type will be for clusters of hosts that have different CPUs. To maintain the ability to migrate you need the cpu features to be the same. in VMware this is the EVC settings.
Hey, would you consider doing a video on an in place migration from ESXi to Proxmox. I like many installed ESXi to get the Guest OS off the hardware, making migration to new hardware a little easier when the time came, now I have a number of boxes that need to go from ESXi to Proxmox VE. How about it?
I managed Esxi for years at my company (5 hosts, 35 VM) last summer we moved to Proxmox: 100 % satisfied, just a few small issues (solved). The only problem I still have is when I want to download and deploy a virtual appliance but it is available only to vmware. It works on proxmox after conversion but it's not ideal.
If you have not tested running multiple Proxmox nodes in a cluster with ZFS snapshot replication running between them, then you have not seen a huge part of the usefulness of Proxmox for small environments. Using that simple setup, you get the ability of almost instant HA or manual failover of VMs/LXCs between Proxmox nodes without needing any shared storage required. This is huge for small business or home users, to be able to have this VM migration capability without requiring any san or external storage.
Proxmox now features easy import from Vmware. Don't know when it was available, but the newest version is capable of that and it works very well (as long as you dont have something like a Server 2003 you need to migrate)
LXC containers are a lot more light weight than full vm's, they start up in ms and use less than 100 mb of ram. If your usecase does not require extreme security then they're great.
At 8:03 you say all management meeds to be done frol the gui. I'm a novice proxmox user and my feeling generally is that sometimes in proxmox you have to use cli. That said I believe you _can_ do everything in cli that you can from the gui. And there's a lot of documentation around it. I agree a wizard for basic setup would make certain things easier though
Most of the cli is just normal ol' linux cli. I would say the ZFS stuff in particular is probably still a situation of very much needing the cli. Save a copy of the admin guide on where ever you are usually doing your stuff and you'll have it down before long. If you are going ZFS, look for a book by Michael Lucas called FreeBSD Mastery: ZFS. It's well worth it.
Just spent the last year of my life migrating from nutanix ahv to vmware. Looks like im gonna be spending the next year moving elsewhere. Really wish i would have purchased 5 years of support to push this out
@@bouzidoussama496 Our hardware was EOL, so we started pricing out upgrading or replacing it. For the price, we were able to get 3x more in terms of performance, storage, compute, and networking by going back to VMware non-hyperconverged. AHV did work pretty well; however, there were times when it was frustrating. Some vendor appliances only support VMware, and we ended up having to run a small cluster alongside the entire time due to this. Simple things like pulling a report of guest OS were nonexistent, simple but a lot of small things that are easy in VMware do not exist. Going to AHV is pretty easy; their migration tool works great. Moving back to VMware was not as easy but doable with some tools.
I’m interested about similar comparison between Nutanix, Scale Computing, Oracle VM, Red Hat alternatives with RHEV and Openstack VM, Microsoft solutions for onprem VMs and containers.
I completely hear the point on the benefit of a menu on the console. However, more and more system admins are so reliant on menus and GUIs, that they are not taking the time to learn the CLI. I have been in technology for 25 years and some of the best platforms that I have supported have been on Linux. Getting to "know" the OS and how to maneuver within it is a necessity that the next generation of administrators need to embrace. Just my 2 pennies... :)
i can't agree more! having a simple cli menu for basic tasks would be nice, but knowing linux gives you so much more power on the console. as example: we are moving from vmware to proxmox in my company and i do all migrations on the console because i can convert/move vmdk files directly to proxmox ceph storage. the GUI can not do this.
yep, the whole world pretty much runs on linux now, so if you're a "serious" sysadmin and the best you can do is navigate an ncurses menu or a webui, then it's time upskill
Vsphere is also available as a standard version, includes vsphere std hypervisor and vcenter std. Also essentials and essential plus are available. These will suite for SMB and will defenetily cost no more than before Broadcom. Only difference is that lowest support is now production support which will of course be more expensive than basic.
reli nice video, I helped my company to migrate all VMware VM to Azure last year and removed all the on-prem hardware. I think the trend will go on that VMWare is not in a priority in the coming future. Personally, i will learn some Proxmox for the on-prem alterative. Just an interest, will talk a bit more about the Proxmox Backup Server? (as for the VMWare, I am familiar with using Veeam Backup and Restore. How about in Proxmox?) Thanks and cheers.
very useful video, thx. never used vmware, trying out pve now. i like to learn the under the covers troubleshooting, but agree they coulda shoulda provide a mgmt interface on host. i struggled to get the connection working to get the web ui, which required navigating local conf files to even get going. i'm wiser for it, but in a production setting i'd like a ui pointing me in the right direction to potentially save hours of downtime.
I'm using Proxmox for 3 years now. The switch from VMWare was extremely easy and I dindn't need to use the documentation ever. They are very similar. But i also manage a lot of headless Unix(FreeBSD and AIX) and Linux servers. So the command line is pretty easy for me.
Do you have a one file, one click config backup/restore? If it crashes. How long will it take you to get tte same or similar machine 100% configured and ready? (More than 10 minutes?). Did you implement and configure a UPS for power outages/spikes?
@@pepeshopping I mostly work in datacenters all over the Benelux. Off course snapshots and backups are made regularly. Also most server racks are equipped with at least 2 UPS’ses. At home I only have 7 dual CPU Xeon V4 servers and a small SAN/NAS Backup solution(2x NetApp FAS2246+ and a Synology DS3622xs+) off course also with 2 UPS’ses.
Hi. It is possible to do a M.2 IOPS test on a virtual machine on proxmox? if you have a performant one? I used ISXi paid version and with a 980 pro i only got about 600k IOPS, and with hyper-v i got 990l IOPS; which is the very reason a switched from ESXi7 to hyper-v. I am wondering if proxmox allow full speed on m.2 ssds, i would love to give it a try, but i don't have any server available. Otherwise, from your video i see that proxmox is way better than ESXi considering that is free. I am enclined to believe that it might be a better solution than hyper-v also, since is a type1 hypervisor and not 2. Also, it is possible to have a folder sharred between multiple windows virtual servers? like to make them a D to share that folder? And a shared one for linux servers? I believe a folder shared between different OS types its impossible. I hope i get an answer from you. Thank you.
Thank you for this great review. The biggest issue with Proxmox is no realistic self service option because there is no way of setting CPU, Disk and Memory Quotas. Resource Pools only limits what you can do and not how much. VMWare Cloud Director has no equivalent and there is no Resource Pool monitoring either; you need to trust your users not to eat up your entire Datacentre's resource.
You can use those repos, because they exist and are needed for pve updates. Instead of disabling, you need to change the apt sources with the community repo.
This was a great overview, I'll have to move my lab soon, I'd love to see some guides on moving (exporting) VM's and other config from esxi and importing into Proxmox where possible
I’d be very interested in a comparison aimed at enterprise usage. Which doesn’t necessarily mean large deployments, but in terms of support, features and life expectancy - nobody wants to migrate to a platform that’s not financially backed and may stop being maintained at any time. Nutanix comes to mind, and RHE virtualisation for example.
Hey man, i have to say your X-CPNG and ProxMox vs ESXi has been great and super informative. I appreciate you taking the time to do this.. I run ESXi 6.7 in home lab and, given the state of the company going forward, i am looking to change that. Have you done any videos on running ProxMox in mini PC's? I know it is pretty compatible on desktops and would run fine on my R630's but looking to downsize the power usage footprint a bit. have you done video's on Proxmox Backup Server?? Wondering if that is a worthy addition to the PVE environment ... Either way, great video and i am looking forward to catching some more of your content - Liked & Sub'ed!! keep up the great work!
I'm going to be putting a lot more effort into videos around both Proxmox and XCP-ng in the near future, and they will definitely take into account their ability to run on a variety of different hardware! Thanks for the comment!
Containers on linux has always meant LXC. LXC are terribly useful for use cases where you want containerized solution but need more access to hardware. Also as you mentioned it helps tamp down the ram required too. Don't disagree about full virtualized environment provides more security benefits, but for many use cases LXC is the perfect compromise. Plus most modern applications don't need the container itself to he HA to migration is not normally required. One last thing for LXC is with zfs copy on write file system you can spawn new containers off an existing one and not only does it use minimal RAM it uses minimal storage.
Great video. Btw, LXC live migration is coming. Docker comes from LXC. LXC (and the previous "original" container technology OpenVZ was around before Docker was a concept
The PVE VM CPU host section is just telling PVE how to simulate/emulate the chosen CPU family's instruction set. Choosing "host" is the most performant, becase it's basically just a CPU passthrough from the host to the VM. This could cause problems with HA, though, if you have a cluster of say an x86-64 node and an ARM64 node. Choosing anything else than "host" will lower your performance because PVE (or rather QEMU) either has to simulate or emulate the chosen instruction set, depending on the host CPU's virtualization capabilities for the chosen CPU. I think your example of choosing an AMD Epyc CPU while the host is an Intel one, this would probably result in emulation, so very slow performance. Choosing a desktop type AMD CPU while the host is a desktop type Intel CPU, this could probably become simulation, which would be faster but still slower than "host". I'm not on expert on this specific topic though, so take it with a grain of salt.
I've been running a PVE cluster for a number of years now. I like it, and I'm not planning to move away from it, though I'm also playing with xcp-ng just a bit too. But I think a number of the comments here trying to defend PVE kind of miss the point: * Sure, a system admin should be able to configure a network interface from the shell (and I've had to do it on a variety of occasions). But a console menu like xcp-ng's would still be a welcome addition. And yes, you could get around this by just installing a full graphical desktop environment onto it, but that really isn't the way the product is intended to be used. * Yes, the subscription nag screen is obnoxious. It's easy enough to disable, but still obnoxious. I get it, devs gotta eat, but... * Agree on the enterprise repo being enabled by default. Like with the nag screen, it's easy enough to fix--easier now that repository management is included in the GUI--but it doesn't make sense as a default.
I'm a little shocked you did not mention that it's possible to import vmdk files from vmware into proxmox and basically migrate your vm's as-is into proxmox with one command in the terminal.
Is someone has some experience with proxmox in a cluster and storage? I need live migration a d snapshot and dont want ceph. Iscsi was a pain in the ass so far😅
Should create, "Exploring Proxmox and TrueNAS & UnRaid from a VMware User's Perspective" as they're the top three hypervisors (outside VMWare). I feel like TrueNAS is the baby of Unraid and ProxMox xD
When I was reviewing Proxmox VE, I really wanted it to succeed, but I couldn't get promiscuous mode working for network packet capturing for security onion.
Where I work will probably just pay the cost difference and stay on VMWare; but I’m watching these type videos just in case. One issue I have is we have a set of host with Nvidia L40 cards in them for AutoDeskand other graphic intensive software. Linux is notoriously horrible or horrendously complicated with PCIe pass through. What I’d like to see covered in one of the type of videos is how these alternatives deal with hardware like video cards.
In version 8, they made PCIE pass through very easy, tons of instructions online. I passed through a couple of ports on a 4 port nic with a few clicks.
thank you for the video, for sure a straight migration path from VMware to Proxmox is something that has been needed for a while. also, LXC container is targeting the home lab I think. as some people have very lightweight workloads.
I've been using mail gateway for a few years and the confusing mess of an interface is one of the main reasons I'm a bit on the fence sbout VE. Also the nagging screen comes up for users clicking release or whitelist in the report emails. That's just irritating and I'm sorry but I'm not going to pay gor the software for myself snd ten family members when the Community Edition is s whopping €175 per year. Some kind of middle ground here would have been nice.
You know that you can do basically anything in pvr from the console via command line once you're logged in? If you want you can avoid the gui entirely and perform everything the gui can plus stuff the gui cannot do in the terminal? Yeah yeah you know it. But esxi loses. You can't do anything in esxi there even if you want. Let's say network fails, you're lost. You can't even check vm status... Sorry you're absolutely wrong
8:06. No, that is not true. In fact, almost everything the host does via web gui can be done via the console. The only thing that IS correct is that there is no console menu, so if you don't know what you're doing, it's easier to start with the gui.
I'm a sysadmin at a decently sized European university. While our large faculty server is stuck with ESXi for the near future (which I'm hoping to change), our smaller servers all run PVE. This includes our storage/backup/mass-deployment server, a cluster of several machines that deal with high compute power for AI and machine learning, a cluster that runs *several hundred* concurrent VMs for networking and white hat hacking exercises, and smaller roles too numerous to mention. I've never had a single "I wish PVE had x..." moment. The only thing that's a bit difficult to work with is the user and permission management, but it's not a deal-breaker.
As for LXCs - I'd rather have them than not. My home server has very little memory, and having my services run inside Debian or Alpine LXCs that each consume ~20 megabytes of RAM is pretty baller.
Also LXC could support the same almost-everything live migration if someone put in the work, because LXD supports it using CRIU and it works and uses the same APIs.
I'm basically the same at my company (which is ~100 employees but growing decently mostly at a main location but we have two satellite locations with 10-20 employees each). I put vcenter/esxi at the main office but when we expanded to the other locations and it looked like vmware might get sold I gambled on proxmox at those locations (I was using the free version at home in the home lab) and I haven't looked back. By the end of this year the main area will be transferred off esxi/vcenter to proxmox as well. Having the free version only minorly annoying reminded me at the right time and has turned into a few thousand a year in licence fees for them from my day job so it's basically the best advertising program they can have. The only thing I miss is the multi location management through a single pane you can do with vcenter and I'm hoping sometime in 2025 they release that feature, there is talk of them working on it on the forum on roughly that timeline.
I am a teacher at high school in Norway. My students that were new to virtualization, they prefer Proxmox over VMware when it comes to user friendliness. And LXC containers are very useful, when you're low on resources.
Hello,
@Wampa842 , @barfnelson5967 , @2GuysTek
I'm going to start building my minilab (especially for AI) and was looking at PVE.
My first challange is to port a "custom" vmdk to PVE (to free up a server).
The thing is that this one has:
1.bitlocker enabled (windows 10)
2.(virtual)TPM device present (Virtual Machine Settings->Hardware->Trusted Platform Module=Present)
3.VM is fully encrypted (Virtual Machine Settings->Options->Access Control=Encrypted)
My question(s):
a)Can I migrate it as is / how ?
b) if not,do I really need to remove/disable all 3,2,1 ?
Thanks!
Lxc containers are better designed and more secure than the mess than docker is, I used them a lot
regarding to the cpu types, if you choose "x86-64-v2-AES" you can live migrate VMs between Intel and AMD physical CPUs. :)
Agreed, I'm assuming that's why it's the default selection!
Thank for letting me know that. I am new to proxmox and I was thinking about the diffrence between host and x86-64-v2-AES.
@@2GuysTek ultra charged EVC. :)
@@desiengineerplays3544 "host" replicate host cpu model.
"host" copies the motherboard CPU model. You have to be careful if you have different CPU models in a Proxmox Cluster and then do live migration. Therefore, the default version assigned is intended to secure against issues. If every CPU in the cluster is the same, then you can keep it at Host and in some cases (newest CPU's) you'll experience better performance 😊
8:05 "all management needs to be done via web gui"
completely incorrect. Proxmox makes virtually everything and more available via the command line. "pct" program and "pve" prefixed commands, tons of them. You can enter hosts via the "pct enter hostID" etc
in /etc/pve/ all machines are available for config edit as well
Agreed, it's not as neat as a menu where any ape can just navigate and press enter, but the functionality exists for sure.
There are loads of scripts that you run in the terminal that does entire VM or CT setups.
@@watvannou Yea you can tell he didn't do much research and just spat out his video to get views in this high traffic time.
The docs etc are filled with command line stuff to use, yet he claims it "NEEDS" to be done via GUI lol. Unfortunately (for those apes) there are still many things not even available in the GUI so sometimes you have to get into the cli and files. One day it'll probably all be in GUI though.
Sure you *can* go into the shell and manage things directly from there, but his perspective is talking about for general use and deployment. Would I trust a L1 or L2 tech to go down to the machine and navigate through the menu of XCP-ng to start a VM? Sure, absolutely. Would I trust one of them to go log into a shell and start running commands that directly manage the entire VM structure and config of the house without prior training? Probably not, no.
@@1armbiker no it isn't, otherwise the word "needs" wouldn't have been used
@@1armbikerI mean....how is, "qm start (insert VM machine number here)" something one of your tech's wouldn't be able to handle? Train them on the dark arts of --help. The comment section of this video has suddenly made a lot of past interactions make sense...lol
LXC containers are one of the main reasons I use Proxmox. They are extremely lightweight and fulfill all my requirements in my homelab. Compared to traditional VMs (Virtual Machines), their resource footprint is significantly lighter. I have only 16 threads available on my CPU, which means I need to be particularly cautious with thread reservation. LXC containers, on the other hand, offer great flexibility in terms of resource usage. I really love Proxmox and plan to continue using it in the future.
The LXC containers are amazing, fast, easy to configure and use less of those valuable recourses.
What proxmox devs are doing is little different, it is a place where all the opensource virtualization technology meets.
Yes, it might not be general user intiuitive, but it provides well on core virtualization.
And as a linux nerd I love it.
tbh proxmox's interface seems far *_far_* better to me. Yes, there is a lot, but it's all very clearly formatted and hierarchically displayed. Being able to see everything you can do means you'll be able to sumble across features that solve your problems, it keeps things organized, and it keeps things simple. The Windows route of submenus that open other submenus that do the job of that other submenu but are actually different and can also open this other submenu which is technically deprecated by that submenu over there but actually still lets you do this thing a lot easier, just isn't good. The only reason why people don't complain is they're used to it, but being used to something doesn't mean it isn't bad UX.
FFS I literally had to google how to even empty my trash on windows because my windows partition was low on space and there isn't a clear option to do it anywhere. It takes several submenus deep in the system settings window to empty your trash. If your UI is so convoluted that I need to google how to do basic tasks, don't bother making a UI at all, it's more work than if you just made me google the commands to do the same thing.
I haven't done much virtualization, (I've only really used virt-manager for local-virtualization, though I've always kept an eye on higher scale virtualization since I've been planning to build a bigger homelab long term) but without any experience, I'm picking the proxmox UI without question. I don't need to look for anything, it's just there, I can see it, it's there. If I have something I need to do I can know where it is without having to memorize it because there is a simple and consistent design philosophy.
Life after VMware 🤣 perfect playlist for me
Re proxmox management from the console: The gui available via the web browser is only a think overlay over the command line tools, available from the command prompt. All VM and container related functionality is available from the CLI, and in some specialized cases, use of the CLI is still required to complete the task.
The big advantage of the GUI is that it is much easier to use. The "short" version of the help text listing only available command line options for "qm" (the VM management tool) is several hundred lines long, spanning many pages. Super intimidating for a beginner, but a life-saver for an advanced user, who wants to automate routine tasks.
I mentioned in the video something to the effect of how you can do everything you need to do from the command line, but the point is, that makes Proxmox less of an option for a homelabber or business that doesn't have that deep knowledge in linux command line. I think that when a company is considering their options in terms of their future replacement for VMware, they're going to have to take into account their IT staff and engineers and how much retraining/learning they're going to have to do, and I think for Proxmox, that's a concern.
@@2GuysTek8:04 You’re saying all management from the host has to happen from the GUI, that’s not at all true. Anything you could do from the GUI is possible from that prompt.
You then followed it by saying that kind of access is useful when you have no network connectivity, but again, you have far more access to modify the HV config than you’re letting on.
I’m fine with giving ESXi the win for ease of use, but the phrasing is bad dude. Completely blowing past the amount of power in that console because admins don’t know their tools is a cop out.
@@bigjaydogg3384 I don't think the phrasing is bad. He has a point. Perhaps what Proxmox can implement is a CLI menu similar to TrueNAS whether you can drop into Bash
@@zpariharThats the thing, you don’t need a menu option because you’re already there. Thats why the video is so wrong, its not that the prompt it drops you into is incapable of anything, its that the prompt is so open ended simple tasks require at least some level of Linux knowledge.
@@bigjaydogg3384missing the point.
"can" do it is different to "how" do it.
if i run a small business and nobody in it knows linux nor proxmox, it doesn't matter how much you tout that it can be done if we don't know how to do it, especially if it's an emergency and we need to do something from the console.
if i want a linux expert, i need to also pay their salary. i've personally helped several small businesses without any in-house expert team to do these tasks and having a more layman useable interface could save them big bucks hiring a technician to do emergency maintenance on an hourly fee.
how much work would it really be to have a TUI where you can change hostname and network options? for example, they could use NetworkManager and simply call nmtui for the user, which is a massive life saver.
Good video, fair comparison, and good points, especially around the major version upgrades, such as going from 7 to 8.
I can understand the the immediate thought of Docker and kubernetes when it comes to containers, but remember that the container support in Proxmox actually started with support for OpenVZ containers, and moved to LXC even before docker even existed. Today I'm not sure that LXC makes that much sense anymore, but go back 10 years, and LXC provided significantly reduced overhead in contrast to a full VM. I've been using Proxmox since version 1.0, and with 3GB of ram on a server, shaving the memory usage by a few 100mb actually made a difference. As for the upgrade, it's actually not that hard to do. In fact I find it more unnerving to upgrade a vcenter, especially if NSX is involved.... Keeping my hands off the hypervisor and don't changing anything I've never had any issues upgrading a Proxmox host. That's not something I can say about upgrades of ESXi hosts...
I still do not fully understand the upgrade process on standalone ESXi, like why do I need do download some sort of file to start the update? Can't I just push a button or run a command?
15 lxc containers in my homelab and one full VM for docker. :) Love the lxc contaiers as they are lightweight.
I'm still on the fence as to which hypervisor to migrate to, so I appreciate the videos. However, I wanted to say that I appreciate the presentation style of these comparison videos. I've watched both, and they are fantastic examples of how to do an X vs Y comparison. Nice work!
Tom Lawrence (Lawrence Systems on UA-cam) has a new video on XCP-ng and Xenorchestra out today. Might take a look to see the Proxmox vs. XCP-ng differences (even if his video doesn't really demo that per se). I use Proxmox on "small systems," and XCP-ng on the big iron.
Been swinging between Proxmox and XCP-ng on my test system. XCP-ng feels more of an 'Enterprise-ready' but Proxmox is pretty strong and powerful out the box without over complicating things. I think for me the built-in LXC is going to make Proxmox the homelab solution. I'd probably want to do XCP-ng if we had to drop ESX at work though.
I haven´t made any experience on EXI because as a non professional user I don´t want to pay monthly for a Virtualization. I am running a mac mini but sometimes I have to use a Windows machine and I love Linux. Parallels for mac is so expensive and so a bought a mini PC for about 200 € and can now virtualize Windows and several Linux on that machine by using proxmox. Although I am 71 years old I was able to manage all this because there is a great community supporting proxmox users. And you will find big help on UA-cam also. I think proxmox is a good software and free to use.
would love comparisons with Proxmox and XPG-ng vs Virtual Center. Our VMware bill went from $132k/year to 1.225 mil/year and we need to find some alternative.
I would love to see Nutanix
thank for the video.
As a linux nerd I say yes to a full featured gui with all options but a fallback to a shell for scripting helpfull tools is a great thing.
Converting VM to proxmox works with simple scripts but every Vm should be testet after a transfer.
As you just stated, the reason Proxmox CLI is minimal is the fact the CLI is debian with KVM enabled. I have Ubunutu servers with KVM installed making them TYPE1 hyperivosors and now I just manager them with their gui tools like Cockpit or Virt-Manager. yes I can get in to the hosts via ssh but I don't run additional services on the local host at this point and just manage VM's runing said services. KVM was designed to be open source and full customizable to your liking.
Hi, it's important to emphasize that Proxmox leverages KVM for VM functionality. Great video, keep on home-labing!
Not sure if this has been mentioned but you have multiple CLIs availabla: qm, pct, pvecm, etc, to launch, stop, enter, reboot, etc., vms, lcxs and managing the cluster. Also all the networking and virtual bridges (switches) can be managed from the linux terminal.
LXC have a lot of uses. It's basically for services that would best be run on bare metal, but you don't want to pollute host OS with it (samba server, nfs, ..)
Thank you for making these videos! My organization was just talking about this very thing, and this will help me explain options to my management team.
This is exactly why I’m making them!
Good video my dude. I would like to see a follow up video on clustering, vim networking, vm storage, virtual disks, migration, fault tolerence and "DRS".
Thanks! I think that's my target after getting through these initial comparison videos.
I'm interested in this as well with Fault Tolerance considering he didn't list it, only that it was feature parity was complete between the two and I do not think that is correct.
I migrated ESXi VMs to Proxmox using the OVFtool from VMware. It was quite easy and where are guides out there. Having used VMware (ESX) since it was running on top of Linux (ya, I'm old) professionally and personally, I've been quite happy with the switch to Proxmox. I switched when ESXi 7 came out and made it harder to run on the same hardware as 6.5 did. Learn Linux. You'll be glad you did.
Yeah, I upgraded via scripts and my machine ran fine. Then did a new install and was told my CPU had aged out. I shut it off and have used a KVM based machine ever since but am interested in ProxMox :) Large customers I work with are moving away from VMware as FAST as they can and the renewal costs I'm hearing about are eye watering.
LXC is really nice to have, I wouldn't complain about it being there. If you need Docker or k8s, throw a VM with them in there .
You can throw Docker or/and k8s in an LXC container! Proxmox rocks!
What about openstack / Ubuntu micro stack?
Ubuntu microcloud, I think you are referring to this one the latest version the install fails, microovn doesn’t install and microceph bootstrap fails and you need to install all manually. Also the lxd project leader just left canonical and forked lxd into incus. Microstack bootstrap process fails more than it installs successfully.
I moved from Esx to HyperV then to Proxmox a few years ago. VM migration was a nightmare. However I’ve not regretted moving to Proxmox. It works well. Particularly good when you set up a Ceph cluster. There are easy ways to disable the nagging. I have a script that removes the nagging that executes on every boot. Not seen the nag in a very long time. The LXC containers don’t live migrate but they do migrate. They just shut down and start again on the destination host. They’re very useful for lightweight tasks such as running ddclient, letsencrypt, cloudflared etc. Generally, I’m glad I moved to Proxmox. Free HA, good for Linux and Windows VMs, an acceptable built-in VM backup system, Ceph for cluster storage. What’s not to like?
since it is debian based, couldn't you install any GUI of choice to interface with the host on the same machine?
I started using proxmox last year and I have to say that it is clearly very good
I agree concerning the anoying repository
I must also add that to have the possibility to make snapshots of your vm you have to pay attention to the dis format of your VMs
however up to now it is working like a charm
I believe the reason Proxmox has the VMware features, the pvscsi, vmxnet3, etc is because if installed Vmware Player on a Linux host, it has to compile kernel modules. So they may have the modules precompiled and ready, not to mention there is Open-VM-Tools, which is an open source version of VMware Guest Additions for Linux. When the Open VM Tools became standard, VMware pretty much dropped support for their own proprietary guest additions for linux, especially for newer kernels. Maybe having these for guests is for compatibility should you wish to migrate from ESXi or any other VMware product.
Is there a "Distributed Switch" like functionality built in, or does it require 3rd party SDN?
I´m not exactly sure how user an group management is in esxi, but have you noticed that you can also add other realms in proxmox like from LDAP/AD or even OIDC?
Yep, that’s also available in ESXi and vCenter.
Should have really talked more about the ability to GPU and other pass through capabilities as well without paying an arm an a leg for Nvidia licensing. It's really nice for Plex or Jellyfin transcoding and running cheaper LLM/AI solutions. Also the Proxmox backup solution is truly amazing and probably needs to be talked about more. It doesn't just give you VM backups but it also does file backups as well. For me this was such a better and easier solution than going with Commvault or Veeam.
Ick. CommVault. What a pile of steaming crap that thing is. Thank God we got rid of that a few years ago. We went to Veeam, but someone else manages that, so I don't know how good or bad that is. But, yeah, backup capabilities being built into the platform vs 3rd party is so practical.
I'm sure you will hear linux ops folks talking about how great the cmd line is for admin'ing... and I don't disagree... IF you're doing it frequently enough to remember how to do everything. For the rest of us who don't admin linux machines, we waste a LOT of time looking up commands and that goes away with a basic (even text based like with esxi) ui.
I just started playing with Proxmox as a potential migration path from vsphere for my roughly 500 core cluster, the new vGPU non-prod release gives me hope that by the time I need to migrate there will be production gpu support which would make proxmox fully capable of replacing my vsphere e+ environment.
Good overview but again I'd like a deeper dive in to advanced features like backup... upgrade workflow, software defined network and so on.
oh and yea I DID assume containers meant app containers too.
Host with API access only. Let the community write the front-ends they need. Makes the host smaller as well.
@@brodriguez11000 Hosts need more than an api... they need a capable diagnostics interface (that should allow admins to manage the local machine). An api only host would be a problem in certain disaster recovery scenarios.
Could be console based, ssh or web... but should not rely on external custom software to work. (everything needed on the box).
this is an amazing video. I learned more than expected and am very comfortable with my decision to go the proxmox path. Thank you !!
Comments like this are why I continue to make videos! So glad it help you!
One thing I noticed in your XCP-ng comparison was the mention of the limitation of VM pausing during snapshot creation if memory is included in the snapshot. However I don't remember hearing about Proxmox snapshots in this video. Does it have the same VM pausing issue as XCP-ng for memory-inclusive snapshots? It looks like the snapshot capability is dependent on a number of factors such as the virtual disk format or the file system used to hold VM images (so you need to use QCow virtual disk files if your VM storage is networked using NFS or Samba), and that would have been good to mention.
I can't give a comprehensive answer but the short answer for taking a snapshot with memory. Yes, the VM state is paused for the duration of taking the snapshot.
You can both pause or stop to get all the memory in snapshots. Those limitations are the differences between block and file storage. Not due to proxmox.
Since when can Proxmox automatically shift VMs to other hosts in the cluster to balance out workloads?
It has HA so that in the event a host goes down, the VMs should power up on a host that's online.
18:40 what about ESXi's EVC?
That's kinda what I was hinting at in terms of being able to emulate down to the VM any number of different CPUs. EVC only allows back compatibility from the procs in your host, and when EVC is enabled, you can't migrate a live VM between hosts with different generation processors. It's entirely possible that choosing a different CPU ID in Proxmox is meaningless to KVM, but it doesn't feel _right_ to me, you know?
@@2GuysTek It is not meaningless. If you choose a cpu type that is the base common denominator such as qemu64 and (even older less performant) kvm64 you can plop it on pretty much anything and it will work. There are also the vulnerabilities that come with something that can run on anything. So the named models are a way to get close to the cpu types your system uses, add in vulnerabilities/mitigations and still be safe to live migrate to a new host, unlike cpu type pass through which would require the identical cpu on the host you are migrating onto. ABI Compatibility if you can google.
Wonder if the CPU type selection when creating a VM has to do with something similar to EVC on VMWare, where the VM is limited to a specific instruction set regardless of physical CPU, so you can move VM's around between hardware from different generations, so that if you set the CPU type to the lowest common denominator in a cluster of machines, it shouldn't have any issue moving a vm between those hosts.
Yes to being able to select a cpu type that is the lowest common denominator as well as the reverse case of utilizing something about the actual type of cpu(s) you are using on your hypervisors.
You pretty much hit the nail on the head. It does indeed choose a common instruction set so that you may do live migration from one node to another even if they have different physical CPUs. _(also enables high availability for the same situations.)_
You can still use ESXi, assuming you have kept the ISO's and your keys, which i have. I decided to give Proxmox a go last month an after being a VMware admin for 9 years i found it fairly easy to get to grips with it. There were a few things i had to google such as the ISO storage !. Personally i have found it very good and on comparable hardware im noticing lower CPU use and lower RAM use compared to ESXi running exactly the same VM's.
Any news on what they plan to do with VM Workstation Pro?
I use Workstation Pro for testing and for keeping older OS's available so I can offer support for older OS's
I have over 100 OS's from desktop to server to Appliances like pfsense
Any word on those licenses or some sort of alternative?
Didn't the part of VMware that handles Workstation just get sold off a couple days ago? I think changes are likely.
@@thegrossmeyerHmmm this is an amazingly strange move.
I absolutely agree with all of the comments on the lxc containers in proxmox. The first time I started my proxmox virtual server, I started with VMs. However, I realized that it takes more memory and resources to run the virtual machines, so I changed to creating lxc containers in proxmox and it uses very little resources when loading a lxc container. I only use VMs if it calls for using them in proxmox. I am now getting ready to run a second proxmox VM because I love that they added the backup server to backup virtual machines. however, I do want to try other VMs, such as XCP-NG with Xen orchestra since I am very familiar with citrix, and also SUSE harvester. I want try VMware ESXI, but I am glad I did not go further since they got rid of the free edition.
I totally agree! If proxmox made the GUI a little easier, made the upgrade more seamless, and made it possible to migrate from vmware to proxmox smoothly, it would be a no brainer. But these points remains and a couple more you mentioned in the Video. Really hope to see these changes! Thanks for the video, good summary!
Just to add to this, Proxmox recently released a conversion tool to make migrating from VMware to Proxmox easier: news.itsfoss.com/proxmox-vmware-migration/
Thank you for this and your other companions. How about a comparison with Hyper-V? My company is thinking of going with it as we have server licenses
It’s in the works!
The main question is, is PVE reliable, i mean rock solid so you would run your essential SAP Environment on it, 24/7 ?
Similar to esxi really. Build a 3 or 5 node PVE cluster with Cepth.
one of the critical feature linux based hypervisor (kvm) lack is storage offload API like VAAI
With VMware screwing the pooch I'm betting lots more dev is going to be pushed into KVM based stuff and OEMs of storage solutions are going to want to support it too. Hopefully it comes along!
I just wiped my drives last night from Vmware ESXI 7u03 after using it since 2022. I ended up going with proxmox for now because due to having way more support in terms of DevOps tools such as Ansible and Terraform as well as installing nested hypervisors. I tested xcp-ng but the support base is too small esp for DevOps and API automation stuff at the moment. Also the RAM usage is slightly higher.
Maybe cover Nix Shell.
thanks for the comparison, i noticed that with proxmox unlike esxi, you can't install a promox host to a proxmox cluster if that host has vm's on it.. even if they are shutdown.. the promox most MUST be clean.
It depends on the host's storage. If the VMs are on a separate storage then installing ProxMox on the host's boot drive is not an issue.
Great video! I'd love to see a comparison of xcp-ng and proxmox now!
The VM CPU type will be for clusters of hosts that have different CPUs. To maintain the ability to migrate you need the cpu features to be the same. in VMware this is the EVC settings.
Hey, would you consider doing a video on an in place migration from ESXi to Proxmox. I like many installed ESXi to get the Guest OS off the hardware, making migration to new hardware a little easier when the time came, now I have a number of boxes that need to go from ESXi to Proxmox VE. How about it?
I managed Esxi for years at my company (5 hosts, 35 VM) last summer we moved to Proxmox: 100 % satisfied, just a few small issues (solved). The only problem I still have is when I want to download and deploy a virtual appliance but it is available only to vmware. It works on proxmox after conversion but it's not ideal.
tbh the major upgrade from 6to7 and 7to8 was so painless and easy. just a small check-script and the go for it. no much hassle
VMWare or Proxmox?
@@TechBench I´m about Proxmox. Unfortunately I never got the chance to major upgrade a vmware...
I’ve been hearing about Scale Computing - do you know anything there? Would love to see a breakdown
What about solusvm from webpros?
If you have not tested running multiple Proxmox nodes in a cluster with ZFS snapshot replication running between them, then you have not seen a huge part of the usefulness of Proxmox for small environments. Using that simple setup, you get the ability of almost instant HA or manual failover of VMs/LXCs between Proxmox nodes without needing any shared storage required. This is huge for small business or home users, to be able to have this VM migration capability without requiring any san or external storage.
Does Proxmox have the stringent CPU requirements as ESXI does?
No
No. Proxmox will run on a potato.
And on the other end of the potato scale, proxmox supports big little ecores too, which esxi still has some troubles with
Proxmox now features easy import from Vmware. Don't know when it was available, but the newest version is capable of that and it works very well (as long as you dont have something like a Server 2003 you need to migrate)
LXC containers are a lot more light weight than full vm's, they start up in ms and use less than 100 mb of ram. If your usecase does not require extreme security then they're great.
are you gonna make a video about harvester? it's a hypervisor from and meant to be integrated with suse's rancher platform
At 8:03 you say all management meeds to be done frol the gui. I'm a novice proxmox user and my feeling generally is that sometimes in proxmox you have to use cli. That said I believe you _can_ do everything in cli that you can from the gui. And there's a lot of documentation around it.
I agree a wizard for basic setup would make certain things easier though
Most of the cli is just normal ol' linux cli. I would say the ZFS stuff in particular is probably still a situation of very much needing the cli. Save a copy of the admin guide on where ever you are usually doing your stuff and you'll have it down before long. If you are going ZFS, look for a book by Michael Lucas called FreeBSD Mastery: ZFS. It's well worth it.
And killed the VMWare nonprofit program. Great video, thanks for putting together
Just spent the last year of my life migrating from nutanix ahv to vmware. Looks like im gonna be spending the next year moving elsewhere. Really wish i would have purchased 5 years of support to push this out
Why have you decided to migrate away from Nutanix AHV, and what factors are preventing you from considering it as an option?
@@bouzidoussama496 Our hardware was EOL, so we started pricing out upgrading or replacing it. For the price, we were able to get 3x more in terms of performance, storage, compute, and networking by going back to VMware non-hyperconverged. AHV did work pretty well; however, there were times when it was frustrating. Some vendor appliances only support VMware, and we ended up having to run a small cluster alongside the entire time due to this. Simple things like pulling a report of guest OS were nonexistent, simple but a lot of small things that are easy in VMware do not exist. Going to AHV is pretty easy; their migration tool works great. Moving back to VMware was not as easy but doable with some tools.
I’d love to see a harvester and hyper-v videos
I’m interested about similar comparison between Nutanix, Scale Computing, Oracle VM, Red Hat alternatives with RHEV and Openstack VM, Microsoft solutions for onprem VMs and containers.
Dayum, seems like I will have to move back to Proxmox due to ESXi not being free anymore.
I completely hear the point on the benefit of a menu on the console. However, more and more system admins are so reliant on menus and GUIs, that they are not taking the time to learn the CLI. I have been in technology for 25 years and some of the best platforms that I have supported have been on Linux. Getting to "know" the OS and how to maneuver within it is a necessity that the next generation of administrators need to embrace. Just my 2 pennies... :)
i can't agree more! having a simple cli menu for basic tasks would be nice, but knowing linux gives you so much more power on the console. as example: we are moving from vmware to proxmox in my company and i do all migrations on the console because i can convert/move vmdk files directly to proxmox ceph storage. the GUI can not do this.
yep, the whole world pretty much runs on linux now, so if you're a "serious" sysadmin and the best you can do is navigate an ncurses menu or a webui, then it's time upskill
Vsphere is also available as a standard version, includes vsphere std hypervisor and vcenter std. Also essentials and essential plus are available. These will suite for SMB and will defenetily cost no more than before Broadcom. Only difference is that lowest support is now production support which will of course be more expensive than basic.
You should checkout Rancher Harvester! Power of k8s applied to VMs with strong defaults to make it just work OTB.
reli nice video, I helped my company to migrate all VMware VM to Azure last year and removed all the on-prem hardware. I think the trend will go on that VMWare is not in a priority in the coming future. Personally, i will learn some Proxmox for the on-prem alterative.
Just an interest, will talk a bit more about the Proxmox Backup Server? (as for the VMWare, I am familiar with using Veeam Backup and Restore. How about in Proxmox?)
Thanks and cheers.
very useful video, thx. never used vmware, trying out pve now. i like to learn the under the covers troubleshooting, but agree they coulda shoulda provide a mgmt interface on host. i struggled to get the connection working to get the web ui, which required navigating local conf files to even get going. i'm wiser for it, but in a production setting i'd like a ui pointing me in the right direction to potentially save hours of downtime.
Proxmox LXC system containers I find super useful in a homelab situation, for a production environment I use Incus (LXD fork by LXD devs).
I'm using Proxmox for 3 years now.
The switch from VMWare was extremely easy and I dindn't need to use the documentation ever.
They are very similar.
But i also manage a lot of headless Unix(FreeBSD and AIX) and Linux servers. So the command line is pretty easy for me.
Do you have a one file, one click config backup/restore?
If it crashes. How long will it take you to get tte same or similar machine 100% configured and ready?
(More than 10 minutes?).
Did you implement and configure a UPS for power outages/spikes?
@@pepeshopping I mostly work in datacenters all over the Benelux. Off course snapshots and backups are made regularly. Also most server racks are equipped with at least 2 UPS’ses. At home I only have 7 dual CPU Xeon V4 servers and a small SAN/NAS Backup solution(2x NetApp FAS2246+ and a Synology DS3622xs+) off course also with 2 UPS’ses.
How about nutanix?
The smaller Hypervisor players are just ok, but the smarter people will go with standards based, open source solutions like KVM + ZFS.
Hi.
It is possible to do a M.2 IOPS test on a virtual machine on proxmox? if you have a performant one?
I used ISXi paid version and with a 980 pro i only got about 600k IOPS, and with hyper-v i got 990l IOPS; which is the very reason a switched from ESXi7 to hyper-v.
I am wondering if proxmox allow full speed on m.2 ssds, i would love to give it a try, but i don't have any server available.
Otherwise, from your video i see that proxmox is way better than ESXi considering that is free.
I am enclined to believe that it might be a better solution than hyper-v also, since is a type1 hypervisor and not 2.
Also, it is possible to have a folder sharred between multiple windows virtual servers? like to make them a D to share that folder?
And a shared one for linux servers?
I believe a folder shared between different OS types its impossible.
I hope i get an answer from you.
Thank you.
Thank you for this great review. The biggest issue with Proxmox is no realistic self service option because there is no way of setting CPU, Disk and Memory Quotas. Resource Pools only limits what you can do and not how much. VMWare Cloud Director has no equivalent and there is no Resource Pool monitoring either; you need to trust your users not to eat up your entire Datacentre's resource.
GREAT video will Hardware Vendors maybe start to OEM or sell Proxmox installed is that known?
Yes! In fact, we're working on video as we speak about a company that's doing that right now!
@@2GuysTek GREAT news!
You can use those repos, because they exist and are needed for pve updates.
Instead of disabling, you need to change the apt sources with the community repo.
This was a great overview, I'll have to move my lab soon, I'd love to see some guides on moving (exporting) VM's and other config from esxi and importing into Proxmox where possible
Their is now a Native Tool that proxomox has that allows you to import VMware ESXi Images directly to proxmox
@@lachlanstone282 yep I used that as soon as it came out, goodbye VMware
I’d be very interested in a comparison aimed at enterprise usage. Which doesn’t necessarily mean large deployments, but in terms of support, features and life expectancy - nobody wants to migrate to a platform that’s not financially backed and may stop being maintained at any time. Nutanix comes to mind, and RHE virtualisation for example.
Hey man, i have to say your X-CPNG and ProxMox vs ESXi has been great and super informative. I appreciate you taking the time to do this..
I run ESXi 6.7 in home lab and, given the state of the company going forward, i am looking to change that. Have you done any videos on running ProxMox in mini PC's? I know it is pretty compatible on desktops and would run fine on my R630's but looking to downsize the power usage footprint a bit. have you done video's on Proxmox Backup Server?? Wondering if that is a worthy addition to the PVE environment
...
Either way, great video and i am looking forward to catching some more of your content - Liked & Sub'ed!! keep up the great work!
I'm going to be putting a lot more effort into videos around both Proxmox and XCP-ng in the near future, and they will definitely take into account their ability to run on a variety of different hardware! Thanks for the comment!
as a data point for you: my home lab is a mini pc (minisforum HX90) running proxmox. It started life as proxmox 7.x and is now 8.x.
I’d love a video on harvester.
Prox doesn't have nmtui?
root@proxmox:~# nmtui
-bash: nmtui: command not found
@@2GuysTekapt update
apt upgrade
apt install nmtui
I also thought it was only a network manager for Red Hat distros like RHEL, Centos and Rocky
@@ernestoditerribile Now it just needs to be part of the base Proxmox install, and when you log into the console, it pops up automatically!
Can you do Sangfor? Only if you can.
Can you do Incus? It looks very promising.
Containers on linux has always meant LXC. LXC are terribly useful for use cases where you want containerized solution but need more access to hardware. Also as you mentioned it helps tamp down the ram required too. Don't disagree about full virtualized environment provides more security benefits, but for many use cases LXC is the perfect compromise. Plus most modern applications don't need the container itself to he HA to migration is not normally required.
One last thing for LXC is with zfs copy on write file system you can spawn new containers off an existing one and not only does it use minimal RAM it uses minimal storage.
Yeah baby!
The beard is back!
That is good VM omen
Great video. Btw, LXC live migration is coming.
Docker comes from LXC. LXC (and the previous "original" container technology OpenVZ was around before Docker was a concept
The PVE VM CPU host section is just telling PVE how to simulate/emulate the chosen CPU family's instruction set. Choosing "host" is the most performant, becase it's basically just a CPU passthrough from the host to the VM. This could cause problems with HA, though, if you have a cluster of say an x86-64 node and an ARM64 node. Choosing anything else than "host" will lower your performance because PVE (or rather QEMU) either has to simulate or emulate the chosen instruction set, depending on the host CPU's virtualization capabilities for the chosen CPU. I think your example of choosing an AMD Epyc CPU while the host is an Intel one, this would probably result in emulation, so very slow performance. Choosing a desktop type AMD CPU while the host is a desktop type Intel CPU, this could probably become simulation, which would be faster but still slower than "host". I'm not on expert on this specific topic though, so take it with a grain of salt.
Except for cases where the VM needed specific hardware support direct on the CPU, I have personally not see a visible difference.
@@michaelcollins1530 True, but I think it's better to use host whenever I can, so the VM's CPU has all the features available to it.
I've been running a PVE cluster for a number of years now. I like it, and I'm not planning to move away from it, though I'm also playing with xcp-ng just a bit too. But I think a number of the comments here trying to defend PVE kind of miss the point:
* Sure, a system admin should be able to configure a network interface from the shell (and I've had to do it on a variety of occasions). But a console menu like xcp-ng's would still be a welcome addition. And yes, you could get around this by just installing a full graphical desktop environment onto it, but that really isn't the way the product is intended to be used.
* Yes, the subscription nag screen is obnoxious. It's easy enough to disable, but still obnoxious. I get it, devs gotta eat, but...
* Agree on the enterprise repo being enabled by default. Like with the nag screen, it's easy enough to fix--easier now that repository management is included in the GUI--but it doesn't make sense as a default.
Great videos! Thank you. 😊
I'm a little shocked you did not mention that it's possible to import vmdk files from vmware into proxmox and basically migrate your vm's as-is into proxmox with one command in the terminal.
Is someone has some experience with proxmox in a cluster and storage? I need live migration a d snapshot and dont want ceph. Iscsi was a pain in the ass so far😅
Cepth is a good path, but you need minimun 3 hosts. Alternatively you could use a NAS with NFS. I
dont like too to use iscsi.
Great comparison. Thank you.
Great video!
Should create, "Exploring Proxmox and TrueNAS & UnRaid from a VMware User's Perspective" as they're the top three hypervisors (outside VMWare). I feel like TrueNAS is the baby of Unraid and ProxMox xD
When I was reviewing Proxmox VE, I really wanted it to succeed, but I couldn't get promiscuous mode working for network packet capturing for security onion.
Where I work will probably just pay the cost difference and stay on VMWare; but I’m watching these type videos just in case.
One issue I have is we have a set of host with Nvidia L40 cards in them for AutoDeskand other graphic intensive software. Linux is notoriously horrible or horrendously complicated with PCIe pass through. What I’d like to see covered in one of the type of videos is how these alternatives deal with hardware like video cards.
In version 8, they made PCIE pass through very easy, tons of instructions online. I passed through a couple of ports on a 4 port nic with a few clicks.
@@eilanbarak thank you for this input. Seems I should get one of my dinosaurs out of the garage and install v8 or later and do some testing.
thank you for the video, for sure a straight migration path from VMware to Proxmox is something that has been needed for a while.
also, LXC container is targeting the home lab I think. as some people have very lightweight workloads.
I've been using mail gateway for a few years and the confusing mess of an interface is one of the main reasons I'm a bit on the fence sbout VE. Also the nagging screen comes up for users clicking release or whitelist in the report emails. That's just irritating and I'm sorry but I'm not going to pay gor the software for myself snd ten family members when the Community Edition is s whopping €175 per year. Some kind of middle ground here would have been nice.
You know that you can do basically anything in pvr from the console via command line once you're logged in? If you want you can avoid the gui entirely and perform everything the gui can plus stuff the gui cannot do in the terminal?
Yeah yeah you know it. But esxi loses. You can't do anything in esxi there even if you want. Let's say network fails, you're lost. You can't even check vm status...
Sorry you're absolutely wrong
8:06. No, that is not true. In fact, almost everything the host does via web gui can be done via the console. The only thing that IS correct is that there is no console menu, so if you don't know what you're doing, it's easier to start with the gui.
Super well done video.