I think some people are totally missing the point of this video.. Of course if you have a few local servers/old pcs at home, use those! But for only 1$ and 1hr of your time you can learn and apply this skill without having a multi server homelab to play around with! Awesome video Patrick!
That is exactly the point. Sometimes people want to try something quickly like a new feature. If you do not have a second cluster on hand, this is a fast and inexpensive way to try it.
@@lululombard Totally. Think about when there is a new cluster feature you want to test and need a test cluster. 20 min to setup the cluster and get VMs / containers online, 40 min to try out the new feature and you only spent $1 to test on a 5x node cluster.
@@wsv8818 Totally, but then you need to be able to handle nested virtualization, install the OSes, and setup networking. You are still burning a little bit of electricity, and you may not have the same full network in/out bandwidth you have in a data center. You also need the RAM, CPU, and disk to bring up 5 machines. We have another Proxmox cluster setup that I built last week and it took much longer than 20 minutes because of the above. For short duration test, this is a very fast setup.
Hi, I like Proxmox. Maybe you can do a second part for the video with Ceph as shared storage. It just works. Migration within sub seconds, don’t have to download images one every node. Cephfs for files. Shared storages can scale with the number of drive and server. Regards, Frank
This is very informative indeed. I was surprised that the prices for a bare metal cloud system are quite affordable. It allows you to check the latest BMC/IPMI features of Supermicro hardware, which is good for a buying decision of actual hardware.
Hi, congratulations on your video, it's great and very well explained. It would be interesting if you made a high availability video of 3 nodes with different public IPs in different locations and a CT virtual machine with a webserver installed and that migrates from one node to another and that the public IP assigned to the CT machine swings between the nodes . Thanks a lot
Why don't you make a video (much slower than this LOL) on network side of Proxmox? I just wanna know how to assign public IPs to the VMs automatically when we create. I hope you will consider this soon. 👍
Hi @ServeTheHome any chance of a piece on security best practices and secure management strategies when using services like these in production? Something like a 1-2-3 hit list of exposure (or management need) and best practice to accomplish. Love the content!
You could to test. I would probably pick servers with more than 1 SSD so you can dedicate a SSD to Ceph. Also I would use all 10GbE nodes. Finally, for production you would want more nodes/ SSDs as Ceph does better with larger scales
So does a proxmox cluster increase availability and stability or reduce the time for task completion by separating out the tasks, like, let's say, machine learning and data science calculations? Or is it both tasks are separated across the node, and high availability is there by design?
Couple questions, if you have different public IP subnets, when you migrate a server to a different node will it be reachable? I assume there 5 x /29's would have to be overlaid on the same broadcast domain in order work correct? Also without shared storage, can you migrate KVM nodes or just containers?
On the public IP migration - you are correct. What I would do is use a proxy and then route it to the internal machine IP. You can also do something like put pfSense in VMs on different hosts and route traffic through them, then to the internal private IPs (10.x.x.x.) Totally right on those though if you do direct to public IP on a VM/ container. On the shared storage bit, you need that for live migration, but Proxmox can do offline VM migration to local storage. That is where the 10GbE nodes do much better. You can also do things like make a LXC container with NFS and import that as shared storage to the cluster. Lots of functionality. Again, this is not the right setup, as shown, for production, more of for a quick lab setup.
TrueNAS Core is FreeBSD and feels more storage focused. Think of it as an alternative to Synology/ QNAP. Proxmox VE is more trying to be a VMware alternative. TrueNAS Scale will go after Proxmox's core market but it is still not even at its first official release.
Assuming you had the space, didn't mind the power bill, we're willing to maintain the hardware, and you could get similar or better hardware. Still... I don't think it would be as much if you took his hint of having a single backup bare metal machine and you backup when done for the day and restore only when you need to work on the lab again. The bulk of the time you would have one backup machine idling and holding multiple labs that you can use to restore dozens of differently configured labs.
You seem to misunderstand the use case. Cheap rapid testing environment. Set it up, experiment / validate as necessary, tear it down. Well worth a couple of bucks! Given the limited hardware in my home lab (and use of HyperV), this is a great way for me to start experimenting with Proxmox's advanced workflows. Extremely helpful and something I've never considered.
why clickbait? it is $1 per hour which is nearly nothing if you want to play a little bit with 5 DEDICATED hosts NOT vms ..most of the time you'd have to commit and pay setup fees like crazy.. some time ago just to have little POC you'd have to pay couple hundred bucks now you can do it with one dollar.
No, it's not important for test instances like these. You still get updates, just not access to the enterprise repository. The enterprise repository is only important if you have something in production, but this video was clearly a tutorial on how to spin up instances for a couple of hours for testing purposes.
too bad there is no terraform support.. but I found a provider in Europe who is actually even cheaper and has terraform support on no limits on the OS.
you really should establish a sth bench based on price/perf/iops for a small scalable cluster (1m iops vm?)#netfs #mysql cluster #scale iops #redis 200m iops aws #diy - best deals are moving targets esp with zen4 and new sbc #ram #nvme #network upsells #opnsense + qemu + 24/7 pkt cap - do a ha opnsense build, do a cheapest 10gbe router and compare a 100g vs bluefield dpu which is still ent #sonic #opnsense #vm bridge-utils #smb mkt recovery #zen4 has such ipc gains but also so much more efficient - it will drive upgrades #tco #roi #erp #productivity gains #etl jobs
We have been using Proxmox VE since the first STH colocation in 2013 which was a quarter cab in Las Vegas. It is a lot more popular now as features and maturity have grown.
Honestly, as easy as it is, when you don't full control of the hardware and networking you really shouldn't build Proxmox clusters. This results in people with really crappy setups calling people like me.
I think some people are totally missing the point of this video.. Of course if you have a few local servers/old pcs at home, use those! But for only 1$ and 1hr of your time you can learn and apply this skill without having a multi server homelab to play around with! Awesome video Patrick!
Exactly
That is exactly the point. Sometimes people want to try something quickly like a new feature. If you do not have a second cluster on hand, this is a fast and inexpensive way to try it.
Title might mislead a few people, because they might expect to actually only pay $1, not $1 per hour continuously
You can do a lot in 1 hr especially if the setup is
@@ServeTheHomeVideo So you pay 1$ to set it up and then shut it down?
@@lululombard Totally. Think about when there is a new cluster feature you want to test and need a test cluster. 20 min to setup the cluster and get VMs / containers online, 40 min to try out the new feature and you only spent $1 to test on a 5x node cluster.
@@wsv8818 Totally, but then you need to be able to handle nested virtualization, install the OSes, and setup networking. You are still burning a little bit of electricity, and you may not have the same full network in/out bandwidth you have in a data center. You also need the RAM, CPU, and disk to bring up 5 machines. We have another Proxmox cluster setup that I built last week and it took much longer than 20 minutes because of the above. For short duration test, this is a very fast setup.
@@ServeTheHomeVideo I guess why not!
Great video! I I use Proxmox at home on 2 Dell R610 Servers. I like the enthusiasm that just gets one fired up and exited for virtualization!
While I think the the title is a little misleading, paying
loved your energy man, just also starting my homelab, keep it up!
Thanks!
Hi, I like Proxmox. Maybe you can do a second part for the video with Ceph as shared storage. It just works. Migration within sub seconds, don’t have to download images one every node. Cephfs for files. Shared storages can scale with the number of drive and server. Regards, Frank
Oof, describing Ceph as "it just works" is a bit misleading ;)
Love your energy :)
This is very informative indeed. I was surprised that the prices for a bare metal cloud system are quite affordable. It allows you to check the latest BMC/IPMI features of Supermicro hardware, which is good for a buying decision of actual hardware.
i watched this on 1.5x speed so it only took about 15 minutes! A new record! :)
So you paused?
Lol, Patrick, I think you're reaching LMG level clickbait-y titles. Love it! I assume you're going to change it in a week.
Have to try new things.
LMG changes it after a day or 2 🤣
@@kenzieduckmoo Ah. Never noticed.
@@ServeTheHomeVideo If you never noticed, then the plan worked perfectly :)
you mean for 720 bucks a month…
It is a cloud setup. Why run it for a month?
@@ServeTheHomeVideo this!
Hi, congratulations on your video, it's great and very well explained. It would be interesting if you made a high availability video of 3 nodes with different public IPs in different locations and a CT virtual machine with a webserver installed and that migrates from one node to another and that the public IP assigned to the CT machine swings between the nodes . Thanks a lot
Why don't you make a video (much slower than this LOL) on network side of Proxmox? I just wanna know how to assign public IPs to the VMs automatically when we create. I hope you will consider this soon. 👍
Guuuuuuuuuuys!!
This is Patrick from STH 😎
Hi @ServeTheHome any chance of a piece on security best practices and secure management strategies when using services like these in production?
Something like a 1-2-3 hit list of exposure (or management need) and best practice to accomplish.
Love the content!
Yes, this would be wonderful, especially when it comes to networking issues/design.
Hi, did you try or would you recommend clustering the local storage using Ceph on these servers ???
You could to test. I would probably pick servers with more than 1 SSD so you can dedicate a SSD to Ceph. Also I would use all 10GbE nodes. Finally, for production you would want more nodes/ SSDs as Ceph does better with larger scales
So does a proxmox cluster increase availability and stability or reduce the time for task completion by separating out the tasks, like, let's say, machine learning and data science calculations? Or is it both tasks are separated across the node, and high availability is there by design?
1$/hr is a lot.
Couple questions, if you have different public IP subnets, when you migrate a server to a different node will it be reachable? I assume there 5 x /29's would have to be overlaid on the same broadcast domain in order work correct? Also without shared storage, can you migrate KVM nodes or just containers?
On the public IP migration - you are correct. What I would do is use a proxy and then route it to the internal machine IP. You can also do something like put pfSense in VMs on different hosts and route traffic through them, then to the internal private IPs (10.x.x.x.) Totally right on those though if you do direct to public IP on a VM/ container. On the shared storage bit, you need that for live migration, but Proxmox can do offline VM migration to local storage. That is where the 10GbE nodes do much better. You can also do things like make a LXC container with NFS and import that as shared storage to the cluster. Lots of functionality.
Again, this is not the right setup, as shown, for production, more of for a quick lab setup.
@@ServeTheHomeVideo Thank you. I appreciated the reply and clarification.
looks like a good playground for testing out new stuff, not going to use it daily
That was the idea. Then again, these instances are actually priced pretty well (much less than public cloud instances) if you ever did deploy
How does Proxmox compare with TrueNas?
TrueNAS Core is FreeBSD and feels more storage focused. Think of it as an alternative to Synology/ QNAP. Proxmox VE is more trying to be a VMware alternative. TrueNAS Scale will go after Proxmox's core market but it is still not even at its first official release.
I think for ~$9k/year, or $45k/5yr, I'd rather just have hardware.
Assuming you had the space, didn't mind the power bill, we're willing to maintain the hardware, and you could get similar or better hardware. Still... I don't think it would be as much if you took his hint of having a single backup bare metal machine and you backup when done for the day and restore only when you need to work on the lab again. The bulk of the time you would have one backup machine idling and holding multiple labs that you can use to restore dozens of differently configured labs.
@@mos7wan7ed_yt Hardware lifespan is 5 to 7 years. This is 9k/yr. How much server and power budget does $56,000 buy?
Awesome!!
Clickbait title. I didn't expect it from you.
He did exactly what the title said.
You seem to misunderstand the use case.
Cheap rapid testing environment. Set it up, experiment / validate as necessary, tear it down.
Well worth a couple of bucks!
Given the limited hardware in my home lab (and use of HyperV), this is a great way for me to start experimenting with Proxmox's advanced workflows.
Extremely helpful and something I've never considered.
That is exactly the point.
why clickbait? it is $1 per hour which is nearly nothing if you want to play a little bit with 5 DEDICATED hosts NOT vms ..most of the time you'd have to commit and pay setup fees like crazy.. some time ago just to have little POC you'd have to pay couple hundred bucks now you can do it with one dollar.
Add in another €95/year per CPU socket if you want updates, which most will agree are important when you have anything hosted.
No, it's not important for test instances like these. You still get updates, just not access to the enterprise repository. The enterprise repository is only important if you have something in production, but this video was clearly a tutorial on how to spin up instances for a couple of hours for testing purposes.
You forgot to factoring in your price for time. I ask €120 an hour for jobs I do.
too bad there is no terraform support.. but I found a provider in Europe who is actually even cheaper and has terraform support on no limits on the OS.
There is a Terraform plugin on github. v0.9.1 released last month.
you really should establish a sth bench based on price/perf/iops for a small scalable cluster (1m iops vm?)#netfs #mysql cluster #scale iops #redis 200m iops aws #diy - best deals are moving targets esp with zen4 and new sbc #ram #nvme #network upsells #opnsense + qemu + 24/7 pkt cap - do a ha opnsense build, do a cheapest 10gbe router and compare a 100g vs bluefield dpu which is still ent #sonic #opnsense #vm bridge-utils #smb mkt recovery #zen4 has such ipc gains but also so much more efficient - it will drive upgrades #tco #roi #erp #productivity gains #etl jobs
That is somewhat hard to do at a cluster level, but I see the general idea.
I find it hilarious that you're doing proxmox over vmware.
We have been using Proxmox VE since the first STH colocation in 2013 which was a quarter cab in Las Vegas. It is a lot more popular now as features and maturity have grown.
delusional kid
Honestly, as easy as it is, when you don't full control of the hardware and networking you really shouldn't build Proxmox clusters. This results in people with really crappy setups calling people like me.
You should disclose that this is bullshit advertising bullshit.
Such bullshit.