The one thing I got caught out by, is that the 2.2GB boot image doesn't give you too much room to install anything You can add an extra 10 gigs to the image with:- qm resize 8000 scsi0 +10G
Yeah he really explains things clearly without going long into the explanations making you want to speed the video up to 1.5x like I do some other channels. There's a reason I always come to this channel
Great video Tim! I would generally suggest to use the 'releases' builds as you get the latest LTS build inside there (22.04), instead of choosing the daily builds like in the video. Also please note that these images do not contain the same packages as the standard server, since some features are not relevant in the Cloud, like printer drivers and such. There is a manifest file next to the img where you can see the packages installed. I would suggest booting the image and making sure everything you need is there before setting it up as template. Note that some of the images don't use the 'linux-generic' kernel which can cause issues down the line when trying to isntall packages such as 'nfs-server' for example. And lastly, the VMs created by the template have a disk the same size as the image by default, which is probably not desired in most cases. However, these cloud images are usually designed to auto expand the disk upon boot, so with Proxmox you should be able to use the "Resize disk" button in the Hardware section to set the desired size.
Hey Tim, great video 👍 Before creating the template, I would recommend adding in the qemu-guest-agent to the cloud image with the following command: virt-customize -a {{IMG}} --run-command 'apt-get update && apt-get upgrade -y && apt-get install qemu-guest-agent -y'
Hey just wanted to note that this does something you wouldn't want with your image which is creating those machine-id that he mentions. Be aware of that, cheers.
You can run the command (change the image to whatever you're using, 22.04 LTS is out so I'm building using jammy): virt-customize -a jammy-server-cloudimg-amd64.img --install qemu-guest-agent --truncate /etc/machine-id
I’m an extremely picky and impatient open source home labber. I made it through this entire video. Really excellent work. All online instructors should watch this. Thorough, proper explanations, simple easy to follow commands without rushing. Beautiful! I didn’t even have to copy and paste. Paused and typed! (I don’t recommend that.) I just ssh’ed into a brand new Ubuntu server with no password! And it’s on my ZFS. Ready to be replicated and added to HA and backed up if I start caring about it. That’s so fast! And I can do it with other builds! The commands should be the same except for the img URL! That’s really powerful stuff. I have to stop and think about what I could do with that! I need to bake docker compose and portioner into the img. How would I do that without booting it? I’ll have to work on that. Thank you Tim for this excellent video.
This is a great video tutorial on how to create a Proxmox template with Cloud Image and Cloud Init. I found it very helpful and informative. The presenter is very clear and concise in his explanations, and he does a great job of walking through the steps involved. I would definitely recommend this video to anyone who is interested in learning more about Proxmox
This is perfect. I also adjust the disk to my needs per VM. Besides installing qemu-guest-agent as one of the pkgs in my Ansible script I added the cloud-initramfs-growroot pkg, so when I do make the disk larger I just reboot the VM to get the file system expanded.
@@mitchross2852 Before you fire it up first time, click on the disk then resize, it's a button above. The amount you put in will be added to the existing size will not be the new size.
Another Great Video!!! 👏 I have found this way is a lot easier than using Packer to create my templates. I did run into a few snags. When creating the VM from the command line, I see that the SCSI Controller is set to the Default (LSI 53C895A) and when you import the disk image it is added as an Unused Hard Disk 0. So when you run the next command to add the disk to the vm it fails. All that is needed is to change the controller to VirtIO SCSI and then change the Unused Hard Disk 0 setting to make it a SCSI disk and then everything is fine. 😀 Now on to learn Ansible.👍
ok, wild coincidence. i was revisiting doing cloud init with proxmox on the very same day right before you uploaded this. now i have a handy guide to help me out. thanks dude :)
recent find after using these images for a bit, if you plan on using terraform to provision vms in proxmox, having the guest agent in the template will allow terraform to clone these cloud init images successfully . a quick way to do that is to run the following commands after downloading the cloud init image using wget: sudo apt install libguestfs-tools -y virt-customize -a ubuntu-22.04-minimal-cloudimg-amd64.img --install qemu-guest-agent
A perfect video explaining basics of cloud-init cofiguration and minute details of proxmox internals. And the constructive comments in the document blog page has helped me get the jammy cloud image created. Almost perfect, one minor manual step to remove machine id after cloning VM from template. It could be taken care of while deploying using terraform. thank you Tim, for all your efforts in educating us. I didn't make a first stop at your site and paid the penalty of many hours loss.
I like cloud init a lot. Pitched it at work for a new way to deploy consul clusters. Only real draw backs I took away from it was to keep it simple. If your init gets to complex it can take several minutes to execute and do all the things, It can als be a little difficult to debug. Good you touched on it. Next video should be about service mesh!
This is a good way to understand how to use cloudinit cloudinit with proxmox. But if you want to provision the best way i found around there is using terraform with proxmox provider :-).
Cool! I'm doing the same, but without UI, just bash script to batch configure it with correct VLAN and IP. Of course would be simpler to do it with ansible pointed to the proxmox host
This is what I do but using terraform or/and ansible depending my needs. Easy to do, fast and you can optimize / customize the cloud init to fit your needs.
Hey Tim, probably a late comment, but in your docs instead of removing the file: /etc/machine-id just remove its content ( I use echo -n > /etc/machine-id), that works every time! For me this helps a lot with automations, I can grab later the ip of the machine with the proxmox api. Lot easier than doing some other hack!
very cool, thank you! While following the video i put these commands into bash scripts to make it super easy to repeat if i mess it up. Some things were different but easy to adjust to my version. I really love the implementation of cloud-init like this, saves SO much time. edit2: the vm's show up as "ubuntu" on first boot, but will advertise the correct hostname after a reboot.
Great stuff. I love cloud init. One other thing you could do is use cloud init to kick off an initial ansible script that did the after provisioning bootstrapping too.
To provision a Kubernetes cluster in my Proxmox homelab, it was long and tedious. Before: - I had to create the virtual machines manually from an ISO, - Then I accelerated the process with Cloud-Init images, - But i had to install the packages manually, - Same for the configurations - I make sure to make snapshots of my initial configuration, - If the configuration fails, i delete my machines and we start again... Duration : 15-30 minutes and more.... Now : a Terraform declarative file is enough to get my kubernetes cluster with the desired configuration, Duration : 3 to 5 minutes maximum and a simple "terraform apply" is enough to destroy and rebuild my cluster,
I'm also running Cloud Images in Proxmox and they are set up through Ansible. Love it! On all my virtual machines i have a second drive which contains all my data so the Cloud Image drive is just a "throw away" drive, just like docker images. To switch between different versions of ubuntu i only need to change 1 line in an ansible file and run 2 ansible scripts.
Not sure whether this is related to Proxmox version (8), but this doesn't work for me when I create a template without password (SSH Key only). It always gives me publickey persmission denied, even though I do have everything configured correctly (been doing this in public clouds for a while now). So I imagine there might be more tweaking to do with a custom cloud init yaml file.
My only gripe about using cloud images directly from a vendor is that the partitions themselves are not in LVM. So if I want to expand those partitions, I cannot.
I was making a 5 node k3s cluster and got so sick of installing Ubuntu 😂😅😂 Decided there had to be a better way. I wonder if vagrant can work with proxmox...
Recent proxmox actually have a download-from-link feature in the UI where you would normally upload the iso, which is a very practical addition. Would you consider making a video on doing this entire setup via Terraform? I've been playing with it via the API, and it seems like an elegant way to setup K8s clusters.
I've written Ansible to spin up K8s using k3os and their takeover script. It's pretty wild to start with an Ubuntu template and end up with however many k3os machines as you want all clustered together. I even create DNS entries in Pihole and DHCP reservations on my Mikrotik router. That said, I only did it that way because I use Ansible for my day job and I'm familiar with it. I do think terraform is probably the better way to do it. Terraform even has a Proxmox plugin for it.
@@FlexibleToast Thanks, I normally use k0s with k0sctl, which is relatively easy to add a node to a cluster when you can ssh to it. I'd love to see Tim's take on a setup like this. I guess there's many roads to rome this way.
@@TweakMDS I've never heard of k0s. The k0sctl kind of reminds me of openSUSE Kubic and kubicctl. Which is really awesome, but more resource heavy than k3s. I really wish OKD supported Proxmox for IPI installations.
@@FlexibleToast I'm running both k3s and k0s, and for my use cases, k0s has the edge, especially ease of install / adding nodes and upgrading with k0sctl.
@@FlexibleToast so much on resource usage XD , but okd is freaking awesome, especially with ipi. Proxmox really needs a Kubernetes cloud controller and CSI driver. That would be dope.
Getting cloud init working in proxmox is step 1 of 2 for full infrastructure as code using Terraform which has a proxmox module that relies on images having cloud init support
This kind of thing is where you really start to see the lack of polish in Proxmox. Compared to oVirt, this is incredibly complicated. OVirt can be connected to OpenStack glance repositories and by default comes with one attached. From that repository you can import the cloud image directly. You can then build VMs from those cloud images directly, no need for any cli. On top of all that, Ansible AWX (open source version of Ansible Controller, formally Ansible Tower) has built in inventory sync for oVirt. So the new machine can get automatically added to an inventory and imports a bunch of facts about the VM that can be used as variables. It's honestly the thing I miss the most about oVirt. What I don't miss is that oVirt hci was super fragile and I could never rely on it. Proxmox hci with CEPH is rock solid even on my old, affordable gear.
Yeah true I love oVirt as well but as you said it's not the most stable product... It's very unfortunate... I also miss the fact that oVirt has a CSI driver for Kubernetes... This I'd love to see on Proxmox so that I don't have to run any storage provider...
@@LampJustin rook-ceph can be used with an external CEPH cluster. The Proxmox devs said they aren't doing anything special with their CEPH implementation. So you could probably use rook-ceph as your storage operator connected to the Proxmox CEPH cluster. Just give rook its own pool. This would make a potentially very cool hypercoverged setup.
@@FlexibleToast sure but I don't currently have the resources/nodes to get a Ceph Cluster going... And on oVirt you can use any kind of backing storage
Until now i've just been using a base Ubuntu-Server machine as a template. I think i'll look at the cloud image and see if i can use that for a more efficient home docker stack.
Thanks Tim! Just a remark, I installed a cloud debian 11 template. After a few attempts I realized that dhcp was installed in the image by default! I I verified this in the .json file of the debian cloud repository. Indeed when I added a dhcp to the template, every vm generated by the clone had two ip addresses! My lesson, better to know what exactly you put it in making templates. Thanks anyway for the video.
Very helpful! One question: if I want to update the template with a new image can I just replace the image in storage? Or do I need to do something to the VM?
Nice one. Been doing this with the terraform provider for proxmox for some time but recently been switching to Flatcar Container Linux and their ignition config system. It's a bit hacky getting up and running with images (gotta use args) but pxe booting and installing is magic (with matchbox and terraform provider)
why not use the KVM optimised kernel version "QCow2 UEFI/GPT Bootable disk image with linux-kvm KVM optimised kernel"? I think you are using proxmox --> ubuntu VM host --> docker/k8. Is that what this is for? what benefits are there, specifically?
Tim, great content!. Question: if I want to add some custom info, such as: adding a repository by default, modifying .bashrc, creating some folders, adding a tar.gz file inside a folder... Cloudinit would be the option?
I created my own cloud-init iso (with the user-data and meta-data files) and attached it to the vm as an additional drive, but when i go to the cloud-init tab in the web gui it's all greyed out. I am stumped. EDIT: I figured it out. One needs to blindly follow your instructions and attach a cdrom with an image that does not exist yet with the special name "cloudinit", implicitly instructing proxmox to generate it in that location. Also, I found out that you can not use the ISO storage for that. It is quite annoying that all of this is never spelled out anyhwere.
2 роки тому
You mentioned showing the post-deployment configuration (ex: QEMU agent) via Ansible later. Is that coming in another video? (I know you have a getting started with Ansible video already, but I was hoping you'd go into more details on how you use it in your lab)
@@TechnoTim yeah im from brazil, im devops and app engeneering. I spent most of my time using Hyper-V but now with proxmox everything changes. Thanks for the excellent tutorial.
@@LampJustin I believe proxmox allows you to specify a custom yaml config, but it can only be done via the cli. I remember something about that on the wiki.
I’ve had issues where the VMs keep getting different IPs from the dhcp server every time they are restarted even though the DHCP has a lease of 364 days
Great video. I was curious if when a new LTS comes out you should just create a new template or if there is a way to update your template with the new LTS?
Great video! What I am missing is on what are those small vm's used. They have a fixed disk storage and were 70ish% used. Are they meant to use with microservices with data in a NAS/SAN? What would you run in them? I am honestly asking 😊😊. Thanks!
I tried doing the same thing a while ago, but using ansible only... it resulted in unbootable/non configured VMs 😓 ansible's proxmox_kwm module is not good for this tasks! One thing I can suggest is that the qm set commands can be written as a single one (I keep seeing them separated on all the guides I found online, but the documentation says it can be done as a whole)
Nice, is their a way to set a script to run in all of the vms and or host to edit the hard drive sizes, install 'my default" packages, ect. (on the template side so everything is done when its previsioned)
Hey Tim! Great video. When following the video to setup a cloud-init template I am see an issue when starting my cloneed VM. When the VM boots is also boots me into the grub resue prompt. Any help would be greatly appreciated.
Well, I had a different idea of cloud-init than just providing default user, keys and network info - but rather the idea of specifying a run-once script based on a YAML config like the User-data in EC2 instances in AWS. I get that with MTU on my Mac - so I was initially excited to see Proxmox supported Cloud-Init until I realized how limited it was. Also, I thought server Ubuntu image already came with cloud-init.
Question for ya. I've been running the Ubuntu cloud image for a while now. Previously I've just been using a standard Ubuntu install, making it generic, and then making it a template in Proxmox. I then use Ansible to deploy the template, and rework it with new ssh keys, host name, etc., when I need a new Linux box. This has worked out well so far. Now I'm on to playing with the Ubuntu cloud image, and I have to say, the disk access performance is pretty awful. I've got a pretty ho-hum application that I run, and it uses a MariaDB back end. I make an update to the application, and then need to wait nearly 30 seconds for those changes to populate. Using iotop says that the EXT journaling process is constantly the busiest process showing anywhere from 8% IO to 30% IO. On the same version of Ubuntu (20.04 server) using the standard installer and same application, the journaling process is not always the top process eating up disk IO, and when it does take up disk IO, it's like 2% at the highest. Something to note here is that this is definitely NOT a well provisioned Proxmox machine. It's a desktop box with a single hard drive in it, so a certain amount of performance agony is to be expected. That being said, the Ubuntu cloud image is the first time I've experienced any disk related performance agony on this box. And given that the standard Ubuntu images run just fine, I think it's safe to say that there is something very wrong with the Ubuntu cloud image. The mediocre specs of the physical box are just amplifying the numbers a bit. Is this something you've seen in your environment? I've got a Proxmox machine at home that's significantly beefier. I intend to try all this out on that machine soon. I suspect I'll see similar results, but given that it's running a ZFS pool of mirrors, rather than a single EXT4 disk, I think a lot of the performance hit that is apparent in the Ubuntu cloud image will be masked.
@@TechnoTim I've never really had to update the virtio drivers. I always just use whatever comes with the distro I'm using. But one thing I did find out later today is that I am using a newer version of the application. So when I get in to work tomorrow, I'm going to have to check to see if the performance hit comes with the new version of the software. I wouldn't think so. But it's always good to be sure, amirite.
been using this feature for a couple of months, however I find this problem when I'm building it for making my home-assistant. Somehow using cloudinit and building the supervised version will create problem with network. Somehow the HA script changes its network into 2 multiIP and did not work well. Unsure what happen with it. Running it on Manual Debian install did not cause any problem with network, so wondering why? Maybe you have a solutions?
Instead of booting the image to customize it, you could just use virt-customize to add packages like qemu-guest-agent to your image.. Which you need to get the IP out of a new machine that uses dhcp when you deploy with terraform. Edit: Don't install libguestfs-tools (which contains virt-customize) on your proxmox.
This works really well, but the Ubuntu image needs the filename changing to .qcow2 for this to work Then use the CLI or the GUI to enable the Qemu Guest and fstrim to clean up the disk image a little qm set 8000 --agent enabled=1,fstrim_cloned_disks=1
@@TechnoTim It was in reply to Hoover's suggestion of using virt-customize to add in the qemu-guest into the image. Setting it outside of the image manually works fine
I haven't the foggiest idea if I mist anything, so let me know if I did ☁
Well you did misspell missed ;-)
@@MikeCarr07 I once watched a film about clouds but it was way over my head. :)
Perhaps resizing the boot drive to be larger than ~2 1/4 GB?
The one thing I got caught out by, is that the 2.2GB boot image doesn't give you too much room to install anything
You can add an extra 10 gigs to the image with:-
qm resize 8000 scsi0 +10G
Fantastic addition. The major thing now is to add or continiue on is at least a bit of hardening, topic that is fairly missed. Other then that bravo!
Okay, now I can officially begin my Saturday
Your videos are legit really good. There can literally never be too much home lab content and yours is super palitable and informative.
Thank you so much!
Yeah he really explains things clearly without going long into the explanations making you want to speed the video up to 1.5x like I do some other channels. There's a reason I always come to this channel
@TechnoTim hey can you share the link to download .img images
thanks in advance
Great video Tim!
I would generally suggest to use the 'releases' builds as you get the latest LTS build inside there (22.04), instead of choosing the daily builds like in the video.
Also please note that these images do not contain the same packages as the standard server, since some features are not relevant in the Cloud, like printer drivers and such. There is a manifest file next to the img where you can see the packages installed. I would suggest booting the image and making sure everything you need is there before setting it up as template.
Note that some of the images don't use the 'linux-generic' kernel which can cause issues down the line when trying to isntall packages such as 'nfs-server' for example.
And lastly, the VMs created by the template have a disk the same size as the image by default, which is probably not desired in most cases. However, these cloud images are usually designed to auto expand the disk upon boot, so with Proxmox you should be able to use the "Resize disk" button in the Hardware section to set the desired size.
Hey Tim, great video 👍 Before creating the template, I would recommend adding in the qemu-guest-agent to the cloud image with the following command: virt-customize -a {{IMG}} --run-command 'apt-get update && apt-get upgrade -y && apt-get install qemu-guest-agent -y'
Thank you!
Didn't know about virt-customize. Really handy for setting up VM templates. Saves me the hassle of cleaning up mashine-id and ssh-keys etc
Hey just wanted to note that this does something you wouldn't want with your image which is creating those machine-id that he mentions. Be aware of that, cheers.
You can run the command (change the image to whatever you're using, 22.04 LTS is out so I'm building using jammy): virt-customize -a jammy-server-cloudimg-amd64.img --install qemu-guest-agent --truncate /etc/machine-id
@@pewter77 Is this run inside the image or on proxmox?
Just used these today actually! Messing around with setting up my environment using Ansible, and cloud-init is great for this as well.
I’m an extremely picky and impatient open source home labber. I made it through this entire video. Really excellent work. All online instructors should watch this. Thorough, proper explanations, simple easy to follow commands without rushing. Beautiful! I didn’t even have to copy and paste. Paused and typed! (I don’t recommend that.) I just ssh’ed into a brand new Ubuntu server with no password! And it’s on my ZFS. Ready to be replicated and added to HA and backed up if I start caring about it. That’s so fast! And I can do it with other builds! The commands should be the same except for the img URL! That’s really powerful stuff. I have to stop and think about what I could do with that! I need to bake docker compose and portioner into the img. How would I do that without booting it? I’ll have to work on that. Thank you Tim for this excellent video.
Wow! Thank you so much! I am glad you like it!!!
@@TechnoTim FYI I found it best to leave out your serial terminal command and leave it on default. I’m on Proxmox 8.0.3.
This is a great video tutorial on how to create a Proxmox template with Cloud Image and Cloud Init. I found it very helpful and informative. The presenter is very clear and concise in his explanations, and he does a great job of walking through the steps involved. I would definitely recommend this video to anyone who is interested in learning more about Proxmox
This is perfect. I also adjust the disk to my needs per VM. Besides installing qemu-guest-agent as one of the pkgs in my Ansible script I added the cloud-initramfs-growroot pkg, so when I do make the disk larger I just reboot the VM to get the file system expanded.
Twinsies!
@@TechnoTim So is this the content for next weeks video? Loved this one...but feels like the ansible stuff should be part 2!
yea disk and disk resizing wasnt covered, please do a short video on it! interested in cloud-initramfs-growroot pkg
@@mitchross2852 Before you fire it up first time, click on the disk then resize, it's a button above. The amount you put in will be added to the existing size will not be the new size.
@@enderst81 thanks for this!!
Another Great Video!!! 👏 I have found this way is a lot easier than using Packer to create my templates. I did run into a few snags. When creating the VM from the command line, I see that the SCSI Controller is set to the Default (LSI 53C895A) and when you import the disk image it is added as an Unused Hard Disk 0. So when you run the next command to add the disk to the vm it fails. All that is needed is to change the controller to VirtIO SCSI and then change the Unused Hard Disk 0 setting to make it a SCSI disk and then everything is fine. 😀 Now on to learn Ansible.👍
Great stuff Tim. This is exactly what I wanted to set up at my studio/homelab!
Glad it was helpful!
Great video. Tried it with ansible, it works great. If you want to add qemu-guest-agent just customize the image you downloaded.
This is exactly the method I personally use to create my templates and create Vms very quickly. Thanks for popularizing this practice Tim ;)
Glad it was helpful!
ok, wild coincidence. i was revisiting doing cloud init with proxmox on the very same day right before you uploaded this. now i have a handy guide to help me out. thanks dude :)
Enjoy!
recent find after using these images for a bit, if you plan on using terraform to provision vms in proxmox, having the guest agent in the template will allow terraform to clone these cloud init images successfully . a quick way to do that is to run the following commands after downloading the cloud init image using wget: sudo apt install libguestfs-tools -y
virt-customize -a ubuntu-22.04-minimal-cloudimg-amd64.img --install qemu-guest-agent
A perfect video explaining basics of cloud-init cofiguration and minute details of proxmox internals. And the constructive comments in the document blog page has helped me get the jammy cloud image created. Almost perfect, one minor manual step to remove machine id after cloning VM from template. It could be taken care of while deploying using terraform. thank you Tim, for all your efforts in educating us. I didn't make a first stop at your site and paid the penalty of many hours loss.
This man single-handedly giving me a career path
Most clear video about the subject i found it in internet, now a must see video for new employees in our enterprise.
I just installed proxmox and was looking to do something like this. Thanks a lot for this
No problem 👍
Ya know I went back and watched one of your older videos yesterday for something rancher related. Crazy how much your background has changed haha
It's changed quite a bit! Everything has! Thanks for noticing!
I like cloud init a lot. Pitched it at work for a new way to deploy consul clusters. Only real draw backs I took away from it was to keep it simple. If your init gets to complex it can take several minutes to execute and do all the things, It can als be a little difficult to debug. Good you touched on it. Next video should be about service mesh!
Agreed! I handle everything after it get's an IP address with Ansible
Really good video, I'd just add a qemu-img resize command to make the image 32G or something that is more usable for a default VM setup
This is a good way to understand how to use cloudinit cloudinit with proxmox. But if you want to provision the best way i found around there is using terraform with proxmox provider :-).
Thanks Tim, never done it before and worked perfectly.
Cool! I'm doing the same, but without UI, just bash script to batch configure it with correct VLAN and IP. Of course would be simpler to do it with ansible pointed to the proxmox host
This is what I do but using terraform or/and ansible depending my needs.
Easy to do, fast and you can optimize / customize the cloud init to fit your needs.
Do you have any recommendations for content on integrating terraform and ansible with cloud init images?
Damn, I literally setup my cloudinit images 4 days ago. It's simple, but I'd prefer if I had this video for that.
Hey Tim, probably a late comment, but in your docs instead of removing the file: /etc/machine-id just remove its content ( I use echo -n > /etc/machine-id), that works every time!
For me this helps a lot with automations, I can grab later the ip of the machine with the proxmox api. Lot easier than doing some other hack!
I’ve been looking forward to this topic! Another great video!
Glad you enjoyed it!
Just what I was looking for! Now we need a Terraform/Proxmox video 👀
very cool, thank you!
While following the video i put these commands into bash scripts to make it super easy to repeat if i mess it up. Some things were different but easy to adjust to my version.
I really love the implementation of cloud-init like this, saves SO much time.
edit2: the vm's show up as "ubuntu" on first boot, but will advertise the correct hostname after a reboot.
Thank you!
Very useful and worked a treat. I followed some other guide and it didn't work, as always yours is very clear and easy to follow thank you
This is great, works like a charm thank you
Great stuff. I love cloud init. One other thing you could do is use cloud init to kick off an initial ansible script that did the after provisioning bootstrapping too.
That's a great idea!
To provision a Kubernetes cluster in my Proxmox homelab, it was long and tedious.
Before:
- I had to create the virtual machines manually from an ISO,
- Then I accelerated the process with Cloud-Init images,
- But i had to install the packages manually,
- Same for the configurations
- I make sure to make snapshots of my initial configuration,
- If the configuration fails, i delete my machines and we start again...
Duration : 15-30 minutes and more....
Now : a Terraform declarative file is enough to get my kubernetes cluster with the desired configuration,
Duration : 3 to 5 minutes maximum and a simple "terraform apply" is enough to destroy and rebuild my cluster,
This is much more efficient than the docs. Thanks.
I'm also running Cloud Images in Proxmox and they are set up through Ansible. Love it! On all my virtual machines i have a second drive which contains all my data so the Cloud Image drive is just a "throw away" drive, just like docker images.
To switch between different versions of ubuntu i only need to change 1 line in an ansible file and run 2 ansible scripts.
Nice!
awesome - very useful! Planning to use this template with Terraform next.
didnt even know i could do this, it will save me alot of time because for some reason i reset all my proxmox nodes monthly
Not sure whether this is related to Proxmox version (8), but this doesn't work for me when I create a template without password (SSH Key only). It always gives me publickey persmission denied, even though I do have everything configured correctly (been doing this in public clouds for a while now). So I imagine there might be more tweaking to do with a custom cloud init yaml file.
Yeah, I ran into this as well!!! You end up with a VM you can ONLY use on the Proxmox GUI.
What am I doing wrong here???
Another amazing tutorial - Straight o the point yet detailed. Man you are an asset
Thank you!
always saving my life brother... thanks for this video... cheers from costa rica
My only gripe about using cloud images directly from a vendor is that the partitions themselves are not in LVM. So if I want to expand those partitions, I cannot.
I was making a 5 node k3s cluster and got so sick of installing Ubuntu 😂😅😂
Decided there had to be a better way. I wonder if vagrant can work with proxmox...
Thanks for the demo and info, have a great day TT
Thank you, you too!
Recent proxmox actually have a download-from-link feature in the UI where you would normally upload the iso, which is a very practical addition.
Would you consider making a video on doing this entire setup via Terraform? I've been playing with it via the API, and it seems like an elegant way to setup K8s clusters.
I've written Ansible to spin up K8s using k3os and their takeover script. It's pretty wild to start with an Ubuntu template and end up with however many k3os machines as you want all clustered together. I even create DNS entries in Pihole and DHCP reservations on my Mikrotik router.
That said, I only did it that way because I use Ansible for my day job and I'm familiar with it. I do think terraform is probably the better way to do it. Terraform even has a Proxmox plugin for it.
@@FlexibleToast Thanks, I normally use k0s with k0sctl, which is relatively easy to add a node to a cluster when you can ssh to it. I'd love to see Tim's take on a setup like this. I guess there's many roads to rome this way.
@@TweakMDS I've never heard of k0s. The k0sctl kind of reminds me of openSUSE Kubic and kubicctl. Which is really awesome, but more resource heavy than k3s. I really wish OKD supported Proxmox for IPI installations.
@@FlexibleToast I'm running both k3s and k0s, and for my use cases, k0s has the edge, especially ease of install / adding nodes and upgrading with k0sctl.
@@FlexibleToast so much on resource usage XD , but okd is freaking awesome, especially with ipi. Proxmox really needs a Kubernetes cloud controller and CSI driver. That would be dope.
This was great! thanks Tim
This is great! Will set this up first thing tomorrow.
Thx!
Hope you enjoy it!
Getting cloud init working in proxmox is step 1 of 2 for full infrastructure as code using Terraform which has a proxmox module that relies on images having cloud init support
This kind of thing is where you really start to see the lack of polish in Proxmox. Compared to oVirt, this is incredibly complicated. OVirt can be connected to OpenStack glance repositories and by default comes with one attached. From that repository you can import the cloud image directly. You can then build VMs from those cloud images directly, no need for any cli. On top of all that, Ansible AWX (open source version of Ansible Controller, formally Ansible Tower) has built in inventory sync for oVirt. So the new machine can get automatically added to an inventory and imports a bunch of facts about the VM that can be used as variables. It's honestly the thing I miss the most about oVirt. What I don't miss is that oVirt hci was super fragile and I could never rely on it. Proxmox hci with CEPH is rock solid even on my old, affordable gear.
Yeah true I love oVirt as well but as you said it's not the most stable product... It's very unfortunate... I also miss the fact that oVirt has a CSI driver for Kubernetes... This I'd love to see on Proxmox so that I don't have to run any storage provider...
Thanks for your insight!
@@LampJustin rook-ceph can be used with an external CEPH cluster. The Proxmox devs said they aren't doing anything special with their CEPH implementation. So you could probably use rook-ceph as your storage operator connected to the Proxmox CEPH cluster. Just give rook its own pool. This would make a potentially very cool hypercoverged setup.
@@FlexibleToast sure but I don't currently have the resources/nodes to get a Ceph Cluster going... And on oVirt you can use any kind of backing storage
@@LampJustin you can have a one node CEPH cluster. You'll want to change the failure domain from host to osd.
You had to say something... Getting error qm set 8000 --scsihw virtio-scsi-pci --scsi0 NAS_NAME:vm-8000-disk-0 and its the right # ... 😑
[FIX]-> qm set 8000 --scsihw virtio-scsi-pci --scsi0 NAS_NAME:8000/vm-8000-disk-0.raw
To others with same issue. Mine was local:8000/vm-8000-disk-0.raw
Did i heard Ansible ?!
Really good video again thanks ^^
Until now i've just been using a base Ubuntu-Server machine as a template. I think i'll look at the cloud image and see if i can use that for a more efficient home docker stack.
Nice and easy. I always just clone one I setup how I like it. Guess it is time for me to finally start learning Ansible
You can do it!
Great video. This will save some time when pushing out new VM's 👍
Thanks Tim! Just a remark, I installed a cloud debian 11 template. After a few attempts I realized that dhcp was installed in the image by default! I I verified this in the .json file of the debian cloud repository. Indeed when I added a dhcp to the template, every vm generated by the clone had two ip addresses! My lesson, better to know what exactly you put it in making templates. Thanks anyway for the video.
Glad I found you! Great stuff.
Very helpful! One question: if I want to update the template with a new image can I just replace the image in storage? Or do I need to do something to the VM?
Very nice work but I have issue when I try ssh say public key error
Sounds like you are using the wrong public key. Check yours and try again.
Yeah, same here. Definitely using the right public key. I imagine there might be something else to do with a custom YAML cloud init file.
Nice one. Been doing this with the terraform provider for proxmox for some time but recently been switching to Flatcar Container Linux and their ignition config system. It's a bit hacky getting up and running with images (gotta use args) but pxe booting and installing is magic (with matchbox and terraform provider)
Hi mate, do you have a link for a how-to that describe what you did?
why not use the KVM optimised kernel version "QCow2 UEFI/GPT Bootable disk image with linux-kvm KVM optimised kernel"? I think you are using proxmox --> ubuntu VM host --> docker/k8. Is that what this is for? what benefits are there, specifically?
Thank you this help's me will thank you again !
Yet again another great tutorial. Thanks
Thank you!
Tim, great content!. Question: if I want to add some custom info, such as: adding a repository by default, modifying .bashrc, creating some folders, adding a tar.gz file inside a folder... Cloudinit would be the option?
Sorry how do we change the drive size?
Your video save me !! Thanks
I created my own cloud-init iso (with the user-data and meta-data files) and attached it to the vm as an additional drive, but when i go to the cloud-init tab in the web gui it's all greyed out. I am stumped.
EDIT: I figured it out. One needs to blindly follow your instructions and attach a cdrom with an image that does not exist yet with the special name "cloudinit", implicitly instructing proxmox to generate it in that location. Also, I found out that you can not use the ISO storage for that. It is quite annoying that all of this is never spelled out anyhwere.
You mentioned showing the post-deployment configuration (ex: QEMU agent) via Ansible later. Is that coming in another video?
(I know you have a getting started with Ansible video already, but I was hoping you'd go into more details on how you use it in your lab)
That's included here too! ua-cam.com/video/w9eCU4bGgjQ/v-deo.html along with all of my playbooks, one of which includes qemu-guest-agent!
this is awesome and mind blowing. tks tim
Glad you enjoyed it!
@@TechnoTim yeah im from brazil, im devops and app engeneering. I spent most of my time using Hyper-V but now with proxmox everything changes. Thanks for the excellent tutorial.
It’s too bad proxmox doesn’t expose the yaml for the cloud-init drive. That’s one thing I love about Xen-Orchestra.
You could just mount the logical volume in Proxmox or /dev/sr0 in the VM ;)
mount -o ro /dev/sr0 /mnt
@@LampJustin I believe proxmox allows you to specify a custom yaml config, but it can only be done via the cli. I remember something about that on the wiki.
@@GrishTech yes you can
I’ve had issues where the VMs keep getting different IPs from the dhcp server every time they are restarted even though the DHCP has a lease of 364 days
Great video. I was curious if when a new LTS comes out you should just create a new template or if there is a way to update your template with the new LTS?
I would just create a new template! Thank you!
Thanks for this!
Great video! What I am missing is on what are those small vm's used. They have a fixed disk storage and were 70ish% used. Are they meant to use with microservices with data in a NAS/SAN? What would you run in them? I am honestly asking 😊😊. Thanks!
They can be used for anything that you need for Ubuntu - a Docker host, k3s nodes, or really any Ubuntu vm! Just expand the disk after a clone :)
Also cloud init has built-in option to expand the disk on reboot when you change size on proxmox.
Thanks, super useful.
I tried doing the same thing a while ago, but using ansible only... it resulted in unbootable/non configured VMs 😓 ansible's proxmox_kwm module is not good for this tasks! One thing I can suggest is that the qm set commands can be written as a single one (I keep seeing them separated on all the guides I found online, but the documentation says it can be done as a whole)
Nice, is their a way to set a script to run in all of the vms and or host to edit the hard drive sizes, install 'my default" packages, ect. (on the template side so everything is done when its previsioned)
I used cloud init + terraform in my kvm hypervisor it help me to build my VM when it first boot
love me some cloud-init
me too!
I currently use packer and a preseed.cfg to build my own "clouinit" Debian Images. That way I can bake in all the packages I need and more.
Nice! I keep mine vanilla and lean and then and run ansible afterwards depending on the role! So many options!
It's imported to the set the disk with the correct path using qm set 8000 --scsihw virtio-scsi-pci --scsi0 Synoimages:8000/vm-8000-disk-0.raw
Great vid,
will You create same video about LXC template?
Doing the lords work
Do you have any plans to install OpenShift cluster on proxmox? I love your videos.
seems like no matter what i do the vm wont accept the login detials in the cloud init tab, even tried default ubuntu logins and non seem to work
Hey Tim! Great video. When following the video to setup a cloud-init template I am see an issue when starting my cloneed VM. When the VM boots is also boots me into the grub resue prompt. Any help would be greatly appreciated.
Well, I had a different idea of cloud-init than just providing default user, keys and network info - but rather the idea of specifying a run-once script based on a YAML config like the User-data in EC2 instances in AWS. I get that with MTU on my Mac - so I was initially excited to see Proxmox supported Cloud-Init until I realized how limited it was. Also, I thought server Ubuntu image already came with cloud-init.
There is a custom parameter in Proxmox. You can specify the path to the yaml file in it. But it doesn't work at all...
Ive been planning on deploying okd4 in my home lab... and waiting on some hardware.
Question for ya. I've been running the Ubuntu cloud image for a while now. Previously I've just been using a standard Ubuntu install, making it generic, and then making it a template in Proxmox. I then use Ansible to deploy the template, and rework it with new ssh keys, host name, etc., when I need a new Linux box. This has worked out well so far.
Now I'm on to playing with the Ubuntu cloud image, and I have to say, the disk access performance is pretty awful. I've got a pretty ho-hum application that I run, and it uses a MariaDB back end. I make an update to the application, and then need to wait nearly 30 seconds for those changes to populate. Using iotop says that the EXT journaling process is constantly the busiest process showing anywhere from 8% IO to 30% IO.
On the same version of Ubuntu (20.04 server) using the standard installer and same application, the journaling process is not always the top process eating up disk IO, and when it does take up disk IO, it's like 2% at the highest.
Something to note here is that this is definitely NOT a well provisioned Proxmox machine. It's a desktop box with a single hard drive in it, so a certain amount of performance agony is to be expected. That being said, the Ubuntu cloud image is the first time I've experienced any disk related performance agony on this box. And given that the standard Ubuntu images run just fine, I think it's safe to say that there is something very wrong with the Ubuntu cloud image. The mediocre specs of the physical box are just amplifying the numbers a bit.
Is this something you've seen in your environment? I've got a Proxmox machine at home that's significantly beefier. I intend to try all this out on that machine soon. I suspect I'll see similar results, but given that it's running a ZFS pool of mirrors, rather than a single EXT4 disk, I think a lot of the performance hit that is apparent in the Ubuntu cloud image will be masked.
I haven’t noticed anything. Have you installed the virtio drivers?
@@TechnoTim I've never really had to update the virtio drivers. I always just use whatever comes with the distro I'm using.
But one thing I did find out later today is that I am using a newer version of the application. So when I get in to work tomorrow, I'm going to have to check to see if the performance hit comes with the new version of the software. I wouldn't think so. But it's always good to be sure, amirite.
How do you pre-install qemu guest agent into cloudinit image before creating template out of it?
Great content, thank you.
Thank you so much
Terraform and Proxmox. pls
been using this feature for a couple of months, however I find this problem when I'm building it for making my home-assistant. Somehow using cloudinit and building the supervised version will create problem with network. Somehow the HA script changes its network into 2 multiIP and did not work well. Unsure what happen with it. Running it on Manual Debian install did not cause any problem with network, so wondering why? Maybe you have a solutions?
Interesting, I'm aware cli is quicker but I presume this can be done via the webui also?
CloudInit CD is shared between every VM?
I want to have multiple CloudInit Template in my cluster
Instead of booting the image to customize it, you could just use virt-customize to add packages like qemu-guest-agent to your image.. Which you need to get the IP out of a new machine that uses dhcp when you deploy with terraform.
Edit: Don't install libguestfs-tools (which contains virt-customize) on your proxmox.
This works really well, but the Ubuntu image needs the filename changing to .qcow2 for this to work
Then use the CLI or the GUI to enable the Qemu Guest and fstrim to clean up the disk image a little
qm set 8000 --agent enabled=1,fstrim_cloned_disks=1
Thanks but I didn't need to change the extension at all. Are you running 7.1?
@@TechnoTim It was in reply to Hoover's suggestion of using virt-customize to add in the qemu-guest into the image. Setting it outside of the image manually works fine
Yooooooooooo. I just setup MinIO and it's amazing if you can give some thoughts on it or something I think it would be awesome
Does deleting the machine-id also cause other identifiers, such as the host ssh keys, to be regenerated?
Super helpful!
Glad it was helpful!
Follow up, show the same functionality in Harvester :)