Linux VM Templates in Proxmox on EASY MODE using Prebuilt Cloud Init Images! Proxmox Quick Tips
Вставка
- Опубліковано 15 чер 2024
- Have you ever wanted a nice, easy way to create new VMs to play with in Proxmox using your favorite base distro, without doing a lot of work to configure basic settings like your account, networking, hostname, etc?
Cloud-Init can do all of that and more, but it's designed more for big cloud providers and not the easiest thing to setup. But, what if we could take a generic cloud image, and use it with Proxmox's built-in Cloud-Init automation, to provision easy bare VMs without having to build our own templates? That's what I've done, and I've written a script to automate downloading these templates on new Proxmox systems from the major distros which provide them (Debian, Ubuntu, Fedora).
Of course, once cloud-init is installed and configured, there's no reason we can't clone a template, install software on it, and then clone the clone to have a newly-configured VM with more software installed. From here, you can build out your library of useful application templates for whatever you do regularly.
See the script on my blog:
www.apalrd.net/posts/2023/pve...
My favorite private cloud distributions - Ubuntu and Debian
Debian - cloud.debian.org/images/cloud/
For Debian, you usually want generic or genericcloud (same as generic but with less drives for hardware, good for most virtual machines)
Ubuntu - cloud-images.ubuntu.com/
For Ubuntu, you can use the standard .img, or the -disk-kvm.img. The KVM version has fewer drivers, and I found that the size difference was so small that using the basic .img was best for me.
The script sets up the VMs to have a serial display instead of VGA, this is more convenient since you can copy and paste in the console window. In general, all of these templates will work with serial since OpenStack requires it, but they seem to work fine with VGA consoles too if you're into that.
Link to my Discord to chat about related topics:
/ discord
If you find my content useful and would like to support me, feel free to here: ko-fi.com/apalrd
Timestamps:
00:00 - Introduction
00:40 - Manual Template Creation
04:39 - Scripted Template Creation
06:49 - Using Templates
10:06 - Conclusion
#proxmox #virtualization #cloud-init - Наука та технологія
This is the kind of videos I like the most. Short (no "youtube shorts" please) and packed with a lot and accurate info. Thank you!!
Love this video. Doesn't dance around the subject, just 100% useful all around. Keep it up good sir!
Probably the most concise video on Proxmox cloud-init templates, with a handy bonus bash script to automate the process. Thank you for making this!
Glad you like it!
Proxmox doesn't have such good accessible information like this, so definitely a good subject to cover!
great stuff again! certainly quite some tips which make server management much easier! cloutinit is surely something I need to look into more thoroughly! thanks for the goodies! 👍
Nice one! This will give me something to play with over the weekend. Thanks, man!🖖
Way cool! Love using scripts to make things easier with reproducible results. Love how QEMU/KVM/ProxMox constantly improves over time.
Great stuff man! Helped me to save a lot of time creating new vms.
Great video. Powerful way to always be creating new VMs. Thanks.
Woohoo so fun. Cloud init is the best thing since sliced bread. And now I finally know how to say Apalrd
lol thanks
Great tute - love you work. Thanks!
Excellent work, Apalrd. Cool name.
Great work! Keep up the videos!
Just found your channel. Nice work. Subscribing.
good quick tips. a couple things though, before finalizing your template, it's good practice to sterilize your machine id and also wipe any ssh_host keys as well as bash history before converting to template, otherwise that will carry over. you will want different machine ID's otherwise you can run into issues later down the road that are hard to diagnose
My general preference is to start from a fresh cloud image each time and build it up from there, either a one-time setup for testing or with automation.
I love you man! lol You are the best!!
Thanks for this!
This is awesome, thank you bro
Finally went through and started working on my template. A note which may help some users. I'm on a slightly older version of Proxmox on my main host, 7.1-7, so I had to use "qm importdisk" rather than "qm disk import" when adding the img to the VM. I really need to upgrade to 7.3
Yep. Upgrades are pretty easy these days. I haven't had any issues between versions.
@@Darkk6969 It's on the list of stuff to do. Even have all the proper backups made. Just have to actually do it. lol. You know you're too serious about your home lab when you have difficulty trying to schedule your own maintenance window! lol
Awesome content. Thanks
This channel is great.
Really nice! Thanks! I want to know more about subnetting and network segmentation in a proxmox cluster. Best practices from a security standpoint putting VMs and continers in different subnets/VLANs along with firewall rules. Also I want to know more about virtualizing the router OS within proxmox, preferrably OpenWrt. How to create and let OpenWrt manage the virtual interfaces. :)
So damm useful! TY VERY MUCH!
best video out there
very helpfull :)
Cool short video. Any tips on how to manage templates on a cluster? Lets say I create a template on node1 on the cluster. I can't seem to create a VM on node2 using that template on node1.
You can do the initial creation on the node which has it, and then migrate it immediately afterwards to it's new home (before you even start it).
The way I avoid losing the shell is to connect to the Proxmox host via ssh from one of the local VMs. That's my go-to solution most of the time.
or just use tmux or screen
@@nbensa Tmux for the win.
@@Darkk6969 @Norberto Indeed! That's the method I use when not connecting through a vm.
Really hope there will be a video tutorial on setting static IPv4 for Debian cloud image in the future. 😢 I have been trying in this for a long time, but all my attempts have failed.
The Debian 11 release template isn't working for me, its not running the cloud init at boot. Ubutnu 23.04 does work though.
Have you used Hashicorp Packer before to build custom VM templates? I've used packer for ESXi and for KVM/QEMU/Nutanix, but not specifically with Proxmox. At work I currently use packer to automatically build new Ubuntu and Windows images once every month with all current updates and put them into our Hypervisor. It means when we clone them to create new VMs they're always pretty much ready to go. Is such an integration feasible with Proxmox? E.g. via the API?
With cloud-init like this video, you can run a script periodically to pull new images from Ubuntu and replace the template with an updated one. No need to build new images since Ubuntu provides them regularly with updates. Presumably, you'd have automation (such as Ansible) start by creating a new VM clone, letting cloud-init do the bootstrap configuration (user/ssh keys for Ansible), then letting Ansible do the package installs from there to setup the full configuration.
On Windows, Microsoft doesn't provide images unfortunately, but cloudbase-init is a Windows-compatible cloud-init client that will read the cloud-init drive and reconfigure on first boot of the clone.
All of the commands I've used in the script can be done over SSH, and all of them can be done over the API except importing a disk img/qcow2. It might be possible to import the disk via the API if the path is accessible by the server, such as on an nfs/smb mount (or if you copy it first with scp), I haven't tried that.
Definitely cleaner than how I was doing it before. Think i'm going to have to set up a new default Ubuntu image. I like doing stupid little config changes on my distros like default bash format, etc. Made the mistake of trying out a bunch of distros one day, lol, and found one that had a bash setup that I REALLY liked so I change my .bashrc file every time I deploy a new VM. lol
So you can take any VM and make it a template, and if it has cloud-init installed (which all of the cloud images do) it will re-run init when it's cloned. So you can start with a cloud image, add the bashrc, and then make that the template.
@@apalrdsadventures That's what I had done in the past but using a full ISO with a bunch of pre-installed packages, etc. If I use a cloud init image it's probably a little bit slimmer on resources, especially HDD space. Which I guess isn't as big of an issue but every now and then I get this itch to want to see if i can squeeze a little bit more resource efficiency out of my VM's. lol.
Every distro has their own size for the default disk, but you can resize it pretty easily. The script automatically expands them all to 8G, which is usually enough to start with. The cloud images usually lack a lot of the kernel modules for real hardware, since they are intended for VMs.
If you really want to do some fancy cloud-init, it's possible to write your own yaml file (which can do things like run scripts on first boot, or write arbitrary files) and have Proxmox add it. It's the --cicustom command in proxmox.
@@apalrdsadventures Just finished setting it up and it went great. Was able to make my changes to the original cloud-init VM, regenerate image, converted to template, and then cloned to a new VM and the changes persisted along with having the fresh stuff that should be fresh such as new IP addresses, etc. since I use DHCP most times with labs
@@apalrdsadventures I'm not real comfortable with scripting yet. It's by far my weakest area in my I.T. tool bag. I like how Proxmox lets you resize the HD's. And more so how you can do it on the fly in a container. I did use the resize feature to add 1gb to the default image size for temp files, etc while I got all my desired stuff added in.
Great one!
Proxmox Video Request: VxLAN
;-)
nice job, thank's! i just improve Your script, to add some features like: qemu-guest-agent,ncat,net-tools,bash-completion in debian and ubuntu, and auto install libguestfs-tools if is not installed :))
Save's me at least $5 in time. Thanks
Thanks!
I thought there were things you needed to clean up when you booted ubuntu instance creates machine specific data like machine-id and etc. Is that no longer a thing?
Starting from a cloud template and cloning results in a unique machine-id for each machine.
Starting from a clone and cloning again does not regenerate the machine-id.
Hi, highly appreciated your PVE tricks, I don't know if I missed any step, it seems the disk size just same as cloud image size after I cloned the template, only 3G more or less, may I know how to resize and expand the disk size afterward?
The script I wrote expands them all to 8G I think, but my intention was you'd resize from there as needed since you can't shrink later.
What would be the difference with CT templates?
vaichuong:)). Thanks
i am getting a no cloudinit drive found error
Hello apalrd, Great tutorial thank you and the blog post contains updated images! Could you please explain to me what happens when you use cloud-init and leave the password blank like you do?
I'm not able to login using the serial console like you hint at in the video. Is it creating a random password that we don't know? Is it an empty password but ubuntu does not allow entering an empty password? Since i can SSH in and the user we created has sudo privileges it is fine i suppose but what is actually happening behind the scenes? I can do sudo passwd -d user so it obviously has some kind of password assigned to it.(tried with the ubuntu 22.04 image) Thanks again
It depends slightly on the distro, but not adding a password means the cloud-init should not allow password login of that user, only ssh keys. Depending on who built the image, it may have a well-known root password instead (i.e. debian has some images which are for testing the build process and not for normal use).
@@apalrdsadventures Appreciate the response, thanks! Users, groups and permissions are easily the area i spend the most time debugging.
What kind of monster uses ipv6 at home??
Hahhahaha!
Only the biggest and baddest on the block!
Any chance you could show us the steps to create Windows Cloud-Init (CloudBase-Init) templates to automate provisioning?
I'm working on an equivalent for Windows, but it's more work since Windows doesn't publish prebuilt images like this.
@@apalrdsadventures Thanks, I'll keep an eye on your videos
8:40 "The purpose of this video is..."
Why after installing the bunch of software you didn't execute "cloudinit clean" ? This will rerun cloudinit and reapply values from proxmox. Do it make sense?
I just tried to install k3s and it's saying i don't have disk space. I tried adding a second drive, but no luck. Looks like the drive is being treated as read only. Is there a way to make this work?
Ah yes. By default the drives are pretty small, you can click on the Drive, the go to Hardware -> Disk Action -> Resize, and add the number of gigs to add to the drive. Then shutdown / start (not reboot).
@@apalrdsadventures Thanks. I kept double clicking on the drive and wondering why there's no options at all. This was very helpful
PSA, if you keep the ipv6 settings on SLAAC default on Ubuntu 22.04 in the cloud-init setting at the VM in the host, a network job never finishes and the boot gets delayed for 2 minutes every time in the guest. The workaround is to set it to static for ipv6, no need to enter anything. @apalrd I guess with you loving ipv6 so much you never encountered this issue? From looking at your configuration in the videos it looks like you assign static ipv6 that matches up with your ipv4 or what system do you use? I once turned ipv6 on at my router, I was not that impressed. Everything got an ipv6 address, but stuff seemed slower. I'm not even sure if my ISP will always give me the same ipv6 address, on ipv4 it changes after the router reboots sometimes. So I don't understand how assigning static addresses work. Maybe you could do some kind of follow-up video on your other recent ipv6 video. Thanks and keep the videos coming!
I didn't do non-IPv6, but you can select 'static' and leave the box blank to not assign an IPv6 address.
Generally I do use static for servers, using the last word only. I do often match the IPv4's decimal as a hex number (i.e. 192.168.1.130 becomes ::130). I wish I just had an IPAM system which would generate a completely random one each time I create a new container. Future video I guess.
Oh, and the images -- most of them stall following cloud-init, failing to take commands from Proxmox host.
Return TASK ERROR: can't lock file '/var/lock/qemu-server/lock-101.conf' - got timeout when trying to reset / restart / shutdown / STOP.
Great templates, if you need to reboot your production enviroment ASAP lol
Have the same issue like you.
Around 9:50 both virtual machines show the same iPv4 address ⚠️
I realized based on some later comments that it's an issue with machine-id not getting regenerated. The base cloud images don't have this issue (a new machine-id is generated from each clone of the cloud image), but duplicating a VM from an existing clone will not regenerate machine-id and Ubuntu (but not Debian) uses machine-id as the DHCP client ID and potentially other minor things.
@@apalrdsadventures so how do we avoid it ? or fix it? i want to use templates with some packages i need in them , how do i make sure they are clean like the cloud init images themselves?
Not working.
Debian 12 boot loops.
Ubuntu22 uses VNC, no copy paste, no guest agent.
Futhermore, logging in with proxmox console doesnt login without password despite creating a fresh key ssh-keygen -b 4096
within /root, and updating script to point to /root/.ssh/..
All in all, didnt enjoy deleting the 8 broken templates, with missing SCSI mounts and failing resize warnings during execution. But if someone else appreciates this rutine, go right ahead.
It sounds like your setup has issues unrelated to the templates. The Proxmox console shows the VM's 'physical' screen, not an SSH session, so of course it won't login with ssh keys on a serial or vga terminal.
This is fine and good, but i use Packer for my templates. This way i can automate templates, create different ones and images with software already installed.
Than i use terraform for actual vm deployments. This way it is configured in code and i can recreate them as i change proxmox hosts
If you start with a distro image you don't need to keep your own templates at all, you start from the bare cloud distribution + cloud-init configuration for the initial network and user for terraform, then terraform can take over and do the configuration.
@@apalrdsadventureshey, thanks for your good video!
i tried the ubuntu cloud image, but it is just an ext4 partition mounted on /
i then downloaded a full iso, set up a ubuntu installation with a few lvm LVs, so that I have different partitions for /home /var /var/lib,...
i guess i would have done this with Packer too - but do you have other ideas to do a specific partitioning?
In general there's no reason to create LVs on a virtual disk for a server VM. You let the hypervisor manage storage.
@@apalrdsadventures i don't understand... when someone spams /home, because he saved lots of ISOs there, the whole server is in a critical state, when / is at 100% storage. better use a partition for /home
Also, what would be the difference between the cloud init images here vs. the LXC templates that you can download directly from within the Proxmox GUI?
These run as VMs, not containers. So you have a full OS with a full kernel and modules and everything.
@@apalrdsadventures
Yeah...I'm still a little fuzzy about what the difference is because I have used the LXC containers, for example, as a Steam cache server before and it seems to perform that task/role just fine.
It seems like that LXC is a little bit more "heavier weight" than a Docker container (where it will ONLY run the application and nothing else).
But with the LXC container, I was able to run different applications with it, so I am not entirely sure what the difference between LXC vs. a "full weight" VM would be.
Thank you for answering my dumb questions. (I've tried googling this before and it's still about as clear as mud).
LXC containers use the same kernel primitives used by Docker to run a complete Linux system in a container. So, you share the kernel with the host, but have your own network addresses, services, user ids, ...
VMs are emulated at a more bare level, so they are running a full x86-64 kernel with drivers and modules of their own, on an emulated x86-64 system with emulated hardware. This is a more universal approach, works with all x86-64 operating systems (as it can emulate real hardware like network cards and graphics cards), and is generally used when you need the functionality of the full operating system and kernel. So, you have the option of running Windows, BSD, or Linux versions based on appliance installs or using different kernel versions. They have the flexibility to live migrate in clusters as well.
LXCs have limits in what they can do even though they are Linux-based, you cannot load your own kernel modules or configure anything in the kernel (so no Wireguard for example).
@@apalrdsadventures
Ahhh...okay...got it.
Yeah, I am going to have to do more research in terms of what things that I use "needs" "configuration in the kernel" vs. "it just runs".
(I didn't know that wireguard needed configuring the kernel, but that would make sense though.)
@@apalrdsadventures
Sidebar:
Would you mind testing out the CentOS 7 and/or CentOS 8 LXC container templates from the Proxmox templates download?
I tried them last night and CentOS 7 said that the cgroup v2 is too new for it.
And then CentOS 8 - I couldn't get it to run `yum update`.
Thanks.