The FASTEST Way to run Kubernetes at Home - k3s Ansible Automation - Kubernetes in your HomeLab
Вставка
- Опубліковано 19 чер 2024
- Setting up kubernetes is hard. That's why we made it easy. Today we'll set up a High Availability K3s cluster using etcd, MetalLB, kube-vip, and Ansible. We'll automate the entire process giving you an easy, repeatable way to create a k3s cluster that you can run at home in a few minutes.
This is a great way to set up Kubernetes in your HomeLab.
Video Notes: technotim.live/posts/k3s-etcd...
Support me on Patreon: / technotim
Sponsor me on GitHub: github.com/sponsors/timothyst...
Subscribe on Twitch: / technotim
Become a UA-cam member: / @technotim
Merch Shop 🛍️: l.technotim.live/shop
Gear Recommendations: l.technotim.live/gear
Get Help in Our Discord Community: l.technotim.live/discord
Tinkers channel: / @technotimtinkers
A HUGE THANKS to our sponsor, Micro Center!
New Customers Exclusive - Get a Free 240gb SSD at Micro Center: micro.center/1043bc
You can find the repo here
github.com/techno-tim/k3s-ans...
Videos mentioned:
Ansible Setup Video
• Automate EVERYTHING wi...
Proxmox Cloud Image Video
• Perfect Proxmox Templa...
Thank you to Jeff Geerling for the video clip and his work on Ansible + K3s!
• Raspberry Pi Cluster E...
Thank you for forking and adding kube-vip support to the k3s Ansible Playbook!
github.com/212850a/k3s-ansible
This will also be included in the documentation
l.technotim.live/docs
(Affiliate links may be included in this description. I may receive a small commission at no cost to you.)
00:00 - Intro
00:58 - Micro Center Advertisement
01:49 - How Did I get here?
03:28 - What did I automate with Ansible and k3s?
04:23 - k3s Load Balancers and Architecture
05:58 - What is the architecture
06:06 - Creating machines
06:36 - Configuring Ansible Playbook
11:03 - Setting Hosts
11:34 - Running the automation
12:31 - Verifying the k3s install
13:31 - Deploying a workload to k3s
13:53 - Deploying a Service + Load Balancer
15:25 - Testing kube-vip and k3s control plane
16:31 - Tearing down k3s
17:15 - What problems did we solve?
18:09 - Stream Highlight - "If it's not DNS, it's a firewall rule"
#k3s #Ansible #OpenSource
"Mime Dance" is from Harris Heller's album "Arcade"
"Wrong Way" is from Harris Heller's album "DANSU"
"Sunday Pancakes" is from Harris Heller's album "Low-Key"
l.technotim.live/sb-music-lic...
Thank you for watching! - Наука та технологія
This setup is pure gold. I can't thank you enough, within a day I've understood how MetalLB is the LB alternative for self-hosted / bare-metal kubernetes deployment and this playbook has saved me many many expensive hours which would have been needed to get my test lab up. Can't thank you enough!
Thank you!
I second this! This made me join as a member.
Woah... I really gotta go through this video and apply it to my setup, was totally trying to do this the other day. Ended up without it being automated. Elegant and beautiful solution man
Inconceivable!! This worked really well right out of the box as promised. I had a k3s 3-node Raspberry Pi cluster up and running in minutes - and I love the Ansible add in. I was vaguely familiar with Ansible from a introduction about a year ago, but this took my understanding to a whole new level. Thank you very much!
Thanks Tim! Got a technical interview on Tuesday and you've just helped me prep for it!
So so awesome, just tried this out and works so well. Thanks for the supporting documentation as well!
Wow I set up this cluster having almost no idea what to do with it.
After setting up the cluster I relied on various blogs to get services running. I'm now at a point where I've set up services using only docker documentation, docker-compose files, and Kompose.
My latest project has been delving into using BGP on metallb to be able to direct traffic from certain pods to my VPN.
Thank you so much Tim!!!
I absolutely love you. I was following all your old stuff and just beating my head into my desk trying to get it all to work right for my situation. Thank you so much for this. This needs to be the first result when you search UA-cam for k3s setup, for real.
Thank you!
Thanks! This will help spin my raspberry pi clusters. Not having to use the external load balancer for kubernetes is awesome! Thanks again.
As someone whom has been working on deploying an OVA (including application setup after the vm deploys) with ansible... i can appreciate how much work you've put into this.
Thank you! I am standing on the shoulders of giants who have built most of this out before me!
Hey Tim. I did see this some time ago and tried to run it but had some issue. Don't remember what it was now so went on the back burner. Just updated the repo and run and works perfect. Lots of thanks for this.
I just got my environment setup with ansible and docker compose files and then I run across this....lol. I need a break before I embark on this journey and just enjoy the homelab. Sometimes I think I enjoy tourting myself.
I love that you OSS guys are using each other’s work and shouting each other out. Stream on!
Really killing it content wise... Your videos have been so helpful lately.
Thank you!
Great talk dude. I am exactly on this. I want provide a full secured k3s cluster for airgapped environments ( for industiral production for example ) .
The final setup should like this:
1. Private registry with SSL setup
2. Provide docker on a builder node for remote builds
3. Sec Stack
3.1 Cert manager
3.2 Keycloak as ODIC provider
4. Monitoring Stack
4.1 Grafana
4.2 Loki
Currently my repo is private beacaue it is in dev. After my first relase will share with you. Maybe can do more together ;-)
Hello from czech republic, thank you for your work, I am so glad that i have found your channel.It's very helpful.
Dobry den :^)
Great job like always TIM :) thx for all you hard work!
Amazing vid as always Tim! I did the same setup but with a debian cloud image instead and it works great
Can you please share more? Did you set it up remotely?
WOW, you make really good content, detail and well explained, thanks.
Awesome Tim!, Thanks for automated the entire process
Glad it was helpful!
Hi Tim thanks for sharing definitely very helpful and great work there! Just forked your repo as I need my CNI to be Calico instead of Flannel. Thanks a lot!
This is fantastic! Thank you so much!!!👍
Thanks for the video. I started the journey with kubernetes for my Homelab thanks to your videos.
I ended to the same results. Having it automated. I use the pretty good template from k8s-at-home. They have All setup including sops and flux.
Super awesom, thanks Tim. 🖖
This is a super helpful video, thank you for putting this together! It would be great to have a followup where you add in some applications or perhaps even a container of one of your own apps. Thank you for all these great and helpful videos!
Great suggestion!
I agree with OP, full prod WordPress / email server would be great...
This is an awesome video. I have understood how HA and K3S works, but never understood how you could access the webserver from a single IP. Keep up the awesome work!
Thank you!
Keep up man your channel is just excellent
A masterpiece!! Thx for sharing
Hardware Haven sent me to a real expert 👍
It just worked from the first time! complete awesomeness!!
That's awesome news!
Mate, thanks for this, now I need to go figure out how you wrote this playbook so I can understand how it all operates and how k3s works. I plan to migrate my entire docker-compose stack to HA k3s and this is perfect. Thanks again!
No problem 👍
Amazing as always
My issue was the wrong lan network! Mine is a 10. Network not a 192. The three masters would work and join but it'd hang on joining the agents. Changed everything including vip and flannel ip ranges to my lan and it worked like a charm! Also I was using -K for become password but I tried it without that and it worked. Hope this helps anyone out who may have the same problem. Thanks for the work to get this going!!!
did you make k3s api accessible from the internet?
Yay, This video helped me learning ansible, it feel really good to make everything automated XD
Thank you!
The k3s cluster runs very smooth, thanks for your effort Tim. ik only can't install rancher on it. Everthing else works great. But Reancher does the hole cluster crache. the expose of metalLB ip to rancher Works. but the installation never ends before the cluster crashes. I looked different video how to instal rancher on a k3s cluster but it never works
Just want to provide a testament to how good and useful Tim's work is. I messed my K3s cluster pretty badly so decided to reset from scratch. 15 minutes and 2 commands ("ansible-playbook reset.yml -i inventory/my-cluster/hosts.ini" and then "ansible-playbook site.yml -i inventory/my-cluster/hosts.ini") and I have a fresh start. Another 15 minutes of "kubectl apply -f" to reinstate my deployment yamls and everything is back to its original working state.
Thanks a lot mate!
👍
Glad you liked it! Thank you!
thanks you for this video ! 😊
Great job Tim... would be great the following follow ups:
Rancher install on this cluster and how about some longhorn? Cheers
This truly is pure gold. The only thing i would add to this is to also have FORKs for different hypervisors. Ansible is very friendly with all hypervisors and can create the K3s VMs automagically.
Thank you! This is hypervisor agnostic and even works with bare metal!
Tim, great work here. I'd really like to show you a similar way I did this with custom k3os images, proxmox and terraform.
I find this interesting. I have been working on a combination of Terraform and Ansible to spin up a k3s cluster on the Oracle Cloud Free Tier. Would love to see your ideas on how to get this working.
Me too. I'm working with a weird cluster of raspberries, x32 laptops, and an x64 mini-pc, with proxmox, K3s, terraform, cloud-init, and ansible.
And I'm interested on your project too, could you leave us a link to your project? (github, blog, even a google doc could be good)
Thanks!
Thank you! Sounds awesome. I’ve made this to be a building block that can fit into any infra automation ☺️
@@chrisa.1740 okie, i'll make a video outlining how i did it :)
@@zoejs7042 Sounds good
Great video sir!
Welp, you've done it now, Tim. Great job!
DUDE! I have not watched this video... But this seems exactly like a video I have needed for ever.... 4-ev-er.
Your my boy blue!
👏👏👏 it'd have been amazing if you could make a video on how we can run Hadoop and pyspark on top this kube cluster to have some data transformation at home 🤩
thansk so much for doing this, i started working on this exact problem like a year ago but had to shelve it because i didn't have time anymore :( thanks so much!!!!!!!
Thank you! Happy to help!
So quick! Only spent a week to get it to work in a few minutes 😂😅😂
Please add Rancher to this
Great idea
This! I wonder how this works in Rancher. Your preconfigured nodes, were they created in rancher or by hand?
Love your work. Just managed to set all this up, but I'm still clueless about how to use it. Would be amazing if you could do a video on how you're using and deploying your stuff on this cluster. TA
Thank you! I have tons of videos on kubernetes apps
Automated K0s and RKE2 ansible deployments the other day with some pretty barebones playbooks. It's kinda fun trying to automate and architecture everything.
Want to get to a point that I can just setup a new installation of Proxmox using ansible, and have it create VMs and a cluster (or join existing).
O.M.G this was amazing, in a single night. I setup a Ubuntu Server cloud init template in ProxMox, built 9 VMs (3 masters, 6 workers) and ran though this video to get a fully HA k3s Kubernetes cluster installed.
The best part, I am a freaking n00b at all of this. Such a great teacher, love your work and I am looking forward to consuming more of your content.
Nice work!! Thank you!
dude!!
I'm using Rocky Linux and this installed like an absolute dream.
I'm ready to stop having all these pets in my homelab. Time for the cattle.
Thank you
I use that helm installer CRD that k3s offers and just have Ansible drop a yaml file in the respective directory to install kube-vip, personally. This approach of yours is equally valid but one lets me use the stock upstream module which is nice. It also lets me install my CD of choice (Argo in my case, saw you had a Flux guide too), and I just drop everything else to install including MetalLB into an set of app-of-apps Argo CD Applications. I find I prefer k3s only handling absolute minimum to make the control plane HA to be the easiest strategy for me and then let my CD system take it the rest of the way.
Great work! Have you looked into gitops with flux or argocd? I find it quite useful to simply push to git and have the cluster pick up the manifests and deploy them automatically. The first thing I do after installing a vanilla K3S install is to connect the cluster to a git repo (using flux) and send all my manifests by pushing to Git. The cluster automatically configured itself, including MetalLB. That makes it really easy to tear down a cluster and build it up again.
Yes, I have! I've done a video on flux! ua-cam.com/video/PFLimPh5-wo/v-deo.html
Thank you so much for your assistance in setting up k3s using Ansible. Could you possibly create an updated video on how to install Rancher along with Traefik + cert-manager? Additionally, could you demonstrate how to use this k3s cluster with a GitLab CI/CD pipeline? It would be of great help.
Awesome Tim -- just setup on 4 raspberry pi CM4s using Deskpi (need 2 more pis to fill!!!). I also have some NVME storage (overkill, I know), so I put k3s & containers on the storage instead of the emmc for speed/safety. I plan on watching your video on Longhorn to do that next! Huge Thanks!
at the end, you had a reboot script -- I couldn't find that - and checked, i don't see an obvious project in your github repos where it may be. I wanted to use that as a basis for a reboot script. Thanks!
sad. replying to myself. :( I found it in launchpad! Somewhat anti-climactic in that it was / looks sooo simple. I think setting serial to 1 should work. :) testing time :)
I have 8 old Mac minis that I have been working on to make into a K3s cluster using rancher, Ubuntu 20.04, and a ton of trial and error. I am just about to spin this up, but should I scratch that and go with Ansible? Dang, as soon as you think you have a grip on something, someone awesome like TechnoTim comes along and throws a new solution right at you. Thanks for all the great videos!
Thanks! Haha! This is automating what you would otherwise copy and paste from docs and adds load balancers so you don't have to. :)
Great video as usual Tim. "I love open source". It would be great if you add LongHorn support at that script. Additionally, it would be great if you can do a video on how to migrate a docker install that already have some data on a local volume to kubernetes...
Thanks man for your videos!
Thank you! I’ve considered adding longhorn and rancher to the script but many may not need it. My other videos shows how to install these with a few commands! Will consider it in the future!
There are a few assumptions in this video that a beginner will bang their head against the wall for days. I finally got this all working. This is awesome.
Nice work!
Out of curiosity, what were they?
@@TechnoTim
If open for discord DM's ill send you a friend request. I have some general feedback I think could be used to help your channel/others troubleshoot. Else I can comment here, let me know!
Is there a follow-up video for rancher and how to install k3 apps? being that this is HA/Vip Etc it would be good to have a video on how to utilize all of this to deploy trafik, maybe pihole/adgaurd HA, etc. I know you have some topics on this already, but I don't know if they still apply with this setup. If they do can you link to the next proper video for rancher, HA adgaurd/pi-hole, etc...
Hi, first of all thanks a lot for such a great tutorial! Can you please elaborate why the netaddr dependency is needed? where exactly is it used?
Thanks!
You bet!
Can't wait to watch this one!!
Saved...will watch and probably have a go at this coming Monday
Thx Tim!! ✌️
Really appreciate this video. I definitely need to research your blogs and understand them, I know what I want but the order of execution eludes me. I've got a HA SQL cluster already (so want to use that instead of etcd), I do want Longhorn, and Rancher and Traefik 2.... if I'm right I can just add the datastore param to the global_vars and it should use that SQL db, but how do I stop it installing etcd? And I'd assuming the best order of events would be the ansible playbook, then longhorn, rancher and traefik 2 (in that order)... as for cert-manager.... I guess between longhorn/rancher?
Watching again 😊 any chance of one with rancher/longhorn added?
Although this setup was great and worked really fast, I don't think I want to use it right now. The reason is that I won't learn anything from it. So if something goes wrong or I would like to tweak something, I would not have any clue how to do it. For example I noticed in the settings that there was an entry for setting up bgp and ASN, since I use pfSense I figure that would be a great way of getting the routing for the K3s cluster working. But no matter what I tried I could not get it working. But it has inspired me to actually start learning Kubernetes and start building my first k3s cluster from scratch.
It would actually be great if you would break down in a more in depth video or series the different parts that you used to get this running and options one could use to tweak things. Because it is really cool how fast it goes to get the cluster up and running.
Gear video! Thanks!
About deploying Traefik. Can you explain or point to an article that show how to set the traefik to work with k3s?
Thanks for the vid and appreciate you publishing your repo! :) Very helpful and I was able to use them along with k3s-ansible upstream, Lempa's vid, and the k3s docs to pull it all apart, figure it out, and get my own k3s setup codified. However, I skipped all the Metal LB as I found it trivial to get kube-vip to work as the load balancer for both the control plane and services. Curious as to what you got caught up on?
Would you mind sharing your setup that uses only KubeVIP? I would like to compare that with this, on my way to modifying the Ansible playbook to use Traefik for these tasks if possible.
Thanks Tim! I was able to get my cluster up and running easily with this. Question, I installed Rancher and now need to access the UI. How can I config the nginx ingress to route to the Rancher UI?
Hi!! Thank you so much!! This is just a work of art. Dude: how could I add a new node without removing and restarting the existing service?
Hey Tim. I've been looking for a while now. Are there any load balancers for k3s other than klipper(svclb) that work over a layer3 VPN like wireguard?
I've tried getting both metallb and kube-router working, but they won't route between nodes that are only accessible across the VPN. Probably because there's no layer2 connection between the nodes, only layer3.
I'd love if there was some way to get metallb or some other lb working that can assign VPN-internal addresses to services.
I'm working around it now with klipperlb and just using different ports and network policies for my internal services but it's not optimal.
Wow! Another solid content! Thank you very much.
Is it possible to use rke2 instead of k3s?
Not with this playbook, but one might exist
Hi Tim, is there an easy way to add more master nodes with etcd later on? Thank so much.
I gotta take some time to debug this to my use case, as the kibe-vip LB is not working o my odroid-c2 (armbian arm64) cluster. But thanks for the hard work to put it all together
12:50 "Super, super awesome" Tim's smile says it all. 😁👍Well done just downloaded to try out now. Thanks Tim
Thank you so much! I am glad you liked it. Let me know how it works out!
Now wouldn't it be nice to have hypervisor support, like Proxmox?
Spin up VMs on one or more hosts, retrieve the IPs for ansible, then deploy k3s on them automagically...
Down the rabbit hole we go!
What do you think about talos kubernetes ? thanks
First of all thank you for the video and the code. I have one question, i was trying to set ha kubernetes until i saw this video. I was doing this setup with some VPS. My question is if i have to create a net between them to run this ansible playbook. That was what i do configurating ha kubernetes. In the other hand, ther is a way to do it without a VPN? What VPN would you recommend to me, i was using wireguard with 2 servers (one master) and other the nodes as clients. This because i need to create a virtual ip and that i think that is only posible with a /24 net. Thanks for reading, sory if you dont understand my problem at all.
It would be great ifyou could begin tesing these setups in a real world environment. For instance, putting each VM on separate physical proxmox nodes. Then testing both performance and data integrity of the HA mysql/mariadb, and maybe some kind of php web application - and start powering off VM's mid use (while writing and reading lots of data). Any load balancer can show a static nginx page - but when you have fast changing application data tied to user sessions (you'll want redis for this) all of a sudden you realise these magic clusters aren't as magic as they make out. Also sql databses have clustering capabilities baked in (sharding) - and the sql agent itself is aware and in control of ensuring the integrity of the cluster. How do containers in a cluster ensure the database integrity? I personally love the idea of a self scaling, self healing auto magical high performance container cluster - but all you ever see are demo examples that developers show off and then destroy. I guess what I'm getting at is often the applications themselves need to be written to be cluster aware/compatible, and the architecture needs to be manually configured more often than not to make this stuff work - you can't generally just spin up 10 containers with a stock OSS container application image and have it 'just work'
100% agree and this topic comes up a lot, RE scaling. I discussed it on stream today too. Applications need to be written and architected with scaling in mind, most of them too stateful to run more than one. I'd love to dive deeper into more topics like this in the future if there's appetite for it!
@@TechnoTim I would love to see you cover something like this
The repo works with raspberry pis (I just test my cluster up!), so you could in theory just like 5 pis and just unplug them from the network stack whenever while the performance tests are running
can you add zerotier or eireguard clustering? also how about metallb external address detection internal address detection for nodes?
finally would it be possible to have pods run in control plane?
I want halb but don't want to lose workernodes
I’m thinking on using this model but for 2 mini servers
Helpful video Tim. Thank you so much. I have a question : Is this prod ready or for testing only ? Should I adjust some params for prod deployment ?
it's prod ready!
@@TechnoTim thanks
Awesome, JUST what I was looking for, perfect timing!. Since the playbook works with CentOS, it should also work with Rocky Linux right?
Perfect! It should but please let me know!
@christian braun I used almalinux. Half the tasks it didn't run(blue/didn't match the centos or redhat facts) but the only one that mattered that I needed to force was the secure pathstuff for sudo.
Worked great on alma.. I imagine your experience with rocky would be identical. But honestly I think I'll be making my own playbook that's half the size and use defacto kubeadm instead.
you are just awesome!
Wow, thank you!
Thank you so much, As a beginner, I was expecting to see how you add Traefik and configure it to proxy requests to the example service you had.
Thank you! I have docs on Traefik, I might break it out soon into its own playbook because not everyone will use Traefik!
@@TechnoTim Second this. Looking forward for Traefik install with Helm guide
This is great content but could you tell me how could i create a k3s cluster with Cilium cni using this setup instead of flannel ?
Truly awesome, as always Tim!
Im wondering, 5 nodes seems to be a bit much for my server, CPU spikes. If I want to try this with only 3 servers, is it enough to remove nodes from the hosts.ini? Or is there taints on the servers so they cant act as workers? Had a look, and I think it should work, but confirmation wouldn't hurt :)
Trying to deploy rancher right now, but it seems to be running into some issues. Takes forever to deploy, thought it might be because I removed the nodes..
Thanks! All you need to do is update the host.ini and put them under the right header!
@@TechnoTim yeah got it to work by removing all hosts from the node header. But k3s is really killing my CPU man.
Im running a Xeon E3-1226 v3, only 4 cores, but still. If I add 3 server nodes, with 2 cores each, it really struggles. Sure thats over provisioning, but with only Rancher as load, I thought I would work.
I dont see how IoT devices would survive running k3s and a workload. What am I missing? :)
You are awesome 😎
Hardware heaven sent me
I keep getting an error for my masters when attempting to provision the cluster. Something about 'no usable temporary directory found in /tmp, /var/tmp, /usr/tmp or my home/user directory'. The directories exist, not sure what it means they're not usable. I tried pasting the full error here but my post keeps getting deleted.
Any idea what might be causing this and how to resolve it?
It s possible to run longhorn in lxc container ?
I managed to install it with SSH, what an absolute head f*ck that was. after 5 complete vm and cloud-int removals, I finally got it up and running.
I have no expertise in using any of these programs, and my networking knowledge is little to none. Basic networking setups.
I must say if you could provide maybe a section for ssh setup in the tutorial paper that would be great for others with no expertise.
So thank you very much im having great pleasure making your face do weird pause poses as i get through this video :)
Hi Tim, I've discovered and been following your channel since a year and basically watched all your videos. So well explained every time!
If I were to try your Ansible script to test things out at a small scale in a first time, would the script work if I were to put the same few IP addresses both as Masters and as Workers?
(I know it's not best).
Also, one thing I always notice in your video is how many IP addresses you have, more precisely all the different subnets you use. It would be very useful to get a video on the segmentation logic you use. Because in the case of deploying this script, I really don't have a clue on which IP (and ranges) to use so that it does not interfere with other devices, VMs, services, etc. and so that I don't have to redo the whole deployment in the future.
Thank you.
Thank you! Yes, you can have all nodes use all roles! About networks, yes! Coming soon!
@@TechnoTim Thank you for your answer! Can't wait to watch it! Continue the good work!
Love the video Tim. I'm struggling to get this up and running on 5 ubuntu 22.04 machines. I've noticed that the args in your video don't match what's in your repo. Any reason for the change or are the original args listed anywhere? Wasn't sure if I should open an issue on GH or not.
The default args in the repo should work! If you want to see what was used in the video just look at the first few commits, however the latest repo is what you want!
@@TechnoTim Thanks! I think I found my issue. I'm running ansible from an RPI 4, which was running Python 3.7 and ansible core 2.11.5. I think that 2.11.5 is not new enough for split to work. I'm upgrading and hopefully that will fix it!
How do I set this up with a FQDN in place of ip address for joining a cluster off network?
For work I use Rancher + ansible for a very similar approach as yourself. As I've replicated my work environment (modified) in my homelab, I'm having trouble understanding the benefits to this over Rancher?
This is just a plain k3s cluster without any management. It's not an alternative to Rancher, it's a Kubernetes cluster than anything can run on, even Rancher itself since it needs a kubernetes cluster to run on. It's more foundational.
Amazing work, Tim! I was looking so long for a fully automated solution.
Have you found a way to overcome the k3s certificate expiration after 1 year? I know that restarting the k3s service, 3 months before the expiration, will automatically renew it, but doing it on an HA cluster...
Thanks! RE the expiring certs, don't you ever patch and reboot any of these nodes? It should auto renew if the certs are within 90 days the next time k3s starts (I thought)
@@TechnoTim LOL. I was thinking the same thing. My next plan is to make a safe patching script that walks through the nodes 1 by 1 and makes sure k3s is back alive before moving to the next one...
By the way, Tim, could you show us some provisioning from scratch. Some bare-metal way to install it all with ansible (and maybe terraform, cloud-init, or something like that).?
Your videos have inspired me to start my own cluster of servers (I have low end hardware, but a bunch of it, so I'm trying to make it all work together).
Thanks!
There is a nice Proxmox Terraform plug in available to provision LXC or VM. Similar to cloud init but using TF. It is not perfect but a nice way to learn TF without having to spend money on cloud provider. I should reach out to him.
@@Irish2086 Yes, thanks, I saw those before, but... so far, for what I understood (and what I need):
- Cloud-init plugin is only a "read" plugin. Meaning, I can read the file, store values read from that file into vars, then use them with other plugins.
- The proxmox one, is great for the mini-PC that has proxmox. ☺️
- But for the raspberries, I think is best, not to use any type of VMs (proxmox or ESXI). I know there is a TF plugin for K8s and for docker, but at that point, I think I rather set it up with Ansible.
@@Irish2086 This is my "set up" path right now. I want to provision all the machines from scratch. But PXE is an awful, unsecure, and obsolete technology. So:
1) Copy Linux image to the SD card.
2) Copy, manually, Cloud-init to the card.
3) Let the Raspberry, boot, install, and apply all the settings on the Cloud-init file (usually some basic accounts, IPs, Hostname, SSH, Firewall, and install Terraform).
4) Download from a URL Terraform tf files, and apply. (These would finish setting the device, with secrets, better security (key-pairs, etc.), Install Ansible.
5) Setup everything else with Ansible.(K8s, images, Jenkins).
6) any future changes on the system, would be applied by Ansible (if it is setting the "apps"), Jenkins (if it is setting the contents of the apps (a website, microservice, webapps, Dbs, etc), TF (if it is the system itself).
And that's as far as I have gotten so far! 🤷🏻♂️
(With the concepts, but I haven't applied them yet, and still 'designing' the whole system in Terraform/Ansible files.
You can definitely combine them! This is just one building block, a LEGO if you will!
Hey Tim, awesome guide and repo. This helped expand my Ansible knowledge and produce a useful outcome of a k3s cluster with which I can mess around. Question for you, when I deploy Rancher using Helm (following another guide of yours, thanks) it doesn't seem to be accessible externally. Is this a MetalLB and Rancher conflict? Any guides I can look to that would help me resolve this? Thanks!
Hi. Thanks! It shouldn’t be as long as you disabled the service load balancer using the arg
Did you happen to find a way to expose rancher? I am currently trying to figure this out as well
@Tristen I was able to get it exposed using a MetalLB address by running "kubectl expose deployment rancher -n cattle-system --type=LoadBalancer --name=rancher-lb --port=443" that should work for you too assuming you've already got rancher installed and it's running in the cattle-system namespace. I found this on another guide of Tim's on installing Rancher.
@@motu1337 I think I just cam across the guide you used! lol
Thank you sir for the help! That worked!
@@motu1337 is it work ? without traefik
Thanks a lot for sharing, Tim! One question, Is it considered HA when all of this is done one just one Proxmox server? Or is it only HA when also Proxmox has different nodes?
It depends on the point of view, the cluster itself is ha but obviously it's not fault tolerant to a Proxmox node failure. He should've at least run the control plane distributed between his 3 nodes. :P
And it is soo much nicer to do k8s node updates without downtime. So I think it definitely is worth it
@@LampJustin what is worth it exactly? K8s or running this k3s HA install?
Agreed! With Modzilla. This is an HA k3s cluster. If you want HA through and through your server nodes would have to be spread across hosts.
Thanks for the Videos. I have a 5 node clustered server rack and will be trying harvester out on it. But in order to utilize it do I need to set up K3S on a RPI then deploy workload loads to the harvester cluster?
Set up your clusters first then create vms, then use this ansible playbook to target all machines. Or create the k3s cluster in Harvester and join your Pis to the cluster manually
@@TechnoTim Thank you for the help cant wait to try it out
Why did you go with metalLB over klipper, which comes with k3s? Was it just preference, or were there specific reasons you chose to deploy your own load balancer?
I prefer MetalLB over klipper. I've used it many times outside of k3s so it's familiar and battle tested.
Tim, I have a question or two. First, your 1U SuperMicro servers aren't available anymore. Can you recommend a similar server (1U 4-bay)? I'll need to populate it with 4 4-8TB drives. Secondly, and this is crucial, using Proxmox (which I love), how would one install it on an SSD and have 2 additional HDD's for all the VM's? There is no video out there that I have found that goes over this. Please consider doing one. Bonus question, how would one having the scenario above (SSD+HDDs) also backup not only the Proxmox drive but all VM data? Thanks.
I have many recommendations on my kit site. kit.co/TechnoTim