This homelab setup is my favorite one yet.
Вставка
- Опубліковано 23 сер 2024
- Hosting my own services using a homelab has been an absolute dream. However, my first homelab setup had some mistakes, and so I decided to rebuild it from scratch, and share with you how I did so.
Links to my hardware:
Beelink EQ12: amzn.to/3zK4Pbu
32GB Memory: amzn.to/3Lkotgy
2TB SSD: amzn.to/4eV117n
Ubiquiti Switch: amzn.to/3W5sJWp
Desk Whiteboard: amzn.to/3LogIGw
Keyboard: ZSA Voyager
The above are Amazon Affiliate Links which means I get a commission if you decide to make a purchase through them. This comes at no additional cost to you and helps to support the channel.
Other Links:
Github Repo: github.com/dre...
NixOS Installer: nixos.org/down...
NixOS Anywhere: github.com/nix...
Discord Community: / discord
i did not understand anything past nixos but i still watched until the end
`!!`
literally me haha
same
Yup
I hear I'm supposed to like NixOS but I still don't understand it
We are really interested in a crash course about homelab computing. How is a cluster useful? What is the purpose of a DNS? Load balancer? Namespace? All of those questions are a search away, but a "getting started" video would blow up for sure.
I shall add this to my backlog!
Second this, I work in devops so I have some understanding of what those things are (I'm still a junior) and how they are useful in a company. But for personal use? I can envision a usecase for hosting your own movies and maybe website.
tbh, very limited you can learn from video like this, it's like a showcase of how he setup his home lab, to achieve that you need to learn lot of knowledge to has foundation to start to do same setup. From your question, can see that you're lack of very basic knownledge on computer networking, and it's more nightmare if you put your feet on thing like IaC as use Helm, complex system like Kubernetes.
@@ChaiRuou That's why, a getting started video could cover the standard concepts without using advanced tools
@@Otakutaru try look at devops road map 🙂
It's funny this video found me. I just purchased 3 of those Mini-Pc's 3 days ago, and now this is sitting in my video feed. I guess the algorithms are on the job.
You got the beelink? Which model did you get?
Devops engineer here. Ubuntu server + Ansible is how I roll, both for homelab and at work.
I'm envious! I've never been able to get into the mindset for ansible. Now I've got a spare cluster (my old one) I may find some time to give it a go
@@dreamsofautonomy To be honest, even though it is simpler and easier to use, I feel like Ansible is a downgrade from NixOS, in regards to the control you have over the state of the machine. Nix allows you to rollback and specify the exact state of the machine, while Ansible only allows you to declare what you want to be performed on the machine, it doesn't give you any assurances that it is the state you wished for.
@@obvionaoe Aren't these different tools? I am not sure you can achieve the same ideas ansible can bring to a bunch of a machines while having good compatibility.
But please explain if it's possible, I am intrigued to know.
@@areweevensomeoneoutthere the benefits of nixos being a declarative system, rather than a declarative tool used with a normal system, are the full system reproducibility (if you set it up right) and freedom from dependency hell (every program and library is installed to its own directory, allowing multiple installs of the same program/library)
ansible is a great tool for reproducibility, but nixos can replicate all of ansibles functionality and a bit more
@obvionaoe nix is great for the initial set up phase but ansible is good for controlling multiple servers
Budget tip #574: Buy 8GB version of Beelink EQ12 + 32GB RAM stick. You can save some money especially when buying more nodes. Reselling the 16GB stick is not easy (and not worth it).
I have 6 EQ12s in a cluster, fantastic little machines to play with!
the main question i have is, what do you use it for? hosting websites? hosting game servers?
unless you are renting out these in form of k8 compute, its really hard to justify the cost.
@@DogeMultiverse Three control-plane nodes, three compute-nodes? It's always good to have options to try out different things.
Do I need a justification for a hobby?
I use them to test different technologies: hypervisors, hyperconverged storage, high availability.
Currently running vSAN cluster, purely for learning purposes. When I’m done, I’ll tear it down and build something else.
@@TheSoberPirate That's the same reasoning I used to justify a 4U storage-server with 160TiB usable storage and dual 10G fiber.
@@SnorreSelmer hahahah
neat to see some examples of k8s based homelabbing/self hosting!
I did learn nothing from this video - too complex 😅 but it did spark interest for trying kubernetes stuff myself
I'll definitely go through concepts in more detail! This one was much for of a devlog/homelab log. I'm glad it inspired some interest though! Let me know if there's anything you'd like me to dive in more about and I shall do so :)
@@dreamsofautonomy what are your thoughts on k3s vs Proxmox?
Relieved to see that I am not the only one.
@@dreamsofautonomy I thought longhorn was the Win Vista codename, but here it is what exactly?
what makes it a bit "complex" is the fact that he is using kubernetes heavily, but in the end its all the same, DNS is DNS, TCP/IP and networking are the same, its just kubernetes introduces some new concepts to people not familliar but its not hard to learn and understand..
i would recommend some videos on kuberentes and helm on youtube then return to this video.
Great in-depth and detailed explanation for setting up a cluster from scratch, especially nice that in the end you explained why you needed two additional charts to get the final pieces of the puzzle working. Thanks!
Nice to see more NixOS around! I am using colmena to keep a config of my nodes, so I don't need to SSH into them to apply changes to my NixOS nodes.
Keep it coming!
Great video, just note. You should never writte a secret in a nix things even if you don't use git. Almost everything you put in secret will end up in the nix stores so beware of leaving behind secret there
so where do you put it?
@@John-x3l7m I can't explain here the rabbit hole of nix encryption but basically you need to store in the nix store already encrypted secrets files and in your NixOS configuration a module that can at the start of your server decipher your encrypted secrets and expose them in some runtime temporary file only specific program can have access.
Personally I use nix sops and sops to do that but like I said it's a rabbit hole and there more tools for the same jobs but here the ideas.
Last week in office cleanup I was asking for 2-3 cpus and two guys from leadership gave me their own old cpus.
I was thinking about setting a kubernetes cluster. Lo and behold your video comes now I know what to do this weekend.
How did it go?
The rest of the video was good enough to excuse not saying "i-skuzzy". Took me a second to realize that MetalLB was required for DHCP since its layer 2 and would not be routed in or out of the cluster without an outside relay.
I still need to learn more about how Kubernetes works, but I really appreciate your approach of setting up part of it, encountering an error, then fixing that error. Moving along in small steps, showing the error and how to fix it, I find really helps me to understand how the pieces fit together, rather than just giving a config file that works for you and leaving it up to me to figure out what I need to change for my environment.
this is so awesome and exactly the kind of video i was hoping for from you. i'm here for all the homelab content all day.
never heard of helmfile until today, thanks.
It's pretty great for managing helm files without setting up any gitops or similar!
me neither! flux and argocd are everywhere but normally a bit much
Tried a similar setup about a year ago for some light homelabe stuff I wanted highly available. This would have made the exact thing I was trying, much simpler. I got flash backs multiple times in this video. The iscusi dependancy abd fix, metal lb crds with nginx as the ingress, yea identical and this is cleaner still. Add cert manager and you'll have everything I needed at the time.
If you plan on continuing with this project and series I'd be interested to see, meanwhile I'll take a look at you're other videos. Good stuff, nice work.
Thanks! I'll definitely be continuing with this setup and have added in cert-manager etc as well 😁. Should be more content coming out soon!
I didn't understand everything in this video, but it was good to watch, and certainly something to refer back to when i want to learn more about setting up my own home lab. I subscribed to see more videos about this and what you do with this home lab in the future.
This was excellent. Although I would love to have seen the HA feature testdriven; perhaps builidng out the k3s stack and THEN upgrading the RAM & storage iteratively. I would also love to see cert-manager integration, as well as acme/letsencrypt.
Bookmarks for the video would be a *really* nice thing to have. Such an awesome setup! Thank you for sharing it!
This is a really nice channel.
I hope it grows.
Great video! As a follow-up, I'd be interested in more specifics on HA and disaster recovery, especially regarding Longhorn.
i'm impressed by the clear and concise instructions you give in this video. to be honest, the whole kubernetes with helm with helm charts with helm values with ingresscontroller etc, sounds like a lot of work to setup. it doesn't give me the feeling this setup is stable, but don't really know any details beyond that first look.
It would be interesting to hear about the usual breakage scenario's, and how updates look like.
Great Homelab Setup Video, will also implement in my Homelab system
I mostly enjoyed how clean your UI looks.
Great video and you have a talent to very cleary explain complicated concepts in a simple way. As a devops engineer working with this stuff daily I can say your setup is very nice and clean. But honestly I would not spend that much just to run a few containers! $200 worth of used laptops would be good enough for 10 times this workload. But I can also understand the desire to have a perfect setup. Looking forward to videos where you utilise that horsepower, maybe with Frigate and local LLMs?
This is the first time your channel has been recommended to me, and I subbed before I even watched the video.
The DNS automation is huge! Thank you.
I understood maybe 10% of that, but enjoyed every second of it!
I'm glad! Is there anything you'd like me to do a deeper dive on?
@@dreamsofautonomy I could watch a deeper dive on any of the individual services that you covered. Would love to have a better understanding of longhorn for example.
Thank you so much for this video! This pretty much exactly encapsulates my vision of what i am currently in the process of building! The helmfile and kustomize were two Lego bricks i felt like i was missing in my current attempt.
I'm excited to see more!
😮 ngl the way the tooling is presented reminds me of the Microservices skit
Can't wait to see your video on Talos Linux
I just got my framework 16 recently. Loving it so far :). ⚙️
Your automation tools and workflow is amazing, really interested if you make like crash courses for the tools you used.
Great video, thanks for making! 🙏
Am curious to learn more about nix in upcoming videos
You can use avahi to broadcast hostnames to the network so you can refer to all nodes by their hostname and not have to remember ips.
Great video! love how detailed it is
I manage some Kubernetes clusters in the cloud and I learned a few things. Very interesting video. I bumped into your video 2 days ago because I did some searches on mini-pcs with n100 Intel cpu. I wanted to setup a Proxmox server with a Home Assistant VM as a starter. I also already copied some of your zsh config and moved my prompt from PowerLevel10Kto oh-my-posh (I already used it for powershell with the default config).
To test some Kubernetes configurations without deploying to the cloud I use K3D (in WSL on my laptop). Nodes are docker containers. It worked well for my use cases, I managed to validate some istio (service mesh) egress load balancing setup. But yes it is not a permanent home lab setup...
This is the tutorial I wish i had a couple of years ago trying to figure all this out
Who all saw yesterdays stream,
nice thumbnail!
That was a fun stream! Crowd sourced design!
Great video as always, boss
This seems fun, once i move to a bigger place i will definitely try to do something like this
excellent video! I appreciate your CLI-to-video approach and you cover interesting content.
01:08 British accent and American plug. Brain went "huh?" 😀
Wow, I learned a few things and it gives me some ideas to try in my clusters
Im so jealous you had the knowledge to be able to execute this project. I would love to make your acquaintance and learn your background.
I understood everything you just did and that’s honestly a fantastic process you laid out but I do NOT feel confident that I could do it myself. You ran into extremely difficult issues that I don’t know how I would have figured out on my own. Great video though.
That's some overkill setup for a pihole ;)
Excellent! Thank you very much for sharing!
Good video!
Wow, that's brilliant !
Overcomplexifying things that could be done with bachelor's degree networking knowledge.
Do tell!
Thanks for the video! And bonus points for having the same laptop and keyboard as I do
👍
It's a great keyboard!
MORE HOMELAB PLZ!!
Highly relevant to my interests! I was dabbling with RKE2 but I'm not really in that mindset yet, so I think I'll go back to K3S. My nodes are running on RK3588 CMs, so I'm not sure if I can use Nix though. Will investigate further.
This video is like ASMR to me.
Great video!
I would love to see a kube install of jellyfin with hardware acceleration. This is easy on raw-docker, but not so straightforward on kube.
I have it set up! If you join my discord and drop me a message I'll send you over my configuration.
Nix mentioned 🙏
This is the most alt way Ive seem of setting up high availability. To to it from scratch like this seem overly complicated but fun
Loved the video after nixos anywhere I am lost but still saw the whole video at least I will sound smart when i will use the terms I learned here in front of my noobs friends. Hope to see many more videos in homelab series. ❤
Let me know if there's anything I can break down easier and I'll do some more dedicated videos!
Been there, done it.
I would NOT use really small PCs, unless they are Enterprise grade (used, cheap) so you never have to worry about dangling cables.
Talos user here. If all you are interested in is k8s, Talos is a perfect set and (almost) forget solution (remember to update it!)
I also run Talos on production clusters, I can vouch for the ease and stability.
Awesome! I'm really looking forward to giving Talos a go.
Talos definitely would have been a quicker way of getting to a Kubernetes cluster. My firewall provides DHCP, which sets the hostname for each of my Talos nodes. This means I can use a single config for all nodes, only diverting controlplane and worker nodes. If you take a look at the quick start docs - yes, it is actually this easy.
But kudos for going the long way, it builds experience ;)
Would love to see how you migrate homelab typical services to NixOS, such as Plex, Nextcloud, ollama, vaultwarden...
Nixos master race
We are rebuilding our Kubernetes clusters at work to use Talos, highly recommend a look! Its the first time servers i deployed on-prem truly felt lile cattle instead of pets.
Terraform + Talos config + Helmfiles + Gitlab CI pipelines is life changing lol. I can deploy a full production ready cluster from Gitlab in a matter of minutes
I'll definitely be giving it a go!
We use tf and talos for our ka8s cluster as well. talos was definitely a bit of a learning curve, as it is a bit different than a more traditional distro.
@@solido888 yea learning to maintain the nodes in an API driven way is a learning curve for sure. But having essentially two config files to configure the nodes with a single command is wild. Plus the immutability drastically cuts down on maintenance required, and upgrading is as simple as changing the image in the file, and applying the config.
It's great
Appreciate you documenting your homelab renovation! Any chance you have a public facing repo with some of the IaC shown here? ty!
I should do in the description! At least I hope so.
Awesome video. I do have a question: Why do you use Kustomize for a file that doesn't need any templating? Just use kubectl apply -f file.yaml instead of using kustomize and adding more complexity.
been into homelab for about a year. now wfh is never been the same
or even everywhere thanks to cloudflare tunnel
What do you use cloud flare tunnel for currently? I've been seeing some good things about it! Might try to find an excuse to add it to my homelab.
what do you do for wfh?
@@dreamsofautonomy I use it for ssh to my computer. So wherever I work I can just ssh to my computer and use my fast computer with tmux+neovim instead of 8GB laptop provided by my company lol
@@DogeMultiverse I make docker swarm cluster with my 2 server to replicate my office stacks (postresql, redis, timeseries db, prometheus, kafka, etc) and use them instead of try to run them all in my main working pc/laptop
NixOS is sooooooooo goooood!
Pretty dope! But just some remarks: with Talos you'd have spend much less time setting up your cluster (I totally get why you had done this, but still maintenance is a lot better on Talos), also you should check out the piraeus storage operator, it is muuuuch better than longhorn in terms of draining nodes, speed and maintainability as It's using drbd kernel module under the hood. (Talos has an extension for the kmod btw). Still great setup, I'm pretty jealous. Maybe try out kubevirt and hosting some VMs;) but you'll probably want to use piraeus/drbd for the storage then 😅, Longhorn is just hella slow
Definitely looking forward to trying out Talos in a future video! I'll be decommissioning my old cluster and using it for dedicated videos once I get everything migrated which should be end of this week!
Kubevirt is also something I want to check out as well! So much fun k8s stuff to jump into.
@@dreamsofautonomy awesome! But really do try out Piraeus on that test cluster, it is much much faster and I am pretty confused why nobody is taking about it. Everyone's just using Longhorn and many complain about it's performance but still choose and use it
@@LampJustin Longhorn has been a thing for longer than Piraeus, that's definitely the reason. Most people prefer stability of a known tool over faster performance or extra features, especially when running services with high SLOs
@@obvionaoe while it's true that Piraeus is a younger project, the technology is not drbd is pretty dang old at this point and very much proven. Longhorn is known to have problems with lagging volumes that are out of sync, so slo is definitely worse. I also didn't mean for all people to talk about it, but I've seen no UA-camr talk about Piraeus.
i had a *fun time setting up pihole's recursive DNS server via docker on my NAS. in hindsight, recursive dns was a nice to have but not worth it for the extra container instability/troubleshooting during a reboot.
Going to start calling it Kuberneddies from now on
Thanks for the inspiration. Tried out NixOS and borrowed heavily from your conf now when I converted my cluster to 3-node-HA. Did you keep or disable the "longhorn-iscsi-installation"-pod? Not sure if it is because I run a multi-arch cluster but on the x86_64 nixos node that container crash loops trying to use yum. I would like to disable that install-pod entirely since I already ensure I have iscsi on all nodes.
Ugh, as soon as you started talking about helm my eyes glazed over. Then again I'm really not a DevOps guy, I use proxmox for my home lab cluster. I just know enough to get everything going and keep it running. But then I also run a lot of my own custom software on my homelab. Modified versions of pihole and plex.
The last time I heard about home lab was from breaking bad
Hi, loved the video. I might be wrong, but I believe you have your cluster set up such that you have all three nodes as part of both the control plane and hosting worker pods. Is this correct? If so, are you not concerned about a possible container escape?
Great video! I know it's a bit off-topic, but I'd love to see a video of you showing how you setup your Hyprland, it looks so nice and clean at 5:00 😊
Absolutely!
I just want you to make actual in-depth courses, workshops and alike, to really push this sort of stuff forward. I'd be first on queue.
Soon! I promise. I'm looking for video editors currently so I can spend more time creating :)
Nice video! It’s nice to see some alternative choices.
You could also check out rancher and harvester. I’m currently building a new homelab with rancher, Traefik (feel the pain), cert-manager, longhorn and metallb.
I'll check it out!
Great video, and a lot of cool ideas for me to now try over the next month or so. I setup a k3s "cluster" just this morning, the timing for this video could not be any more perfect. All I wanted was a postgres database for one of my side projects, so I just use an old rpi 3b that I had lying around and installed k3s on it just for fun. helmfile looks interesting though, I might give it a shot. I just made 5 different yaml configs to deploy postgres, followed a guide from digital ocean. A beelink eq12 cluster seems a little overkill for my use case, but I might add some more PIs to my cluster and then add kubernetes to my resume.
This is fantastic. I used RBpis 4Bs with k3s for this. How do you get rid of that annoying 'update available' link in the bottom of the pihole UI? I deploy the helm upgrade but the link doesn't move, and I wonder if this is why my adblock score has dropped from circa 90% to 30%.
Great video! Interested about your jellyfin setup
Awesome video. Can you kindly explain your terminal setup? Especially what terminal based file explorer that was? I know about and use zen but your configuration was 👌
You can use sops nix to point to the k3s tokenFile in a secure way. The problem lies in setting up sops-nix to play well with nixos-anywhere.
Just get a bunch of used optiplex with 10th gen intel for fraction of the cost. up-to 64GB ram. Can even upgrade to 2.5/10/25/40 GbE anytime.
This is awesome. I also have own the Beelink EQ12. Really fantastic device.
A thing to note is that you'll need at least a 6.6.x kernel to get the wireless interface(s) functioning. I wanted to use Ubuntu Server, but of course I purchased the device just before 24.04 dropped, so that meant more work. Instead I went with Pop! OS ... Definitely not ideal, but the machine is running pretty well.
Next up, I would love to replicate what you've done here. We both need the rack-mount tray thing that "Techno Tim" used for his mini pc build. Though, he went with much more powerful Intel NUC machines.
Ohhh that's awesome I'm gonna check out the rack mount, currently I have both clusters in operation as I'm migrating and my tiny shelf is a mess.
Good to know on the kernel as well! I'm pretty spoiled for ethernet runs so I forget to test out wifi capabilities!
Exactly what I was looking to do on my homelab (except I'm using debian/terraform/ansible, instead of Nix/helmfile). Two small things missing I think : `ingressClassName: "nginx-internal"` in `pihole.yaml`, and remembering to configure the router to use pihole as the default DNS. Otherwise, thanks for the base config!
I just use a basic server with SystemD, worked fine for years now.
I love your terminal and desktop! What do you use for these?
Another hardware to consider is the Odroid H3 or H4 mini boards. They can be more affordable depending on where you live (edit: spelling)
Oh awesome! I'll add this to my backlog and take a look.
Maybe try installing home assistant on this cluster.
Great video, would love to see the next episode. I think I prefer Ubuntu server though
You just got a new subscriber 👍
Glad to have you on board!
I've been thinking about using nixos on the server side as well for a while. But I felt like it was more suited for clients. But after seeing this, I may should try it on the serverside as well..
I've got a couple of plans for server side nixOS videos coming soon!
Man NixOS kicked my ass the first time I confidently decided to install it , had a small issue , no biggie ill just check the wiki ... 2 hours later and 30 tabs open I knew I was in over my head ! i will try again sometime though.
neatly explained, nice setup!
one question - aren't you creating potential bottleneck/inconvenience by using DRAM-less QLC SSD? compromising both performance and lifespan while not really saving much money or power draw?
would docker swarm be good too?
This is an amazing video on so many levels.. finally a knowledgeable peer that has a functioning brain and talks about interesting stuff without all the useless stuff (how to open your browser), bad advice (install via `curl | sudo`) and non-sense that ALL the other channels do. Thanks for existing Elliott, you just won a loyal subscriber.
Also TIL systemd tmpfiles.. nice tip :) .. and that externa-dns supports pi-hole as a provider! neat! \o/
Really interesting video thanks so much. Would have watched it even if it would have been an hour just to understand everything behind.
I probably would have died during the editing if it was an hour long haha!
I'm really glad you enjoyed it. Looking forward to doing more in the future!
I'd love to see how to get HTTPS in Kubernetes with local auto renewal. Like Kubernetes Cert-manager or something!
Dude, you inspired me to get my Lab inorder :D
Got two of those Belkins together with some HP Mini G3s I have
One thing thought, Would you consider making a video abt NixOS?
I am running MacOS with Nix installed ( dont know if I can do what you suggesting with nixOS anywhere )
Pretty decent lab you have there !
Thanks! You can do nixos anywhere with macOS and nix!
I have a couple of videos planned for nix! Going to start with one looking at Nix Darwin and then on nixos.