Designing My Multi Site HomeLab
Вставка
- Опубліковано 11 чер 2024
- After moving some of my HomeLab servers into the new colocation I have so many choices to make when it comes to self-hosted services and architecture! From networking, to VPN, to security, to hypervisors, to backups, and even DNS! I NEED YOUR HELP! Help me decide if I have created a solid foundation for my new HomeLab in a Colo!
Network diagram created with Figma l.technotim.live/figma (affiliate link but they have a free option)
Video Notes: technotim.live/posts/homelab-...
Support me on Patreon: / technotim
Sponsor me on GitHub: github.com/sponsors/timothyst...
Subscribe on Twitch: / technotim
Become a UA-cam member: / @technotim
Merch Shop 🛍️: l.technotim.live/shop
Gear Recommendations: l.technotim.live/gear
Get Help in Our Discord Community: l.technotim.live/discord
Tinkers channel: / @technotimtinkers
(Affiliate links may be included in this description. I may receive a small commission at no cost to you.)
00:00 - Intro
00:46 - Network & Firewall Review
02:10 - New Network & VLANS
03:16 - Site to Site VPN & Firewall Rules
05:05 - New Network Diagram
06:28 - Proxmox Servers and DNS
08:05 - Proxmox Virtual Machines
09:52 - Proxmox Backup & Restore
11:46 - Storage Services: HomeLab vs Colocation
13:14 - Colocation Services
15:54 - GitOps with Flux made this easy
17:29 - Longhorn migration with Object Storage
19:01 - I'll "Git Off" GitOps
19:33 - New Rancher Cluster
21:02 - Migrating workloads from HomeLab to Colocation
21:53 - Rancher Network Diagram
22:44 HomeLab Services & Changes
25:14 - Site to Site VPN vs VPN Mesh (Tailscale)?
26:55 - I NEED YOUR HELP - Next Steps
Thank you for watching! - Наука та технологія
My boyfriend has fallen asleep watching you several times now. Your voice is apparently very relaxing. He claims to enjoy the content of your videos too.
I'm sitting here staring at the Dark Mode logo that you have covering something...I assume it's an upcoming project or you forgot to put something personal away before filming =P
It's a PoE dildo
same, its just.... there. its not subtle but at the same time it kinda blends in. lol.
Hmm. New Unifi Dream Router SE Pro Max? (Embargoed until May?)
@declanmcardle need to throw an "Ultra" in that name
I really hope there are new UDMs coming soon
Hi Tim, I know you said that there are better solutions for DNS out there. But I will recommend, switching DNS (and maybe DHCP) servers before your infra gets too big. I had to do it, and it's better earlier than later. Windows DNS, Technitium, BIND9 etc.
I currently use two Technitium DNS server that are authoritative for three zones (A hosts zone for dynamic A records from DHCP, a 'base' zone that contains my apps cnames to hosts zone entries and a conditional forward for the Samba ADDC zone). One of which is a "hosts" zone that gets updated via a couple of KEA Dhcp servers via RFC2136.
Meaning that as devices connect via DHCP they get an A record that is then zone transferred (using native DNS tech rather than hobbiest scripts) from standard primary to secondary zones that each "site" has.
As leases expire and are not renewed, the records are removed.
+100 for Technicium, i hit it with 450k reqs/sec and it didnt care.
Another vote for Technitium here. Handles all my DNS/DHCP/Ad Blocking now.
Run PBS in both locations and use replication to pull down the backups to local. At least I think that should work 😊
This what I was thinking exactly. PBS is btw great and saves a lot of storage.
Agree! The virtual PBS inside Proxmox PVE at Colo for BU, and virtual PBS inside Proxmox PVE at Home, connected to PBS at Colo and pulls remote BUs from PBS Colo. This setup only requires port 8007 open at Colo. Home env no ports required to be opened. Very nice and avoids NFS/Firewalls.
Outstanding video. Good coverage of the what, why, how, and where you're going. Well done, you.
LAN-IN is traffic coming into a LAN interface and transiting through the device, eg traffic from a LAN subnet going to the WAN
LAN-OUT is traffic coming out of the interface after transiting through the device. eg traffic already through the device and now going out of a LAN interface to a LAN subnet
LAN-LOCAL would be traffic coming into a LAN interface destined for the device itself
IN/OUT is for transit traffic, eg traffic not destined for the device itself, LOCAL is traffic destined to the device itself
Thank you! Why does VPN traffic coming in to the network apply to LAN OUT and not LAN IN?
@@TechnoTim LAN IN is outgoing traffic, LAN OUT is incoming traffic. I like to think of it as LAN IN is traffic going IN the LAN port, while LAN OUT is traffic coming OUT the LAN port.
So your incoming VPN traffic hits LAN OUT because its coming from the port to your LAN
@@TechnoTim Unlike WAN interfaces, you cannot target VPN interfaces in UNIFI (other vendors you can), so you need to target interfaces which your firewall knows about. LAN OUT is going TO your LAN, and LAN IN is come FROM your LAN.
@@TechnoTim Because, as far as I understand it, the VPN traffic basically comes out of the device itself and tries going out the LAN interface to the device, but get blocked by the rules. It never goes IN that interface, because it's already sort of at LOCAL.
Ubiquiti doesn't show, when I last looked, the actual VPN interface. Usually, in something like pfSense, you have a separate interface to which you attach the rules themselves, but you also have the IN/OUT behaviour for both interfaces. Or you can directly create floating rules, that apply just to subents and IPs, irregardless of the interface (you can choose all or some of the interfaces together).
Basically, it's colo --> lan IN ----(vpn)----> lan OUT --> home, since you can't target the vpn directly, you either target the lan in at the colo or the lan out at home (which is the safer option).
Of course if you want to block traffic that goes from your home to the colo you need to target the lan IN at home.
@@TechnoTim You may want to look at the "iptables process flow" as unifi is build on edgeos which is build on linux and iptables.
Its looks complicated but once understood, it will answer your question.
VPN traffic into your network is still leaving a LAN interface to get to devices. A LAN IN rules would only work on the LAN side of the colo.
If you had higher access to the interfaces of the firewall (like in edgeOS or vyos) you could add a fw rule on the VPN interface IN direction.
I have a bunch of stuff connected on different locations and Tailscale makes all connect so easy. Tailscale ACL configuration is a beast. You can control A LOT with Tailscale ACL.
Definitely try out Tailscale connection between COLO and Home
16:30 GitOps with Flux is absolutely the way to go for ANY kubernetes cluster, you are 100% right there Tim!
I prefer ArgoCD, but both are good.
@@Aruneh I use ArgoCD and FluxV2, and, apart from the nice UI, I think that Flux beats Argo hands down, especially when it come to reaching a "desired state". I think that ArgoCD just likes to give up when CRDs don't exist yet (or any other dependency) and register an error.
Would you use this infra-as-code solution for even local k3s clusters and deployments?
@@tommytigerpants Absolutely.
@@tommytigerpants Yes, that is exactly what I do.
I really like your'e content Tim! Thanks for sharing your'e knowledge. It's a really great way to improve people's skills at large. Keep it up!
You definitely need to look into having an sd-wan for your infrastructure. A lot of the rules you might be having trouble with in the firewall can be addressed completely by using an sd-wan.
That was awesome!!!! Makes it easier for me to jump into things when I see others having fun doing it though I wonder how much fun I'll have if I can't get enough cooling hardware to not fry myself while working.. Am dreaming of having a mobile server/lab someday that can assist me in helping others with different projects as well as having a team of geeks doing all sorts of fun stuff including Astronomy, Air/Water+Radiation testing, and other things that would help people get into advanced subjects of activity~
Hey Tim. Thanks for the video and the content.
I personally would be interested in how your storage of the pve clusters looks like. Do you use ceph/clusterfs or replication with zfs pools ? If so, how do you deal with asynchronous mirroring?
Thanks for everything and have a nice sunday
I would definitely make your infrastructure so that the colo is your primary "production" environment with no reliance on your home infrastructure. The likelihood of your colo going down is very low, while your home infrastructure has a higher risk. You dont want your colo infrastructure to be degraded because your home internet went out, otherwise you lose out ok the benefit of having the colo.
I would also recommend creating a "jump box" of some kind in the colo and install tailscale at least on that. That way if your site to site goes down (because that definitely happens a lot) you habe a secure "back door" in to your environment to repair the connectivity without having to load upand go to the data center.
- this is all coming from a guy who has worked as a systems infrastructure engineer with multiple datacenters for awhile now.
I'm definitely going to look more in to GitOps, because that looks slick!
You should configure the NFS Shares for your Backups so, that the pve hosts can’t delete their backup, or do at least snapshots of them, witch the host can’t access. Elsewhere, if someone compromised your pve hosts he deletes the backups and that would be fatal.
The ideal option would be to use the Proxmox Backup Server and maybe pull with a other backup server the backups, because than you have 3 Locations that hold your backups, but can’t delete backups from the other system. (Each system should have other ssh-keys and passwords etc.)
But nice Project. 👍🏼 Because of your Channel I dived into Kubernetes and some other thinks. 👀
I think a deepdive into your IAC Workflow and file-structure would be great. 😁
Witch OS do you use for Kubernetes? I experience some stability issues on Ubuntu.
Good work 👌
I recommend you put secondary DNS in home site, because if your cloud has cut down traffic you lost all DNS resolution about your domains
Why NFS for Backups of proxmox and Not a dedicated proxmox Backup Server? You could even run it in truenas/proxmox as a VM. Big benefit is you can sync between multiple instances and the Backups are incremental.
Also Automatic Verification and reverification for each Backup. Cheers Paul
For backing up your colo servers, I would absolutely back them up to another cloud service and then have a machine on your home network pull those down for a copy once complete. I only have a couple of servers in the cloud, and they back up encrypted files to an object storage provider, which a server on my home LAN pulls down periodically for storage here. I only have the keys on the source (for encryption prior to upload) and my home systems (for decryption after download). In my opinion, having your colo system only able to connect to your home LAN where it is absolutely essential is what I would aim for. So if one of those gets compromised, the possiblity of someone pivoting into your home network is as low as practical.
Looks like it's been a while since there was a wholesale "wipe, no chance of recovery, no restore from backup, recreate/re-engineer only the needed parts, from scratch" either literally, or as a thought exercise.
While it's certainly understood (and expected) that you play with, toy with, engineer and demo quite a bit, that's adding a lot of complexity, so the "whoops, all burned down" can be a good mental filter to strip and refine.
The other thing I didn't see highlighted is engineering around local-remote failure assumptions/expectations. Example:
When you have two routers and 3 (or more hops) between home and remote for something like home etcd for a remote k3s/k8s/whatever, that's adding 4+ potential over-network failure points.
Very nice setup. Almost like mine I just dont have anything in a datacenter. I use argo for the cd paired with my own gitea instance as I can see everything at a glance via UI from wherever I am.
How do you like Longhorn? I use OpenEBS on my home Raspberry Pi Kubernetes cluster (3x) worker nodes and (1x) master. The big problem with OpenEBS how I have it configured in openebs-hostpath is that pods have to be created on the original worker node it was provisioned on. I.E. no support for migrating pods with PVs across worker nodes.
I want to setup some sort of overlay VPN like tailscale or some sort of site to site thing like you have going on. Only downside is that from the little research I've done, there don't seem to be good options for configuring that stuff as infrastructure as code.
I tried getting headscale setup at one point, but their only recent releases are alpha releases. It sounds like you're gonna go with tailscale instead of headscale, but I wondered if you considered headscale at all.
Hey Tim! Great content, as usual. Do you use HA on proxmox? You have enough nodes for an official quorum, but you mentioned your naming scheme includes the host # in the guest name. Maybe those with specific host numbers are not meant for migration?
Hey! I do not use HA in Proxmox, I have them in a cluster though for shared backups, config, etc. Typically don't make my VMs HA, I make my services HA with Kubernetes ;). This numbering system also helps me to know which I can take down for patching, ie. I can patch and reboot all the 1's at a time. Then wait, then all the 2, etc...
@@TechnoTim This makes sense from a management perspective (security and documentation). However, that hardware strategy doesn't scale very well in much larger environments. It might be better to mentally abstract the hardware: "There is no spoon." Focus on the data and treat the infrastructure as a commodity. There are some security considerations to worry about, but most of them are administrative and not technical controls.
Not the familiar with your entire stack but have you dealt with IaC before and cluster deployment?
for backups, instead of allowing the Colocation PVE nodes to have access to your NAS and push backups from the outside in, you could deploy a Backup server at home (maybe Veeam free version), grant it access to the NAS via a Vlan with some rules and allow the Backup server to Pull backups inward.
You could also use the Proxmox backup server but it still relies on the PVE nodes pushing backups to it, but it could act as a proxy of sorts between the nodes and your NAS.
I just don't like the idea of anything outside the local network having access to a NAS regardless of s2s vpn.
I am actually building a nas that I am colocating so that I have proper 321 for multiple networks I manage… so I’ll have a few sites connecting to each other. I just have a nas at home (and on several other networks) that can backup to the nas in the data center. I’m actually adding duplicates of services for high availability for some DC’s, etc. vs the migration you are doing.
I like the firewall on opnsense/pfsense much more than unifi, I moved away from my dream machine. Some networks I’m using site to site with are still unifi though and it depends, unifi is fine for me until I start hitting more advanced features.
Tim, you madman. Would love to know where the conversation of "can" vs "should" took place!?
Thanks Tim.
Tim I think someday we'll have to have a beer to discuss the definition of home network. 😂😂😜
Great video! Have you looked into hosting Proxmox Backup Server (PBS) in a VM? You could run that at home without exposing NFS to your colo (you have to mount your NFS share in your PBS VM and then create a datastore in PBS pointing to that folder). Or you could host two PBS VMs, one in each location, and then sync your colo PBS to your home PBS. Or vice versa.
And with that you even get incremental backups, with deduplication. Running PBS is so much better than direct Proxmox backups.
Thank you! I need to look at PBS again. I ran it in the past and I think it might work if I can use an NFS share for the backing data!
@@TechnoTim 👍 It required going into the CLI, but I have it working at home. PBS VM in proxmox with autofs installed to mount my Unraid share over NFS. Storage usage reporting isn't great because the Unraid NFS mount reports the usage of the entire cache pool instead of just the share, but having a nearly 60x deduplication factor is pretty nice!
very nice its pretty tough to configure tons of stuff connecting it like this
Do you have a cloud key at the Colo or are you having the UDM at the Colo talk back to your network application at home?
The UDM is a Unifi Console, it is its own Cloud Key, and as such cant be connected to another cloud key or SHC.
He has UDM running in both location.
I'd like to see you try Nomad.
volsync would have saved you alot of time on backup/restore :) you just point at a s3 repo and volsync does everything else for you, as long as you point at the same volume.
Awesome.
I needed something like this. I am currently designing my network that can scale and this video has helped me see certain things i overlooked. Thanks for making a this amazing video.
@TechnoTim "I know there are better dns servers than pi-hole, but". What would you recommend more than pi-hole?
Powerdns all the way
How will you deal with disk encryption?
twingate and make it fully granular?
So im not sure if I missed something but why run proxmox at the colocation at all? Sense you can run pihole in kubernetes, rancher you can have that running on your home network(sense you also have your nas there) to manage the colocation cluster and you can run github runners in kubernetes as well. For disaster recovery, try something like kasten k10(haven't tried it myself) but I think with flux and if you are running something like longhorn, you have essentially a state of your cluster in git and then restore the data with longhorn.
As for server recovery, does the supermicros have a ipmi or do you have a kvm in the colocation?
Also im not sure why you would want tailscale sense you already have a site to site vpn and if you want to open up some services or things like that you can use an host an ingress there instead and then just route it to your nas. Otherwise you would just have 2 vpns and you just trade site to site vpn rules with tailscale config? But try tailscale on your phone, it's great!
But regardess of that, a really fun setup!
Yea a site to site wireguard connection with perhaps a reverse proxy is more secure and efficient
Good video, great to see so much documentation on so much infrastructure!
i would think of different locations as prod/stage/lab envs
so that they don`t have anything in common, be sepparate and self sufficient
good thing to use gitops for all of them, also you dont need to copy stuff between clusters, create "base" config and change it between clusters with kustomize overlays or helm values
also why not use PBS for backups? there you can have thin provisioning for backups and put it on tailscale, so all proxmoxes can backup to same location
WHAT IS ALL THIS USED FOR ?
Please, create a course to ill helps learn how to setup a server with all the systems needed for developers. From the 0 to full automation, Gitops, etc..
Cool
I think you are adding to much risk using your vpn as the security layer to your home data services.
I would try to restrict access from colo to home with an Nginx proxy or full flesged Kong gateway. Kong provides more auth plugin option but is a lot heavier
Hopefully you don't run those Databases inside Kubernetes.
If you do keep the following in mind:
- Is my application really HA? (Zero downtime upgrades)
- Does the DB gets rebuild/restartet when the application gets updates.
- How can I scale my Stateful Set. (Dublicated DBs etc.)
nice
Tailscale has seperate acls and it will bypass the internal firewall.better to stick with the site to site vpn.
Home Lab... Away Lab... as a Home Lab scientist... away usually means 'offsite backup' or 'cloudflare DNS or Tunnels' ...are you still Home Lab?
As a networking person, we don't accept the Unifi Firewall workflow experience. It's definitely the same reaction in our knowledge domain. I absolutely wish for a traditional zone based firewall experience.
Why use VM's instead of LXC?
They are a little more flexible than LXCs
I thought i knew computers but i am so lost. I understood maybe half the words used in the second half of thia video.
Weird question. What brand and model are your glasses frames? lol
Old Warby Parker! It’s time for a new pair 😅
Highly recommend ditching port forwarding and traefik and going all cloudflare tunnels.
Proxmox backup server is something you should take a closer look at
I only set up PBS recently, wish I'd done it years ago. Very impressive.
I guess you really have to switch your tool box to the dark side so you don't have to censor it. 😉😇
IMO since this is just for homelab / personal use.... alway always ALWAYS think "How can I make this simpler with minimizing additional security risk"?
Ask yourself, is what I"m doing going to save me time, money, or frustration? If not, then it's going to be for personal reasons, like fun/learning, and those 3 other reasons do not matter.
In regards to Lan IN and Lan Out... since encrypted traffic crosses over the VPN, it does not cross the WAN interface. Both Mikrotik and UNIFI do not make interfaces for pure IPSEC tunnels, so you have to use firewall rules on what interfaces you do have. Now luckily, if you had mikrotik, you don't even need to target an interface, you could just target the subnet. Source / Dest subnet - allow / block, done.
To be fair, saving time, money and frustration do also matter if it's just for personal reasons.
- Money probably doesn't grow on his back.
- Time wasted on fixing stuff is time wasted not playing with other stuff.
- Frustration is just never good.
@@FinlayDaG33kI"m not discounting those 3 reasons, just prioritizing them. SOme people are willing to put up with more money, more time, or more frustration because of learning / fun / hanging out with others.
You remind me of Johnny Depp so much 😂
Caddy... Try Caddy instead of NGINX Ingress Controller
Figma is quite the name. Very bold of them.
Geezus Peezus!
Holy Cow time, you have more compute power than Heinz has Pickles!
I'm not sure what country, but there is a country in the world that has less set up!
Very Amazing, thank you for sharing!
As always an interesting video!!!. I have more background on Network and I want to share some tips:
- Don't over engineer creating additional VPNs with Tailscale. Makes troubleshooting more complicated and figuring out what system / apps goes via which Remote VPN.
- You have 2 macro network segments (Home and Public) but I think you need to be more granular on your segmentation inside of each site (Home and Public).
- Define simple rules for network flows based on these additional segments: An idea is all directly internet facing is the red zone (i.e., the interface of your UDM Pro), All traffic that is accessible from the open internet should land on a DMZ (i.e., Yellow Zone), and all the internal traffic should be on your internal zone (i.e., Blue Zone). Using color coding will help you identify easily the type of security controls and also what and where to put workloads (and why).
- Traffic from Red to Blue is not allowed directly (except for your Remote Access when you are on the street and want to connect to any of your sites). That means that any other traffic that comes from the internet should terminate in your DMZ (Yellow Zone) on a Proxy Server (i.e., your Traeffik or NGNIX) running CrowdSec Bouncer for extra IPS security.
- Any traffic into your Internal / Blue Network should be allowed from the IPs of your Proxy Server ONLY.
- Traffic from your Internal Network (Blue) is allowed to any other zone (Yellow and Red) without additional security controls.
- Sensitive Data should be stored only in your Blue segment(s).
- Use different VLANs (at least 1 per Zone: like for Yellow, one for Red, one for Blue) and any traffic between VLANs should be inspected by a Firewall (i.e., UDM and/or even a virtual one like PFSense or OpenSense) following the rules above.
- Replicate this zonification in both Home and Public.
- Make each site a DR of the other site (Depending on your end goals).
- Identify what data / workloads are critical so you get backups locally in each site and also in the remote site.
To follow these simple segmentation rules will probably trigger the need to have at least 2 set of clusters of K3S/R2K per site: 1 for the Yellow Segment (DMZ) and for the Blue Segment (Blue) to honor the security flows based on the security zones and allowed flows.
My 2 cents on this.
15:00 Rule of thumb from working in the cloud, is all about redundancy.
As a system engineer I gotta say, the first network is built over complicated and a lot of unnecessary stuff.
first
Just start saying you dont have a homelab. You have two, just one in a "cooler" home
A lot of folks like to overcomplicate their networks for that wow factor. Just remember one word when your designing it, KISS!
There's an irony here. In other videos you tell us to ditch the cloud and self host, but then you show the most complex thing, literally scary!!
Not really ironic. It's not like he ditched self hosting. (And to be honest, this applies to everyone: Running a business server/service from home can cause HA (high availability) issues. As soon as something starts making you money, making sure it doesn't go down/offline should be priority #1) And it's only complicated because he wants 2 different sites to behave as if it's one location, so it does complicate things rather than treating them as 2 different networks. If he did that, or choose to use the remote site as either the backup or main, and vice versa the local site, it would be a lot easier to manage, as you'd only need seldom contact, instead of constant contact.
He is not using the cloud. He is hosting his machines in a data center. The cloud you use another person's computers, in colocation you use your computers in someone else's space.
@@kevikiru Thanks! Yup, 100% self-hosted, I own the hardware, my own private cloud.... just someone else's power and internet connectivity!
He still runs the „same“ architecture.
The only thing that makes it complicated is tunneling everything through a VPN tunnel.
3 home0s are in your house
If you're going to make this about you, I'm going to unsubscribe.
What do you mean?
Please tell me, what should my channel be about 😅