You are now my personal hero, thank you for this explanation, and most of all for the clearity, context and the way you did the edit. I looked for many videos and stumbled upon yours, and after all those others, i didnt learn as much as i did in the video you made thats shorter, more to the point and well great. So yes make more of them 😜
Very nice video and useful for me. I had a Talos OS cluster already setup on 3 Intel NUCs with Mayastor as storage solution. However, I did the interactive Talos OS installation of each node. As I messed up the configuration, I had to redo the setup. This time with your approach. It work well. Looking forward to more content from you.
Glad it helped! I tried Mayastor in the past and the performance was good. Sadly, the lack of snapshot support made me look for other solutions. I'll probably try to cover rook-ceph in the future as well, and take another look at Mayastor if/when they implement snapshot support
Thank you very much! I do plan to cover that as well at some point. I just gotta figure out a way to fit it all into my schedule somehow. My video release schedule is abysmal right now :( It's on the list though!
Great video thank you.. does your talos cluster commit to disk after doing this steps ? What happens if one of the nodes in the cluster is rebooted does it automatically scale and cluster goes to healthy state after node comes back online.
In the machine configuration which you apply during the install, you specify the disk which Talos will use. Once the machine config is applied, Talos is written to disk, yes, meaning that you can remove the ISO and the machine will boot into the os you configured. If a node is rebooted, the Talos cluster behaves much like a K8S cluster, it will return to a healthy state once the node returns/is replaced
Yep, it's definitely an option. My main gripe with this approach is that you now either rely on TechnoTim to update and maintain that playbook or just assume that responsibility/workload yourself. Also, you now have the underlying OS to update and manage as well
How does Talos compare to k3os and a few other similar solutions? I rolled my own K3s playbooks based on earlier work by Jeff Geerling. I quickly found out that k3s was filling up storage with logs. Technotim 's playbooks were similar. I ran k3s on Proxmox in a LXC container and on a pi all configured with my own Ansible roles. Talos looks interesting to try on bare metal, I don't even want to bother with Proxmox. I also think about running Talos on a VM in TrueNAS scale since K3s is about to be removed from TrueNAS. I would love to see Talos with Truecharts and a nice Web ui and maybe Rancher.f
Thank you! It's not really designed for "mixed" clusters, but they do provide cloud-specific images. Migrating should be doable with a tool like Velero or volsync for example
I have tried Rancher, both in my Homelab on top of Talos as well as professionally, using it to manage multiple rke clusters. It's an interesting tool and dashboard, but I don't personally like it that much. I don't have it deployed in my infra as I have no need for it. As for ingress - I am using ingress nginx for now with Talos and at work too. It's great and I recommend it. I'm not really a huge fan of Traefik, even though it seems popular in the Homelab community
Well... yes and no. In my opinion, the HTTP server is a smaller attack surface than SSH. If you were to break through SSH and gain access to a machine, you have the familiar environment of a shell which would allow you to do more or less whatever you want. With the API server on the other hand, if you were to get the certificates to authenticate you are still relatively limited as to what you can do. The API server only exposes certain functionalities, not an entire shell with all of the tools and utilities. There is also the "security by obscurity" argument since the API server definitely provides a less familiar interface than a shell. But in essence, yes. Neither the API nor SSH are perfectly secure and every solution is susceptible to vulnerabilities. I would also argue that having the OS config in a YAML file makes it easier to just nuke the node/cluster and restore it if something happens, but this is not really related to the API vs SSH discussion, but rather to the traditional OS vs API-driven immutable OS.
@@mirceanton APIs don't restrict capability of a bad actor just because it's not a shell environment, one or more webshells are likely going to be deployed once compromised. CVE-2021-26855 for one example.
I definitely think both have their own place. Talhelper makes a lot of things much easier, though at the cost of an added layer of complexity. I want/plan to make a video about it as well to showcase how it can simplify some things and make more complicated configs easier to manage. One analogy I like to make is that using talhelper for talos configs is similar to using kustomize for k8s manifests.
I tried the same but also had 6 worker nodes. It complained about the VIP, and that worker nodes can't have them. I commented out that part in the worker.yaml file. Now the 3 control planes are stuck in a bootloop...
Are you sure you used `--config-patch-control-plane` and not just `--config-patch` when specifying the `vip.yaml` patch? Since you removed it, your controlplane nodes are stuck in a boot loop because they need the VIP to find each other. That VIP configuration should not have made it to the worker nodes though, so that makes me think you either passed in the controlplane configuration to worker nodes as well or just used the VIP as a generic patch, instead of it being controlplane specific.
Thank you! To expose workloads running inside the cluster to the outside, you would need an ingress controller such as ingress-nginx. To the further expose that to the internet you can either do some port forwarding on your router or use something like cloudflare tunnels
Great tutorial. It worked perfectly the first time, however, I started noticing some weird issues and decided to reinstall the cluster. As of today, after the talosctl apply -f rendered/controlplane.yaml, the install hangs and /dev/sda disappears. Anyone have some ideas?
@@mirceanton Hi! I am running on vSphere 6.7. When I check the disks before running the apply command, I see my 32 gig /dev/sda drive. After I run the command, it goes away and my Talos stays in maintenance mode, after that /dev/sda seems to be gone. I am running the newest ISO. Could this have something to do with it?
amazing video!!! Please make the next video how to add storage, networking, and other stuff, and most importantly how to add applications during bootstrap, to make deployment of k8s cluster as easy as possible - end result, running cluster with apps and everything :D
Thank you! Absolutely! I plan to cover a couple more things on the Talos install and bootstrap part so that we can get a proper cluster up and running easily using FluxCD
Thanks for the heads-up! I recently migrated my blog from Hugo to Jekyll and apparently I didn't do a great job at preserving all the links. I updated the description so now the URL should work. Thanks!
Yes, that's great, but you are basing the instructions on two directories with files (patches) that each person must create manually and there is generic content. Keeping in mind that the intention is for us to learn, you could try to make it easier to increase the chances of it being successful. In any case, I think it's an excellent video and of course I subscribed. It has been very useful to me. Good job, thank you very much :)
Hi, @camu2be! Thank you for the feedback! I'm quite new to making tutorials and videos so I'm still trying to figure it all out. I'm glad you liked it and found it helpful. I'm curious if you can expand a bit more on "try to make it easier to increase the chances of it being successful". Would a GitHub gist/repo be easier to follow along with? IIRC I have a link in the description which points to my blog which also has these patches there to copy paste.
Great video and great explanation why to choose Talos. But thee music is horrible if you want to listen and focus. A tutorial with that complexity, I do not believe that anyone is interested in a distracting and annoying noise pollution 😮 Now, we can continue with the promised video series about automation 👍
The instructions are clear but what you receive in the end is not working. Machines stuck on Booting state, you can't do anything. There is no explanation what is host 10.0.10.10 doing and i see a lot of error messages related to my VIP ip (in my case different IP). If i run: talosctl get members rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: cannot validate certificate for IP.IP.IP.NODE3 because it doesn't contain any IP SANs" If i repeat this message, IP.IP.IP.NOD1, NODE2, NODE3 are cycling, but otherwise the same message.
Around 9:45 I explain that 10.0.10.10 is the IP address I am using for the VIP that essentially acts as a load balancer in front of the nodes. Did you set up your talosconfig properly? Maybe you have the wrong endpoints or perhaps an old secrets bundle
Oh, I'm sorry to hear that. It may be my Balkan accent. I can try adding captions in UA-cam so you get subtitles as I speak for my next videos. Would that be helpful?
i'm experiencing vip missing (removed). this how its produce: 1. using 2 CPs (192.168.11.100,192.168.11.101), using 1 vip 192.168.11.88 2. both running healthy and ready 3. deploy php apache 4. testing with 192.168.11.100:30004, works 5. testing with 192.168.11.101:30004, works 6. testing with 192.168.11.88:30004, works 7. shutdown node 192.168.11.101 8. testing with 192.168.11.100:30004, works 9. testing with 192.168.11.88:30004, works 10. testing with 192.168.11.101:30004, NOT working (due to shutdown) 11. at this point, node 192.168.11.100, missing its vip (vip is removed), controller-manager unhealthy, and never recovered since then. to restore the condition, i have to reboot all. unfortunately, the process took at least 30 minutes 🤷♂
That is the expected behavior. You are running a cluster with an even number of controlplane nodes which is NOT recommended. At step 7, the cluster officially lost quorum and is in a degraded state. From this point you will run into undesired behavior since components will start getting in an unhealthy state. Pods will no longer be scheduled, and, as you have noticed, the VIP will start misbehaving as well. The way I see it, you have 2 options here: 1. add another controlplane node to the cluster so that you reach an odd number of 3, which is the minimum number of nodes for a HA controlplane, 2. run the 2-node cluster as a controlplane and a worker Note that option 2 will not assign the talos built-in VIP to the worker node, so you will need some other solution for that, such as MetalLB or Cilium. I suggest option 1 as it is by far the simplest and it is what I would personally recommend for a small home cluster.
You are now my personal hero, thank you for this explanation, and most of all for the clearity, context and the way you did the edit. I looked for many videos and stumbled upon yours, and after all those others, i didnt learn as much as i did in the video you made thats shorter, more to the point and well great. So yes make more of them 😜
An excellent POC. I would like to see in the upcoming videos: ingress controller, statefulset and backup&restore. Keep up the good work!🎉🎉
Very nice comprehensive video, looking forward to your next one! ( Kubeti )
awesome video! cannot wait to find a way to apply this to my craft :) keep at it
I was looking for an installation guide for Talos Linux. The title doesn't say that, but the video is great. I got what I wanted.
Excellent video. Liked an subscribed. Looking forward to your future videos.
The topics were really well explained, even for someone who is very new to these concepts. Loved it
Nice and very instructive video. Thanks a lot !
Looking alse forward to your future videos
Thank you! Will setup this and test. Seems way simpler than my present cluster based on AlmaLinux with Kubernetes installed using kubeadm :)
great video!
yes I would be very interested if you have more videos about storage and networking
Very nice video and useful for me. I had a Talos OS cluster already setup on 3 Intel NUCs with Mayastor as storage solution. However, I did the interactive Talos OS installation of each node. As I messed up the configuration, I had to redo the setup. This time with your approach. It work well. Looking forward to more content from you.
Glad it helped! I tried Mayastor in the past and the performance was good. Sadly, the lack of snapshot support made me look for other solutions. I'll probably try to cover rook-ceph in the future as well, and take another look at Mayastor if/when they implement snapshot support
This was a superb video. Mircea, Thank you so much for this. It would be super if you can do video about Talos + Cilium CNI
Thank you very much!
I do plan to cover that as well at some point. I just gotta figure out a way to fit it all into my schedule somehow. My video release schedule is abysmal right now :(
It's on the list though!
I appreciate your effort! Si astia pe site zic ca il instalezi cu o singura comanda, ha :))
M-ai făcut curios. O sa încerc Talos in HomeLab-ul meu. Poate ne-om cunoaște când voi trece prin Bucuresti, la un schimb de experiența
Great video thank you.. does your talos cluster commit to disk after doing this steps ? What happens if one of the nodes in the cluster is rebooted does it automatically scale and cluster goes to healthy state after node comes back online.
In the machine configuration which you apply during the install, you specify the disk which Talos will use.
Once the machine config is applied, Talos is written to disk, yes, meaning that you can remove the ISO and the machine will boot into the os you configured.
If a node is rebooted, the Talos cluster behaves much like a K8S cluster, it will return to a healthy state once the node returns/is replaced
K3s is definitely the easiest I've found. Particularly techno Tims ansible playbook. So satisfying watching it run too
Yep, it's definitely an option.
My main gripe with this approach is that you now either rely on TechnoTim to update and maintain that playbook or just assume that responsibility/workload yourself. Also, you now have the underlying OS to update and manage as well
How does Talos compare to k3os and a few other similar solutions?
I rolled my own K3s playbooks based on earlier work by Jeff Geerling. I quickly found out that k3s was filling up storage with logs. Technotim 's playbooks were similar. I ran k3s on Proxmox in a LXC container and on a pi all configured with my own Ansible roles.
Talos looks interesting to try on bare metal, I don't even want to bother with Proxmox. I also think about running Talos on a VM in TrueNAS scale since K3s is about to be removed from TrueNAS. I would love to see Talos with Truecharts and a nice Web ui and maybe Rancher.f
Super info about Talos!!
Can I add a Talos/Kubernetes node to other K8S clusters on GKE/AWS/Azure/...
Thank you!
It's not really designed for "mixed" clusters, but they do provide cloud-specific images. Migrating should be doable with a tool like Velero or volsync for example
Thank you....
nice video. thanks a lot... have you ever try rancher, ingress on talos kubernetes ?
I have tried Rancher, both in my Homelab on top of Talos as well as professionally, using it to manage multiple rke clusters. It's an interesting tool and dashboard, but I don't personally like it that much. I don't have it deployed in my infra as I have no need for it.
As for ingress - I am using ingress nginx for now with Talos and at work too. It's great and I recommend it. I'm not really a huge fan of Traefik, even though it seems popular in the Homelab community
It doesn't have ssh but it does expose an HTTP server right? So it's removed one attack surface and introduced another.
Well... yes and no. In my opinion, the HTTP server is a smaller attack surface than SSH. If you were to break through SSH and gain access to a machine, you have the familiar environment of a shell which would allow you to do more or less whatever you want. With the API server on the other hand, if you were to get the certificates to authenticate you are still relatively limited as to what you can do. The API server only exposes certain functionalities, not an entire shell with all of the tools and utilities.
There is also the "security by obscurity" argument since the API server definitely provides a less familiar interface than a shell.
But in essence, yes. Neither the API nor SSH are perfectly secure and every solution is susceptible to vulnerabilities.
I would also argue that having the OS config in a YAML file makes it easier to just nuke the node/cluster and restore it if something happens, but this is not really related to the API vs SSH discussion, but rather to the traditional OS vs API-driven immutable OS.
Ah that's a really good point
Thanks
@@mirceanton APIs don't restrict capability of a bad actor just because it's not a shell environment, one or more webshells are likely going to be deployed once compromised. CVE-2021-26855 for one example.
Crazy quality!
much cleaner and visual than using talhelper in my opinion
I definitely think both have their own place. Talhelper makes a lot of things much easier, though at the cost of an added layer of complexity.
I want/plan to make a video about it as well to showcase how it can simplify some things and make more complicated configs easier to manage. One analogy I like to make is that using talhelper for talos configs is similar to using kustomize for k8s manifests.
I tried the same but also had 6 worker nodes. It complained about the VIP, and that worker nodes can't have them. I commented out that part in the worker.yaml file. Now the 3 control planes are stuck in a bootloop...
Are you sure you used `--config-patch-control-plane` and not just `--config-patch` when specifying the `vip.yaml` patch?
Since you removed it, your controlplane nodes are stuck in a boot loop because they need the VIP to find each other.
That VIP configuration should not have made it to the worker nodes though, so that makes me think you either passed in the controlplane configuration to worker nodes as well or just used the VIP as a generic patch, instead of it being controlplane specific.
@@mirceanton i did solve it somehow, but did not know that there was a confog-patch-control-plane command, thank you!!
More videos please
Amazing video! How can we expose nginx to internet?
Thank you!
To expose workloads running inside the cluster to the outside, you would need an ingress controller such as ingress-nginx. To the further expose that to the internet you can either do some port forwarding on your router or use something like cloudflare tunnels
Great tutorial. It worked perfectly the first time, however, I started noticing some weird issues and decided to reinstall the cluster. As of today, after the talosctl apply -f rendered/controlplane.yaml, the install hangs and /dev/sda disappears. Anyone have some ideas?
Thanks! Are you installing on bare metal, or VM? Are you sure `/dev/sda` is not the USB you are booting from?
@@mirceanton Hi! I am running on vSphere 6.7. When I check the disks before running the apply command, I see my 32 gig /dev/sda drive. After I run the command, it goes away and my Talos stays in maintenance mode, after that /dev/sda seems to be gone. I am running the newest ISO. Could this have something to do with it?
@@mirceanton Got it working using the ova instead :)
amazing video!!!
Please make the next video how to add storage, networking, and other stuff, and most importantly how to add applications during bootstrap,
to make deployment of k8s cluster as easy as possible - end result, running cluster with apps and everything :D
Thank you!
Absolutely! I plan to cover a couple more things on the Talos install and bootstrap part so that we can get a proper cluster up and running easily using FluxCD
@@mirceanton Great videos, when are the rest of the videos coming :) - would be great to see how to add persistent storage like longhon/openebs
Your blog post link doesn't work.
Thanks for the heads-up! I recently migrated my blog from Hugo to Jekyll and apparently I didn't do a great job at preserving all the links.
I updated the description so now the URL should work. Thanks!
Yes, that's great, but you are basing the instructions on two directories with files (patches) that each person must create manually and there is generic content.
Keeping in mind that the intention is for us to learn, you could try to make it easier to increase the chances of it being successful.
In any case, I think it's an excellent video and of course I subscribed. It has been very useful to me.
Good job, thank you very much :)
Hi, @camu2be!
Thank you for the feedback! I'm quite new to making tutorials and videos so I'm still trying to figure it all out. I'm glad you liked it and found it helpful.
I'm curious if you can expand a bit more on "try to make it easier to increase the chances of it being successful". Would a GitHub gist/repo be easier to follow along with? IIRC I have a link in the description which points to my blog which also has these patches there to copy paste.
@@mirceanton
If so, I should have been more attentive. I didn't see them! Again, congratulations for the tutorial 👌
Great video and great explanation why to choose Talos. But thee music is horrible if you want to listen and focus. A tutorial with that complexity, I do not believe that anyone is interested in a distracting and annoying noise pollution 😮
Now, we can continue with the promised video series about automation 👍
Thanks for the feedback! Would you say it would be better to have the music quieter or just no music at all during tutorials?
The instructions are clear but what you receive in the end is not working.
Machines stuck on Booting state, you can't do anything. There is no explanation what is host 10.0.10.10 doing and i see a lot of error messages related to my VIP ip (in my case different IP).
If i run: talosctl get members
rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: cannot validate certificate for IP.IP.IP.NODE3 because it doesn't contain any IP SANs"
If i repeat this message, IP.IP.IP.NOD1, NODE2, NODE3 are cycling, but otherwise the same message.
Around 9:45 I explain that 10.0.10.10 is the IP address I am using for the VIP that essentially acts as a load balancer in front of the nodes.
Did you set up your talosconfig properly? Maybe you have the wrong endpoints or perhaps an old secrets bundle
@@mirceanton I have the same issue. Did you install VIP separately? My firewall doesnt forward this IP to anywhere.
I appreciate your effort. I just struggle to understand your English.
Oh, I'm sorry to hear that. It may be my Balkan accent.
I can try adding captions in UA-cam so you get subtitles as I speak for my next videos. Would that be helpful?
@@mirceanton Yes, the auto generated captions weren't very accurate, but if you added them yourself that would help a ton. Keep up the good work!
i'm experiencing vip missing (removed). this how its produce:
1. using 2 CPs (192.168.11.100,192.168.11.101), using 1 vip 192.168.11.88
2. both running healthy and ready
3. deploy php apache
4. testing with 192.168.11.100:30004, works
5. testing with 192.168.11.101:30004, works
6. testing with 192.168.11.88:30004, works
7. shutdown node 192.168.11.101
8. testing with 192.168.11.100:30004, works
9. testing with 192.168.11.88:30004, works
10. testing with 192.168.11.101:30004, NOT working (due to shutdown)
11. at this point, node 192.168.11.100, missing its vip (vip is removed), controller-manager unhealthy, and never recovered since then.
to restore the condition, i have to reboot all. unfortunately, the process took at least 30 minutes 🤷♂
That is the expected behavior. You are running a cluster with an even number of controlplane nodes which is NOT recommended.
At step 7, the cluster officially lost quorum and is in a degraded state. From this point you will run into undesired behavior since components will start getting in an unhealthy state. Pods will no longer be scheduled, and, as you have noticed, the VIP will start misbehaving as well.
The way I see it, you have 2 options here:
1. add another controlplane node to the cluster so that you reach an odd number of 3, which is the minimum number of nodes for a HA controlplane,
2. run the 2-node cluster as a controlplane and a worker
Note that option 2 will not assign the talos built-in VIP to the worker node, so you will need some other solution for that, such as MetalLB or Cilium. I suggest option 1 as it is by far the simplest and it is what I would personally recommend for a small home cluster.
@@mirceanton yes, i skip the documentation why to at least 3 CP. now, the cluster is working great. thank you...