just started using it bc docker-desktop+kubernetes just SUCKS big time - with k3d we finally have an option that runs for most local dev on any local OS as long as it can somehow run docker
Docker made a huge mistake by first ignoring Kubernetes, and then by trying to create a system that can run both k8s and Swarm. Kubernetes in Docker Desktop sucks mostly because they did not focus on making it great, but in being able to run Swarm in parallel with k8s.
I ran k3d for a while, but I'm not so much a fan from a productivity point of view. I'm in the situation now where I have a few k8s clusters I can use as a playground which makes k3d/ Kind less appealing. Instead, using Argo CD with a few test namespaces has been far better for my development (btw your videos were an awesome inspiration for that setup!). k3d was cool for testing basic kube ops, but I found testing my real-world use-case would make my laptop catch on fire if it had to do real processing. k3d/kind worked okay for toy-apps/ experiments, but for testing my actual apps I found I really needed a cluster with some real horsepower.
On a modest 2017 i5 ThinkPad with 4Gb, "k3d cluster create my-cluster" takes about 12s and uses about 400Mb. Of course that's a bare cluster with no workload, still that's amazing. Thank you! K3d is terrific. Setting up clusters has been a bottleneck to training, but now I can scrap the Pi's I was going to use.
Yeah. K3d is great or, to be more precise, k3s is great. You might want to check out Civo as well (there's a video somewhere in my channel). They use it for their Cloud K8s offer with the same benefits. It's fast (90 sec to create a cluster), has low system-level resource usage, and it's cheap). Since I discovered them, I'm using it all the time for demos, trainings, etc. The cost is almost negligible.
Great! Another great video Viktor! And I fully agree adding the support of config files is key for its wider adoption. Keep it up! And btw, K3s has added full support for embedded etcd by default as of release v1.19.5+k3s1.
I'm thinking of using this to give my devs a local cluster that would resemble what we use in eks! Very lightweight, easy to configure, and it just works. I've had problems setting up metrics-server in Kind, but here you get it from the creation. Also, loadbalancing, ingress all works without any hassle whatsoever! Great recommendation, Victor!
Hi Victor, great video. Subscribed. For timing your commands while recording you can just use the `time` UNIX command. Example: $ time k3d cluster create .....
Hello Viktor, your videos are great as always, keep it up! I'm building my home lab. I've been doing my research and I decided to go for MicroK8s. Have you been able to see it? What do you think about it? Regards
I haven't used microk8s much. It's limited to Ubuntu and I'm always trying to find solutions that are more agnostic. That does not mean that microk8s is not good (it probably is) but that I skipped it. I would be curious however to get an independent opinion about it. Please try it out and let me know what you think.
Hey Viktor, your videos are so informative. Thank you 🤘 Is it possible to deploy it on VM and expose over internet so that the team can use it for testing.?
I used time ago, but had issues because I use ZFS on my main desktop. Tested this v4 and seems is staill have them: time="2021-04-08T20:58:40.647686221Z" level=error msg="Failed to retrieve agent config: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: invalid argument"
I never used ZFS on my desktop/laptop. Do you think it's common? If it is, I might need to add it to my list of "permutations" I use for testing my work.
@@DevOpsToolkit I think it's not common nowadays. The only easy setup is Ubuntu 20 or Linux Mint 20, have an option at install to do with ZFS and is marked as beta. I installed myself to see how ZFS is going on linux (on freeBSD worked very nice) since it was listed as supported for docker. Since last summer I only did found issues with k3d/k3s and with Tilt.dev, but this one was due to using docker + buildkit in the background to build the containre images , but improved in the latest versions.
Deploying Helm charts to k3s is the same as to any other types Kubernetes clusters. A `helm upgrade --install` or a similar command should work there just as anywhere else. The major thing to consider is Ingress. When running locally with k3d, and assuming that you set the `loadBalancer` in the k3d config (like what I did in the video), you need to make sure that your Ingress hosts are either `*` or, when you use a service like nip.io or xip.io when you need a "real" domain. In the later case, an Ingress host could be my-app.127.0.0.1.nip.io. Other than Ingress, everything else should be the same with k3d as with any other k8s cluster.
I have limited experience with metallb. From what I saw, all you need to do is fetch Traefik IP, create a confirg for metallb to use that IP, and apply the manifests.
I had trouble getting the config cluster to start up (k3d cluster create --config k3d.yaml). I kept getting a CA error. I was able to make it work with this setup:
It seems that the manifest/setup you wanted to paste got lost or that it was deleted. UA-cam tends to delete comments with links or code automatically. If you want, you can DM me the config and I'll post it. You can reach me on Twitter (@vfarcic) or LinkedIn (www.linkedin.com/in/viktorfarcic/).
@@fwungy4285 That should be enough. Unfortunately, I cannot try it out myself, at least not right away. I removed Docker from my computers a while ago and switched completely to Rancher Desktop. What I can do is install it again over the weekend and see whether there is an issue in the latest release. If it works, we'll know that the problem is not global and is limited to specific use-cases (e.g., your laptop). If that's the case, we can debug it together. Does that make sense? Is this weekend OK or that's too late?
Assuming that you are not trying something that requires more cpu and memory than what you have on your laptop, the answer is yes. The only limitation that comes to my mind is if you are trying out something that requires external access to your laptop. For example, you might not be able to trigger pipelines (e.g. tekton or Argo cd) from github since it would not be able to access your laptop. Other than that, you should be fine. P.S. there might be some other minor quirks that do not come to me right now but, in general, you should be able to do (almost) anything.
this solution doesnt actually work for mac if you intend to run a local server (like dedicated .. like with a mac mini), does it? k3s requires virtualization to run on mac, like in docker so k3d. k3d requires docker. Docker desktop requires you to _log in_ to start docker, hence, not a valid server (and, I find it _really odd_ that Docker chose to provide a /Library/LaunchDaemon instead of a /Library/LaunchAgent since all they were doing was _scheduling_ it to be run when the user logs in)
I'm not sure I would use anything but Linux for a server... The truth is that containers cannot run on macos so you need a VM one way or another. That being said, after I made that video, rancher Desktop was released and that is the one I am using on Mac. It does create a VM, but that is unavoidable.
@@robertotomas You should go to rancherdesktop.io Docker Desktop is only one of many ways to run containers. It could also be minikube which supports various virtualizations, Rancher Desktop which is based on Lima, etc. What they all have in common is that they all create Linux VM(s).
@@DevOpsToolkit thank you for highlighting the difference from just rancher! With Rancher Desktop installed, and rebooting to a different user, I do not see the kubernetes cluster running. Is there a LaunchDaemons plist script to get this to run as a system daemon so all develoeprs can benefit from it?
@@robertotomas Rancher Desktop is a "desktop" app with some things being managed through the UI. If you're looking for something that is without a UI (as you probably are since you're setting up a server), you might want to check minikube. However, since you're setting up a server using Mac and given that server needs Linux for containers to work, you might just as well ignore all the Kubernetes distributions meant to be running on laptops and go straight for a VM with Linux and k8s installed on top of it. That could be k3s, or Docker with KinD or k3d (Docker on Linux is a daemon), etc. Essentially, if it's a server, it should be a "real" k8s (even though it'll be small). ... or just use something like Civo. It's cheap.
@@DevOpsToolkit that brings me full circle to the "there's no k3s install that works out of the box on an m1 mac" :( Are you telling me that rancher desktop doesn't create the vm and k3s in a manner that I can use externally from the desktop?
Unfortunately not. It's been a long time since I worked with browser testing. At that time (probably more than 10 years ago) it was all about Selenium and headless browsers. I'm sure that the tech changed a lot since then.
That depends on where I am. In my home office, I use iMac with 64 GB RAM. When on the road, I have the new M1 MacBook with 16 GB RAM, and when I'm in the village I visit over weekends, I have the old MacBook Pro with 32 GB RAM. However, all that is not that important. What matters is that k3d is more lightweight than any other similar solution and that it hardly uses any RAM or CPU. What matters is how much CPU and RAM the applications you run inside the cluster need. Also, the limit is not set by your physical RAM and CPU but by how much you allocated to Docker Desktop (unless you're using Linux).
Hi, would it be possible to run it using podman instead of docker? And if it is possible, could someone pass links or give hints on how to do it, please.
I'm guessing no, but I haven't tried it so I might be wrong. On a side note, I stopped using k3d. I believe that Rancher Desktop is a better choice now. It's also based on k3s and it was made by the same team but, unlike k3d, it does not need Docker. Rancher Desktop is a full replacement for Docker Desktop.
Assuming that you mean to install or upgrade Helm charts into that cluster, the answer is yes :) But, before you start using k3d, please watch the videos on this channel about Rancher Desktop. I believe that it is a better option than k3d.
@@DevOpsToolkit I am provisioning virtual machines with k3d and helm for a migration project I'm working on. My team will split the repository (monorepo) in exclusive repos dor each container application. I only have 4 months of experience in devops hahah your videos are helping me a lot. Thanks!
@@DevOpsToolkit thanks for reply. But on macbook it is not accessible.. Any workaround to get ingress working on mac for k3d. Accessing ingress of k3d from Mac host
Ingress worked without any issue on localhost. However, since rancher Desktop was released, I haven't been using it so I'm not whether they something changed in k3d since then.
What do you think about k3d? Are you already using it? If you're not, do you think it's a good idea to switch to k3d?
I never used k3d. Looks worth using it for testing applications on local kubernetes. Will give it a try once. Anyway good explanation. Thanks again !
just started using it bc docker-desktop+kubernetes just SUCKS big time - with k3d we finally have an option that runs for most local dev on any local OS as long as it can somehow run docker
Docker made a huge mistake by first ignoring Kubernetes, and then by trying to create a system that can run both k8s and Swarm. Kubernetes in Docker Desktop sucks mostly because they did not focus on making it great, but in being able to run Swarm in parallel with k8s.
I ran k3d for a while, but I'm not so much a fan from a productivity point of view. I'm in the situation now where I have a few k8s clusters I can use as a playground which makes k3d/ Kind less appealing. Instead, using Argo CD with a few test namespaces has been far better for my development (btw your videos were an awesome inspiration for that setup!). k3d was cool for testing basic kube ops, but I found testing my real-world use-case would make my laptop catch on fire if it had to do real processing. k3d/kind worked okay for toy-apps/ experiments, but for testing my actual apps I found I really needed a cluster with some real horsepower.
@@flakes6992 agree - but I am only talking about CI - and test driven approaches - and for that k3d is a good one, supporting the TDD development
Finally a knowledgable channel where the creator does not ask for likes and subscriptions before even saying anything.
Who disliked this video!? The guy is awesome. Great explanation, with no faff. I love it.
very nice! great video
On a modest 2017 i5 ThinkPad with 4Gb, "k3d cluster create my-cluster" takes about 12s and uses about 400Mb. Of course that's a bare cluster with no workload, still that's amazing.
Thank you! K3d is terrific. Setting up clusters has been a bottleneck to training, but now I can scrap the Pi's I was going to use.
Yeah. K3d is great or, to be more precise, k3s is great. You might want to check out Civo as well (there's a video somewhere in my channel). They use it for their Cloud K8s offer with the same benefits. It's fast (90 sec to create a cluster), has low system-level resource usage, and it's cheap). Since I discovered them, I'm using it all the time for demos, trainings, etc. The cost is almost negligible.
Great! Another great video Viktor! And I fully agree adding the support of config files is key for its wider adoption. Keep it up! And btw, K3s has added full support for embedded etcd by default as of release v1.19.5+k3s1.
I'm thinking of using this to give my devs a local cluster that would resemble what we use in eks! Very lightweight, easy to configure, and it just works. I've had problems setting up metrics-server in Kind, but here you get it from the creation. Also, loadbalancing, ingress all works without any hassle whatsoever! Great recommendation, Victor!
Great as usual! Support of config files in k3d is something that I’ve been waiting :) Now this is really a pleasure to use k3d !
Hi Victor,
Very nice , i just started using this , and it is awesome, hats off to your work and nice demo .
great video! keep 'em coming! I'm a longtime kind user but this encouraged me to try out k3d!
Awesome video and every single min in the video is worth watching . Thanks a lot for this :)
Hi Victor, great video. Subscribed.
For timing your commands while recording you can just use the `time` UNIX command.
Example:
$ time k3d cluster create .....
Beautifully explained, thanks!
Another excellent video. Compare k3s, kind, ninikube and microk8s would be a great video.
Adding it to my TODO list
great short explanation, thanks!
Thanks!!! As usual, simple, and clear, awesome..........................
Great content keep going 👍
Nice one. Thanks for this one!
Glad you like it!
Just FYI: "INFO[]", so the second cluster took >17,
👌
Hello Viktor, your videos are great as always, keep it up! I'm building my home lab. I've been doing my research and I decided to go for MicroK8s. Have you been able to see it? What do you think about it? Regards
I haven't used microk8s much. It's limited to Ubuntu and I'm always trying to find solutions that are more agnostic. That does not mean that microk8s is not good (it probably is) but that I skipped it.
I would be curious however to get an independent opinion about it. Please try it out and let me know what you think.
Great! now run rancher and control your clusters:) , and what about harvester tutorial?
Great suggestion! Adding Harvester to my TODO list...
I saw the the cover photo of this video and thought, is that Igor Girkin, and he has a Kubernetes tutorial? Lol
Hey Viktor, your videos are so informative. Thank you 🤘
Is it possible to deploy it on VM and expose over internet so that the team can use it for testing.?
I wouldn't use k3d for that. Instead, I'd either create a k3s cluster (k3d is based on k3s) or, even better, use vCluster inside a "real" cluster.
Great video. Been using KinD for a while, now I have to look at k3s. Any thoughts on k0s?
I never got around going through it in depth so I cannot say much about it (yet). Adding it to my TODO list...
thx for this one - what do you think about keel - and would you combine keel + argo-cd for your dev-pipe CI + pre-prod/prod deployment?
I haven't used Keel. Need to get my hands dirty first before commenting. Adding it to my TODO list...
I used time ago, but had issues because I use ZFS on my main desktop. Tested this v4 and seems is staill have them:
time="2021-04-08T20:58:40.647686221Z" level=error msg="Failed to retrieve agent config: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: invalid argument"
There was a wrapper for usage ZFS docker volumes i used for v3 (first search on google)
I never used ZFS on my desktop/laptop. Do you think it's common? If it is, I might need to add it to my list of "permutations" I use for testing my work.
@@DevOpsToolkit I think it's not common nowadays. The only easy setup is Ubuntu 20 or Linux Mint 20, have an option at install to do with ZFS and is marked as beta. I installed myself to see how ZFS is going on linux (on freeBSD worked very nice) since it was listed as supported for docker. Since last summer I only did found issues with k3d/k3s and with Tilt.dev, but this one was due to using docker + buildkit in the background to build the containre images , but improved in the latest versions.
very nice content.
could you please share a video where we can deploy a helm chart using this k3s and access the services?
Deploying Helm charts to k3s is the same as to any other types Kubernetes clusters. A `helm upgrade --install` or a similar command should work there just as anywhere else.
The major thing to consider is Ingress. When running locally with k3d, and assuming that you set the `loadBalancer` in the k3d config (like what I did in the video), you need to make sure that your Ingress hosts are either `*` or, when you use a service like nip.io or xip.io when you need a "real" domain. In the later case, an Ingress host could be my-app.127.0.0.1.nip.io.
Other than Ingress, everything else should be the same with k3d as with any other k8s cluster.
Thanks for the great video. Are there any specific things I need to run metallb in k3d cluster ?
I have limited experience with metallb. From what I saw, all you need to do is fetch Traefik IP, create a confirg for metallb to use that IP, and apply the manifests.
I had trouble getting the config cluster to start up (k3d cluster create --config k3d.yaml). I kept getting a CA error.
I was able to make it work with this setup:
It seems that the manifest/setup you wanted to paste got lost or that it was deleted. UA-cam tends to delete comments with links or code automatically. If you want, you can DM me the config and I'll post it. You can reach me on Twitter (@vfarcic) or LinkedIn (www.linkedin.com/in/viktorfarcic/).
@@DevOpsToolkit I set the image to rancher/k3s:latest and servers/agents=2. I could not get 3 servers/agents to work on my mbp. :shrug:
@@fwungy4285 Maybe there wasn't enough CPU/memory. How much do you have assigned to Docker Desktop?
@@DevOpsToolkit cpu: 4 memory: 8.25gb swap: 1gb
@@fwungy4285 That should be enough.
Unfortunately, I cannot try it out myself, at least not right away. I removed Docker from my computers a while ago and switched completely to Rancher Desktop. What I can do is install it again over the weekend and see whether there is an issue in the latest release. If it works, we'll know that the problem is not global and is limited to specific use-cases (e.g., your laptop). If that's the case, we can debug it together.
Does that make sense? Is this weekend OK or that's too late?
Hello, can you please compare k3s to microk8s? thanks
It's already on my TODO list. Bumping it closer to the top... :)
Is it good for small cluster (4 nodes) in production?
It's not. Use k3s in production. K3d is just a version of k3s inside docker containers.
@@DevOpsToolkit thanks
Hi, is k3d enough to learn K8s with all its bells and whistles like CI, canary releases, chaos engineering, etc. ?
Assuming that you are not trying something that requires more cpu and memory than what you have on your laptop, the answer is yes. The only limitation that comes to my mind is if you are trying out something that requires external access to your laptop. For example, you might not be able to trigger pipelines (e.g. tekton or Argo cd) from github since it would not be able to access your laptop. Other than that, you should be fine.
P.S. there might be some other minor quirks that do not come to me right now but, in general, you should be able to do (almost) anything.
@@DevOpsToolkit thank you, Viktor!
this solution doesnt actually work for mac if you intend to run a local server (like dedicated .. like with a mac mini), does it? k3s requires virtualization to run on mac, like in docker so k3d. k3d requires docker. Docker desktop requires you to _log in_ to start docker, hence, not a valid server (and, I find it _really odd_ that Docker chose to provide a /Library/LaunchDaemon instead of a /Library/LaunchAgent since all they were doing was _scheduling_ it to be run when the user logs in)
I'm not sure I would use anything but Linux for a server...
The truth is that containers cannot run on macos so you need a VM one way or another. That being said, after I made that video, rancher Desktop was released and that is the one I am using on Mac. It does create a VM, but that is unavoidable.
@@robertotomas You should go to rancherdesktop.io
Docker Desktop is only one of many ways to run containers. It could also be minikube which supports various virtualizations, Rancher Desktop which is based on Lima, etc. What they all have in common is that they all create Linux VM(s).
@@DevOpsToolkit thank you for highlighting the difference from just rancher! With Rancher Desktop installed, and rebooting to a different user, I do not see the kubernetes cluster running. Is there a LaunchDaemons plist script to get this to run as a system daemon so all develoeprs can benefit from it?
@@robertotomas Rancher Desktop is a "desktop" app with some things being managed through the UI. If you're looking for something that is without a UI (as you probably are since you're setting up a server), you might want to check minikube.
However, since you're setting up a server using Mac and given that server needs Linux for containers to work, you might just as well ignore all the Kubernetes distributions meant to be running on laptops and go straight for a VM with Linux and k8s installed on top of it. That could be k3s, or Docker with KinD or k3d (Docker on Linux is a daemon), etc. Essentially, if it's a server, it should be a "real" k8s (even though it'll be small).
... or just use something like Civo. It's cheap.
@@DevOpsToolkit that brings me full circle to the "there's no k3s install that works out of the box on an m1 mac" :( Are you telling me that rancher desktop doesn't create the vm and k3s in a manner that I can use externally from the desktop?
Does K3d supports podman?
I don't think so. K3d is a way to run Kubernetes clusters as containers inside Docker while Podman is a replacement for Docker.
hey, do you have any suggestions about browser testing software?
Unfortunately not. It's been a long time since I worked with browser testing. At that time (probably more than 10 years ago) it was all about Selenium and headless browsers. I'm sure that the tech changed a lot since then.
@@DevOpsToolkit thank you
@@amabamo5769 Nope, still (mostly) about Selenium and headless.
@DevOps Toolkit by Viktor Farcic
Could you please share your system configurations . Like, RAM, Processor etc..
That depends on where I am. In my home office, I use iMac with 64 GB RAM. When on the road, I have the new M1 MacBook with 16 GB RAM, and when I'm in the village I visit over weekends, I have the old MacBook Pro with 32 GB RAM. However, all that is not that important. What matters is that k3d is more lightweight than any other similar solution and that it hardly uses any RAM or CPU. What matters is how much CPU and RAM the applications you run inside the cluster need. Also, the limit is not set by your physical RAM and CPU but by how much you allocated to Docker Desktop (unless you're using Linux).
Hi, would it be possible to run it using podman instead of docker? And if it is possible, could someone pass links or give hints on how to do it, please.
I'm guessing no, but I haven't tried it so I might be wrong.
On a side note, I stopped using k3d. I believe that Rancher Desktop is a better choice now. It's also based on k3s and it was made by the same team but, unlike k3d, it does not need Docker. Rancher Desktop is a full replacement for Docker Desktop.
@@DevOpsToolkit thanks for answering! Ohhh, well I might check Rancher desktop .
It's awesome and is the only one I'm using for a while now.
You might want to check out ua-cam.com/video/bYVfCp9dRTE/v-deo.html and ua-cam.com/video/evWPib0iNgY/v-deo.html
Can we use it to check helm charts?
Assuming that you mean to install or upgrade Helm charts into that cluster, the answer is yes :) But, before you start using k3d, please watch the videos on this channel about Rancher Desktop. I believe that it is a better option than k3d.
@@DevOpsToolkit I am provisioning virtual machines with k3d and helm for a migration project I'm working on. My team will split the repository (monorepo) in exclusive repos dor each container application. I only have 4 months of experience in devops hahah your videos are helping me a lot. Thanks!
Thanks for this video !
Have you try k0s (k0sprojet dot io) which is another light k8s distribution from Mirantis ?
Adding it to my TODO list...
Can you publish something on Koneveyor?
That's a good idea. Adding it to my TODO list...
How to access k3d ingress on mac
Assuming that you installed Ingress, it should be accessible on localhost.
As a side note, I suggest using Rancher Desktop instead.
@@DevOpsToolkit thanks for reply.
But on macbook it is not accessible..
Any workaround to get ingress working on mac for k3d.
Accessing ingress of k3d from Mac host
Ingress worked without any issue on localhost. However, since rancher Desktop was released, I haven't been using it so I'm not whether they something changed in k3d since then.
@@DevOpsToolkit ok
Let me try rancher desktop and let you if that resolved issue for me
Thanks again for your response
Make a video and compare ingress controllers.?
Good idea. Adding it to my todo list.