You're an excellent teacher Marcel, thank you for making these videos. Your vlog format is groundbreaking in tech. While there are plenty of others on youtube offering fantastic content, you've introduced (at least to me) an element that has made it so much easier to take the information in. Well done man!
Man this is just awesome, I'm beating myself for not understanding, after watching this I can assure myself I got the idea I remember this quote, You can't connect the dots forward - only backward - Steve Jobs Thank you for the lecture marcel, You made my day.
Hi Marcel Thank you so much for this simple explanation with a REAL WORLD example of persistent volumes. I have been wracking my brains trying to tie together pv, pvc, and statefulsets. You explained and demonstrated in 20 minutes what other people couldn't get across in hours of demos. Love your channel, keep it up mate 👌👍🙂
Bro out here making every DevOps engineers life so much easier, thanks the school system around the globe should take you as an example on how to transfer knowledge....
Marcel, your stateful and persistance videos demos are brilliant, thank you for putting in the work to share your knowledge, your delivery is brilliant
Good video generally, but there's a misconception about containers. A container is not a process, it's a sandboxed runtime environment in which you can run one or more processes. You can stop a container (`docker stop `) which terminate all processes inside of a container and then run it again (`docker start `) and the files will still be there. Kubernetes never bothers to restart a container - when it restarts a pod it recreates containers with the same IP address assigned, etc. so you might think that containers are completely stateless (and they're typically treated as such) but a container keeps its filesystem state after all of its processes terminate which is pretty useful when it comes to retrieving logs and inspecting what caused the termination.
Marcel you're a good man. You always explain things very clearly and dough you type as fast as the light you repo in GitHub shows everything. So many many thanks. I was stuck trying to bind a local PV to a mongo db StatefulSet , looking at your code(dough for Postgres ) made me find a way to debug my pv.yaml.. it results that was a typo in the nodeAffinity key.. thank you so much again. Vinny
Thanks a lot, Marcel this is the best video about the concept that I've found on UA-cam, your explanation coming from docker volumes creates a solid ground to easily understand the kubernetes part :smile:
Thank you so much for this. I’ve been in class all week and studying my ass off and just have NOT understood the pv claim until now. Now I can get some sleep. Lol
Thanks alot Marcel, I finally understood the concepts of PV, PVC and how to use them with Postgres thanks to you. Massive thanks! Also I would like to request if you could teach how to do authentication and authorizations from nginx-ingress using Keycloak, as I am sure many people struggle with it... And there arent any UA-camrs like you, who teach from the basics and build up concepts upon that, rather they usually just assume that we just know stuff. Also, since KeyCloak is really popular Auth as a Service, so I am sure it would be very helpful for others too, so please consider doing a video on it, please. Thanks a lot in advance.
Great Video, breaking down the content in ingestible bytes. One improvement I'd suggest is timestamping your video so that people can skip ahead to the exact part they are looking for. Keep doing you man your content is great
How does it work, then, if I have multiple apps that need to share the same physical storage. From what it sounds, I still need to create a pv for all of them, but can mount them to the same physical path in the storage class, nfs for example. But then how does the storage amount come into play? How can a pv reliably tell a claim “hey, I have this much storage available, use me.” If there is overlap in the physical storage?
Hi Marcel and thanks you for the great job you do! Can you give a hint or direction please. I've deployed wp with pvc using nfs-provisioner on bare-metal. nfs share is located on the neighbouring vm host in the same subnet and basically both k8s nodes and nfs node use the same physical disk. What I see is that i can easily store anything on this share manually from any pod but WP + mysql is very slow when using this nfs (e.g. it take up to 15 sec to read a few KB files). perms on the share are 777. Access mode readwritemany if it makes any difference. I've googled for this issues and it turns out quite common but there is no specific solution or reason that I could find. Any advices please?
Hi, i got a qusetion, if my volume is stored application's log file, this volume should be oversized, how could i logrotate in this volume to manage the log file size? do u have any suggestion? thank u so much
I would recommend looking into Fluentd for shipping logs and not store logs in a volume. K8s has pod logs go to a container runtime folder and Fluentd mounts that directory to forward logs out. The container runtime is designed to rotate logs automatically. ua-cam.com/video/6kmHvXdAzIM/v-deo.html
What is the best way to setup storage server and its configuration for kubernetes cluster persistence volume for production system? NFS or ISCASI any hardware brand specific. Thanks !
Hi Gaurav, is this end of K8s series, please add if few more topics pending. and i must saym your tutorial made my concep clear and handson experianced.
but If we have a three node cluster one is master other two are worker first time our Postgres pod is schedule on worker1 now its PV on worker1 and second time after deleting the pod k8s schedule it on Worker2 now pod create its PV on Worker2 but now data is not persistence. because data on worker1 can not allocate to worker2 pod? What's a solution?
This is because you are using a deployment with PVs which is not meant for stateful workloads. You'll need to learn about StatefulSets ua-cam.com/video/zj6r_EEhv6s/v-deo.html
Hi Marcel, I've created a PersistentVolume using the hostpath StorageClassName and I mounted it in a mysql pod. The Persistent Volume hostpath is "/var/folders/data", and it's mounted in "/var/lib/mysql" in the Container. What's strange is that when I list the PV hostpath "/var/folders/data" it's empty. But when I create a test database and then delete and recreate the deployment the test database is still there. You know where the container "/var/lib/mysql" files could be found? I was expecting to find them in the PV hostpath "/var/folders/data". I'm using docker for mac.
Thanks! I have a single tennet app that will have multiple copies of the same pod running in different namespaces with different persistant volume claims. Many of these resources have the same name, but are just deployed in different namespaces. It would be great to see a video about how a single pod .yaml could be used in that way to create different volumes.
Thank you, the guide was very clear and concise. Do you have a guide on how to handle a "real world" project in the different phases? For example I imagine one would develop in local on minikube, then deploy on the cloud (AWS, GCP, etc), maybe having TEST/PRODUCTION as different clusters. How do you manage all this? Does it make sense to have in the same git repo different kubernetes folders, one for each environment, containing the appropriate yml? Or would you rather have different branches? A guide on this would be great.
Hi Thanks you for great video.! I did the same and tried to deploy jenkins but when i create pv and pvc . mnt/data is empty. and i used volumemount to var/jenkins_home.
For logging, I have a separate video in the pipeline, You would rather export logs from the app and not store it in a volume. Take a look at the EFK\ELK concepts, where basically every pod in K8s writes logs to stdout. It is best practise not to write logs to file and rather use stdout\stderr. These logs will automatically end up in the host docker logs which you can push out of the system using Fluentd\FluentBit. This allows you to centralise logs for all deployments\statefulsets running in your cluster under something like Elastic Search or 3rd party application.
Hey Marcel, love the videos! I think there is a small mistake with your explanation on PVCs claiming storage from PVs. You cannot have several PVCs bound to one PV, once a PVC is bound to a PV the PV will not be able to bind to another PVC but many pods can use the same PVC. You create PVs to configure a certain way to claim storage and to use it you bind a PVC to it. If you aren't using PVs the storage class dynamically creates and provisions a PV for the requested PVC. Keep up the great content!
Great video Marcel! However, I do have a question. I am able to persist data through the lifetime of the container / pod, but not able to do so if I restart my cluster. How do I achieve this? I am planning to use local storage class
Hi Marcel, my setup is a bit unique, I have created a K8S cluster on a few VMs in our own private cloud and all the nodes are mounted with a single NFS drive (say /data), can I define a storageclass of hostpath type using /data? Does PV has to be mounted to all the nodes?
That depends on the app you're planning to run and how it uses the storage. If you plan on using a statefulset its recommended that each pod has its own storage. StatefulSet will use a storage class to provision a PV per pod and not share the same storage. Hostpath will work, but if you mount a shared storage to all the nodes, they may override each other's data. So unless you mount unique storage to each node, it may not be ideal. I'm new to NFS, but i would see if its possible to provision NFS dynamically using a storage class and see if the statefulset can issue PV + PVCs per pods using NFS dynamic provisioning. That way each pod will get its own NFS storage provisioned by k8s when you scale up and down
Checkout NFS provisioning using Rook rook.io/docs/rook/v1.4/nfs.html Seems interesting. If you have not worked with Statefulsets I'd highly recommend to ua-cam.com/video/zj6r_EEhv6s/v-deo.html
Hey Marcel, great videos. After I altered the ConfigMap to use a different POSTGRES_USER and deleted/recreated everything including the PV, the old database user & table existed. Can you explain why that happened even after I deleted the PV? Is there more to "scrub" here? I assumed recreating everything would result in a fresh database with the new user credentials. Thanks!
This highly depends on the storage class you use. In this demo I use host path which means data is persisted on the node at a path (which can be changed). The get a fresh storage, you either have to change the path to get a new folder on the host. Remember your old data would remain. Or SSH to the node, clean it up, Or run another container with a host mount and carefully run "rm" command to delete the content
@@MarcelDempers Thank you for such a quick response! I used hostPath/your tutorial code too. If you have any advice on ssh-ing to the docker-desktop (Mac) node to clean up, I'm all ears. Thanks again for these videos, very helpful.
Understanding StatefulSets for beginners👉🏽 ua-cam.com/video/zj6r_EEhv6s/v-deo.html
You're an excellent teacher Marcel, thank you for making these videos. Your vlog format is groundbreaking in tech. While there are plenty of others on youtube offering fantastic content, you've introduced (at least to me) an element that has made it so much easier to take the information in. Well done man!
Man this is just awesome, I'm beating myself for not understanding, after watching this I can assure myself I got the idea
I remember this quote,
You can't connect the dots forward - only backward
- Steve Jobs
Thank you for the lecture marcel, You made my day.
Hi Marcel
Thank you so much for this simple explanation with a REAL WORLD example of persistent volumes. I have been wracking my brains trying to tie together pv, pvc, and statefulsets. You explained and demonstrated in 20 minutes what other people couldn't get across in hours of demos. Love your channel, keep it up mate 👌👍🙂
Possibly the best tutorial on the topic!!
Bro out here making every DevOps engineers life so much easier, thanks the school system around the globe should take you as an example on how to transfer knowledge....
You teach in a clear and easy to unsertand manner, really important to learn about storage classes.
Marcel, your stateful and persistance videos demos are brilliant, thank you for putting in the work to share your knowledge, your delivery is brilliant
Great explaination!! After watching at least 10 other tutorials, yours was the first one that really worked!
You're on another level mate, so concise, so clever, so well spoken, love your videos
Thanks Marcel! Would really love to see one on Argo Workflows/Events for setting up and triggering pipelines in kubernetes!
You are too good at this, your videos have helped me understand and breakdown those heavy jargons on K8s doc pages. Thank you.
You sir are amazing! Love your style and your editing and your teaching ability. You're like the Kubernetes god! Keep up the great work!
Good video generally, but there's a misconception about containers. A container is not a process, it's a sandboxed runtime environment in which you can run one or more processes. You can stop a container (`docker stop `) which terminate all processes inside of a container and then run it again (`docker start `) and the files will still be there. Kubernetes never bothers to restart a container - when it restarts a pod it recreates containers with the same IP address assigned, etc. so you might think that containers are completely stateless (and they're typically treated as such) but a container keeps its filesystem state after all of its processes terminate which is pretty useful when it comes to retrieving logs and inspecting what caused the termination.
this devops guy LIFTS!!
💪🏽
This video is almost as good as you triceps. Respect
Marcel you're a good man. You always explain things very clearly and dough you type as fast as the light you repo in GitHub shows everything. So many many thanks. I was stuck trying to bind a local PV to a mongo db StatefulSet , looking at your code(dough for Postgres ) made me find a way to debug my pv.yaml.. it results that was a typo in the nodeAffinity key.. thank you so much again. Vinny
excellent and well detailed! thank you for the explanation dude!
Thanks a lot, Marcel this is the best video about the concept that I've found on UA-cam, your explanation coming from docker volumes creates a solid ground to easily understand the kubernetes part :smile:
@13:40, How is the PersistentVolumeClaim referencing the PeristentVolume (example-volume) ? How does the pvc know which PV to use?
Thank you so much for this. I’ve been in class all week and studying my ass off and just have NOT understood the pv claim until now. Now I can get some sleep. Lol
Thanks for this amazing video, I was really struggling with this part in Kubernetes. It helped me a lot :)
Just Amazing ! Thank you very much man ! Keep Going !
Thanks alot Marcel, I finally understood the concepts of PV, PVC and how to use them with Postgres thanks to you. Massive thanks!
Also I would like to request if you could teach how to do authentication and authorizations from nginx-ingress using Keycloak, as I am sure many people struggle with it... And there arent any UA-camrs like you, who teach from the basics and build up concepts upon that, rather they usually just assume that we just know stuff. Also, since KeyCloak is really popular Auth as a Service, so I am sure it would be very helpful for others too, so please consider doing a video on it, please. Thanks a lot in advance.
Thank you for this video, please make a video on the production-ready Postgres database.
Great Video, breaking down the content in ingestible bytes. One improvement I'd suggest is timestamping your video so that people can skip ahead to the exact part they are looking for. Keep doing you man your content is great
how would I add nodeselectos and affinity rules for statefulsets?
How does it work, then, if I have multiple apps that need to share the same physical storage. From what it sounds, I still need to create a pv for all of them, but can mount them to the same physical path in the storage class, nfs for example.
But then how does the storage amount come into play? How can a pv reliably tell a claim “hey, I have this much storage available, use me.” If there is overlap in the physical storage?
Thanks for the great content Marcel
Hi Marcel and thanks you for the great job you do!
Can you give a hint or direction please. I've deployed wp with pvc using nfs-provisioner on bare-metal. nfs share is located on the neighbouring vm host in the same subnet and basically both k8s nodes and nfs node use the same physical disk. What I see is that i can easily store anything on this share manually from any pod but WP + mysql is very slow when using this nfs (e.g. it take up to 15 sec to read a few KB files). perms on the share are 777. Access mode readwritemany if it makes any difference. I've googled for this issues and it turns out quite common but there is no specific solution or reason that I could find. Any advices please?
Hi, i got a qusetion, if my volume is stored application's log file, this volume should be oversized, how could i logrotate in this volume to manage the log file size? do u have any suggestion? thank u so much
I would recommend looking into Fluentd for shipping logs and not store logs in a volume.
K8s has pod logs go to a container runtime folder and Fluentd mounts that directory to forward logs out.
The container runtime is designed to rotate logs automatically.
ua-cam.com/video/6kmHvXdAzIM/v-deo.html
@@MarcelDempers thank u so much! its useful to a kubernetes beginner user.
Thanks for sharing. Gives me better understading than k8s doc.
Thank you kindly for this video. One question though. Why would a pvc specifies a storage class? isn't it already defined in a pv?
What is the best way to setup storage server and its configuration for kubernetes cluster persistence volume for production system? NFS or ISCASI any hardware brand specific. Thanks !
You are really good at explaining things. Can you do a tutorial on tricep extensions? Thanks!
super underrated comment lol
Best in class 🙌✌️
Your channel is a godsend 🙏🙏🙏
Suffering for the past week figuring out how to persist data on a database container has finalldy led me here to find the solution...
Hi Gaurav, is this end of K8s series, please add if few more topics pending.
and i must saym your tutorial made my concep clear and handson experianced.
Excellent video It helped me to clear my concept Thanks.
Every sec has information in your videos.
but If we have a three node cluster one is master other two are worker first time our Postgres pod is schedule on worker1 now its PV on worker1 and second time after deleting the pod k8s schedule it on Worker2 now pod create its PV on Worker2 but now data is not persistence. because data on worker1 can not allocate to worker2 pod? What's a solution?
This is because you are using a deployment with PVs which is not meant for stateful workloads. You'll need to learn about StatefulSets
ua-cam.com/video/zj6r_EEhv6s/v-deo.html
The local file system path you specified, you would see files in there after creating the table right?
Hi Marcel, I've created a PersistentVolume using the hostpath StorageClassName and I mounted it in a mysql pod. The Persistent Volume hostpath is "/var/folders/data", and it's mounted in "/var/lib/mysql" in the Container. What's strange is that when I list the PV hostpath "/var/folders/data" it's empty. But when I create a test database and then delete and recreate the deployment the test database is still there. You know where the container "/var/lib/mysql" files could be found? I was expecting to find them in the PV hostpath "/var/folders/data". I'm using docker for mac.
Thanks! I have a single tennet app that will have multiple copies of the same pod running in different namespaces with different persistant volume claims. Many of these resources have the same name, but are just deployed in different namespaces. It would be great to see a video about how a single pod .yaml could be used in that way to create different volumes.
Thank you, the guide was very clear and concise.
Do you have a guide on how to handle a "real world" project in the different phases? For example I imagine one would develop in local on minikube, then deploy on the cloud (AWS, GCP, etc), maybe having TEST/PRODUCTION as different clusters. How do you manage all this? Does it make sense to have in the same git repo different kubernetes folders, one for each environment, containing the appropriate yml?
Or would you rather have different branches? A guide on this would be great.
Very informative
where is /var/lib/postgresql/data located? Is this location inside pod?
Hi Thanks you for great video.! I did the same and tried to deploy jenkins but when i create pv and pvc . mnt/data is empty. and i used volumemount to var/jenkins_home.
If you need help with Jenkins checkout my CI/CD playlist, there are two Jenkins guides with source code for you 💪🏽👊🏽
The tutorial doesn't work on my end. Keeps getting /var/lib/postgresql/data invalid permission.
Im running docker with k8s on windows 11
how to backup and restore database in this case ?
Hello. I have a question. What if you would want to save and mount the logs of a server application? Should you still use deployment or statefulset?
For logging, I have a separate video in the pipeline, You would rather export logs from the app and not store it in a volume. Take a look at the EFK\ELK concepts, where basically every pod in K8s writes logs to stdout. It is best practise not to write logs to file and rather use stdout\stderr. These logs will automatically end up in the host docker logs which you can push out of the system using Fluentd\FluentBit. This allows you to centralise logs for all deployments\statefulsets running in your cluster under something like Elastic Search or 3rd party application.
@@MarcelDempers oh. interesting. I'll look it up! Thank youuu so much. been watching your videos. so much helpful for me. great videos btw :)))
Hey Marcel, love the videos! I think there is a small mistake with your explanation on PVCs claiming storage from PVs. You cannot have several PVCs bound to one PV, once a PVC is bound to a PV the PV will not be able to bind to another PVC but many pods can use the same PVC. You create PVs to configure a certain way to claim storage and to use it you bind a PVC to it. If you aren't using PVs the storage class dynamically creates and provisions a PV for the requested PVC. Keep up the great content!
very good video, tks Marcel.
Great video Marcel!
However, I do have a question.
I am able to persist data through the lifetime of the container / pod, but not able to do so if I restart my cluster. How do I achieve this?
I am planning to use local storage class
Great video!
Hi Marcel, my setup is a bit unique, I have created a K8S cluster on a few VMs in our own private cloud and all the nodes are mounted with a single NFS drive (say /data), can I define a storageclass of hostpath type using /data? Does PV has to be mounted to all the nodes?
That depends on the app you're planning to run and how it uses the storage. If you plan on using a statefulset its recommended that each pod has its own storage. StatefulSet will use a storage class to provision a PV per pod and not share the same storage. Hostpath will work, but if you mount a shared storage to all the nodes, they may override each other's data. So unless you mount unique storage to each node, it may not be ideal. I'm new to NFS, but i would see if its possible to provision NFS dynamically using a storage class and see if the statefulset can issue PV + PVCs per pods using NFS dynamic provisioning. That way each pod will get its own NFS storage provisioned by k8s when you scale up and down
Checkout NFS provisioning using Rook rook.io/docs/rook/v1.4/nfs.html
Seems interesting.
If you have not worked with Statefulsets I'd highly recommend to ua-cam.com/video/zj6r_EEhv6s/v-deo.html
@@MarcelDempers thanks Marcel, I have deployed Rook Ceph RBD in my cluster and now the applications now works like charm
Hey Marcel, great videos. After I altered the ConfigMap to use a different POSTGRES_USER and deleted/recreated everything including the PV, the old database user & table existed.
Can you explain why that happened even after I deleted the PV? Is there more to "scrub" here?
I assumed recreating everything would result in a fresh database with the new user credentials. Thanks!
This highly depends on the storage class you use. In this demo I use host path which means data is persisted on the node at a path (which can be changed).
The get a fresh storage, you either have to change the path to get a new folder on the host. Remember your old data would remain. Or SSH to the node, clean it up, Or run another container with a host mount and carefully run "rm" command to delete the content
@@MarcelDempers Thank you for such a quick response! I used hostPath/your tutorial code too. If you have any advice on ssh-ing to the docker-desktop (Mac) node to clean up, I'm all ears. Thanks again for these videos, very helpful.
thanks a lot , very informative and useful
Why does the table already exist @4:45 if you just created the volume. I'm guessing you edited out the part when you first created it.
Greate, thanks a lot for this video it cant be better
Please a video on storage class for aws
Thanks brother
You're great. Good job!
great video thank you
awesome !!
That Brilliant Guy who is Awesomeeeeee.....
HERO!
Thank you
Hey Marcel, once again thanks for another excellent video. One question though. Would the Postgres data survive a Docker Desktop restart?
Technically, it should since the example used host path persistence so unless the host is destroyed, it should persist yes
@@MarcelDempers Hey Macel, thanks for the video. unfortunately the data being lost when docker restarts.
@@MarcelDempers data loss after restarting the docker desktop
Thx Great Video 👍👍👍👍
Excellent
Perfect thanks! Super .......
You requested for 50mb you got 1gb :P :D
I believe capacity stated on the PVC is that of the PV and not the requested of the claim :)
Do people really prefer the background music ? Maybe you should poll your subscribers, just distracting.
Great video!