you definitely slowed down from the last time i saw your videos! before i had to change the playback speed but now it's perfect for me lol keep up the good work Marcel
Hey marcel, great video, really appreciate your efforts 👍🏼👍🏼, i think you should take this prom setup to next level by creating a detailed video on Thanos and it’s components( humble suggestion). 🙂
Would be great to see a video on how you are using prometheus, grafana, and alert manager for visualizations and alerting. I am assuming you are customizing the system you generated in this video, to some degree. It would also be great to see how you are monitoring workloads. Is it just as simple as creating service monitors?
I only customise verbosity of what metrics are sent out using Prometheus remote write. I dont store metrics in-cluster and send it to a hosted platform, so i dont have to deal with data retention, state and availability. Other than that no customisation is needed since all cluster metrics work out the box. Combine this with fluentd for app logs and developers have all they need to monitor. For custom metrics, only service monitors are needed, yes
Muchas gracias por explicar tan claramente tantas cosas indispensables para entender cómo trabajar con estas tecnologías tan nuevas, complejas e impresionantes
besides blackbox exporter is there an exporter that we can use to get the performance of the applications? im used to Dynatrace and NewRelic APMs to get latency of functions being called but im not sure what to use to get this data inside the containers apps
It's a little more complex than dropping NewRelic in there and it's a different type of architecture. Take a look at Jaeger Tracing. ua-cam.com/video/idDu_jXqf4E/v-deo.html
Thanks Marcel, this video helped to get prometheus installed. But now how do I monitor my custom apps in their namespaces? I would like to start with just a basic scraping.
Hey, what do you think of using the annotations on a Pod/Service/Deployment etc to tell Prometheus where and if it should scrape? Any downsides to this over the ServiceMonitors?
Hey Marcel, I followed this guide to run prometheus & grafana in k3s with an ingress instead of port-forward and I did not have to do the datasources fix, it just works. Not sure if they have fixed it or because I'm running it in k3s.
hi hi, I deployed the 0.9 version ... my EKS cluster is still 1.21... so it seems the data source problem does not exist... ;) but now neither does any of the dashboards... any ideas? do yo maybe have a video that goes after this, that addresses how to add more selectors... trying to instrument a Python Flask app deployed on EKS. thanks
Great video! One question , does these manifests needs to be deployed on same namespace as our kubernetes cluster, if not then how does it know to look into namespace into which kubernetes resources are?
I wonder that Which one in the monitor shows it can get the network metrics ? Cpu and ram is normal but how about the network ? How this monitor operator get the network metric?
You can persist data in prometheus by using a persistent volume, you can also persist data in Grafana I believe (have not looked into it) Grafana Loki can definitely persist data too
My prometheus pods are using increasingly more RAM up to the point it kills OOMKilled and stop monitoring until i restart the pods, is there any way to minimize this? I was wondering if i add some persistency to the deployment it would do any good
As far as i know by default, Prometheus is an in-memory database and persistence is only to write data to disk to prevent loss during restarts You may need to research config options if its possible to offload some data to disk to reduce memory usage. Alternatively you can shard by running more Prometheus instances (one per namespace etc) to reduce it too
Hi Marcel What is difference Prom+Grafana vs Kubernetes Dashboard if we want to monitor k8s cluster? Or in which scenarios Prom+Grafana should be used over Kubernetes Dashboard
The kubernetes dashboard mainly relies on basic API metrics used by HPA for example. They are highly aggregated and limited, I.E memory+cpu only. Prometheus solution provides way more in depth telemetry via kube-state-metrics
Love the video... making me think of changing my setup, this looks so much more stitched together... Any chance you can do a similar video, but where the Prometheus and Grafana is off the K8S cluster, imagine a environment with multiple K8S clusters, where Prometheus and Grafana is #1 on a dedicated cluster or where #2 Prometheus & Grafana are on a dedicated EC2 hosts. PS: you did not mention where the prometheus.yaml file is stored to inform Prom about off cluster targets.
You'll need to run a prometheus instance in every cluster for the service monitors to work, and use remote-write to push telemetry out to your central EC2 Prometheus.
@@MarcelDempers hi hi, was starting to think the same, was actually thinking Thanos might be a good way of pulling it together onto a central Prometheus/Grafana stack. With EKS and multiple AZ's the persistent storage etc of course becomes a much more interesting discussion, I got one cluster, spanning 3 subnets, A-App, B-Database, C-Management, and these are then spread over the 3 AZ's. so for the prometheus we want to pin them to Management in A, Management in B, and so on...
@@MarcelDempers hi, can you share your experience on how to configure remote-write in prometheus inside k8s cluster and get metircs on another central prometheus?
Nothing fancy, I have an Intel i9 CPU, any i7 would do. Memory is key, especially running bunch of containers, virtual machines and k8s clusters locally. I have 32GB which is more than enough. And SSD, i have a 1TB Samsung SSD (one of those tiny PCI-exp ones😁 )
I've dug into this before and Windows is a little tricky at the moment since the kubelet and kube-state-metrics do not provide similar metrics as linux pods. I.E they dont export the same metrics. However - metrics-server (used by autoscalers) does get CPU and memory stats for Windows pods. You can use metrics-server and look on github for a project called metric-server-exporter to export pod cpu and memory. Add a service monitor for that and you can make a custom dashboard for it. That should give you observability into windows workloads for now until better support lands
Kubernetes still supports docker images. The deprecation is about the runtime on the kubelet. Kubernetes will use containerd going forward as a runtime for containers and will not use docker as a runtime. Docker images are still a standard supported by containerd, so your pods will and can still use docker images. kubernetes.io/blog/2020/12/02/dockershim-faq/
interesting! i just noticed that. Need to learn what it means. (never used server-side apply) Thanks for pointing it out. Learning something new every day
@@MarcelDempers ty for answer, seem to me my problemm more deeper)) 1. Install from kuberspray 2. k8s 1.24 3. (major cause) CRI wcih I m using in my cluster is a DOCKER, I think DOCKER major cause, ty again
An comment about the ¨bug" on the datasource, its because of those stupid Network Policies they put in the manifests, I particularly just remove it all, so you also will not have problems exporting grafana to a load balancer.
As someone who has been working in Kubernetes full time for the last 6 months, this video is a great learning source. Thanks Marcel.
what does working with Kubernetes full time look like ?
Thanks a lot Marcel. your video are loaded with useful content. No fillers.
As always Big Thanks Marcel. You Sir, are a master at explaining K8s and other stuff.
you definitely slowed down from the last time i saw your videos! before i had to change the playback speed but now it's perfect for me lol keep up the good work Marcel
A big thank you, Marcel! 🙏🏻 Your hard work is much appreciated! 🥇
As always, awesome video! Very helpful!
Wonderful Video. This helps to learn a lot. Your support is awesome.
Hey marcel, great video, really appreciate your efforts 👍🏼👍🏼, i think you should take this prom setup to next level by creating a detailed video on Thanos and it’s components( humble suggestion). 🙂
THanosss let's do it 😃
Quick, succulent, concise. 10/10 , thank you! 👍
Good Better Best - Never Let Them Rest - Thank you.
Thank you so much marcel, this is very informative
This video is pure gold 😀 Thanks a lot
This was fantastic! It would be great if you could do a video on how to integrate Datadog with kubernetes
Great video. Already looking forward to a follow up.
A big thank you it is working .
Kube Prometheus is awesome project. I wish they maintained official Helm chart. Working with vanilla yaml files isn't scalable in the long term.
Would be great to see a video on how you are using prometheus, grafana, and alert manager for visualizations and alerting. I am assuming you are customizing the system you generated in this video, to some degree.
It would also be great to see how you are monitoring workloads. Is it just as simple as creating service monitors?
I only customise verbosity of what metrics are sent out using Prometheus remote write. I dont store metrics in-cluster and send it to a hosted platform, so i dont have to deal with data retention, state and availability.
Other than that no customisation is needed since all cluster metrics work out the box. Combine this with fluentd for app logs and developers have all they need to monitor. For custom metrics, only service monitors are needed, yes
@@MarcelDempers thanks, man. Forgot to mention this was another great video!
Another great video, thanks
Muchas gracias por explicar tan claramente tantas cosas indispensables para entender cómo trabajar con estas tecnologías tan nuevas, complejas e impresionantes
Nice video. What if I don't want to use port-forwarding? What are my options?
Hello everyone! Do any of you know if it's possible to run all these pods on the worker nodes?
Thanks in advance
besides blackbox exporter is there an exporter that we can use to get the performance of the applications? im used to Dynatrace and NewRelic APMs to get latency of functions being called but im not sure what to use to get this data inside the containers apps
It's a little more complex than dropping NewRelic in there and it's a different type of architecture. Take a look at Jaeger Tracing.
ua-cam.com/video/idDu_jXqf4E/v-deo.html
Great, but how to I use Prometheus to monitor the servers outside the K8S cluster?
Thanks Marcel, this video helped to get prometheus installed. But now how do I monitor my custom apps in their namespaces? I would like to start with just a basic scraping.
Great video keep it up
Hey , love the way you explain . Just wanted to check are there any plans to create a video on Node problem detector setup and usage?
really helpful! Save me from tons of docs
What do you consider to be the most useful/important metrics to be monitored in a K8s cluster?
Hey, what do you think of using the annotations on a Pod/Service/Deployment etc to tell Prometheus where and if it should scrape? Any downsides to this over the ServiceMonitors?
What, if anything, are other viewers using to store data collected by Prometheus? TIA
Hey Marcel, I followed this guide to run prometheus & grafana in k3s with an ingress instead of port-forward and I did not have to do the datasources fix, it just works. Not sure if they have fixed it or because I'm running it in k3s.
Hey, new to k3s, could you share the ingress?
When I try to do the port forwarding part, it just times out. Not sure why. It is a remote cluster on digital ocean.
could you please tell the promql expression to monitor POD cpu and memory utilization.
great video, THX !!! :)
... 2nd ask... how do I add new targets... aka labels to scrape ? someone I missed this.
hi hi, I deployed the 0.9 version ... my EKS cluster is still 1.21... so it seems the data source problem does not exist... ;) but now neither does any of the dashboards... any ideas? do yo maybe have a video that goes after this, that addresses how to add more selectors... trying to instrument a Python Flask app deployed on EKS. thanks
Great video! One question , does these manifests needs to be deployed on same namespace as our kubernetes cluster, if not then how does it know to look into namespace into which kubernetes resources are?
Hey, Need one help where I can configure my slack channel to shoot alert messages and how I can bind port permanently.
I wonder that Which one in the monitor shows it can get the network metrics ? Cpu and ram is normal but how about the network ? How this monitor operator get the network metric?
can you pls add one session on Grafana alerts for kubernetes
17:10 Is Loki also a solution for persistent log writing?
You can persist data in prometheus by using a persistent volume, you can also persist data in Grafana I believe (have not looked into it)
Grafana Loki can definitely persist data too
@@MarcelDempers I am learning Loki next.
Can you make videos on how a python application can be monitored using Prometheus + Grafana + Loki?
Hey Marcel, thanks for the video!! With this solution on your source code i can visualize latency of the nodes on grafana?
why not use the helm chart for prometheus?
My prometheus pods are using increasingly more RAM up to the point it kills OOMKilled and stop monitoring until i restart the pods, is there any way to minimize this? I was wondering if i add some persistency to the deployment it would do any good
As far as i know by default, Prometheus is an in-memory database and persistence is only to write data to disk to prevent loss during restarts
You may need to research config options if its possible to offload some data to disk to reduce memory usage.
Alternatively you can shard by running more Prometheus instances (one per namespace etc) to reduce it too
Hi Marcel
What is difference Prom+Grafana vs Kubernetes Dashboard if we want to monitor k8s cluster?
Or in which scenarios Prom+Grafana should be used over Kubernetes Dashboard
The kubernetes dashboard mainly relies on basic API metrics used by HPA for example. They are highly aggregated and limited, I.E memory+cpu only.
Prometheus solution provides way more in depth telemetry via kube-state-metrics
Love the video... making me think of changing my setup, this looks so much more stitched together...
Any chance you can do a similar video, but where the Prometheus and Grafana is off the K8S cluster, imagine a environment with multiple K8S clusters, where Prometheus and Grafana is #1 on a dedicated cluster or where #2 Prometheus & Grafana are on a dedicated EC2 hosts.
PS: you did not mention where the prometheus.yaml file is stored to inform Prom about off cluster targets.
You'll need to run a prometheus instance in every cluster for the service monitors to work, and use remote-write to push telemetry out to your central EC2 Prometheus.
@@MarcelDempers hi hi, was starting to think the same, was actually thinking Thanos might be a good way of pulling it together onto a central Prometheus/Grafana stack.
With EKS and multiple AZ's the persistent storage etc of course becomes a much more interesting discussion,
I got one cluster, spanning 3 subnets, A-App, B-Database, C-Management, and these are then spread over the 3 AZ's. so for the prometheus we want to pin them to Management in A, Management in B, and so on...
@@MarcelDempers hi, can you share your experience on how to configure remote-write in prometheus inside k8s cluster and get metircs on another central prometheus?
Can you share your system configuration of your demo machine. For smooth experience of trying and learning your lessons 🙂
Nothing fancy, I have an Intel i9 CPU, any i7 would do.
Memory is key, especially running bunch of containers, virtual machines and k8s clusters locally.
I have 32GB which is more than enough.
And SSD, i have a 1TB Samsung SSD (one of those tiny PCI-exp ones😁 )
Hey bro why don't you show grafana dashboard
What's your approach for Windows nodes?
I've dug into this before and Windows is a little tricky at the moment since the kubelet and kube-state-metrics do not provide similar metrics as linux pods. I.E they dont export the same metrics.
However - metrics-server (used by autoscalers) does get CPU and memory stats for Windows pods. You can use metrics-server and look on github for a project called metric-server-exporter to export pod cpu and memory. Add a service monitor for that and you can make a custom dashboard for it.
That should give you observability into windows workloads for now until better support lands
thank you
a very newbie question: Kubernetes still suports Docker after all?
Kubernetes still supports docker images.
The deprecation is about the runtime on the kubelet. Kubernetes will use containerd going forward as a runtime for containers and will not use docker as a runtime.
Docker images are still a standard supported by containerd, so your pods will and can still use docker images.
kubernetes.io/blog/2020/12/02/dockershim-faq/
@@MarcelDempers Thanks!!
Why did you do just create and not server-side apply ?
interesting! i just noticed that. Need to learn what it means. (never used server-side apply) Thanks for pointing it out. Learning something new every day
Devops)) need help grafana show no DATA for PODs(cpu network, ram) physical servers monitorungs fine.... where can be problem?
make sure your helm chart you are using matches the compatibility matrix and your cluster version
@@MarcelDempers ty for answer, seem to me my problemm more deeper)) 1. Install from kuberspray 2. k8s 1.24 3. (major cause) CRI wcih I m using in my cluster is a DOCKER, I think DOCKER major cause, ty again
Please show us how to monitor ephemeral and short lived jobs with push gateway!
Do you hv any udemy course ?
Good content.
1 suggestion to put camera infront of you, so don't need to record at such weird angle.
An comment about the ¨bug" on the datasource, its because of those stupid Network Policies they put in the manifests, I particularly just remove it all, so you also will not have problems exporting grafana to a load balancer.
Can devops engineer earns 150k per year after 3 years of experience?
Anything it's possible! Just search for a new job 🥳🥳🥳
@@whiteorchid2023 Thanks sir