Great teaching ability, thanks a lot for all amazing & high quality content, please make videos regarding eks & gke logging solutions with grafana loki or with elasticsearch, thanks
why not automate even the datasources and dashboards setups? both can be deployed as configmaps and automatically discovered based on some labels hint for next video? :) great tutorial, btw!
Hi! Thanks Anton for the explanations Can you give a hint - how to configure Prometheus how much space in the disk to use for metrics store and for how long to keep them?
Usually 7 or 14 days, disk size depends on how many targets you have and scrape interval. In all clouds you can increase volume when you reach a limit, just make sure that your kubernetes storage class has "expandable: true". If you need long term checheap storage use thanos.
Hi bro. Thanks for your content. Especially on EKS. How do you make your presentation. With animation of red arrow for exemple. Do you make it yourself, with which tool, or you pay someonelse. It look really nice.
Anton, do you have any suggestions for how to handle migration from "plain vanilla" Prometheus to Prometheus Operator? For example if I have an existing Prometheus instnace running in my EKS cluster, can I install the Prometheus Operator in that same cluster and then migrate to it?
Yes, you can. Both Prometheus instances can be run in parallel. You need to deploy them using an operator and convert your plain Prometheus config to a service or pod monitor. Make sure that both have the same targets before decommissioning the old one.
Nice video, thanks for the content. In the case I'm using `kind: AlertmanagerConfig`. How do I do to make this work with a `PrometheusRule` operator to send notifications to slack? I already have alertmanager up and exposed with TLS on my URL. Once again, thank you
Would you recommend to use helm charts with terraform, we are on an awkward situation where we want to pass secrets to the helm chart and have it force a reload in the pods like kustomize does with helm inflator generator
it's hard to maintain helm charts over time. if you want it for temporary env, qa, staging, dev environment it's ok. but if you plan to maintain that env for over a year, it can be dangerous
Well, alerts are configured on the local prometheus (recommendation from Thanos itself), it's usually done by using alert custom resource "NOTE: It is recommended to keep deploying rules inside the relevant Prometheus servers locally. Use ruler only on specific cases. Read details below why." thanos.io/v0.8/components/rule/#rule-aka-ruler
Hi! In probe.yaml , if we provide a incorrect or fake domain, prometheus still shows target is up with incorrect domain url, as a result no alert will be generated, if we access blackbox it will correctly shows the errors, can you please guide about the fix, thanks
🔴 - To support my channel, I’d like to offer Mentorship/On-the-Job Support/Consulting - me@antonputra.com
Hello Anton. I loved the video and I sent you an email.
@@Fidellio369 Thank you! I think I just replied to your email.
Extremely well done video and repo. You saved the day!
Thanks!
Great teaching ability, thanks a lot for all amazing & high quality content, please make videos regarding eks & gke logging solutions with grafana loki or with elasticsearch, thanks
Thanks Muhammad, will do!
This is gold
Thank you!!
Anton will become big youtuber one day definitely✌🤟
Thanks for the support Piyush!
FINALLY got it working. Thanks!
welcome :)
Would be nice to provision the target and some dashboard using the helm release if possible
Thanks for the feedback!
Very nice video, any tuto of multiple clusters to monitor with prometheus agent and a global cluster with prometheus operator using remote write?
Thanks, I'll create one soon for thanos
Awesome, Thank you Anton
Thanks Ahmed!
Extremely useful sir!!!
Thank you!
Next time integrate with thanos with compactor and s3 backend
@@MrBharathkumarraju Thanos (Multi Cluster Prometheus) Tutorial: Global View - Long Term Storage - Kubernetes - ua-cam.com/video/feHSU0BMcco/v-deo.html
@@AntonPutra woww you are real k8s monitoring guru!!! and you made them available for free...hatsoff to you!!!
@@AntonPutra any videos on loki-promtail and tempo for tracing?
Thanks for the videos
Can you do persistent storage for Prometheus using helm am struggling with it
Sure, just set this block - github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml#L670-L678
Thanks for this precious content , very well explained .
Thanks Anas!
Best videos as always.
Thanks Rafael =)
Thank you, Anton
That really helped a lot
You're welcome!
Thanks, that's what I needed yesterday
Welcome =)
@@AntonPutra can you explain what is the purpose of prometheus adaptors?
@@ziaurrehman4738 You can use prometheus metrics for horizontal pod autoscaller (by default only cpu and memory), i have a tutorial how to implement
@@AntonPutra thanks. I will try to find that video on your channel
@@ziaurrehman4738 This one - ua-cam.com/video/iodq-4srXA8/v-deo.html
why not automate even the datasources and dashboards setups? both can be deployed as configmaps and automatically discovered based on some labels
hint for next video? :) great tutorial, btw!
Thanks for the feedback!
Hi! Thanks Anton for the explanations
Can you give a hint - how to configure Prometheus how much space in the disk to use for metrics store and for how long to keep them?
Usually 7 or 14 days, disk size depends on how many targets you have and scrape interval. In all clouds you can increase volume when you reach a limit, just make sure that your kubernetes storage class has "expandable: true". If you need long term checheap storage use thanos.
Thanks a lot, this tutorial really helped me!
thanks!
Hi bro. Thanks for your content. Especially on EKS. How do you make your presentation. With animation of red arrow for exemple. Do you make it yourself, with which tool, or you pay someonelse. It look really nice.
Thank you! I do it myself using the Adobe suite.
Anton, do you have any suggestions for how to handle migration from "plain vanilla" Prometheus to Prometheus Operator? For example if I have an existing Prometheus instnace running in my EKS cluster, can I install the Prometheus Operator in that same cluster and then migrate to it?
Yes, you can. Both Prometheus instances can be run in parallel. You need to deploy them using an operator and convert your plain Prometheus config to a service or pod monitor. Make sure that both have the same targets before decommissioning the old one.
Nice video, thanks for the content. In the case I'm using `kind: AlertmanagerConfig`. How do I do to make this work with a `PrometheusRule` operator to send notifications to slack? I already have alertmanager up and exposed with TLS on my URL. Once again, thank you
Would you recommend to use helm charts with terraform, we are on an awkward situation where we want to pass secrets to the helm chart and have it force a reload in the pods like kustomize does with helm inflator generator
Well now that I think about it, maybe could use sops/age for secrets do you have video about it ? We want a solution to remove secrets from state
it's hard to maintain helm charts over time. if you want it for temporary env, qa, staging, dev environment it's ok. but if you plan to maintain that env for over a year, it can be dangerous
Ah do you mean managing helm charts in general or just outside gitops (terraform)?
hi @AntonPutra , how did yo manage configuration for alerts. You did not show anything similar to that
Well, alerts are configured on the local prometheus (recommendation from Thanos itself), it's usually done by using alert custom resource
"NOTE: It is recommended to keep deploying rules inside the relevant Prometheus servers locally. Use ruler only on specific cases. Read details below why."
thanos.io/v0.8/components/rule/#rule-aka-ruler
Awesome work ✨
Thanks!
Thanks for this excellent tutorial, but it is possible to add basic auth? I didn't find the way to include a custom web.config.file.
Try this to setup basic auth - github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#httpconfig-1
I am facing issue when applying prometheus, prometheus-main-0 goes in pending state.
prometheus-main-0 0/2 Pending 0 3m19s
probably there is no pv for pvc, check both and logs from this pod
Hi! In probe.yaml , if we provide a incorrect or fake domain, prometheus still shows target is up with incorrect domain url, as a result no alert will be generated, if we access blackbox it will correctly shows the errors, can you please guide about the fix, thanks
can you try using something like status code in the alert? fake domain should not produce 2xx result
Have you tried VictoriaMetrics Operator?
No, first time I hear about it, but I'll take a look.
@@AntonPutra Try it, it's better than Prometheus
@@rohittiwari5575 I will. Why it's better?
whenver you ssay operator i hear brother
working on my english :)
@@AntonPutra its undrestandable, thanks for the explenation tho
Second this time 😰
?
@@AntonPutra I was the second person to comment 😅 a cultural habit of followers in UA-cam 🥲😅
@@YordisPrieto 😅
wth is all this, this is so freakin confusing, yamls everywhere and weird nonsensical terminology, ugh