► Follow me on IG for behind the scenes content 😊👉🏼 bit.ly/2F3LXYJ ► This video was sponsored by the Cloud Native Computing Foundation www.cncf.io/ 🙌🏼 ► More infos on Fluentd: www.fluentd.org/ ▬▬▬▬▬▬ T I M E S T A M P S ⏰ ▬▬▬▬▬▬ 0:00 - Intro 0:26 - Why we actually need log data? 1:40 - Challenges of collecting and consuming logs 3:40 - Solution - a unified logging layer 4:17 - How does Fluentd work 6:29 - How to configure Fluentd? 9:03 - Built-In Reliability 10:09 - Other Use Cases
Have had fluentd daemonset set up in my kubernetes cluster, pushing pod, service, and other logs to elasticsearch for a month or so now. Really like the simplicity and flexibility it offers.
Very interesting. I heard about fluentd in another place and came here to find some info. Glad I found your video, easy to understand and digest. Thanks!
first of all, you are my favourite in my internet world, I learn a lot from you, thank you mam. sorry for my english. in my language which is my favourite आपको बहुत बहुत धन्यवाद।
Nice, my attention was drawn to fluentd by your Udemy course and I was looking for more information about it a few days ago and now you made a video about it 👌
I will actually make a follow-up video on Fluentbit as well. One of the common setups, especially in bigger clusters is, fluentbit pods as collectors and fluentd as an aggregator. Because fluentbit is more lightweight, so such setup would give you good performance while saving resources.
Great Video Nana! I use FluentD for quite some time already collecting logs and sending them to specific indexes outisde of the cluster so they can easially be observed using query's within observability platforms :)
Can fluentd collect logs using push-model like logstash? It is convenient to just add handler/appender to e.g. logback configuration and forget about log format.
That's a common question. Fluentd is more fit for k8s than Logstash, it integrates better and was created with k8s in mind. Plus it has some advantages, like routing the logs is much easier in Fluentd as well as Logstash needs additional service for data persistence. These are some of the main diffs.
Thanks for this video. is there a way to limit nested parsing in json parser plugin @t ? as current it's parsing json in all levels to lots of field getting created. to wanted to limit josn parsing on first level.
Can you search/parse collect a specific string in fluentD like sumo logic. Can you set up dashboards on FluentD itself without pushing it to a destination system
Yes Fluentbit is more lightweight. Needs much less resources. However Fluentd has more features, plugins. So depends on what you need to configure. If Fluentbit functionality is enough, then you can go with it. I will actually make a video about Fluentbit next month :)
Thanks for the video...Currently I am using Fluentd but it's can't go beyond 2k/EventPerSec/PerIntance. Where I am looking for 20k/EPS/PerInstance with each average event size of 800 Byte. Do you have any suggestion for the same?
Dear Nana, when i try to deploy fluentd using helm chart by default it deploys as root user. i changed to non root user by using security context. But the problem is it errors out "[error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/fluentd-containers.log.pos". Can you please advise in case if you had this issue
Hey Nana, the content of your video is always very good and easily understandable. Except for your audio level, it seems low to me. I have tried many ways to increase your audio and try different devices as well but it didnt increase the sound volume. please make sure your videos are having little high audio quality so that some one can easily listen to it without earphones or headphones. I hope it wouldn`t appear offensive to you.
Dear Nana, I like your videos they are very good with good content. But in this case it was not the quality I was expecting :) your videos are watched by IT pros you need to give more in dept knowledge instead of just high level overview. Please :) But overall I like your vides, keep up the good work :)
Hello.. thanks for taking the time to put together this video. It’s excellent, it really does help with conceptualizing how to utilize fluentd.. I got it up and running and I am capturing logs from several of my back up processes and syslogs from few servers. Something I’ve not been able to figure out is how to dynamically assign labels to my Loki match blocks. I’ve used the rewrite_tag_filter plugin to push log lines to different match blocks where I have different labels but that feels like I’m hard coding the same thing over and over just to get one label to be different. I’d like to store the program or process name in a variable based on the log line and use it as a label. Is that possible? I’ve tried several different ways but I have not been successful doing that as yet. Any info would be greatly appreciated. Thanks again *edit when I say labels I’m referring to labels used in Loki and visible in Grafana
I would say that I don't consume logs, I don't eat them, I don't inhale or inject them into my bloodstream ;-) I normally just view them. I prefer consuming food.
I am having a bad experience for the last 3 days even though I have paid for the course. Please provide me a refund if you cannot help me open the second video of this course....
Hi Ash, can you please send an email to techworld-with-nana@nnsoftware.at and provide email with which you have purchased the course? So we can check. On UA-cam comments there is no support Is this your email: goashtech@gmail.com ?
I don't think the explanation is good. it is so abstracted and I wonder if you really have configured it by your self. Is Fluentd flies along the applications like the animation you put on 4:26?
► Follow me on IG for behind the scenes content 😊👉🏼 bit.ly/2F3LXYJ
► This video was sponsored by the Cloud Native Computing Foundation www.cncf.io/ 🙌🏼
► More infos on Fluentd: www.fluentd.org/
▬▬▬▬▬▬ T I M E S T A M P S ⏰ ▬▬▬▬▬▬
0:00 - Intro
0:26 - Why we actually need log data?
1:40 - Challenges of collecting and consuming logs
3:40 - Solution - a unified logging layer
4:17 - How does Fluentd work
6:29 - How to configure Fluentd?
9:03 - Built-In Reliability
10:09 - Other Use Cases
Have had fluentd daemonset set up in my kubernetes cluster, pushing pod, service, and other logs to elasticsearch for a month or so now. Really like the simplicity and flexibility it offers.
Never tried fluentd before, but I definitively will, thanks to this video. I love this channel. Please keep 'em coming.
Glad to hear, thank you Farid :)
wow, super good quality video, thanks for the great effort!
Very interesting. I heard about fluentd in another place and came here to find some info. Glad I found your video, easy to understand and digest. Thanks!
first of all, you are my favourite in my internet world, I learn a lot from you, thank you mam. sorry for my english. in my language which is my favourite आपको बहुत बहुत धन्यवाद।
Nice, my attention was drawn to fluentd by your Udemy course and I was looking for more information about it a few days ago and now you made a video about it 👌
😊👍
That's amazing 🔥
We need more tutorials on EFK stack 🙏
Dear Nana. Very interesting Your opinion. Fluentd vs Fluent-bit? And which product is better to use in 2021? Thank you 😊
I will actually make a follow-up video on Fluentbit as well. One of the common setups, especially in bigger clusters is, fluentbit pods as collectors and fluentd as an aggregator. Because fluentbit is more lightweight, so such setup would give you good performance while saving resources.
@@TechWorldwithNana Thank you Nana!
Nice intro. I was expecting more of a how to actually install and configure fluentd
Great Video Nana! I use FluentD for quite some time already collecting logs and sending them to specific indexes outisde of the cluster so they can easially be observed using query's within observability platforms :)
Thanks for your videos, I'm a pretty bad english listener and I understood all you said and I learned a lot with !!!!
I strugle with this a few days ago. I wish i had found this video earlier! thanks
This content is so good it should be paid
Great. Nana. You always enlightened my knowledge. EFK becomes my target K8s logging framework. Thanks.
I do really like the way you express and deliver your thoughts, much, much appreciated!
Please go on with your videos! :)
+
BTW - how can we support you, donating anyhow? Thanks!
You are the best woman in the world , I love ur info ❤
Spectacular explanation!!! 👏🏽👏🏽👏🏽👏🏽👏🏽
Nice - short but informative. Thank you.
Thanks Vasily :)
Can fluentd collect logs using push-model like logstash? It is convenient to just add handler/appender to e.g. logback configuration and forget about log format.
The DevOps hero we didn't know we needed :)
As usual Absolutely fantastic content thanks a lot for your wonderful explanation.
Another bang on video thank you for your help please add some slides to take notes.
Nice explaination. Good job. Can you also do a video on OpenTelemetry?
Thank you. The contents are same as your Fluentbit lecture. Could you please give bit precise use case info when to use which one or how to use both.
what about logstash !! thanks for the video .
Fluentd is actually more fit for K8s, it was actually created with k8s in mind.
@@TechWorldwithNana Fluentd was created when such things as Kubernetes does not exist at all
Dear Nana, could you please provide the video on design , how to get the health and pod level metrics for the applications deployed in openshift?
If I want anyone collecting my data... It's Fluentd!
very well broken down, the config snippets and animations help to visualize it a as well, thanks !
should I choose Fluentd over logstash/filebeat? @nana
Could you share your views on how Fluentd is different then splunk for logging apart from the fact that splunk is commercial
Great content (same comment again but this content is really good)
Happy to hear, thank you Daniel 😊
A great way of explanation indeed.
Hi Anna, your video is always perfect and helpful. I wonder how you can make like this video, what are tools to do this
As usual the best!!
Filebit or logstash also does same thing then why we should go with fluentd..
And also plz explain fluentd config file in more details..
That's a common question. Fluentd is more fit for k8s than Logstash, it integrates better and was created with k8s in mind. Plus it has some advantages, like routing the logs is much easier in Fluentd as well as Logstash needs additional service for data persistence. These are some of the main diffs.
Great content as always. Thank you!
Thanks for this video. is there a way to limit nested parsing in json parser plugin @t ? as current it's parsing json in all levels to lots of field getting created. to wanted to limit josn parsing on first level.
great video 🎉
Thank you ! very useful content
Thank you Mark!
It's worth watching and learning too. How it differs from Jaeger ? Please throw some light on that. Thanks in advance.
Thank you so much! Is very interisting
Can u please let me know how to push specific namespace logs only to elasticsearch from fluentd in k8s any specific parameters we need to use ?
great explanation as always Nana, what is the logging solution we can integrate for Prometheus to collect application logs?
Very well explained.
start at 4:22 :)
Can you search/parse collect a specific string in fluentD like sumo logic. Can you set up dashboards on FluentD itself without pushing it to a destination system
Thank you for the nice presentation. What tool would you recommend using when visualizing the log data?
I've used EFK stack for that so I think Kibana is really great for visualising logs!
Thanks for your videos
Hi Nana, Can Fluentd replace #autodiscover feature of filebeat? Which helps to get the logs of #AUTOSCALING environment
Can Fluentd collect pod/containers metrics ? if not, what do so in Docker & Kubernetes environments ?
Thanks for sharing..
Though AWS documents suggesting to use FluentBit instead of FluentD..
Yes Fluentbit is more lightweight. Needs much less resources. However Fluentd has more features, plugins. So depends on what you need to configure. If Fluentbit functionality is enough, then you can go with it. I will actually make a video about Fluentbit next month :)
@@TechWorldwithNana You care for us so much :)
Can this be used for logs from iOS application?
Thanks for the video...Currently I am using Fluentd but it's can't go beyond 2k/EventPerSec/PerIntance. Where I am looking for 20k/EPS/PerInstance with each average event size of 800 Byte. Do you have any suggestion for the same?
Hey how does it compare to cribl ? I cant rely spot wheres the diffences right now.
Hi @TechWorldwothNana when the the DevSecOps course going to be available
Nice video, but I can't get the difference between this tool and a broker tool as for example CAMEL
What's the difference with ELK stack?
Need to know this too.
Do you have lessons on linked in as well ? Couldn’t find ?
I'm new to all this. Is nlog just another logger like fluentd?
Is it worth using fluentd together with Prometheus? Or is it more like an alternative?
which one is most fit for gcp cloud run logs collecting?
Thanks for the video. is it possible fluent-logger-java integrate in android? if possible please provide details?
Dear Nana, when i try to deploy fluentd using helm chart by default it deploys as root user. i changed to non root user by using security context. But the problem is it errors out "[error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/fluentd-containers.log.pos". Can you please advise in case if you had this issue
Amazing video, thanks a lot🔥.Could you also please make one lesson on using ArgoCD for Kubernetes as the GitOps continuous delivery tool?
Hey Nana, the content of your video is always very good and easily understandable.
Except for your audio level, it seems low to me. I have tried many ways to increase your audio and try different devices as well but it didnt increase the sound volume. please make sure your videos are having little high audio quality so that some one can easily listen to it without earphones or headphones.
I hope it wouldn`t appear offensive to you.
It should be somehow marked that this is K8s related instruction. Im looking for plain docker instructions
Dear Nana, I like your videos they are very good with good content. But in this case it was not the quality I was expecting :) your videos are watched by IT pros you need to give more in dept knowledge instead of just high level overview. Please :)
But overall I like your vides, keep up the good work :)
You don’t have patreon or smth like that? You are doing an amazing job helping the rest of the community and I would like to give back smth :)
How to create Fluentd index with application name , Like Fluend-containername , deployed with helm
Hello.. thanks for taking the time to put together this video. It’s excellent, it really does help with conceptualizing how to utilize fluentd.. I got it up and running and I am capturing logs from several of my back up processes and syslogs from few servers. Something I’ve not been able to figure out is how to dynamically assign labels to my Loki match blocks. I’ve used the rewrite_tag_filter plugin to push log lines to different match blocks where I have different labels but that feels like I’m hard coding the same thing over and over just to get one label to be different. I’d like to store the program or process name in a variable based on the log line and use it as a label. Is that possible? I’ve tried several different ways but I have not been successful doing that as yet. Any info would be greatly appreciated. Thanks again
*edit when I say labels I’m referring to labels used in Loki and visible in Grafana
Thank you - Crystal clear - ***** Stars
Thanks Fayaz 🙏
Do you have any plans on creating videos about podman, buildah, skopeo?
I didn't plan it for now, but will definitely consider them! 👍
Thank you !!!
Nana , sound volume is a bit low , :)
Thanks for the feedback!
why not looking at a more capable log management solution such as syslog-ng?
very useful
Subbed👍🏻
Hello.. how is it different from Splunk?
You could compare FluentD to Splunk forwarders. Preparsing, transforms etc. It's not a full product as Splunk, just a part of a stack.
I would say that I don't consume logs, I don't eat them, I don't inhale or inject them into my bloodstream ;-) I normally just view them. I prefer consuming food.
thanx ❤❤
still wonder why logstash is left behind and fluentd becomes more common
Suberb mam
Nana, I need your help,...!!!!
I am having a bad experience for the last 3 days even though I have paid for the course. Please provide me a refund if you cannot help me open the second video of this course....
Hi Ash, can you please send an email to techworld-with-nana@nnsoftware.at and provide email with which you have purchased the course? So we can check. On UA-cam comments there is no support
Is this your email: goashtech@gmail.com ?
დიდი მადლობა მსგავსი კარგი ტუტორიალები არსად არაა)
I don't think the explanation is good. it is so abstracted and I wonder if you really have configured it by your self. Is Fluentd flies along the applications like the animation you put on 4:26?
why pointing the finger in thumbnail ? '-[
it would be much better if you explained how to install and manage it than listening of 10 minuts of bla bla bla fluentd is good fluentd is the best
First
all logs to splunk, period !
many Splunk users use Fluentd to perform the data ingestion to Splunk, there is a good value having the option to control "which" data is ingested.