I keep referencing this video again and again. This is a great beginner level tutorial to basics of getting logs into Graylog, separating them into streams, and searching through logs with Graylog.
Thank you, I spent lots of time configuring ELK from scratch, but the work greylog has done is awesome, its simple and does the job well thank you for showing this
I was hung up on how to identify and separate out logs for a project/application once I send the logs from FluentD to Graylog. Your explanation on streams/indices/rules helped cleared out that confusion. Thank you so much
I've been dragging my feet for about a year now on making a decision with respect to log aggregation from a handful of proxies I manage all over the world. I checked this video out and decided to give Graylog a try and I absolutely love it! The install is super, super easy and I had a Graylog instance running and ingesting data from several of my proxies within 2 hours. Now its just a matter of a tweaking queries and dashboards to let me see precisely what I need. Awesome video, as always, Tom - I for sure would have spun my wheels on the streams / indices / extractors /etc!
I’ve played around a bit now and I’ve found you can really set a single “syslog” input for multiple servers. Then you create the index and streams. But when you create the stream “rule”, you can use the “gl2_remote_ip” field to only filter by certain syslog sources. So for pfSense, it would be the router. And for any unifi devices, it’s the IP of the device itself (AP, switch, etc). You can set the stream to be a so for device, or a group. This way you don’t have to have a separate input with a unique port number for EVERY remote server :)
I've been using Graylog at many of my customers for a few years now. Excellent product. I've been able to setup some really informative dashboards and alerts. It works well after you make a few tweaks. One thing I found is to make sure to adjust the heap size to get good performance. Other than that, it works great. We are ingesting Windows logs, NAS storage logs, WIFI AP logs, Firewall and Switch logs and VMware logs. The difficult part is narrowing down the scope of the data to the things you really need, but once you have it you can build dashboards that provide concise information. I have been using the grok patterns to categorize data from firewalls and it helps to build more informative dashboards and allow greater flexibility in presenting the data. Excellent tutorial.
Thank You for showing this piece of software. I was working on setting up an ELK stack for just syslogs and is has been a few days utter failure and making me question my chosen profession and my proficiency at it. I have chosen to take a different route for logs because of the sheer admin cost. It's just two of use for 4500+ Customers and 100 Employee's.
Thanks for the great video! I have been wanting to get into graylog for a while, this video finally got me to get off my butt. Still trying to figure it all out, but this was a great start. I was able to very easily set up the free enterprise license since it seems highly unlikely I will be ingesting over 5gb/day in my homelab.
Ran a Graylog VM and couldn't figure out why it wouldn't ingest my ESXi, TrueNAS and NetScaler logs. I imagine it was the extractor, stream, index architecture that I didn't understand. Great job of addressing that upfront and not just going thru a procedural next, next, next configuration
Thanks for this video Tom! I was just starting to work on this. I’d love to see a video that is specifically about getting Suricata logs into Graylog if that’s something you’re interested in!
Hi, I'm trying to send the suricata logs on pfsense to graylog and then show into grafana. but not luck yet. Only can show filterlog logs but not from suricata. if someone have this done I appreciate some kind of help. Thanks a lot
I had to use RAW udp (in order to see pfsense logs) and not syslog for some readon for the logs to actually popup, it was a rough start.... I'm at pfsense 2.4.4-p3 and i can see you have other options in yours as BSDlogging in your fw, so updating it might be good/fix things for me ... :)
Very good explaining video. I have one question: I have multiple servers / raspberry pis where i want to get the syslogs, however with 100 raspis, i dont like to create 100 different inputs with different ports. Can different hosts ude the same port and can the graylog distinct between the indiviual sources? In the end, all the data of those devices can land in the same location, with some filter to separate out some specific messages to be saved in another location, however i would need to know where the logentry came from, when using the same port. Is this possible?
Graylog tweeted your "almost done my 2023 Graylog update, need some help with an issue" tweet. Docker seems to be the way these days. I saw that you were able to fix the issue you were having with the config file. Are you going to release a tutorial soon?
You could compare for yourself... install Kibana and visualize the same data for comparison or install two instances (one Graylog and one ElasticStack) to evaluate the two. GrayLog is a bit more intuitive than setting up an ElasticStack instance and is a matter of preference. Here's a ElasticStack alternative to try and compare for yourself: github.com/pfelk/pfelk A video comparison would be great!
Nice tutorial. Thank you for not spending the first 30 minutes explaining your life history, begging to be excused for not posting on UA-cam, a tour of your house, with 10 minutes of please like and subscribe
I subscribed to your channel recently and I am very glad now because of videos like this. Q: I have services sending emails. Can Graylog receive or check email? Also, can I set it up so that it alerts me if an email for a task that is scheduled was not received? Thank you!
@@LAWRENCESYSTEMS I have services and routers sending email messages when an event appears. I was wondering if Graylog can extract those messages via IMAP/POP3 and analyse them?
@Leeroy - Syslog was created in the 1980s and by default does not encrypt, transmitting everything in the clear. However, Syslog-NG is capable of transmitting over TLS via TCP. Additionally, Syslog-NG was developed to add additional security and filtering options. This same setup is feasible (replacement to syslog) and capable of leveraging TLS (e.g. encryption).
How do i send log from different subnet? I created a pfsense vip for graylog server. I can ssh and ping graylog server by this ip but not able to open web gui and send log.
I am completely newbie in logging analysis and this domain, can you please tell me if Graylog is SAAS based solution in any way? I mean when we ingest log data for analysis do we need to ingest it in their SAAS platform?
Question to Master : how can i secure graylog to only receive secured message ? i do not want every one can use my graylog server to send message , please HELP !!!!
I don't know if UniFi would report switch MAC/port changes over syslog?. Try getting a Cisco switch to report MAC table 😅 to graylog, omg. I spent 2 last days trying to.. syslog, nope.. Then SNMP should be possible, but SNMP graylog plugin refuses to play nice, least with cisco MIBS... OK, i get it thru Telegraf then it has options for that. Switch sends snmp notifications to Telegraf, and Telegraf reports back to graylog (it has nice and simple output for that), but nope, at most i can see something changed, but not the MAC... so far that i can see. It's possible however to request the mac table over snmp, but needs some serious parsing to understand which port its on.. but/and then youd haft to do that every minute, instead of just getting a notification.
Yes. Elastisearch is a big problem. One server is not enough even for relative small logs docs.graylog.org/en/4.0/pages/configuration/elasticsearch.html We strongly recommend to use a dedicated Elasticsearch cluster for your Graylog setup
have it running in proxmox as CT container, with 4gb ram and data on zfs mirror with two 6-10 year old HDD. syslog from various vm, physical linux machines, raspberries and opnsense (but without every firewallevent). use only default index set, after about one year there are 6.8GiB of data. query works just fine, nothing to complain. so I would say, give it a try!
Great video.. but on a default Ubuntu 21 Graylog install I've found that using port 514 results in "permission denied" as the lower ports are restricted to all but root users (which the Graylog server apparently is not running as) My pfsense will not send logs on any other port than 514, despite what may be entered in the System Logs settings. Sio I've configured Graylog's input to port 1514 and set the server input like this: iptables -t nat -A PREROUTING -p tcp --dport 514 -j REDIRECT --to 1514 iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to 1514 Oh goody. Now I'm flooded with logs in Graylog :/
@@LAWRENCESYSTEMS thank you for your response. If you happen to change things up and start working with grafana. You can send graylog logs to grafana and have beautiful siem graphs.
The whole idea here is A.I for cyber security. With the graphs and the alerting system. Have scripts programmed in python or an other language to react to it. That's my project I'm working on.
@@MiguelCruzer I live in the Netherlands and my timezone is thus gmt+1 , at the time the docker image only supported utc so I had to modify the dockerfile and re-compile it.
@@ItsQuintFX That sounds like a config issue, I'm not sure if this is new or not but docs say any config option can be passed as an ENV_VAR preceded with the GRAYLOG_ name. docs.graylog.org/en/4.0/pages/installation/docker.html#configuration Hope this helpsl
Very neat product, I actually work with Splunk. This is super cool but doesn't have all the features that Splunk does. You should totally take a look at it, I know it is closed source but it is a damn good product.
i don't know what it is about your voice, but it is attracting my cat and she is trying to smash her face under my laptop, like trying to burrow under it. she is obsessed with my laptop. she has never done this before. nor when i pause.
Related Forum Post
forums.lawrencesystems.com/t/open-source-logging-getting-started-with-graylog/8797
Hey, we are doing some UA-cam clean up and just came across the video! What a great tutorial! Thanks for taking the time to make it :)
Thanks!
ok
I keep referencing this video again and again. This is a great beginner level tutorial to basics of getting logs into Graylog, separating them into streams, and searching through logs with Graylog.
Glad you enjoyed it!
Thank you,
I spent lots of time configuring ELK from scratch, but the work greylog has done is awesome, its simple and does the job well
thank you for showing this
I was hung up on how to identify and separate out logs for a project/application once I send the logs from FluentD to Graylog. Your explanation on streams/indices/rules helped cleared out that confusion. Thank you so much
Fantastic!
I've been dragging my feet for about a year now on making a decision with respect to log aggregation from a handful of proxies I manage all over the world. I checked this video out and decided to give Graylog a try and I absolutely love it! The install is super, super easy and I had a Graylog instance running and ingesting data from several of my proxies within 2 hours. Now its just a matter of a tweaking queries and dashboards to let me see precisely what I need. Awesome video, as always, Tom - I for sure would have spun my wheels on the streams / indices / extractors /etc!
could you share what queries you've used for your dashboards or any free resources available. Thanks
I’ve played around a bit now and I’ve found you can really set a single “syslog” input for multiple servers. Then you create the index and streams. But when you create the stream “rule”, you can use the “gl2_remote_ip” field to only filter by certain syslog sources. So for pfSense, it would be the router. And for any unifi devices, it’s the IP of the device itself (AP, switch, etc). You can set the stream to be a so for device, or a group. This way you don’t have to have a separate input with a unique port number for EVERY remote server :)
Thanks, I was just wondering how to mitigate this problem. Your explanation was perfect.
I've been using Graylog at many of my customers for a few years now. Excellent product. I've been able to setup some really informative dashboards and alerts. It works well after you make a few tweaks. One thing I found is to make sure to adjust the heap size to get good performance. Other than that, it works great. We are ingesting Windows logs, NAS storage logs, WIFI AP logs, Firewall and Switch logs and VMware logs. The difficult part is narrowing down the scope of the data to the things you really need, but once you have it you can build dashboards that provide concise information. I have been using the grok patterns to categorize data from firewalls and it helps to build more informative dashboards and allow greater flexibility in presenting the data. Excellent tutorial.
Thank You for showing this piece of software. I was working on setting up an ELK stack for just syslogs and is has been a few days utter failure and making me question my chosen profession and my proficiency at it. I have chosen to take a different route for logs because of the sheer admin cost. It's just two of use for 4500+ Customers and 100 Employee's.
Thank You Tom. I am looking at implementing Graylog in my home network and your video content was very helpful!
I’ve just been thinking about how there must be something like this out there. Thank you! I’ll play with this!!
Thanks for the great video! I have been wanting to get into graylog for a while, this video finally got me to get off my butt. Still trying to figure it all out, but this was a great start. I was able to very easily set up the free enterprise license since it seems highly unlikely I will be ingesting over 5gb/day in my homelab.
We use this at my work. It’s dope.
Been wanting to move away from Splunk for a while, thanks for hitting the high points!!
So helpful! Great tuto! New sub in here.
Greets from Uruguay.
Thanks! This video helped me to get graylog to start seeing incoming data.
I set this up in 2016, we had 3 customers all sending logs to centralized Graylog server; it was fun!
Ran a Graylog VM and couldn't figure out why it wouldn't ingest my ESXi, TrueNAS and NetScaler logs. I imagine it was the extractor, stream, index architecture that I didn't understand. Great job of addressing that upfront and not just going thru a procedural next, next, next configuration
This is Great Tom. I have been looking for this video on this topic. Thanks.
Thanks for this video Tom! I was just starting to work on this. I’d love to see a video that is specifically about getting Suricata logs into Graylog if that’s something you’re interested in!
I am also hoping to export Suricata events to Grafana for visualization if that's something you're interested in exploring.
Hi, I'm trying to send the suricata logs on pfsense to graylog and then show into grafana. but not luck yet. Only can show filterlog logs but not from suricata. if someone have this done I appreciate some kind of help. Thanks a lot
Thanks again for another really good breakdown using real world and human understandable examples.
This is awesome!! Great video!
Hi beb
I had to use RAW udp (in order to see pfsense logs) and not syslog for some readon for the logs to actually popup, it was a rough start.... I'm at pfsense 2.4.4-p3 and i can see you have other options in yours as BSDlogging in your fw, so updating it might be good/fix things for me ... :)
can't wait to use this !!
I see you also have Elastic Search running. would be interesting to hear what your pro's cons are vs using Greylog.
Graylog is much easier to setup and maintain.
vAppliance no longer available/supported from Graylog: 4:00 "no time commitment loading a VM." ☹
Good Timing considering the ElasticSearch license melt down
ElasticSearch licence meltdown? What happened?
It’s no longer open source and they blamed Amazon for it.
GrayLog utilizes ElasticSearch - what do you exactly mean?
Very good explaining video.
I have one question:
I have multiple servers / raspberry pis where i want to get the syslogs, however with 100 raspis, i dont like to create 100 different inputs with different ports. Can different hosts ude the same port and can the graylog distinct between the indiviual sources?
In the end, all the data of those devices can land in the same location, with some filter to separate out some specific messages to be saved in another location, however i would need to know where the logentry came from, when using the same port. Is this possible?
You could use one port and then parse the data by host name.
Nice video. I'll try this on my network and see what it can do.
Graylog FTW
Been using it since 2013.
Pair it with fluentd and you are done.
Add your manual :)
Ea! que tu haces por estos lares? jajaja Saludos mi pana
Have you ever covered SIEMs? I would love to get pointed to the best ones out there, OSS if possible. Thanks!
@S K We use Qradar. It's a great product only very expensive.
Check out wazuh
Thanks for the brilliant video. I'm planning to integrate unifi controller to graylog . do you have any idea where I get the extractors for inputs
github.com/lawrencesystems/graylog_extractors
Graylog tweeted your "almost done my 2023 Graylog update, need some help with an issue" tweet. Docker seems to be the way these days. I saw that you were able to fix the issue you were having with the config file. Are you going to release a tutorial soon?
Yeah, hoping to get it done by early next week.
@@LAWRENCESYSTEMS awesome, thank you!
very interesting! is there a log4j adapter? work?
will there be a AI Threat analysis followup video?
Hey Lawnrence any recommendations on user authentication and navigation logs?
Graylog
@@LAWRENCESYSTEMS thanks i'll try that
Thanks for this! I'm also interested in how this compare to ELK and even Splunk.
You could compare for yourself... install Kibana and visualize the same data for comparison or install two instances (one Graylog and one ElasticStack) to evaluate the two. GrayLog is a bit more intuitive than setting up an ElasticStack instance and is a matter of preference. Here's a ElasticStack alternative to try and compare for yourself: github.com/pfelk/pfelk
A video comparison would be great!
Thank you Tom.
Great & Useful vid
Great video.
Many-many-many-many thanks! Very good tutorial! Still curious, what is in "Memez" bookmark folder xD
My meme stash
Nice tutorial. Thank you for not spending the first 30 minutes explaining your life history, begging to be excused for not posting on UA-cam, a tour of your house, with 10 minutes of please like and subscribe
How does this compare with splunk?
I subscribed to your channel recently and I am very glad now because of videos like this.
Q: I have services sending emails. Can Graylog receive or check email?
Also, can I set it up so that it alerts me if an email for a task that is scheduled was not received?
Thank you!
It can send an email based on parameters that you define. www.graylog.org/features/alerting
@@LAWRENCESYSTEMS I have services and routers sending email messages when an event appears. I was wondering if Graylog can extract those messages via IMAP/POP3 and analyse them?
SYSLOG appears to be a no-brainer. But what about Windows server logs?
Yes docs.graylog.org/en/4.0/pages/sending/windows.html
Interesting video, I decided to create my own laboratory! What about alerts? How do I make an alert for a specific log?
docs.graylog.org/en/4.1/pages/alerts.html#alerts
So it just accepts entries on specific port or is there any auth?
How do I know that gathered data is legit?
Syslog does not use auth, but some of the other support input types do.
@@LAWRENCESYSTEMS Tell me that it is not plain text at least..
@@Mr.Leeroy You should probably read up a bit more on how syslog works.
@@LAWRENCESYSTEMS it has been on 2do list for far too long.
@Leeroy - Syslog was created in the 1980s and by default does not encrypt, transmitting everything in the clear. However, Syslog-NG is capable of transmitting over TLS via TCP. Additionally, Syslog-NG was developed to add additional security and filtering options. This same setup is feasible (replacement to syslog) and capable of leveraging TLS (e.g. encryption).
Awe-some, Tom!
How do i send log from different subnet? I created a pfsense vip for graylog server. I can ssh and ping graylog server by this ip but not able to open web gui and send log.
Ever made a comparison between ELK-Stack and Greylog?
Nope, don't really plan to
very cool.. Thank you!
I am completely newbie in logging analysis and this domain, can you please tell me if Graylog is SAAS based solution in any way? I mean when we ingest log data for analysis do we need to ingest it in their SAAS platform?
As he already told us, it is not, because you can run it on your own server.
Does this integrate with Active Directory?
Yes, via LDAP.
Hi, have you tried integrating syslog-ng to graylog ? Have tried it but the format was bad. If you have tried it, do you have any suggestions ?
I have not, but their forums can be helpful for parsing formats.
Question to Master : how can i secure graylog to only receive secured message ? i do not want every one can use my graylog server to send message , please HELP !!!!
I am listening to someone speak about graylog and said the words star 410 or start up 410 . something 410. Do anyone know what that is?
¯\_(ツ)_/¯
thanks, Thanos, glad you're getting into soft instead of...well...
Good instroduction, even for GL v5 THANKS
I don't know if UniFi would report switch MAC/port changes over syslog?.
Try getting a Cisco switch to report MAC table 😅 to graylog, omg. I spent 2 last days trying to.. syslog, nope.. Then SNMP should be possible, but SNMP graylog plugin refuses to play nice, least with cisco MIBS... OK, i get it thru Telegraf then it has options for that. Switch sends snmp notifications to Telegraf, and Telegraf reports back to graylog (it has nice and simple output for that), but nope, at most i can see something changed, but not the MAC... so far that i can see.
It's possible however to request the mac table over snmp, but needs some serious parsing to understand which port its on.. but/and then youd haft to do that every minute, instead of just getting a notification.
Does it need a lot of ram?
Yes. Elastisearch is a big problem. One server is not enough even for relative small logs
docs.graylog.org/en/4.0/pages/configuration/elasticsearch.html
We strongly recommend to use a dedicated Elasticsearch cluster for your Graylog setup
@@emanuelmilani7976 ah Thanks
have it running in proxmox as CT container, with 4gb ram and data on zfs mirror with two 6-10 year old HDD. syslog from various vm, physical linux machines, raspberries and opnsense (but without every firewallevent). use only default index set, after about one year there are 6.8GiB of data. query works just fine, nothing to complain. so I would say, give it a try!
@@sku2007 will do, thanks
wonder how Graylog compares with solutions such as Kafka
Last time I used elastics, I cried every 3 months having to manually rotate the DB.
Unfortunately Graylog has removed the pre-built Virtual Machine Appliance downloads from the website.
Yup. but they do have docker images
What version on graylog are you using?
GRAYLOG 4.0
Great video.. but on a default Ubuntu 21 Graylog install I've found that using port 514 results in "permission denied" as the lower ports are restricted to all but root users (which the Graylog server apparently is not running as)
My pfsense will not send logs on any other port than 514, despite what may be entered in the System Logs settings.
Sio I've configured Graylog's input to port 1514 and set the server input like this:
iptables -t nat -A PREROUTING -p tcp --dport 514 -j REDIRECT --to 1514
iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to 1514
Oh goody. Now I'm flooded with logs in Graylog :/
Can you do an video on putting graylog logs into grafana. That would be much appreciated.
I don't use grafana so not likely.
@@LAWRENCESYSTEMS thank you for your response. If you happen to change things up and start working with grafana. You can send graylog logs to grafana and have beautiful siem graphs.
Then you can integrate zabbix with grafana and pretty much have a one stop shop.
The whole idea here is A.I for cyber security. With the graphs and the alerting system. Have scripts programmed in python or an other language to react to it. That's my project I'm working on.
Make more vids bout graylog. Keep it up . We will help you out with the subs 👊🙏
tried it, it's nice, but they need to resolve the timezone issue if that hasn't been solved already
Not an issue that I had so maybe it was resolved.
What timezone issue?
@@MiguelCruzer I live in the Netherlands and my timezone is thus gmt+1 , at the time the docker image only supported utc so I had to modify the dockerfile and re-compile it.
@@ItsQuintFX That sounds like a config issue, I'm not sure if this is new or not but docs say any config option can be passed as an ENV_VAR preceded with the GRAYLOG_ name. docs.graylog.org/en/4.0/pages/installation/docker.html#configuration
Hope this helpsl
How is it you always post a video for a solution right when I'm looking for a solution?
Same here!
Can this be run on windows machine?
Don't think so
not finding an ova for use
This all use free or enterprise...?
because all view different with free mode
Looks like they've changed the offering around enterprise and free 5GB. Looks like it might only be 2GB now.
Very neat product, I actually work with Splunk. This is super cool but doesn't have all the features that Splunk does. You should totally take a look at it, I know it is closed source but it is a damn good product.
Expensive as frig
@@foobarturkey You are telling me lol. They have some trial license floating around that lets you do some stuff at home.
thanks
Make a demo bout geo iplocation dashboard. Thsnks!
Tom, regex101.com is my go to for testing out new expressions.
Thanks I will check that out, I have been using regexr.com/
I might be able to help you with some regex stuff, so hit me up if you still have questions.
👍😁 since 2018
Now if it had SNMP too.
t-shirt made me laugh.
i don't know what it is about your voice, but it is attracting my cat and she is trying to smash her face under my laptop, like trying to burrow under it. she is obsessed with my laptop. she has never done this before. nor when i pause.
🐈
"1566 i like that number... " We know... ¬¬
Suggest you check out wazuh :)
I've used it, I'm just not competent enough at it to do tutorials
i don't know why, but you always trigger siri for me. i can't for the life of me trigger it myself though...
Recently configured myself in production. Works perfectly.
github.com/pfelk/pfelk
31:24 ^,
Thanks
better, but is also more professional
"I'm code. I regex."
Seriously, if you [still] need help, let me know.
too complex artich
This all use free or enterprise...?
because all view different with free mode